Connecting to MLflow

Quick tutorial on connecting to MLflow from jupyterhub in CloudSense.

About

In CloudSense, the node that serves MLflow and database that houses the artifacts are preconfigured after deployment. However, if you are attempting to access MLflow from a jupyter notebook, authentication is required. This is accomplished by passing a security token to the MLfLow node and setting your cloud database credentials as environmental variables.

In this guide you will learn how to authenticate from within jupyter so you may interact with the MLflow tracking server and artifact database - enabling you to start logging metrics, models and artifacts for training runs.

Prerequisites

Before continuing with this tutorial please make sure to complete steps in Quickstart.

Procedure

To make the authentication process as seamless as possible the CloudSense team created a public gitlab repo containing example notebooks and modules that you can use get authenticated in any of your jupyter notebooks.

Clone CloudSense / MLflow Repository

First, open a new terminal window in your jupyterhub application. Once opened, navigate to your working directory and clone the gitlab repo.

git clone https://gitlab.com/ulap-public/jupyterhub/mlflow.git

Get and Set MLflow Security Token

Open your jupyter notebook within the same working directory and execute the following cells to import the necessary packages and display the authentication widget.

import ipywidgets as widgets
import mlflow

# Run auth widget
%run /home/jovyan/demo/mlflow/modules/auth

Enter your credentials in the widget and then execute the following cells. This code will instantiate the CloudSense API object, log into the API using the credentials provided in the widget and fetch your cluster information.

# Instantiate CloudSenseAPI object
cs_api = CloudSenseAPI()

# Login to CloudSense API
cs_api.login(username.value,
             password.value)

# Display cluster information
cs_api.cluster_list

From the cluster list printed, identify the cluster_id belonging to the cluster in which your MLflow app is running. Then enter the cluster_id as an argument in the get_mlflow_token method and set your mlflow_uri variable using the following commands.

# Get mlflow application info.
cs_api.get_mlflow_info(CLUSTER_ID)

# Set your mlflow tracking uri
mlflow.set_tracking_uri(cs_api.mlflow_uri)

Set Cloud Database Credentials

Set environmental variables for cloud credentials.

cs_api.set_aws_credentials(aws_key.value, aws_secret.value)