Valohai blog

Insights from the deep learning industry.

All Posts

Valohai's Jupyter Notebook Extension

Valohai is a deep learning platform that helps you execute on-demand experiments in the cloud with full version control. Jupyter Notebook is a popular IDE for the data scientist. It is especially suited for early data exploration and prototyping.

The data scientists tend to waste a lot of time just waiting for the training results when the optimal way is to work asynchronously and iterate with different approaches in parallel. To read more about asynchronous workflows in deep learning, go to Asynchronous Workflow in Data Science blog.

We at Valohai are developing an extension for Jupyter Notebooks, which is specially optimized to provide full version control for deep learning experiments, and a smooth asynchronous workflow without additional technical hassle.

This tutorial describes what the Valohai + Jupyter Notebook combination can do, and is divided into four steps:

  1. Starting a new experiment,
  2. Monitoring experiments,
  3. Checking experiment results, and
  4. Version controlling your trainings.

Let’s take a look at how the extension works.

1. Starting a new experiment

Run deep learning experiments on cloudStarting a new cloud experiment

When we create a new experiment, the Notebook is uploaded to the cloud where new server instance goes up. Once it has downloaded the training data, it executes all the cells in the notebook from top to bottom. A small rectangle appears to top-right corner to signify an experiment running in the cloud.

2. Monitoring experiments

Monitoring deep learning experiments from Notebook with Valohai extensionMonitoring experiment status and logs

Here we have three experiments running. We can monitor the status and log outputs by hovering the experiment gizmo. Blue color means that the experiment is currently running. Green is for finished experiments. If an exception has occurred or the user manually stops the experiment, the gizmo turns red, and the server is immediately shut down in the cloud.

3. Experiment results

image2-1Downloading finished notebook from the cloud

Once the experiment is finished, we can click the Notebook button and download the resulting notebook back to another browser tab. If we like the results, we can continue working on the notebook right away, perhaps make a few changes and start another experiment in the cloud!

4. Version control

Versioned Notebook experiments on Valohai UIVersioned experiments in Valohai machine learning platform

The experiments are version controlled and fully reproducible in the Valohai machine learning platform. Training data, notebook, docker image, hyperparameters, cloud server type, cost, username, and notes are all stored to keep your work safe and reproducible for years to come.

Sign up for the Early Access Program to try Valohai's Jupyter Notebook extension in practice!

Juha Kiili
Juha Kiili
Senior Software Developer with gaming industry background shape-shifted into full-stack ninja. I have the biggest monitor.

Related Posts

Valohai's Jupyter Notebook Extension

Valohai is a deep learning platform that helps you execute on-demand experiments in the cloud with full version control. Jupyter Notebook is a popular IDE for the data scientist. It is especially suited for early data exploration and prototyping.

Asynchronous Workflows in Data Science

Pointlessly staring at live logs and waiting for a miracle to happen is a huge time sink for data scientists everywhere. Instead, one should strive for an asynchronous workflow. In this article, we define asynchronous workflows, figure out some of the obstacles and finally guide you to a next article to look at a real-life example in action in Jupyter Notebooks.

Automatic Version Control Meets Jupyter Notebooks

Running a local notebook is great for early data exploration and model tinkering, there’s no doubt about it. But eventually you’ll outgrow it and want to scale up and train the model in the cloud with easy parallel executions, full version control and robust deployment. (Letting you reproduce your experiments and share them with team members at any time.)