Valohai blog

Insights from the deep learning industry.

All Posts

Announcing Valohai Pipelines

One of the more exciting things we have under development (or, should we say, in the pipeline) right now is our Pipeline system. Since our mission is to enable CI/CD style development for AI and machine learning, there's a logical next step up from just (well, "just" might be the understatement of the year here) running your code in a repeatable manner with Valohai.

Namely, we want you to be able to take multiple steps, say data validation and preprocessing followed by training followed by e.g. quantization and/or compression for mobile devices or deployment to the cloud. Soon enough, you can even have one model figuring out the best hyperparameters for the training of another using Valohai Pipelines!

Valohai-pipeline

As Valohai has been, and will be, API driven from the get-go, our lovely, enterprising users have been able to do this using their own scripts and glue, but as we roll out pipelines as a first-class feature, you'll have a spectacular integrated view of how your data flows through your processes and what has led into the particular model being trained as it has (which, we believe, will be of particular interest to users within more regulated industries such as the financial and medical fields). Pipelines can naturally be triggered and introspected over our API, so whether you already have an external CI/CD system or your enterprise grows to have one, you can rely on being able to still painlessly work with Valohai to manage your computation and data.

Pipelines are currently in closed beta – if you're interested in kicking the tires, get in touch!


valohai demo

Aarni Koskela
Aarni Koskela
CTO and Founder of Valohai

Related Posts

Bayesian Hyperparameter Optimization with Valohai

Grid search and random search are the most well-known in hyperparameter tuning. They are also both first-class citizens inside the Valohai platform. You define your search space, hit go, and Valohai will start all your machines. It does a search over the designated area of parameters you’ve defined. It is all automatic and doesn’t make you launch or shut down machines by hand. Also, you don't accidentally leave machines running costing you money. But we’ve been missing one central way for hyperparameter tuning, Bayesian optimization. Not anymore!

Announcing Valohai Pipelines

One of the more exciting things we have under development (or, should we say, in the pipeline) right now is our Pipeline system. Since our mission is to enable CI/CD style development for AI and machine learning, there's a logical next step up from just (well, "just" might be the understatement of the year here) running your code in a repeatable manner with Valohai.

Automatic Data Provenance for Your ML Pipeline

We all understand the importance of reproducibility of machine learning experiments. And we all understand that the basis for reproducibility is tracking every experiment, either manually in a spreadsheet or automatically through a platform such as Valohai. What you can’t track what you’ve done it’s impossible to remember what you did last week, not to mention last year. This complexity is further multiplied with every new team member that joins your company.