What is MLOps?

MLOps, which is short for machine learning operations, is a set of practices to increase the quality, simplify the management process, and automate the deployment of machine learning (ML) in large-scale production environments. MLOps is similar to DevOps, except it’s specific to machine learning ML systems. As more companies invest in artificial intelligence (AI) applications, there can be a lack of understanding between the data science teams developing machine learning models and the DevOps teams operating the applications that power those models. That’s where MLOps comes in.

Just as good DevOps practices bridge the gap between developer teams (devs) with operations (ops), MLOps practices bring together data engineers, data scientists, and MLOps engineers to build and deploy machine learning models at high velocity.

MLOps provides a tool for monitoring and observing the performance and effectiveness of machine learning models in a production environment. This increases the possibilities for collaboration between data science and DevOps teams, feeding into a continuous process of development, testing, and operational monitoring.

Many of the same principles apply for both MLOps and DevOps, such as the use of continuous integration (CI) and continuous delivery (CD) to keep development cycles and deployments running smoothly. However, there are some significant differences as well.

What are the benefits of MLOps?

While MLOps and DevOps are different practices, they have some similar goals:

• Faster development cycles and rapid innovation
• Reliable, scalable, and highly performant systems
• Improved collaboration between teams

MLOps also includes goals beyond DevOps that are specific to the field of machine learning, including:

• Accurate models in production
• Automated MLOps workflow, from experiments to training models to maintaining models in production
• Reproducible workflows and models, especially because it can be notoriously difficult to replicate a model’s predictive behavior

Ultimately, the goal of MLOps include both general DevOps goals and the more specific goal of faster go-to-market times and lower operational costs, ensuring that learning models are working predictably and accurately, which is a huge challenge in itself.

Why is machine learning important?

Machine learning systems learn or improve their performance based on the data they consume. A well-designed machine learning system can predict behavior based on past behaviors or automatically, and accurately, classify data in increasingly complex environments, augmenting human capabilities and freeing up engineers to focus on other strategic tasks.

What is the difference between MLOps and DevOps?

We can think of MLOps as being comparable to DevOps, but with concerns that are specific to machine learning. Since machine learning is based on both code and data, it isn't just about writing and maintaining code—it’s also about managing and analyzing data. First, through the process of machine learning experiments, data scientists fine-tune algorithms (code) that process data. Once the code is in place, the next step is for data engineers to train the model with large amounts of data.

While an effective DevOps practice includes automating processes around code (such as deployment and testing), with MLOps there are multiple considerations related to the underlying data, including automating the process of training models with relevant subsets. And because real-world data is often messy and constantly changing, MLOps teams and data engineers also need to pay close attention to the quality of the data they use in training. You could create a neural network designed to recognize faces, but if the network only receives pictures of narwhals, it’s not going to work effectively. In other words, your algorithms will only work as expected if you have good data.

It’s hard enough to manage code deployments and ensure that your systems are working correctly in production. Adding data to the mix creates an additional layer of complexity. Here are just a few of the concerns that engineers working with machine learning—and by extension, MLOps practitioners—have to consider:

• Experiments: Data scientists must run procedures that test hypotheses. For example, a data scientist might test an algorithm with different variables to determine which variable leads to the most accurate predictive model.
• Training data: Data is necessary to train models. The wrong training data (for instance, too many narwhal pictures) can result in models performing poorly in production. Getting good training data is even more challenging when data already tends to be messy and incomplete.
• Model accuracy: Just because a model performs well in training doesn’t mean it will perform well in the real world. Models need to be monitored in production to ensure they are accurate.
• Model robustness: Small changes in inputs should only cause small changes in a model’s outputs. A single picture of a narwhal shouldn’t cause a model to expect all foreheads to have horns. That would be a small input with a disastrously large output.
• Model bias: Machine learning models with high bias cannot accurately make predictions.
• Concept drift: Data changes over time, and if machine learning models aren’t updated with recent data, their predictions become less accurate because they are based on outdated information.
• Model explainability: It can be very difficult to understand how a machine learning model learns and makes its predictions. This can make it even more difficult to fix issues when something goes wrong.
• Machine learning governance and compliance: Machine learning is still a recent field with many legal, ethical, and moral considerations.

All of these concerns are specific to MLOps practices.

What is the difference between MLOps and AIOps?

It’s also important to distinguish between MLOps and artificial intelligence operations (AIOps), because they are not interchangeable. Let’s take a quick look at the difference.

• AIOps is about using machine learning to automate your DevOps processes. For instance, you might be using artificial intelligence to find patterns in large data sets, which you could then apply to your DevOps practice. In this case, you are using artificial intelligence to enhance your DevOps workflow.
• MLOps is about using DevOps to improve your machine learning development pipeline and relying on DevOps to enhance your machine learning development processes.

In short, you use MLOps to apply DevOps practices to a machine learning development pipeline, while you use AIOps to apply artificial intelligence to your DevOps practice.

While they aren’t interchangeable, they also aren’t mutually exclusive—you can use both in your workflow.

Who uses MLOps?

MLOps is evolving into an important practice for any company using machine learning models. Machine learning helps businesses deploy solutions that unlock previously untapped sources of revenue, save time, and reduce cost by creating more efficient workflows. Eventually, many companies will—or at least should be—incorporating MLOps to make sure their models are running smoothly. But which teams are using and benefiting from MLOps? Let’s take a closer look at the groups that work with machine learning data in the first place.

• Data engineers build and maintain the infrastructure that is used to collect, store, and analyze data and provide the foundation for developing ML models. Data engineers also create data pipelines that ingest and process raw data, making it usable for data scientists and machine learning models. Managing and automating data pipelines is a key aspect of MLOps.
• Data scientists take large datasets and analyze them. Data scientists that work in the field of machine learning run experiments to determine which algorithms and variables are most effective for machine learning models. Data scientists are often involved in every aspect of the MLOps lifecycle, from gathering data to ensuring that models are working effectively in production. 
• ML engineers are responsible for developing machine learning models and algorithms for the organization to use. They deploy, maintain, and optimize ML models after data scientists have built them.

There’s a lot of overlap between the work that data engineers and data scientists do, so there are also potential issues that can arise. For instance, a team of data engineers might build a system that aggregates data in a way that’s difficult for data scientists to incorporate into an ML model. Or there may be challenges for MLOps engineers to deploy the models that data scientists are building. An effective DevOps practice ensures that teams are collaborating well when their work overlaps, and the same is true for an MLOps practice, too. 

How does New Relic support MLOps?

With the introduction of ML model performance monitoring, New Relic helps data science teams measure machine learning model performance while collaborating with DevOps teams to get additional context on their business impact. Through a number of MLOps integrations, New Relic brings full observability to MLOps practitioners, allowing teams to incorporate MLOps best practices like:

• Monitor machine learning models in production.
• Continuously monitor data across the full MLOps lifecycle.
• Instrument, analyze, troubleshoot, and optimize machine learning application performance.
• View custom dashboards from multiple MLOps observability tools in a central operational dashboard in New Relic.
• Alert on a wide range of issues from data drift to model bias.