Automate machine-learning model monitoring with New Relic One and DAGsHub

Tracking machine-learning (ML) models can be time-consuming, costly, and difficult.  Comparing the performance of models in production to ensure more relevant and meaningful results across versions and datasets is a major challenge unless you have automated status checks and real-time metrics. Imagine a world where you can seamlessly integrate experiment tracking and ML flow, with visibility into model performance.

New Relic is partnering with DAGsHub to help data scientists and machine learning engineers automatically monitor, visualize, analyze, and synchronize model experiments through machine learning. The DAGsHub integration with New Relic One will allow you to set alerts to get automated status checks, create custom visualizations through ad-hoc custom metrics, and track your performance in real-time. 

In the following short demo, you’ll learn how to integrate DAGsHub data in New Relic One,  add an alert, analyze your metrics, and build relevant customized dashboards where you can visualize your model training and evaluation metrics in real-time. 

Why integrate DAGsHub with New Relic One?

DAGsHub is a platform for data scientists and machine learning engineers to version their data, models, experiments, and code. It allows you and your team to easily share, review, and reuse work like you do in GitHub, but with a focus on data science and machine learning models. DagsHub is built on popular open-source tools and formats, like Git, DVC, MLFlow, and Jenkins, making it easy to integrate with tools you already use like New Relic.

The DAGsHub integration with New Relic provides the best solution for data science workflows within one unified monitoring platform. You can set up the integration to monitor experiment performance in real-time, set alerts on specific behavior, receive notifications through various channels, and customize data visualizations. The result is significantly more efficient workflows and extended visibility into ML-powered applications. 

Integrating DAGsHub with New Relic One

To set up the integration, you need to do the following:

1. Log in to one.newrelic.com, go to the account menu, and select API keys. Copy your account ID.
2. Go to the Explorer page and select + Add more data.
3. Select DagsHub from the MLOps Integration section.
4. Create or copy the API key.
5. Log in to DAGsHub, select your repository, select settings, and go to integrations. Now select New Relic.
6. Paste your account ID and API key you copied from New Relic. Finish setting up the integration. 
7. In New Relic, select See your data. This will redirect you to an automatically generated New Relic dashboard powered by DAGsHub.
8. Add a metric by clicking on the + sign and adding a chart.
9. Write your own NRQL query to fetch information or use DagsHub’s loss metric query options. 
10. Set up New Relic policies and conditions based on your machine-learning model metrics by creating an alert condition. To do so, select the three dots on any metric chart in your dashboard and select Create alert condition from the dropdown.
11. Next, set up your notification channels to receive alerts.
12.  You can also correlate your incidents to reduce noise and create custom decisions by training your ML model.

New Relic’s partnership with DAGsHub will provide your data and engineering team full visibility into ML-powered applications and the future of software by allowing MLOps and DevOps teams to seamlessly collaborate.