New Relic One and Amazon CloudWatch streams

In the fourth quarter of 2020, worldwide cloud spending grew 32%, to $39.9 billion—with Amazon Web Services (AWS) grabbing almost a third of the market (31%). That’s exactly why, as strategic partners, New Relic and AWS continue to focus on building and updating integrations for our various services. We strive to optimize cloud infrastructure and lower costs for our mutual customers. We’ve written previously about how we addressed the costs of Amazon CloudWatch for ingesting Lambda logs. Today, AWS and New Relic jointly announce an addition to New Relic One and how it integrates with Amazon CloudWatch Metric Streams.

Amazon CloudWatch Metric Streams is a new way to send metrics for all AWS services and custom namespaces to New Relic One. You get reduced metric latency, enabling more responsive alerting and dashboards, and reducing MTTx. Streaming is the fastest, most efficient approach to keep New Relic One up to date while avoiding API throttling and service quota management as you scale. This post discusses the benefits and outlines how to set up Metric Streams with New Relic.

"Every minute counts with operational incidents, and metric streaming has unlocked valuable minutes to detect, respond to, and mitigate service level-concerns even faster than before!"

—Steven Osborne, Director Of Engineering at Workiva

How Amazon CloudWatch works

Amazon CloudWatch monitors your AWS services in real time and acts as a repository for collecting and tracking metrics. Metric Streams is a major leap in AWS service monitoring and provides a new approach for exporting the metrics that eliminates the need for custom integrations. You can dedicate streams of metric data for delivery through an Amazon Kinesis Data Firehose HTTPS endpoint to the New Relic Telemetry Data Platform. When new AWS resources are created, metrics are automatically streamed. You enjoy the following benefits:

• Speed: Instead of polling the CloudWatch APIs every five minutes or more, Metric Streams will stream metrics to the New Relic Telemetry Data Platform at about two-minute intervals, depending on the architecture and configuration settings.
• API throttling: It’s eliminated.
• Integrations: All metrics from all AWS services and custom namespaces are available in New Relic One. No more waiting for New Relic to build custom integrations for new AWS services or update existing integrations as new metrics are added.
• Cost: Metric Streams offers a 70% lower cost per metric update (on AWS) at $0.003 per 1,000 metric updates vs. the API GetMetricData method, where calls are $0.01 per 1,000 requests.
• Setup: You can set up Metric Streams manually or with AWS CloudFormation.
• Automated: You can use AWS CloudFormation to automate operations with multiple configurations.

The following sections and this video help you get started:

How to integrate Metric Streams with New Relic

Setting up the Amazon CloudWatch Metric Streams integration is a straightforward, two-step process.

Step 1 on New Relic One:

First, create the New Relic One side of the integration using Add more data on the New Relic One home page.

Note: You can also add an AWS account from the Infrastructure > AWS page. Make sure to select the Metric Streams integration mode and follow the steps.

Step 2 on AWS:

There are two ways to set up the AWS side for your next step. You can manually create a Kinesis Data Firehose delivery stream and grant Firehose permission to Metric Streams. Then you create a Metric Stream that sends metrics to your Data Firehose.

Or, you can choose the automated approach using the AWS CloudFormation template. We recommend using AWS CloudFormation with StackSets to deploy Metric Streams across your organization. It gives you the choice of sending all metrics by default or letting you get more targeted with metric filters. For example, you can create a filter rule to stream only metrics in the AWS/EC2 namespace. If you create filter rules, CloudWatch sends metric updates only for the metrics matching the filter rules.

Check the status:

To make sure everything is set up correctly, check the Amazon Metric Streams status page from the one.newrelic.com > Infrastructure > AWS page. You can check configuration using the Account Status dashboard, explore the Metric Streams data or even set up alert conditions. The Account Status dashboard provides a view across metrics, namespaces, metric names, and total metrics by namespace. It also lists any errors you might experience.

Metric Streams in action

All metrics coming from Amazon CloudWatch Metric Streams have the following attributes:

• aws.MetricStreamArn = <the Arn of the metric stream that sent this metric>
• collector.name = ‘cloudwatch-metric-streams’

The New Relic One Telemetry Data Platform stores them as dimensional metrics with the summary type. You can query them using NRQL. Check the documentation for more details.

After you finish setting up, you can bring the data from AWS into dashboards alongside the other New Relic One data available to you. The following example shows metrics from New Relic One’s Synthetic and Infrastructure Monitoring right next to the AWS Application Load Balancer and RDS database Metric Streams data.

For reference, here is the NRQL used for the two charts on the right of the screen capture:

AWS Load Balancer:

SELECT sum(aws.applicationelb.RequestCountPerTarget) as 'Request per Second' FROM Metric WHERE collector.name = 'cloudwatch-metric-streams'

RDS Database CPU Utilization:

SELECT average(aws.rds.CPUUtilization) as 'CPU Utilization' FROM Metric WHERE collector.name = 'cloudwatch-metric-streams' TIMESERIES until 10 minutes ago

The Metric Steams data is stored in the New Relic One Telemetry Data Platform as dimensional metrics of the summary type. You can query it using NRQL and view it in New Relic Navigator and New Relic Lookout.

The following Navigator view is grouped by AWS namespaces:

All your data belongs to you!

More metrics means that you receive a stream of Amazon CloudWatch data from all the AWS services you use as soon as they are available in CloudWatch. Less delay equals faster decisions. You can analyze, visualize, and alert on the Amazon CloudWatch metrics for all of the AWS services your AWS cloud-based applications use—in one unified experience.

Ready to try it out for yourself? Head over to New Relic One and sign up for a free account with platform access and 100GB of ingest per month. Then go to Infrastructure > AWS, and set up your Amazon CloudWatch Metric Streams integration. To learn more about how New Relic One works with Amazon CloudWatch Metric Streams, check out the documentation.