Climate Corporation Helps Farmers Around the World Improve Crop Production with AWS and New Relic

Like nearly every other business these days, farming is driven by data. With the world population projected to grow 50% in the next few decades, the need has become urgent to help farmers improve their yields. Our goal at Climate Corporation is to help them do that by collecting, storing, and visualizing field data that provides insights into how to measure the impact of key decisions impacting crop performance.

We have a solution called Climate FieldView that enables farmers to collect and measure field data to make critical agricultural and economic decisions. They can access the application on an iPad, which they take into their combines or tractors, and as they’re tilling or planting or harvesting, we map their fields in real time. For example, we provide the farmer with a soil composition analysis, so they know the biome within their farm. When they plant, we can track what seeds they’re planting and calculate the expected production based on their specific biome.

Finding the right monitoring solution for the job

From the beginning, we built Climate FieldView on microservices in Amazon Web Services (AWS). We have roughly 180 services at present. One of the challenges we’ve had as the company grew from being a U.S. startup to a global enterprise is scaling up the monitoring capability on all our services. Farmers are no-nonsense people. If we’re going to sell them an application that’s supposed to help them improve their yields and get more value out of every acre of land, it better work the first time and every time. To deliver on that expectation, we need to have scalable, functional observability at all times.

Early on, we had two monitoring tools, New Relic APM and open source Prometheus. While on the surface, open source is free, there are hidden costs because developers need to spend time becoming experts on monitoring in addition to their development work. That was fine during startup time when people had to be a jack of all trades to get the business off the ground. But as the business has grown and matured, we need our developers to be more focused and specialized. We don’t want them to be experts in monitoring.

The first priority was to determine what our requirements were for monitoring and observability going forward, and then decide how best to meet those requirements. I did my due diligence, checked the Gartner Magic Quadrant, and compared other tools. For Climate’s needs, I found New Relic is the best value on the market.

By going all-in with New Relic and getting rid of our open source tool, we now can rely on New Relic to be the experts in monitoring and observability.

Instrumentation baked into the containers

For our company, it’s been important to standardize on an observability platform like New Relic. Climate has been going through a period of hyper-growth, doubling our subscription base in the last 12 months after having doubled it in the prior 18 months. Across all of our environments, including production and QA, we're looking at building over 500 individual services.

Some services are in Amazon EC2, but we are moving to Amazon ECS. In fact, we are containerized to Amazon RDS, AWS Lambda, Amazon SQS, and Amazon EMR, already spanning a lot of the AWS surface area.

To keep up with the monitoring in our development organization, we’ve baked the New Relic APM and Infrastructure agents into the containers, so when developers launch a service, it’s automatically instrumented and pops into our dashboards. We’re also looking into using the APIs to enable infrastructure as code for further instrumentation of the services.

What I really like about New Relic is that I don't have to be an expert in gathering metrics, or even knowing how the data is gathered. New Relic takes care of all that for me. I just have to wire it up, which is extremely easy, and then pull the data out and manipulate it so that my developers understand where the problems are and can build solutions.

For example, we have services that provide satellite imagery to the farmers so they can observe changes that could indicate some threat to their crops, like disease or drought. We have an SLA to deliver those satellite images every three days. They’re huge files stored in Amazon S3 and then pushed out to our customers during off-hours. Therefore, we have to be cognizant of the response time from our S3 bucket.

This image processing goes across multiple services, from ingestion to the S3 bucket through publishing to our customers on their devices. We need all the resources in this process to be operating optimally so we can deliver those images within our SLA. That’s where we rely on distributed tracing from New Relic to track that processing through all the different microservices. We have alerts for the developers if anything is running slow so they can investigate and resolve any issues.

Reduced monitoring complexity = more time for developers

We’re pushing out standards across the organization, making sure we have standardized alert policies and standardized notification channels. For example, we specify alert policy IDs for each batch of services owned by a technology team. We put them into their service manifest file to reduce the chance of human error. The goal is to reduce complexity for the developers so they can focus on their particular services and not have to worry about monitoring.

By providing developers with New Relic, showing them how to use features like service mapping, distributed tracing, and triage, they have the information they need to quickly and appropriately track down an issue. Developers get a lot more time back to spend on developing new features in our solution.

The successes we’ve had with New Relic are just the beginning. This is an ongoing partnership, working closely with my account team, solution architects, and solution engineers to grow and mature our New Relic implementation. Our entire team is excited about continuing to move forward with New Relic.

Watch this AWS and New Relic-sponsored webinar to hear more about how Climate Corporation is delivering DevOps at scale.