ABS-CBN reduces achieves 98-99.5% uptime with automated incident alerts

ABS-CBN Corporation is one of the Philippines' leading media and entertainment organisations. It is primarily involved in content creation and production for television, online, over-the-top platforms, cable, satellite, cinema, events, and online radio for domestic and international markets.

Challenge: Using multiple tools while dealing with streaming spikes

The Philippines is one of the world’s biggest markets for online and digital media, and the fifth-biggest in Asia. Despite some limitations in network infrastructure, Filipinos have embraced digital media both for entertainment and news. As a result, the market has become highly competitive, requiring incumbents such as ABS-CBN to continually innovate and deliver not only the best content but also an exceptional user experience.

Streaming and digital media require a robust and stable tech stack to cope with often volatile demand from event-driven spikes, such as when broadcasting live election coverage. Today's viewers expect a seamless experience with instant, uninterrupted access to media. This expectation puts significant pressure on service providers to resolve problems quickly – preferably before they impact the end user.

ABS-CBN had been using a range of different tools for monitoring, error reporting, tracking resources health, and on-premise infrastructure–each requiring different skills to use and interpret, which created a siloed technology environment.

The company was also struggling with a very reactive approach, where problems weren’t identified until they had escalated and were impacting end-users. Engineers didn’t have enough information to help them "zoom in" to pinpoint a problem.

"We relied heavily on customers complaining and we even had to reach out to them and bother them to get more details. And of course, customers would just elaborate on their experience, and sometimes they couldn’t clearly recall exact details. That made it difficult for us to do our troubleshooting as we didn’t have the right tools, and we we're left with too much guesswork," explains Jane Quiambao, OIC-Head of Enterprise Technology Division, ABS-CBN.

Solution: Automating for speed and efficiency

Realising the pressing need for a solution, ABS-CBN evaluated a few options before identifying New Relic as the best candidate. Another consideration was improving efficiency: empowering ABS-CBN’s tech team to be able to execute faster and move away from working manually. The automation created by New Relic was key in this respect.

“Implementation was relatively simple and didn’t take too much time in terms of training and getting the team up to speed,” Jane recalls.

The output of New Relic is integrated with ABS-CBN’s incident and alerts management platforms. This has enabled the team to filter priority one and two issues from warnings and allows them to address the most serious problems right away.

"Before New Relic synthetics, our team maintained multiple devices to test our streaming platform website. But today our team is able to provide results proactively as we perform a synthetic test every 5 minutes for critical services," Jane says.

“For peak moments ABS-CBN scales up resources before an event, especially when a highly anticipated broadcast is about to take place. This results in alerts being much simpler and less critical than in the past,” she adds.

Impact: A single view with full traceability

With New Relic, ABS-CBN now enjoys a single view and full traceability, integrated from applications and infrastructure (on Microsoft Azure, Amazon Web Services, and Akamai). This has allowed developers to pinpoint, at the code level, which third party is failing by studying the error logs report and the dependency of performance degradation from each component.

"New Relic’s key strength is having a consolidated dashboard for monitoring and alerting. There is full collaboration between the Dev and Operations teams during incidents, as we have a single tool to check and monitor any alerts so we can quickly investigate and resolve issues. Thanks to New Relic, we can easily identify if an issue is with the application or infrastructure layer," Jane says.

As a result, MTTR (mean time to resolution) has been hugely reduced. Instead of taking an average of 4-8 hours or even days to resolve an issue, it now takes no more than 1-2 hours. Availability has also improved, standing at 98%-99.5% uptime depending on the property.

Importantly, ABS-CBN can now conduct proactive monitoring, preventing problems from escalating and affecting users.

"With New Relic, we’re alerted to any performance degradation well before it leads to downtime. New Relic has helped us to provide a much more proactive incident and preventative response. With New Relic monitoring and alerting and our defined Incident Management Process, we can quickly mobilise the team in less than 15 minutes.

"New Relic allowed us to know ahead of time if there are any gaps in our user journeys. This also helped us with detecting and getting evidence from our third-party partners to investigate. We know the issues ahead of time. New Relic has also promoted collaboration in a way that it becomes a healthy discussion instead of a blame game," Jane says.