Australia Post Uses New Relic Observability to Manage Pandemic Challenges

Australia Post, together with its subsidiaries, operate as a postal company in Australia. Founded in 1809, the company offer its services to individuals, businesses, and government bodies. In 2019, Australia Post sent 3.3 billion items to 11.9 million delivery points across Australia and to more than 190 countries around the world.

In early May 2020, it felt like Santa had hit fast forward at a blistering pace.

The hustle and bustle normally associated with the holiday season had come early in Australia, when coronavirus lockdown measures took hold, and demand for the nation's postal services skyrocketed. Parcel tracking volumes were up 40% and authentication requests rose 70% compared with the Christmas period five months prior.

‘The usual Christmas rush was nothing compared to COVID-19’, recalls Australia Post’s Head of Platform Engineering Andrew Nette.

With the entire team now working from home for the first time, and more Australians than ever shopping online due to social distancing restrictions, Nette and his team needed to increase their online collaboration and adapt their ways of working. At the same time, they had to prepare and respond to a massive spike in online transactions driven by changing customer behaviour.

Changing customer behaviour wasn't the only contributing factor to the enormous transaction volumes. There was the regular lease renewal for all of Australia Post's post office boxes, which occurs annually at the end of April. Australia Post also released a new initiative to help the community during the coronavirus pandemic, which allowed pharmacies to send vulnerable customers their prescription medication through the mail network free of charge. This initiative created a surge of account registrations from pharmacies.

‘We saw tens of thousands of registrations each day for this new service, putting our systems under increased demand’, Nette says.

Nette recalls that in preparation for the massive increase in transactions, the team ran a thorough set of tests to ensure that Australia Post were ready to scale to meet this need. The same process is carried out before the Christmas peak delivery period each year.

The organisation used New Relic APM to complete a canary rollout that helped them identify pressure on Australia Post's backend, including unprecedented demand on the Small Business Portal resulting from the high uptake of the pharmaceutical delivery service.

‘We identified an area where we had previously not needed support in the new registration flows on the Small Business Portal’, Nette says. ‘New Relic gave us the ability to view our system at the transaction level and create a clear understanding of which elements needed focus.’

To ensure an exceptional customer experience was maintained, Nette and his team focused on addressing errors as they emerged, while simultaneously mitigating the increased load. Primarily this was done by scaling the systems horizontally. However, they took an opportunity to implement a frontend authentication caching system which had the effect of dropping traffic to the backend by 80%. Introducing this cache allowed both backend and frontend systems to scale down the numbers of servers that were part of the authentication stack.

On top of the pharmaceutical delivery scheme rollout, Australia Post were recording double the number of deliveries than usual, while parcel tracking volumes had also significantly increased.

The volume of transactions in April was 340 million—higher than in any month of Australia Post's history. These transactions were double compared with April 2019 (140 million) and 70% higher than the Christmas peak of December 2019 (200 million).

The number of user registrations for Australia Post's tracking system was 2.5 times higher than in February of 2020 and 30% higher than in December of 2019. The team also recorded parcel tracking numbers at three times higher when compared with February of 2020, and up 40% compared with the peak of December 2019.

‘New Relic gave us the ability to view our systems at the transaction level and create a clear understanding of which elements weren't responding as they should be’

Andrew Nette, Head of Platform Engineering, Australia Post

Nette says that the past months have been an incredibly busy and disruptive period for Australia Post product engineering teams. It has highlighted the need for the teams to scale their applications quickly and effectively so they can significantly reduce issues that affect customers and improve the customer experience.

Nette anticipates that this year's Christmas period will surpass the April 2020 spike that Australia Post recorded. Still, he remains confident that the business's updated plans will place Australia Post in a strong position going forward.

‘We weren't anticipating the kind of levels that we experienced during April for another two years, and it has forced us to create a new road map and bring forward a lot of our capacity planning’, he says. ‘This has been necessary for us to maintain our high levels of customer service and will benefit our ability to meet customer demand in the long run.’

For Australia Post, New Relic have been a long-term, strategic partner, Nette says, recalling that the working relationship began five years ago to 'dip their toes' into microservices architecture. Today, Nette and his teams work closely with New Relic as an integral part of the team.

‘The use of New Relic's technology has enabled greater efficiency for our Site Reliability Engineering team’, he says. ‘It also allows us to monitor the size of our application stack more closely, making sure that we are right-sizing and keeping costs down. New Relic has been a core part of our monitoring and alerting platform for over five years.’

In addition to the peak of the coronavirus lockdown, he recalls that during the Christmas period of December 2018, New Relic provided valuable insights.

‘New Relic's technology provided data that highlighted an emerging issue in our registration process in certain scenarios’, Nette recalls.