All-in-one observability platform for Toyota’s “TORO” to accelerate a full model change to a mobility company

Use Case

New Relic provides an immediately available observability environment to project members as a key service pillar in the TORO shared platform environment, developed by the CCoE team

Toyota Motor Corporation is accelerating a company-wide full model change towards becoming a mobility company. Toyota’s CASE (Connected, Autonomous/Automated, Shared, Electric) movement is beginning to spark a wide range of technological innovation within the company and elicit a major change in how people view cars themselves. Toyota is an industry leader in CASE domain technology development, and is promoting development of mobility related services at the company-wide level. Toyota is leading a once-in-century revolution toward realizing a safer and more convenient mobility society.

“Toyota is driving its full model change from a car manufacturer towards becoming a mobility company at astounding speed. Software projects are accelerating not just within the R&D department, but at departments and teams across the whole company. There is vigorous development in not just the software for self-driving cars, the symbol of self-driving technology, but also applications related to keywords such as mobility and digital transformation,” explains Mitsuhiro Mabuchi, Group Manager of the Cloud Data PF Group at the AI Management Department.

 Mabuchi is a member of the Advanced R&D and Engineering Company, Toyota’s R&D organization and is also  a member of the Digital Transformation Promotion Department since November 2022. Mabuchi leads the company-wide Cloud Center of Excellence (CCoE) team formed across the Digital Transformation Promotion Department and the AI Management Department since April 2022.

Mabuchi continues, “We formed the CCoE team, comprised of internal cloud computing experts, to expand the cloud initiatives started within the R&D organization across the entire company. All departments are eager to take on new challenges and develop new applications. Our goal is to create an environment where all teams in Toyota can immediately start application development and quickly launch a service even if they are not software experts. I am convinced that allowing more ideas to come to life more quickly will accelerate our full model change towards becoming a mobility company.”

Starting in 2022, the CCoE team is focused on designing and implementing a shared cloud platform that can support a diverse range of business needs. The team has already created a public cloud-based environment, primarily Amazon Web Services (AWS), integrated with security that meets Toyota company-wide standards featuring a CI/CD pipeline that enables rapid application development.

“Our goal was to create a platform that freed project members from the burden of provisioning infrastructure and security and allowed them to focus on application development and operations with peace of mind. We are also working to reduce toil at the operational stage, which tends to drop from focus, to continue providing stable service post development. Our environment increases productivity  while reducing security risks. Observability across the DevSec environment is the next key piece required to achieve our goal of DevSecOps,” says Mabuchi. 

Provisioning an AWS account in 2 hours with TORO

TORO, Toyota’s development accelerating cloud platform compliant with company-wide standards, stands for “TOyota Reliable Observatory/Orchestration.” Development of TORO fully launched in April 2022. It leveraged the base of a cloud platform developed by the AI Management Department in 2021 expanded further to meet a wider range of business needs. Experts in agile development, CI/CD, and cloud native technologies such as security, containers, and observability unleashed their full set of skills to launch the first iteration of TORO in just a few months.

Takayoshi Naito, a core member of the CCoE team, says: “TORO is a shared platform that provides the necessary resources and features required for both application development and operational support in the production environment. This allows project members to focus on meeting their business challenges in a fast-paced environment. We integrated the principles of the Toyota Production System (TPS) to allow project members to focus on improving their value-added work. We are focused on improving the developer experience to allow projects to concentrate on achieving success.”

When the need arises, CCoE team members directly support project teams. Administration rights for the project-side AWS account on the TORO can also be granted directly to the project or application development teams.

“TORO integrates jidoka— Jidoka (Automation with a human touch): One pillar of the Toyota Production System. “Stop the machine when an anomaly occurs. Do not produce defective products.” Human wisdom is imparted to the machine, and any problem becomes obvious to humans when the machine stops, thus giving the automation a human touch.  — automation technology that allows us to provision an AWS account compliant with Toyota's security standards on the platform level in just 2 hours after receiving a project application. Developers only need to upload their code, and the CI/CD pipeline can package it into security compliant containers and deploy to production on the same day. That speed is exactly what we wanted to achieve, and reflects the just-in-time (JIT) principle of TPS. TORO allows you to use the environment you need when you need it for just what you need,” says Naito.

In addition to TORO’s shared features of automatic deployment through a CI/CD pipeline and guardrail-style security, the CCoE team also integrates observability into the platform.

“We envisioned that the project members doing application development would also need to monitor operations. We therefore designed the monitoring environment with greater focus on the application instead of the infrastructure. New Relic not only met our requirements as a cloud native observability tool, it also offered a wide range of other advantages, says Naito.

Leveraging New Relic in the TORO Ops environment

Naito comments, “Enabling improvement of the application through observing the user experience and versatility to handle a multi-cloud environment were some of the key features that led to us to New Relic. New Relic’s licensing model was another key advantage for the CCoE team.”

New Relic features a simple licensing model based on the number of users and volume of data ingested, and offers unlimited use of a wealth of features such as APM, browser, logs and, infrastructure.

“Monitoring needs are different for each project. Even if you offer an agile environment, the licensing can become a major bottleneck if each project needs to make a new contract based on the features they require. With New Relic, projects just need a single license to use all the features, matching our sense of speed,” says Naito.

TORO provides automatic deployment for each project right down to the infrastructure level. In addition to meeting Toyota’s strict security standards, the platform also offers automatic deployment of the New Relic observability environment through an Infrastructure as Code (IaC) model. Yuki Murase, lead designer of the CI/CD Pipeline in the CCoE team describes how TORO leverages New Relic in the deployment process.

“Engineers are focused on development, so they sometimes lose sight of the challenges and concerns in an application’s operation. When we offer automation according to the Toyota Production System (TPS) philosophy, it is our responsibility as the CCoE team to provide observability through New Relic from the very start of the project. We are now working on a continuous application improvement system utilizing New Relic’s monitoring of performance before and after version deployment.”, says Murase.

Shared Security, CI/CD and Observability on TORO

Over thirty projects are currently being developed on the TORO platform. It also hosts numerous live applications and services. Once TORO moved beyond its development phase, the CCoE team began actively communicating what can be achieved by TORO  both within and outside Toyota.

Mabuchi describes his vision for TORO and the CCoE team: “We intend for TORO to be an environment that meets a broad range of business needs. Different from the foundational IT department cloud environment and a cutting-edge development environment in the connected application domain, the role of TORO and the CCoE team is to ensure the necessary resources and features are readily available to allow ideas to become reality as quickly as possible. We will continue to improve the TORO developer experience to allow project members to focus even more wholly on increasing their added value.”

Murase comments, “We provide New Relic dashboard templates and are promoting the use of them. The templates allow teams to drill down into the resources and metrics of their application, starting with the SLI/SLO score. We plan to integrate this information into a business KPI/KGI board, which will be a major benefit to all running projects.”

With its unwavering commitment to automation, TORO has reduced the hours required to set up an new AWS account for production environment by 96%, reducing a process that used to take up to two months to just a few days. TORO has earned many avid fans, and an active community of employees engaged in cloud native technologies has grown up around it.

“Within three years, I want everyone at Toyota to know about TORO, and for it to be the first choice whenever they want to create a digital service using the cloud. I would like to reduce individual department-based cloud licenses to zero. New Relic is a critical component in our DevSecOps cycle. New Relic will allow us to normalize the process of driving continuous application and service improvement through observing the user experience, which is key for any project, and moves beyond setting and forgetting after deployment. “ concludes Mabuchi.

“We need to make greater improvements to the developer experience through TORO to accelerate our full model change toward being a mobility company. We would love to hear from engineers who resonate with our passion and want to take on unsurpassed challenges at the absolute cutting edge of technology.”