Case study: a $17 Million DevOps Success Story

Customer success story

Our Impact

  • $17m cost saving (and growing)
  • Over 25,000 engineer days saved
  • New infrastructure provisioned in seconds (not days)
  • Developers onboarded in 30 minutes
  • Improved technology transfer in business
  • Best practice and security embedded in all processes

The challenge

Our client, a leading Oil and Gas company, had formed a large central IT capability after many years of mergers and acquisitions. The team had an established on-premise capability, but it was made up of multiple siloed teams and disjointed processes.

The company had been progressing a modernising cloud-first approach but had duplicated the organisational and process-based model they had on-premise during the process.

The resulting cloud offering improved provision times from 4-6 months to 4-6 weeks but there were still huge inefficiencies that were limiting the ability of business focussed development teams to deliver value quickly.

The head of the trading division product development teams asked RiverSafe to provide a secure, resilient, consistent and scalable solution that would empower developers in their business segment.

The customer requirements were:

  • To be able to provision our own ‘infrastructure’ without relying on handoffs to multiple teams.
  • To accelerate the infrastructure process. “We want hosting of services in minutes not weeks.”
  • To improve our time to market for our products.
  • To minimise downtime.
  • To be able to experiment.
  • All of this with higher levels of embedded security, higher quality and higher resilience.

The solution

RiverSafe was engaged to help resolve these frustrations and pain points through the implementation of a DevOps approach.

We started by value-streaming the existing process, creating automation to replace manual processes, and sizing the application estate to determine what their application needs were.

We chose a kubernetes cluster as our target architecture. In this instance, Openshift was the best fit for their needs as most of the existing application estate uses RedHat products. We worked in two teams – one building the ‘platform’, one building a pipeline that could deploy assets into the platform in a secure manner.

The platform team:

The container platform team delivered a platform that had high availability embedded. This provided the capability to significantly reduce downtime by allowing Kubernetes and our terraform modules to scale up and down nodes, pods, and routes automatically based on events occurring within and outside the cluster.

Chaos testing was embedded to ensure that previously catastrophic events could be created to ensure service architecture could accommodate these failures within the stated non-functional requirements.

The pipeline team:

In the previous working model, developers were using various tools, leading to a lack of control, and making the overall management of the process very difficult.

To resolve these issues, the pipeline team delivered a reference Continuous Integration & Continuous Delivery (CICD) pipeline. This included all the DevOps best practices and ensured fully automated build and deployment flow through environments to production, including security scans, testing and automated change processes.

With this pipeline, developers can quickly and easily provision services and components for their applications in just seconds (rather than weeks). Developers have the assurance that they are consistently delivering compliant and tested code which has undergone stringent security scanning at each stage of the process.

The team also worked to automate the automation. We built a self-service process to allow new users with a new pipeline along with automatic onboarding to all of the DevOps tool chain. This includes accounts and permissions allocated within the container cluster stored in a secrets vault.

Where this process for onboarding new teams and users used to take four weeks it now took 30 minutes.

This new approach is repeatable and consistent and ensures that new team members and projects have the tools they need right from the start, without unnecessary delays or wait times. It also means that if changes to a particular tool are needed, these can be rolled out to all existing teams with minimal impact on project timelines.

With each of these solutions, we started with an initial proof of concept and implemented changes in a small area, iterated to make sure we improved, and then rolled out to the wider business, improving all the time.

The outcome

The impact on the business of these changes has already been huge. The previous time wasted for developers to get the tools and provisions they needed has been eliminated with infrastructure able to be spun up in seconds and new developers able to start building code in under 30 minutes (as opposed to several weeks).  Over the course of the project, this has meant a saving of more than 25,000 engineering days to date.

The standardisation has meant that managing the software delivery process has become much easier. Prior to the introduction there were numerous support requests to the tools and platform teams due to software build issues. We provide well-documented standards and, as everyone was working on the same pattern, the technology transfer across the organisation was vastly improved.

As a result, all of these support requests stopped and the teams could focus on the operational incidents impacting container platforms and tooling.

Currently, 735 services have been onboarded which spanned 56 projects and 1074 repos. Our pattern was applied to 1608 pipelines and those pipelines were run 235,000 times.

But key for the business has been the cost savings which so far have amounted to over £17m over a three-year period. And are still increasing as this is rolled out further within the business.

Book a consultation

Get the perspective and insight you need to create unified solutions and make informed business decisions.