Most people who have worked in software development for any length of time have probably heard of DORA, or the State of DevOps report they publish annually. And for those that haven’t, DORA is a research program that provides insight and guidance into how to measure and improve your software engineering maturity. 

Most of the headlines are given to what they call the ‘4 key metrics’ (although a 5th was later added) for measuring the performance of software engineering teams: 

1. Deployment Frequency – how often a team delivers changes to production 
2. Lead Time for Change – how long it takes a commit to get to production 
3. Change Failure Rate – how often a change to production causes a failure
4. Time to Restore Service – how long it takes to recover when a failure occurs 

But the full report goes into much more detail than simply looking at 4 metrics, and there is a lot more besides that we at RiverSafe have seen contribute to maturity in software development, and DevOps.  

In this post we’ll go into much more detail on some of the things, beyond the 4 metrics above, that indicate the performance of teams and organisations. We’ll also provide you with some actionable insights that you can take away and use to improve your own team or organisation. 

User-Centricity 

Key Metrics: value delivered directly to users, percentage of teams incorporating user feedback into backlog planning, and percentage of SLIs focused on enhancing user experience and satisfaction.

Prioritising the wants and needs of product or service users is essential to organisational, operational, and team performance. While it sounds straightforward, it is often challenging—and sometimes unnatural—for many organisations, where requirements are typically set by individuals within ‘the business’ who may lack a clear understanding of users’ actual needs.

To be user centric, development teams must have mechanisms for capturing qualitative and quantitative user feedback. They must be able to measure and track the things that matter to users. And they must have processes that use this data and information to inform the feature backlog, and the priorities given to each feature.  

A mature DevOps organisation has a high proportion of teams that use user feedback and Service-Level Indicators (SLIs) to inform their backlog of work, and to prioritise it. 

To fully support this approach, the value delivered to users should be tracked and aligned with business targets, allowing for demonstrable value generation. This exercise is valuable in itself, as it prompts teams to explicitly consider how business objectives relate to user needs and satisfaction—and whether adjustments are necessary if they do not align.

Tips for Success: 

• Think explicitly about how you will understand the wants and needs of your users, and how you can measure and track them. 

• Align with ‘the business’ on measures of value that relate to what users care about. 

• Enable teams to be able to implement them features they feel would most benefit users. 

Application Deployability and Loosely Coupled Architecture 

Key Metrics: % of services using shared databases, rate of synchronous vs asynchronous cross-service communication, error propagation rate 

Application deployability is a strong indicator of software delivery performance with a high level of deployability being a good measure of how mature a team or organisation is in its DevOps practices. 

The independent deployability of an application describes how easy it is to deploy in isolation, and that last part is key. Being able to deploy an application is generally, technically, quite straightforward, particularly if you have an efficient CI/CD process.  

Being confident that it will work as expected in an environment where it interacts with many other upstream and downstream services is a completely different matter however. 

If an application has dependencies on upstream applications, or applications share a database or data model, then is it possible that changes may be needed in those applications to facilitate a change. This also means that certain applications may need to be deployed together, making each of them less independently deployable. In turn this will delay releases, or make it more difficult to conduct frequent releases, and value is slower to be realised. 

The antidote to this is to move towards a more loosely coupled architecture in which there are as few dependencies as possible between applications. This is too big of a topic to cover in a single post but some of the things that should be considered to aid in this: 

• Use standardised APIs to provide an interface to an application and don’t introduce breaking changes to the API interface 

• Consider the use of asynchronous communication methods, such as Events or message queues to negate the need for hard dependencies between applications. 

• Don’t share databases between applications 

• Treat the front-end, back-end, and data for a piece of business functionality as a single application or service. 

Getting this right leads to applications that can be deployed independently, allowing developers to be much more productive, and work with a higher level of satisfaction. 

This all only addresses technology concerns though. It is equally important to consider how the development teams are structured. Like applications, creating dependencies between teams leads to bottlenecks, handoffs, and slows down software delivery.  

Teams should be empowered as much as possible to make design decisions about the products they are building, build them, test then, deploy them, and run them. This means centring teams around products, rather than business functions, to reduce handoffs, and increase alignment. 

An organisation’s DevOps processes can be considered more mature if: 

1. Applications can be deployed and run independently of one another 

1. Teams have the autonomy to move from ideation (using user feedback) to operation without going outside the team other than in exceptional circumstances. 

 Use of Cloud Service 

Key Metrics: % services managed by Infrastructure as Code (IaC), service downtime %, average operational overhead cost / service, resource utilisation rate 

Cloud services, such as those provided by AWS, Azure, or GCP can be great enablers for enhanced software delivery performance. We’ve gone from a world where it could take weeks or months to provide infrastructure to one where it takes minutes, and can even be automated.  

Too often though, moving to The Cloud has been viewed, implicitly, or explicitly as ‘enough’. Organisations often take what they have, create some networking infrastructure, and virtual servers and deploy everything there.  

Most of the time though, they realise that they have derived little benefit from doing so, and, actually, there is more overhead for them because now they have a bunch of servers to patch, and networking to manage and debug.  

The real advantage of The Cloud is the scalability, reliability, flexibility, and level of automation that it can bring, if you architect around these principles. 

Scalability – All major cloud providers provide, for most organisations, virtually limitless scalability. Want to scale your application up to run simultaneously across 100 VMs instead of 10, great, you can do this almost instantly. The problem is that you don’t get any of these advantages if you simply do what we’ve described above. To benefit from The Cloud you need to take the time to understand your applications, their performance, and define a set of scaling rules that will allow performance issues to be automatically remediated through scaling. An additional scaling benefit introduced by cloud services is the API-Driven approach they have taken. This has allowed Infrastructure as Code (IAC) tools like Terraform to create, update, and destroy cloud resources quickly and easily. 

Reliability – Similarly, Cloud services can give you a level of reliability that just can’t be achieved using a traditional data centre. Cloud Providers have groupings of independent data centres that your services can run in. Again, simply migrating services to the Cloud won’t make them any more reliable than running them in a data centre. To achieve that you need to configure applications to run in parallel across multiple physical data centres (sometimes called ‘Availability Zones’ or ‘Regions’). 

Flexibility – There are an almost mind-boggling number of choices of Cloud services that can be used to host your applications. Virtual Machines, Containers, Kubernetes, Event-Based systems, Message Queues, Function-as-a-service, hosted APIs, the list goes on and on. And then there’s a myriad of security tooling, data streams, database, and ML/AI services. You get where I’m going with this. Running applications on VMs in the Cloud is not much more flexible than doing so in a data centre. Again, understanding the requirements and profile of your applications can help determine the optimal service/s and configuration for running them. Automating configuration settings is also very easy in the Cloud, and putting in a little bit of effort to think about what changes you might want to make under certain conditions can have a bit payoff. 

Getting all this right can have a massive impact on your ability to deliver and run applications and services. Done right, the use of Cloud can massively reduce the load on teams in operating running applications, allowing them to focus on value generating activities. It can also help deliver a better experience to your users and customers through improved application performance. 

Tips for Success: 

• Go Serverless First – consider serverless as a first option as it often provides for easy scaling options and removes most of the overhead of managing servers etc  

• Understand the key performance metrics for your Cloud applications, monitor, track and find ways to improve them. Make decisions like what architecture to use to host your application based on those metrics. 

• Build in time and capability to go beyond ‘lift-and-shift’ – make sure that your Cloud strategy has continuous improvement built in. 

• Friends don’t let friends do ‘ClickOps’ – use Infrastructure as Code (IAC) tools like Terraform to create testable, auditable, repeatable, and scalable cloud environments. 

 To sum up, although the 4 main DORA metrics can provide great insight into a team, or organisation’s DevOps maturity, there are many things beyond them that indicate DevOps maturity.

The extent to which your teams and organisation take a user centric approach to software engineering strongly impacts organisation performance in terms of value generation. Application Deployability can have a huge impact on productivity, which in turn influences things like agility and time-to-market. And how you use Cloud has an outsized influence on both productivity, and technical debt and developer burnout. 

I hope this has given you food for thought and things that you can action in your organisation.