If you ask ten engineers how they measure the success of their DevOps strategy you are likely to get ten different answers. Success is measured differently by different people, so it’s not unusual to observe these different perspectives. What is the DevOps value proposition? This blog will explore a number of things to consider when you try to answer the question “How do YOU measure the success of DevOps?”

What is the goal of DevOps?

The most important question you can ask yourself when it comes to measuring the success of DevOps is what are your goals, and most importantly, what are the goals of your company? After all, isn’t it the company that needs to benefit the most from DevOps? Without goals, you have no direction and therefore will never know if you achieved anything. The goal for DevOps is simple: Economic Benefit. If the outcome of implementing DevOps doesn’t increase cash flow, raise share price, reduce costs, increase profitability, or improve some other financial measurement then you need to rethink your plans. Ultimately, you want to identify some metrics or KPIs that support the attainment of economic benefit.

DevOps Metrics and KPIs

Gaining an economic benefit is not a simple chore, and anyone who has started or runs a business knows all too well that achieving profitability, or some other measure of economic benefit, can be a daunting task. To measure success, you need to establish some baseline metrics and KPIs before you start implementing any changes. Establishing baseline metrics and KPIs for your people, your processes, and your technology is the only way you can empirically state that you created an economic benefit for your company. Let’s look at the metrics and the KPIs that can be established in each of these areas.

People:  A company is only as good as its people and only as strong as its weakest link. Therefore, when it comes to DevOps, you need to establish how to maximize the throughput of your staff.  The tricky part here is to develop metrics that everyone in the DevOps team can contribute. You can’t focus on one group in the DevOps toolchain because you will create bottlenecks that will frustrate the rest of the team.

DevOps Team.jpgSomething easy to measure across all members of the DevOps team is activity. Where are all the hours that are being logged, or not logged, going? If you ask most people what they do all day, they really won’t be able to give you an accurate estimate. Therefore, it makes sense to figure out how people are spending their time. There are countless tools out there that can track employee activity. Once you know how everyone’s time is being spent, then that can become the baseline for improvement. You may find out that 64% of the development’s team time is spent fixing bugs, but the industry average is 32%. Now you know you need to improve that number by 50%. There is a good chance that instituting this practice can create a Hawthrone Effect. People will change their behavior and become more productive simply because they are being observed.

One problem to avoid is the “Big Brother” syndrome. Micro-managing everyone is not the goal of measuring activity, and that needs to be communicated clearly. Instead, this can be turned into an incentive program. Since everyone is being measured, you can institute rewards for employees that hit certain milestones or achieve personal bests. These successes can be shared with the rest of the team. It will also help identify areas that need attention through additional training, mentorship, and coaching. Aside from measuring activity, you can also monitor attendance, turnover, response time, and employee satisfaction. These will all assist in increasing the DevOps team’s productivity

Process: When we think about a process we envision all the steps that will be completed for a given task. Those steps can be measured along two scales, velocity and quality. In the case of DevOps, velocity is the time it takes to complete each step in the code delivery cycle, and quality measures the rate of defects for each change. There are many metrics you can measure using these two parameters and here are twelve to start.

Velocity KPIs

  1. MLT (Mean Lead Time)
    How long does it take for a bit of code to get built, tested and deployed?
  2. DCR (Daily Change Rate)
    Number of changes getting committed to mainline and tested per day.
  3. MTTE (Mean Time To Environment)
    How much time it takes developers/testers to bring up a testing environment for verifying each delivered change.
  4. MTTD (Mean Time to Detect)
    How much time passes since the original commit of code until the bug it introduces gets detected.
  5. MTTR (Mean Time To Resolve)
    How much time it takes to resolve an issue after it’s detected
  6. MTTA (MeanTime To Approve)
    How much time it takes to approve and verify a release. (Measured from the moment all released content has been delivered and until the release has passed all the defined test and verification cycles)

 Quality KPIs

  1. BFR (Build Failure Rate)
    Percentage of failed buildsKPI Picture.jpg
  2. DFR (Deployment Failure Rate)
    Percentage of failed deployments
  3. IRFR (Infrastructure-Related Failure Rate)
    Percentage of build/deployment failures related to infrastructure issues
  4. RWR (Rework Rate)
    Percentage of tickets being reopened
  5. ADR (Automated Detection Rate)
    Percentage of defects being detected by automated testing cycles
  6. UWR (Unplanned Work Rate)
    Percentage of unplanned issues

By measuring these and other KPIs, you will determine how well you are performing and can you take action on how to make improvements.

Technology: Eventually, everything that was accomplished with all your people, and the processes they followed, end up running on your infrastructure. If your infrastructure has not been correctly architected or lacks the security and redundancy your business demands, you would have wasted your time. Your DevOps value is lost. There are countless DevOps tools that can measure the performance of your infrastructure. Here are 11 KPIs to consider when measuring how well your technology is working. These KPIs are not meant to be an exhaustive list, and there are certainly many more that can be used, depending on primary business requirements.
  1.  Availability (excluding planned downtime) - Percentage of actual uptime (in hours) of equipment relative to the total numbers of planned uptime (in hours).
  2. Percentage of outage due to changes (planned unavailability) - Percentage of outage (unavailability) due to the implementation of planned changes, relative to the service hours.
  3. Percentage of outage due to incidents (unplanned unavailability) - Percentage of outage (unavailability) due to incidents in the IT environment, relative to the service hours.
  4. Percentage of unplanned outage/unavailability due to changes - Percentage of unplanned outage (unavailability) due to the implementation of changes into the infrastructure. Unplanned means that the outage (or part of the outage) was not planned before implementation of the change.
  5. Percentage of Service Desk Availability - Calculation of the Service Desk Availability over the Reporting Period.
  6. Percentage of availability SLAs met - Percentage of availability Service Level Agreements (SLAs) met.
  7. Percentage of (critical) infrastructure components with automated availability monitoring - Percentage of (critical) infrastructure components with automated availability monitoring.
  8. Percentage of critical business processes not covered by a defined service availability plan - Percentage of critical business processes not covered by a defined service availability plan.
  9. Critical-time failures - Number of failures of IT services during so-called critical times. The critical time is the time that a service /must/ be available, for example for financial systems during the closing of the books (at the end of the month, or end of the quarter).
  10. Critical-time outage - Total outage from critical time failures in IT services. The critical time is the time that a service /must/ be available, for example for financial systems during the closing of the books (at the end of the month, or end of the quarter).
  11. Number of business disruptions caused by problems - Number of business disruptions caused by (operational) problems.

Business Value of DevOps

When we began this discussion about the business benefit of DevOps we learned that the ultimate objective is to derive economic benefit. Since that is the case, we ultimately want to prove that all our metrics and KPIs enhance the economic benefit in some manner. By adhering to DevOps best practices, you win by building it better, faster, and cheaper than your competition. But as you strive for these benefits there are many other benefits that you gain along the way.

Some technical benefits include:

 The entire team wins due to the cultural benefits such as:

Overall the business wins because DevOps principals deliver:

Let’s focus on the business value of DevOps a little more deeply. When you organize your business like an “IT Factory” (Incidentally, we have a blog post coming out about this in the future so subscribing to our blog posts will ensure it’s delivered directly to your mailbox) you are essentially stating that you are going to automate everything. Isn’t that why factories exist in the first place? In this IT Factory, the objective is to turn out product. The faster you do it, the faster you can obtain revenue. This concept is known as the velocity of revenue.

When Apple releases a new phone, people line up to buy the new phone, even though their old one is perfectly fine. Apple enjoys a revenue spike each time they release hardware. You read about it every year. Software is no different. On each release of new software, you can measure the revenue spike. If you release software once per year, you will get an annual revenue spike. By employing DevOps automation tools, among other things, you can release software each month, or more frequently, and generate smaller, but more frequent revenue spikes. These smaller spikes accumulate to more than an annual revenue spike. Why? Between software releases your competition is trying to muscle in on your territory. The longer the delay, the more time you give your customers to get dissatisfied with your product and find a product with the features they want now. Conversely, if your competitor just released a highly sought after feature you can’t release for another four months, you just lost four months of potential sales.

The DevOps value formula is simple: Velocity of revenue increases as software deployment frequency increases. Of course, quality can never be compromised, or all is lost. If you release software continuously like Amazon, Google, and Netflix you essentially eliminate all these revenue spikes altogether because the best version of your software is always available. Now you are streaming revenue daily, and the competition will have a hard time competing with an IT factory as efficient as yours.

In summary, the goal of DevOps is to create economic benefit. To understand economic benefit , you need to take a baseline measurement on every key metric that contributes to economic benefit. Improve these metrics by making everyone accountable and incentivizing them for success. Measure , Improve, Measure, Improve. Now you have a finely tuned IT Factory churning our daily software releases which maximize your velocity of revenue. If you don’t know how to start FP Complete would be happy to show you how. We have helped many companies on their DevOps journey by helping them build their IT Factory, showing them how to run it themselves, and being there when they need us. Our DevOps assessment is a quick way for us to gauge what you need to get started.

If you liked this post you may also like:

Subscribe to our blog via email
Email subscriptions come from our Atom feed and are handled by Blogtrottr. You will only receive notifications of blog posts, and can unsubscribe any time.

Do you like this blog post and need help with Next Generation Software Engineering, Platform Engineering or Blockchain & Smart Contracts? Contact us.