How to implement DORA Metrics in 1 day?
In 2010, way before the State of DevOps and Accelerate were published, we were discussing the frequency of releases, failures, recoveries and the time it takes to get something done - in a company in which, we were using an FTP server for doing multiple deployments a day.
I think most DORA implementations waste a lot of time and money for quite a low value of having lots of colourful and fully automated dashboards.
I like metrics. However, fully automated DORA is a low priority on my list.
I see a few common problems when companies decide to implement DORA.
Problem 1: Definitions
We like to theorise more than take action. We may spend half a year and still be at a point in which we are still discussing if:
Is a Lead time for changes the time it takes for a commit to land on production? Or is it the time to get the Jira ticket done, after we move it to In Progress?
Is a Change failure rate more about rollbacks compared to the total number of deployments? Or about customer-facing incidents? Is it a percentage value or a count?
I recommend timeboxing the discussions around the definitions for 1 day and then writing them down somewhere in a source of truth. It might be some knowledge space or ADR (Architecture Decision Record). From that moment, just stop discussing and changing the definitions and focus on the data. You may publish another ADR in the future if you want to change the definition but let’s pause the philosophy for now.
Problem 2: Automation
In my opinion, you can get 90% of the value from DORA metrics without automation.
To get a fully automated dashboard of DORA it takes the time to:
Choose, buy and adjust some SaaS solution. Then you most often need to adjust something in your ways of working to meet the philosophy of a specific tool.
Or, build your dashboard by connecting it with the data from the issue tracker, repository, CI/CD pipelines, observability tools, list of teams and people, and their ownership of codebase areas.
Both of the above solutions may cost from a few thousand dollars to a few hundred thousand dollars a year.
I recommend Google Spreadsheets for the first increment of DORA.
Start with a Spreadsheet
Depending if you want to measure it for 1 product, 1 technical component or the whole organization you may setup rows and columns differently. However, the point is to use one dimension for:
Deployment Frequency
Lead Time for Changes
Change Failure Rate
Time to Restore Service
and the other dimensions for products / systems / technical components.
As a value in the spreadsheet cells, you may use pre-defined drop-downs. For example, in the deployment frequency dimension, you may use “once a month”, “a few times a month”, “once a week”, “a few times a week”, “once a day”, and “multiple times a day”.
Start by filling this table with your leadership. Then start asking the teams to fill it out.
In the next increment, you may want to scale this MVP to more dimensions (e.g. teams), that fit your organization.
Thanks to this simple spreadsheet, from the very first moment, you may start applying the DORA understanding. DORA for me is a perspective, not a dashboard.
If you are an engineering team that starts to use any kind of metrics, and you start with DORA, and you spend a few months to implement a dashboard, then you may get a pushback from some people saying “We knew it without a dashboard that you’re deploying once a week, half of the deployments cause problems and it takes half a day to fix them. Congratz for spending a few months building a dashboard.”.
Start with a spreadsheet. After a few months, if you decide to automate it or buy SaaS - it is fine. However, no matter what you implement, it’s worth applying product management techniques to lower the risks and shorten the time-to-value of DORA.
