Bye Bye DORA: Flaws of the State of DevOps Reports

*(final reply…)*

\
> "No, the outcome is organizational performance, not speed."

\
The book is literally called Accelerate and the outcome metrics are time-based. 😂 

\
"organizational performance" actually has to be defined, in the case of the Four Key Metrics, it's defined by speed.

\
I think it would be far more beneficial for you if when you see definitions you actually think through how they're defined in the literature. 

\
If someone claims to be measuring "organizational performance", actually think through the scientific methodology for how that's measured rather than just taking the claim at face value. Many (if not most) flaws in scientific work come from flawed methodology. This kind of critical thinking is really important when it comes to evaluating research like this.

> "This entire argument is a strawman..."

\
If you read the questions asked, you'll see:

a) they are clearly based on survey questions

b) there is clear scope for cognitive bias as the people who feel better and are more proud about their place of work are undoubtedly going to select "strong agree" to the questions asked, regardless of what dimension they measure.

\
In referendums and opinion polling work, a huge amount of effort goes into ensuring one-dimensional questions don't bias people in a certain direction - that becomes even more tougher when you're trying to correlate between responses to different questions.

> "That’s incorrect. They are indications of batch size, quality process, and stability. Your misunderstanding indicates to me you’ve not read the source material and are assuming from a point of ignorance."

 \n Let's run through them:

* **Change Lead Time** - Time-based
* **Deployment Frequency** - Time-based, as frequency measures a period of time
* **Change Failure Rate** - Time-based, as the rate is measured with total deployments in a given period of time as the denominator
* **(Mean) Time to Recovery** (*MTTR*) - Again, clearly time-based

\
In science, it matters how the research methodology actually measures - not what the authors purport it measures. You claim these measures are of quality and stability - please provide actual evidence to support this claim. Outline the causal links which connect these metrics to those outcomes; "quality and stability" are vague words - when I've conducted representative opinion polling on both the public as a whole and software engineers, they agree that the metrics that matter are those where you'd require some serious mental gymnastics to connect to any of the key outcomes measured by DORA surveys. (And yes, I have read the source material at significant length over a number of years, I have just done so being intellectually critical of the claims which are put forward and not taking them at face value.)

\
*(Continued in next reply…)*

Sorry, I don’t think you actually understood what I meant here. I’m talking about the data tables themselves being made public.

\
For the State of DevOps Reports, the questions will inherently be public because the questions are collected from an online poll anyone can answer. What I’m talking about is actually the computer tables the results are based on.

\
BPC rules mandate that the data tables be shared when research findings enter the public domain, and you’ll find that all UK polling firms who are members follow this practice. See section 2 here: <https://www.britishpollingcouncil.org/objects-and-rules/>

\
You’ll find polling results of any of the studies I’ve done have the raw tables in the annexes of the full report and the results are also published by the polling firm, e.g. J.L. Partners here: <https://jlpartners.com/polling-results> 

Personal Site

Nominated for 2022 - HackerNoon Contributor of the Year - Coding Skills

Nominated for 2022 - HackerNoon Contributor of the Year - Software Architecture

Nominated for 2022 - HackerNoon Contributor of the Year - Code

Dr Junade Ali