How to Build a DevOps Culture Within Engineering Teams

Imagine your developers are the world’s fastest relay team 🏃 When it comes to building, testing, and qualifying they get around the running track faster than anyone else. Unfortunately for them, the finishing line is hidden somewhere outside the stadium. Welcome to regulated DevOps!

How did they get to be running this impossible race? Well, better tools and working practices have meant a dramatic shift from annual software releases to a world where teams can deploy multiple times every day. But!☝️ DevOps teams in places like fintech, automotive, and healthcare still have mandatory compliance obligations.

And the change management processes that worked for annual releases aren’t working now.

The automation we’ve seen in build, test, and security has not been mirrored in change management. That means it’s becoming a bottleneck for regulated teams and it’s time to ask what we should be doing about it.

How do we bring a compliance culture into our DevOps?

Key takeaways

• DevOps is producing volumes of change that IT can’t handle
• External approvals for releasing software are not only slow but also risky
• Building compliance into your DevOps means faster and better releases

A decade of DevOps - from silos to serverless

It’s hard to believe, but we’ve been doing DevOps for over a decade. In that time the way we build and deliver software has changed significantly. Tools that we already take for granted - Docker and Kubernetes, Jenkins, and CircleCI - are all relatively recent inventions. But can you imagine life without them?

You don’t have to be old to remember when systems were built on large, custom-managed hardware. And when you joined a software organization you’d find yourself siloed by specialty. There were different departments for development, analysis, QA, and ops. The software was delivered once a year, maybe once a quarter if you were at the cutting edge. The change was generally seen as a bad thing. It was large, risky, and rare.

But we soon figured out that smaller, services-oriented architectures and independently deployable systems were much better ideas. We started to develop more dynamic and resilient designs. We realized we could use Conway’s Law to organize ourselves around these services.

And these trends continue with the growing adoption of serverless features and functions-as-a-service in the cloud. In a decade of DevOps, we’ve come a long way.

There’s every reason to believe that the next decade will see more innovation and even higher volumes of change. But how do we prepare for that?

What does DevOps mean for change management and compliance?

One of the challenges we have when it comes to understanding the impact of DevOps on technology organizations is that change management and compliance mean lots of different things.

Managing change and ensuring compliance has always been the overall responsibility of the IT department. They typically consult the ITIL framework for guidance on how to manage change in their IT systems. And there are lots of different types of changes to manage.

There are changes to infrastructure, servers, and hardware. Data migrations and enterprise resource planning. Web updates and performance improvements. Traditionally, all of this was categorized in a very general way as change management.

Compliance is a related and similarly broad term covering a range of topics that are also IT’s responsibility. There’s open source compliance, policy compliance, compliance with industry standards, compliance with internal standards, regulatory compliance, etc.

Change management at the software release

But there’s one aspect of change management and compliance where DevOps has an enormous impact - the software release process. In regulated industries, teams need to know what software they have running in production and how it got there. They need to track and document changes to production to ensure their process complies.

Naturally, software releases were also part of the IT department’s change management responsibilities. And that made a lot of sense in the days before DevOps. You’d batch up a bunch of changes, make a change request to a change advisory board (CAB), get their approval, and then release the changes over a holiday weekend in case everything went sideways. There was a lot of risk involved and a prolonged, manual approval process to mitigate it.

And it’s somehow worse than no change process at all

But these processes aren’t even good at mitigating risk. Thanks to extensive DevOps research in Accelerate by Dr. Nicole Forsgren, et al. we now know that:

“External approvals were negatively correlated with lead time, deployment frequency, and restore time, and did not correlate with the change failure rate. In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change failure rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all”

So, not only slow but risky too. Factor in the increasingly dynamic ways we have of delivering software and it’s clear that the old ways of managing change are not only inefficient but potentially disastrous.

DevOps automation meets manual change processes

DevOps, more specifically continuous delivery, produces volumes of change that the IT department was never designed to handle. How is IT supposed to effectively manage software releases when DevOps teams can deliver hundreds of changes every day? 🤔 How is it supposed to ensure compliance in any meaningful way when it’s using manual processes to rubber stamp highly automated software delivery pipelines?

No amount of requests, manual approvals, CAB meetings, and paperwork can manage change at these volumes. As we’ve explored in a previous post: ITIL just doesn’t work for DevOps.

So, what if we take responsibility for managing software releases away from the IT department and give it to DevOps teams? 🙌 Let IT do what IT does best - slow and risky tasks like managing servers, migrations, and hardware. For dynamic ways of delivering software, DevOps teams need change management automation.

Building a DevOps culture around compliance

Modern technology organizations do not have traditional separations between software development, quality assurance, and IT Operations. That has been replaced by cross-functional DevOps teams responsible for the entire value stream of software systems. This new DevOps approach allows organizations to reduce handovers, increase collaboration, and ultimately deliver more innovation.

And as organizations change, the supporting technology approaches also change. Companies are adopting metered cloud infrastructure as a service, automated build, test, and security tooling. With the support of this automated DevOps delivery, high-performing teams can deliver changes 973 times more frequently than traditional teams. 973 times!

If we need to be compliant, but external approvals and management don’t work, what should we do?

Continuous Integration > Continuous Delivery > Continuous Compliance

This vast increase in the number of changes needs new and better approaches to change management and software process compliance. To support the efforts of DevOps teams, we need to move away from manual and gated checks towards continuous, automated checks. We need to take the lessons we learned about continuous integration and continuous delivery and apply them to compliance.

In this way, we can not only achieve more flow in our work but actually get better compliance 🙌

We have seen in practice that the adoption of Continuous Compliance methodology leads to the same sort of benefits we’ve seen with test automation.

Adopting test automation doesn’t make testers obsolete. Instead, it removes monotonous and repetitive work, freeing testers to work at higher-level exploratory testing.

In the same way, Continuous Compliance does not mean the end of compliance work but enables compliance officers to work on higher-value compliance investigations.

Conclusion - the benefits of a DevOps Compliance Culture

The DevOps principles around culture, automation, lean, measurement, and sharing can help teams increase compliance activities while reducing delays and waste. Overall, the benefits of adopting a DevOps Compliance Culture are:

• Culture: gives teams agency to lead risk management responsibilities

• Automation: drives higher compliance conformance and reduces waste

• Lean: results in continuous improvement in risk control posture

• Measurement: ensures compliance becomes data-driven

• Sharing: makes compliance work visible to empower compliance and security functions

  
  

DevOps teams need to build a compliance culture, especially if they’re releasing software in a regulated industry. We can see that DevOps is producing high volumes of change that IT can’t handle. And we know that relying on IT for change management during the software release is both slow and risky.

If regulated teams want to release software with speed and compliance they need to automate change management as part of their DevOps.

Also Published here