Continuous refactoring — what’s stopping you?

Many times over the last twenty years, I’ve had to work on codebases that have made my job difficult and painful. Small codebases, large codebases, small companies, large companies, media companies, tech companies … it keeps following me around. Let’s just call them “messy codebases”. Talking to other developers about this subject always results in people sharing their own experiences, like a kind of therapy session. Google “bad code” and you’ll get back endless results on this topic. So it isn’t just me.

Fixing the code doesn’t fix the habit

A recent Twitter thread by Sarah Mei got me thinking about why messy code bases are so common. I’ve seen people “fix” a codebase by doing a rewrite or large scale refactor. This makes the codebase easy to work in for a while, but soon it starts to get messy again. Sarah describes this making an analogy with a US TV show called “Hoarders” which Wikipedia says “depicts the real-life struggles and treatment of people who suffer from compulsive hoarding disorder.”

“Do you know why Hoarders isn’t on the air anymore?

It turns out that hauling everything away and cleaning up the house doesn’t fix people’s habits that led to the hoarding. Most of the show’s [participants], after the show was over, slowly went back to a hoarded house.

Our hoarded codebases work the same way. If you don’t change the habits and incentives that led you to that point, you’ll end up with a tangled mess …”

Most developers with a few years of experience will be able to identify a messy codebase when they see one. But it’s harder to identify the aspects of the company culture that results in codebases typically ending up messy. Sarah pinpoints the most obvious sign that you can look for:

“The most common “presenting” pathology in the hoarded codebases I’ve seen — by far — is that developers don’t feel they have time and/or permission to refactor code.”

Based on my experience, I would say that’s accurate. I would summarise it by saying that the company’s culture undervalues the benefits of refactoring.

Making the case for refactoring

How important is refactoring though? I’ll leave that to a few people who are a lot smarter than me to explain.

First up, is British programming “celebrity” (my term) Martin Fowler. In his excellent talk about agility and refactoring he says:

[Refactoring is] a very core technique to the whole agile way of thinking because it fits in with the whole way in which we can build software in a way that it can change easily … Refactoring is central to this because refactoring is a disciplined way of making changes. ”

In “Leading Lean Software Development”, Lean “masters” (probably my term) Mary and Tom Poppendieck are blunt not just about the need to refactor, but the need to refactor continuously.

“Code must be kept simple and clear, even as developers understand the problem more clearly and as the problem itself undergoes change. Therefore, code must be continuously refactored to deal with any duplication or ambiguities that were discovered through deeper understanding or introduced by the latest change. An increasing number of tools are available to assist with refactoring, but tools are not usually the issue here. The important thing is that developers must expect refactoring to be a normal part of their work. Like continuous testing, continuous refactoring is not optional.”

Thanks to Agile and Lean practices, the concept of continuous testing seems to be reasonably common these days. But continuous refactoring? I’m not so sure that is.

I’ve worked in a few teams where refactoring comes naturally and normally to developers. I worked in many more teams where it doesn’t. Back to Sarah Mei — she talks about “habits and incentives” as the cause of the absence of refactoring, which is a great description of what culture is.

Anti-refactoring culture

From my own personal experience, I’ve identified four different aspects of this anti-refactoring culture, starting from the most authoritarian.

1. 
   

   Explicit preventionThe need to hit deadlines or churn out features means that developers are told to ask permission to refactor, which is rarely granted. Refactoring tasks are typically allowed to be created for future prioritisation, but these are rarely completed. Developers might do sneaky refactoring here and there where it seems closely related to the change they’re making.

2. 
   

   Too much flakIn Herman & Chomsky’s (news) propaganda model, “flak” is described as “efforts to discredit organizations or individuals who disagree with or cast doubt on the prevailing assumptions”. While nobody is explicitly telling developers not to refactor, it takes developers a large amount of effort and courage to get refactoring tasks complete because it goes against the culture. Developers may be asked for excessive justification of the need for a refactor. Refactoring might be singled out as a cause for delays, rather than being seen as an essential part of a solution.

3. 
   

   Analysis paralysisWhen a code base is already really messy, refactoring seems futile — how is tidying up one part of the code going to help? Or perhaps the architecture is so messy that improving code quality elsewhere isn’t going to address this huge architectural problem. Perhaps there are too many options on where to start, so people go around in circles trying to find out where to start instead of just starting.

4. 
   

   Lack of developer experienceSometimes there’s a culture where developers are encouraged to refactor, but they simply don’t because they’re unable to identify messy code. A lack of technical leadership means nobody is helping these developers to learn how to identify messy code.

Implementing cultural change

What we’re talking about here is the need and the challenges around creating a culture of continuous refactoring. Cultural change often needs to be “top down”. Here are some thoughts on how that could happen:

• Firstly, management/leadership need to be explicit about the need for refactoring and the benefit it brings, creating a vision for the change.
• Secondly, managers/leadership need to give developers autonomy to refactor as is needed, without receiving flak for doing it. In fact, refactoring should be openly praised to show alignment with the vision.
• Thirdly, developers need to be given opportunities to learn and up skill. They need to be confident that they can identify messy code and have strategies around how to refactor it.
• Lastly, once the vision is clear and understood, praise is open, developers have autonomy and the skills — then managers need to hold developers accountable for the quality of refactoring teams are doing.

It’s a long road, but cultural change doesn’t happen overnight.

Do you feel your team refactors enough? Do you identify with some or all of the anti-refactoring causes above? It would be interesting to hear from people who have managed to turn around anti-refactoring cultures.