Continuous Integration: A Merge Story

The real story of how a 6 months project became a Vaporware

Listen to the audio version!

Last time I wrote about Continuous Integration: A War Story. The post describes the Anti-Tank Dog project from World War II and how they only discovered its problems when it was too late.

That reminded me of a real software development situation from the past.

It was at the beginning of my career.

Many years ago…

I was working on a company that built enterprise software for car dealership. It was designed to help with the sales.

Cloud service providers like AWS. didn’t exist. The customer was responsible to buy computers and run the software in their own intranet. Some of them also had to pay a dedicated IT department to configure and maintain their network.

The system was web-based. It was built in 2003 and carried over for a decade.

In the front-end, each part of the screen was created inside an iframe. JavaScript was only used for basic interactions.

In the back-end, we used the Java Platform, Enterprise Edition. All the classes and packages were tightly coupled to each other and there was one PostgreSQL database.

It was a monolith.

There was a Java class so big Eclipse would freeze when it was opened.

Many years ago I worked in a monolithic application that had a Java class so big Eclipse would freeze when it was opened.

One of the high-level modules of this system was the Customer Module. I was tasked to change the design of it and add a lot of requested features that were piling up on the backlog.

The Customer Module was very messy and impossible to extend. I suggested to rewrite it using jquery U.I. so that it would allow us to modify the design by changing one CSS file instead of many and add the requested features.

The task didn’t look simple, but I was confident things would eventually be sorted out. I started changing the page inside the biggest iframe bit by bit and then changing other places that were related to it until the whole module was rewritten.

The one thing I didn’t know was that the page I was changing was coupled to the footer which was another iframe. It was also connected to other pages inside other iframes, which were connected to each one’s listing screen inside of yet another iframe!

The Customer Module was big and very messy. Yet, I suggested rewriting all of it believing that things would eventually be sorted out.

Oh, did I mentioned there were no tests?

10 years of code and it was impossible to verify the functionality. Even those who were there since the beginning didn't remember the use case for many parts of the system.

Despite all of that, I followed up into this endeavor. I was the only one who had the courage to do this rewrite. The Hero, in this case, wasn’t the bug fixer, it was me who naively jumped right into the big monster without knowing the implications.

I definitely got some trust from my peers due to what happened a few months back. I did the same jump to rebuild a small scheduling system into a more feature complete one in JavaScript. From the point of view of the customer, that rewrite was successful. Although, in retrospect, I can see how it had a lot of maintainability problems.

Everyone, including myself, were confident that it would work out the same way.

But it didn't.

The Customer Module had no tests and nobody had the knowledge about how the system worked. Yet, I suggested rewriting all of it believing that things would eventually be sorted out.

We were using CVS at the time. I had written a lot of code and it was reasonably “working” on my local machine.

After some time, I was starting to get asked of how things were going and how much was left. I always repeated the same thing over and over again:

It’s not possible to predict how long it’s gonna take, but it’s going fine.

And in my mind it was!

The system was "working" for the standards at that time, even though there were no tests. The mindset was that maybe I missed some use cases here and there, but they would be rediscovered after some time and I would fix them.

Then 6 months passed…

In the beginning, I started integrating the main trunk into my development branch. However, nobody committed small changes every day, most of the time it was a big change once the work was finished. I was wasting a lot of time solving conflicts instead of coding, so I gave up on it after a couple of months.

In the end, I realized the rest of the team had developed a lot of other features on top of the same codebase. The number of conflicts that would come out of it and the number of features that would have to be rewritten made any merging impossible.

Besides, at the same time, another colleague released a mobile web version of the Customer Module. Many of the business rules that were coded in the new Customer Module were also coded in the mobile web version. Features in the desktop would have to be fully rewritten in mobile, duplicating the work.

It was a nightmare.

But here's the interesting thing: Management wasn't expecting this.

Their expectations were that the mobile version would easily adapt to the desktop needs and the new Customer Module would be released successfully.

However, things didn't go as planned.

The Customer Module project became a Vaporware and was dropped, just like the Anti-Tank Dog. It couldn't be integrated into the main trunk.

Also, the new features were not going to be easily adaptable for the mobile version.

6 months of work were just wasted.

The Customer Module became a Vaporware and its features couldn't be easily ported to the mobile version.

This is a clear example of how important Continuous Integration is.

In the beginning, I tried to merge the main trunk into my branch. However, because there were no small changes committed into the mainline from other team members, that was very hard to keep up.

If I had merged every day something into the main trunk, it would have forced me and the rest of the team to deal with the conflicts early instead of letting them accumulate.

It would allow everybody, including management, to discover earlier that the code was incompatible with how things were done in mobile. That could make them change the overall strategy in the beginning instead of waiting until it reached a state where nothing else could've been done.

You can call it Continuous Integration, Trunk-Based Development, "Release early, Release often", "satisfy the customer through the early and continuous delivery of valuable software", or any other generic buzzword.

In summary, here's the point:

The only way to avoid irreversible code conflict is if everyone tries to merge as soon as possible. Preferably many times on the same day. If anybody takes too long, they'll only see the problem when it's too late.

At that time I didn’t have the knowledge I have today. Today things would be done extremely different and none of these problems would happen.

I made the wrong call. And I’m the one to blame because of that. Nobody else.

In retrospect, besides continuously integrating the code, I can think of a couple of small things that could have been done differently to improve that process.

I could have avoided individualism.

In that team, nobody practiced Pair Programming or Mob Programming. Nobody had enough experience to see the benefits.

If I had suggested and experimented on that or at least understood the communication benefits, then we would probably be aware of what everyone was working on. We would have discovered earlier that the Customer Module rewrite couldn't easily integrate with the mobile version and vice-versa. Maybe that would make us change our strategy and ways of coding.

I could have thought more before coding.

Code less, think more. That’s the kind of counter-intuitive mindset for those seeking the title of a "hacker". If you only think about coding, you can only become a Cowboy Coder.

That's exactly what I was.

I could have optimized my learnings not just for programming, but also for how software development teams work. As a developer, I am the professional that's going to help to shape the organization and I have to be accountable for things that go wrong.

I ended up blaming the process. I ended up blaming the scalpel.

If you have never experienced these problems, then congratulations. You're in a privileged position for having learned, through somebody else's failures, one of the most important things in software development: don't take too long to commit to everybody else's work.

You're not being smart if you're able to solve a complex problem. You're being smart if you can learn from somebody else’s mistakes in order to avoid the complex problems from happening in the first place.

The internet is full of success stories and people that are afraid to expose their failures.

I’m not.

A failure is as good as a success, with the difference that now you have something to learn from it.

I have not failed. I’ve just found 10,000 ways that won’t work. — Thomas A. Edison

No software developer want 6 months of work to be wasted, and I have learned this the hard way.

Hopefully, this will help somebody else not to waste 6 months in something that could have been prevented in 6 minutes.

Hopefully…

See also a technique for how to prevent these problems from happening.

Thanks for reading. If you have some feedback, reach out to me on Twitter, Facebook or Github.