How To Use Scrum With More Productivity And Less Pain

Dealing with Software Development is not the cleanest and easy thing. It encompasses a myriad of issues, technical and non-technical. The technical aspect is surely much more complex and needs hard skills, but at the same time can be managed in some way, using the right tactics and tools. The non-technical world on the other hand has much more degrees of freedom. It has the same variability and unpredictability as human nature.

As with any other product manufacturing process, a number of best practices have been put in place and tested since the early years of the software development adventure. The agile methodologies are a set of different ways to cope mainly with extreme variability of the requirements, and also the lack of focus on the most important goals to deliver the final product.

Sometimes such methodologies go beyond their original purpose and the net result is far from ideal. Scrum is one of these methodologies. It's described more as a framework. It is based on a basic set of principles, rules, events, and roles. We discuss in this article how this framework can be used with judgment and flexibility and avoid some strict interpretations that could be far from ideal.

Very Briefly Introduction to Scrum

The Agile methodologies are based on the following general principles, defined in the so-called "Agile Manifesto":

• "Individuals and interactions over processes and tools"
• "Working software over comprehensive documentation"
• "Customer collaboration over contract negotiation"
• "Responding to change by following a plan"

(Source: Agile Manifesto)

According to the Agile Manifesto, in the above statements, while the items on the right are important, those on the left are more.

From the standpoint of the development process, the agile methodologies all imply an iterative and incremental process instead of the classic Waterfall model: the whole development is made of a number of incremental steps, and each step is made of the same phases that characterize a whole waterfall project: a gathering of requirements, analysis, design, and coding. That is to say, each step represents an increment in the whole development and can be seen as a whole project by itself.

The Scrum methodology can be seen as a particular declination of the above principles. Scrum is a framework within which a team can use its own processes to develop some specific software product. It is characterized basically by Roles, Events, and Artifacts.

The Roles are:

• The Team: it can include analysts, developers, testers, and in general every kind of professional involved in the project. It's supposed to work in a self-organized manner within the boundaries of the Scrum planning sessions.

• The Scrum Master: he/she coordinates all the Scrum processes, organizes the meetings, and so on.

• The Product Owner: he/she has the responsibility for the product. He/she manages the so-called Product Backlog, which contains all the required features expressed in a clear way. He/she can discuss them with the team, and receive help from it, but he/she is the sole responsible.

  
  

Events are:

• The Sprint: represents a single increment in the iterative development. It has usually a duration from two weeks to one month.

• Sprint Planning: it's a meeting with a maximum duration of 4 hours for Sprints of two weeks and a maximum of eight hours for Sprints of one month. It is organized and scheduled by the Scrum Master and includes the Team and the Product Owner. In this meeting, a number of the features of the Product Backlog are selected, evaluated, and discussed to be developed in the current Sprint. Features are selected based on their priority.

• Daily Scrum Meetings: these are daily meetings of no more than fifteen minutes. They are scheduled by the Scrum Master. In these meetings, each team member describes what he/she has done to implement the current tasks, any problems that occurred, and how to overcome possible difficulties. The Scrum master takes care of the scheduling and coordination of these meetings and their correct execution.

• The Sprint Review: it's a meeting at the end of the current Spring. It takes two hours for two-week Sprints and four for one-month Sprints. It's organized by the Scrum Master and the participants are the Scrum team, the Scrum Master, the Product owner, and all the required stakeholders as per the decision of the Product Owner. The current increment is discussed. What went well and what went wrong is outlined, and at the end, the Product Backlog is updated if needed.

• The Sprint Retrospective: it's an hour meeting for two-week Sprints and two hours for one-month Sprints. It takes place just after the Sprint review and before the next Sprint planning. During this meeting, based on the experience of the last iteration, all the actions to improve and add quality to the final product are discussed and planned for the next Sprint.

  
  

Artifacts are:

• Product Backlog: it's an evolving document containing all the requirements of the product. Each item is called a User Story and has attributes such as description, order, estimate, and value. The items with the higher order are clearer and more detailed.
• Sprint Backlog: it contains the Product Backlog items selected for the next Sprint combined with the necessary planning.
• User Story: as we said above, a User Story is an item of the Product Backlog. A User Story follows a common textual structure: "As a <kind os user>, I want to <do some task>, So that <I can achieve some goal>. It also should have a number of Acceptance Criteria, each one with the following schema: Given <some pre-condition>, When <the user performs some task> Then <some system status change occurs>.
• Burn-Down Chart: it's a chart in which the horizontal axis represents the time and the vertical axis the amount of work remaining to be done for each step in time. The amount of work can be measured by so-called Story Points. The Story Point is a measuring unit of work that takes into account all the variables that can influence the development work, such as skills, complexity, tools, and so on.
• Increment: it's the total increment so far, i.e., the sum of the Product Backlog items completed so far plus all the other items already completed in the past iterations.

Don't Confuse Scrum with Some Form of Doctrine

You can consider the above-listed items as a base by which a set of professionals can operate. They can be seen as a useful tool but what really brings cohesion, intercommunication, and effectiveness in a project are the people themselves. The fact is, even with all those prescriptions strictly followed, and all the commitment by the Scrum Master, a project can drift into chaos and fail miserably.

This is because people often confuse rules, methodologies, and frameworks with some sort of divine engine supposed to drive humans to the right path. It is a common psychological flaw. A very important consideration is that reality is different from theories, and it is a very complex and wild animal. If you think you can tame it with a list of rules and patterns you are doomed to failure.

The real purpose of Scrum is to minimize the amount of "background noise" in the development process and improve the focus on the most important things. If used with the right flexibility and modesty, it can be effective in doing so.

In the following section, I describe a real use case, in which I took a journey into the world of Scrum. It was a journey of inexperienced people, including myself, willing to embrace the agile principles in a consistent way for the first time. Many projects in which I worked before were made in an iterative way but without the whole ceremony of a real agile framework.

A Real Use Case

We talk here about a real use case, in which a far from strict adoption of the Scrum framework had been made. The project was about a software system aimed at tracing all the activities involved in an anatomical pathology laboratory, from gathering the tissue specimens, and treating them in various steps until the final distribution of the tissue slides. The system was also supposed to be integrated in specific phases with external automation machinery, by specific software drivers.

I was involved in this project as a software architect. I had to cooperate with the project manager in order to outline the main issues and devise a basic architecture blueprint. If you follow the Agile principles to the extreme, thinking of architecture beforehand is not the best thing. Even the architectural design is seen in an iterative scenario. In most situations though, you still need a base from which to start, and you have to discuss it with all the stakeholders involved.

I approached this preliminary architectural study trying to have a clear separation of the contexts, in a structured approach based on views, viewpoints, and perspectives. Following such an approach is important in order to have a clear separation of problems, and also to customize the discussion based on the particular type of stakeholders.

A part of the discussed architecture was indeed the development phase and its organization. The company itself had not yet adopted any agile methodologies. Nevertheless, in agreement with the manager and the other people involved, we wanted to fill this gap and we chose to get inspiration from the Scrum methodology framework.

We were not trained at all in Scrum, but we were all conscious of its basics. The main directives we had in mind for the project, both methodological and technical, were:

• A basic set of practices to follow, to focalize ourselves on the partial and overall goal in an iterative approach (Scrum in our final decision).
• Tracking activities/sources by automation tools.
• Behavior Driven Development approach coupled with unit and integration testing.

Scrum Practices

Being motivated by the need to enforce some solid methodology framework but forced by our lack of deep skills, we chose to take some of the main rules of Scrum without going too far. First of all an iterative approach was really important in our mind. Scrum covers this through the so-called Sprints, which we can consider the iterative units of work. Each Sprint is supposed to cover a chosen number of features from the backlog and has a specific time duration.

We chose two weeks for the time duration of our sprints. Before starting the real deal, we set up a conventional Sprint number zero of one week to do preliminary work and organizational tasks. In this preliminary Sprint, we also wrote the initial version of the backlog, with all the features listed by priority. We did not embrace a particular method for evaluating task efforts, just a normal discussion between team members.

At the beginning of each Sprint, as for the Scrum rules, we would discuss the work already done, assess all the encountered issues, and choose the features to be implemented in the coming Sprint.

The project manager played the role of the Product Owner, backed up by a domain expert. This made sense in the specific situation because there was no real product manager involved, and the project manager himself had all the knowledge of the client's requirements. As for the Scrum Master, I did that for a while and then pass the role to another colleague, even if both of us were not fully trained in it.

Our team was a heterogeneous one with people working from different cities. We were forced then to organize a virtual version of stand-up meetings by scheduling Skype calls every day at the same hour. The meetings were about 15 minutes long. Some of the team members has some form of resistance regarded this, maybe because they interpreted it as a form of control, which is not.

We spent some time making them aware of the real meaning of the daily meetings: a brief discussion between teammates to focus on the main issues, enhance communication and find ways to improve and make the work easier for all. For a while, they kept saying it was a waste of time and a source of distraction from the real work but in the end, we managed to convince them.

Tracking activities/sources by automation tools

Besides the methodology practices, we also needed the tools to solve these main concerns:

• Versioning the code, tracking its changes, and sharing them across the team.

• Tracking the activities and bug resolution: what has to be done, who is assigned to what, and the status of it.

• Matching source code changes with activities.

  
  

The flow of information in a project is far too complex to rely only on methodology practices to control it. You need a solid ground of tools to make the work easier. The most tasks you automate, the more you can concentrate on important things and produce a better product.

For code versioning, we used Git as it is the most natural choice. Git can be used in a variety of ways, and we personally adopted Gitflow as a workflow pattern.

To track the activities we used Redmine, which is a general platform aimed at project management.

To deal with the third concern, we integrated our Git repository with Redmine in such a way Git commits could be related to specific Redmine tickets by using an issue identification code in the commit comment. This way we had a full mapping between activities and Git units of work.

Behavior Driven Development approach coupled with unit and integration testing

We were strongly willing to base our development process on a Behavior Driven approach. BDD fits very well with Scrum and in general with Agile principles, especially in the part that regards communication. It allows writing test scenarios in a format that can be understood by nontechnical people, and with the right tools gives a visual report of the current status. It draws the logical boundaries of the product and forces the development work within those boundaries.

The BDD functional tests were just the external layer of the whole testing instrumentation. We used unit and integration tests as well. In order not to be overwhelmed by the complexity of such a development environment we had to use the right tools and automation features.

The main technologies involved were:

• Cucumber: integrated into Spring Boot application with REST services

• Other testing tools: JUnit, Mockito, and Spring Boot libraries to support integration tests.

  
  

The following mental map shows a summary of the things discussed above:

Considerations

Was the adoption of Scrum, even if in a light and flexible way, worth it? The answer is yes. Let's be clear though, we cannot see it as the solution to all problems, and if adopted without understanding its very spirit could be even detrimental. The essence of a methodology is to offer a collective mind schema to make people work together with maximum sharing of information and commitment, but if the people are focused on following the rules of an exotic ceremony instead of the actual work, the original purpose vanishes.

A Sports Analogy

You can better understand what was said above with a sports analogy. Not all people like soccer, but almost everyone has at least a minimal idea of how it works. First of all, is a collective game. Two teams confront each other on a game pitch trying to deliver a ball inside a goal. To do this apparently simple task they have to mix individual initiatives with collective strategies and tactics. Those strategies and tactics are taught beforehand by the coach and make up a big percentage of the whole time spent in workout sessions.

To be really effective, the above-described collective rules followed by the soccer players must be executed without effort, and without even thinking they exist. They must be automatic, as the gestures to drive a car for instance. A basic requirement to reach this objective is that they have to be simple enough to be fully absorbed by all the players in the limited amount of time required to prepare for the championship.

If you are focused on following the Scrum ceremony instead of the real work you end up bringing chaos instead of order. On the other hand, if you follow a more pragmatic approach, being flexible and even getting rid of all the things that in our specific experience do not work, you can make the best of it. You can even think of postponing some Scrum approaches if you find them hard to follow, and then try to introduce them in a later phase.

Managing the Managers

Let's stress a concept: agile methodologies and Scrum specifically, can only work if the team people are either willing of adopting it or at least aware of its advantages. It cannot work if it is only introduced by managers in a company to follow the general fuss and tell the world outside: "See? We are agile!". Let's be clear: if the purpose is only marketing it would be better just to pretend of following those methodologies. There is no need to introduce in the internal processes a burden that has only a promotional purpose.

Moral of the story: agile methodologies can have some positive impact only if adopted by the engineers along with the managers and not just imposed by the managers. Most managers, especially those without a technical background, don't know anything about how a methodology could impact the life cycle of a software project, while engineers do, especially senior engineers. Years of experience are better than the best books and the best courses.

Furthermore, based on my weird experience in Italian companies, organizations are often dictated by a culture that seems to come from some sort of medieval heritage. In this context, the Scrum Master and even the Product Owner roles could be simply seen as a source of authority in a career path, rather than operative roles.

I have experimented with this harsh reality recently, to tell the truth. Normally these people do not have the slightest idea of what are the very principles of a methodology, and keep harassing people with rules they don't even understand.

In these horrible environments, Extreme Programming and Scrum are just meaningless titles. In these contexts the managers are sources of entropy to be dealt with, and to reach a decent level of productivity the team must manage its own work and even the managers, to reduce their bad influence. That explains the title of this section: "Managing the Managers".

Balance Between Methodology, Testing Practices, Tracking Activities, and Related Tools

In the use case described in this article, we have talked about methodology but also about testing strategies, tracking activities, and the tools used to support them. The best methodology frameworks cannot solve by themselves all the complex matters involved in software development.

As a matter of fact, BDD, which covers functional testing, is a very effective way of sharing knowledge of the logic of the software system being developed. It builds a strong bond between all the team members and the Product Owner, and it improves the focus on the more important aspects of the project. So, it covers part of the issues that Scrum is supposed to cover.

Unit and integration testing, on the other hand, are more involved in the inner processes of the software developers, but indirectly they make changes easier to deal with, coupling well with the iterative approach of Scrum.

Conclusion

Software development is a complex matter even in minimal projects handled by a single developer. If a project needs a team to be developed, a number of organizational issues arise. Agile methodologies are an attempt to solve those issues but they would be really useful only if taken if a grain of salt and by avoiding any form of senseless magnification.