Agile Development: Past, Present, Future

Observations of an Alibaba engineer with over ten years’ experience in the field

Tao Zhen(陶震), Senior Engineer of Alibaba’s IoT Business Unit:

Almost ten years ago, my product development team began implementing an agile development model. We became, perhaps, the very first agile team in China. As such, my experiences in the industry may prove enlightening, especially in light of the state of agile development today.

Loss of Control

In current discussions on agile development, you’ll likely encounter one of two extreme opinions: “it’s brilliant” or “it’s awful”. This represents a development from when software companies were promoting ISO 9000 and later the Capability Maturity Model (CMM).

For most companies, the introduction of ISO 9000 meant amassing various documents and storing them away in filing cabinets. As a result, engineers never got a real sense of their contents. Moreover, ISO 9000 was created in the era of machine production and is no longer fit to efficiently measure today’s emerging and evolving software companies.

CMM, a model for measuring the capability maturity of a piece of software, was for a time applied extensively, but programmers remained uninterested. In their view, nothing had changed. The stack of documents locked in the filing cabinets had simply turned into a stack of documents relating to the software industry.

At the core of the ineffectiveness of both ISO 9000 and CMM lies control.

Hoping to gain a sense of security for future products, managers often seek to control behaviors and limit information during the development of software, employing various rules and processes to achieve this.

But the software world has changed, something that Linux exemplifies well.

People have long considered an operating system to be the most complicated software system there is. It seems unquestionable that making such a complex system would require the work of vast numbers of programmers in a leading global company like IBM or Microsoft, using a rigorous process management system. But Linux did things differently, engaging hundreds of thousands of programmers located disparately throughout the world. No tedious processes, strict rules and regulations, or tough project managers. In the absence of nearly all traditional means of management, an epoch-making product in the software industry was created. This was an unprecedented feat.

Today, network technologies can link information created by individuals more quickly, which is then presented as a new product. The more the organizational model matches this trend, the more effectively an organization can combine the value of internal and external information to form its own information and product values.

This then begs the question as to which model should be used by a product development team to complete this information link more efficiently.

For both traditional and agile development models, the end goal is the same: develop good products and services. The means are different, however, with the traditional model controlling as much as possible, and the agile one as little as possible. This contrast can be seen in other fields too, including education. Should children be allowed to choose which path to go down, or should we as parents or educators make the choice for them? Is the flexible and more agile approach or the traditional model more appropriate? Naturally, the answer to both these questions is a matter of opinion.

Flattening

As early as the 1960s, the US military had established a world-leading C4I system (which stands for command, control, communications, computers, and intelligence), helping to command and control weapons and soldiers with computer and network communication technologies. The system and its organizational model introduced an efficient and flexible flow of information, allowing the President to command combat troops in 1–3 minutes and enabling soldiers to easily obtain military intelligence. As a result, the United States has been able to quickly and effectively respond to a wide range of threats and challenges around the world, despite fewer active forces.

Due in part to C4I, the government was able to use information technology to make the military hierarchy flatter, maximizing the potential of team members at all levels. The next step, naturally, was to delayer technological innovation and product development.

In early 2007, I met Jeff Sutherland, inventor of the Scrum software development process.

Jeff started out as a soldier, served in Vietnam, and after an 11-year career in the military became involved in IT system development. But warfare requires resourcefulness, organization, and daring, and you can see the impact of his military service in his Scrum methodology. Maximizing the potential of engineers by encouraging the flat organization of product development is the essence of Scrum.

Another example of an agile, flat organizational model is Apple. While Steve Jobs was known for his love of controlling things, he allowed his employees a great deal of independence in their work, and there was an absence of micromanagement in the company structure. The control Jobs exerted was far more about rejecting mediocrity than controlling developers.

While the risks in the development of internet software products are many, they are all rooted in the same thing: uncertainty. However, more control does not necessarily turn uncertainty into certainty. Indeed, uncertainty arguably leads to greater creativity. This is why Donald Knuth and Eric S. Raymond describe programming as an art form.

Value Differences

Of course, flat organization is not a patent for agile development and you can still encounter obstacles. Value differences between Chinese and Western cultures still persist.

I recall one instance when a Chinese engineer in one of the teams I was leading came to me privately and asked for a more formal job title. He didn’t request a pay raise, as many in the West may have done — he simply wanted a new job title. He worried that, after so many years of work, the lack of an ‘official’ title would embarrass his parents. At that moment, the contrast between China and the West could not have been starker.

Differences also have historical roots. For example, Europe and North America have deep-rooted cultures of engineering and craftsmanship. From the Industrial Revolution in Great Britain and the effect it had on its economy, to Benjamin Franklin, a founding father of the United States and world-renowned inventor, to the present day — where every town has at least one home improvement or hardware store and every family owns a toolbox — the culture is deeply engrained.

This is not the case in China, and Chinese engineers are not familiar with this concept. Western engineers, who from a young age have been taught to reconstruct and improve, are perhaps more adaptable to agile development than their Chinese counterparts.

Flow of Information

I was once involved in an agile development project where each of the over 200 participating software engineers was informed of any update to the product design, allowing them to understand and follow its development. At one point, an architecture design was released and one of the junior engineers spotted a flaw. None of the technical experts had foreseen it, and they were unable to propose a quick and effective solution. In the end, it was the same junior engineer that provided the solution through her own research.

From this, we can see that flat organizations creatse a more responsible and better-informed workforce, with the upper layer listening to and understanding what the lower layer is saying, and vice versa. Talented engineers have a greater chance of adding value to the product and are more likely to be inspired through direct involvement. This communication across hierarchical levels is essential to agile development. The flow of information must be unobstructed.

To embrace change is to embrace uncertainty. Many employers might not have taken on a young college dropout who could not code, but then they would have passed up Steve Jobs, one of greatest inventors of his time. Jack Ma, who has been open about his own struggles to find employment — famously being turned away for a job at KFC — went on to set up the world’s largest e-commerce company. Simply put, a flat organization that implements agile development can give those who would be perceived as unlikely candidates for success in a traditional hierarchy the opportunity to thrive. Barriers to information do nothing but restrict your own capabilities and competitiveness.

What the Future Holds

There is a simple but effective algorithm in computing called bubble sort, which goes through an unsorted list comparing items, swapping them out if they are in the wrong order. It is named as such because items that are out of place essentially bubble to the top, not unlike how in a flat organizational structure, the innovation and value of each team member can easily rise to the surface.

In today’s uncertain IT world, whether you employ an agile model or not, the smooth flow of information is key to the success of a team of engineers. And management, as a linker and transmitter of value, serves this flow of information and creativity.

Agile development was formed with a vision for a certain kind of workplace. Whether it works or not depends on whether the employees buy into that vision and embrace the culture it was designed to support. At Alibaba, an open and equal company culture is promoted, and hierarchy and superiority are minimized. Instead of addressing each other in formal tones, everyone calls each other by their English name or nickname. However, the success of agile development — or any other approach — ultimately comes down to whether employees engage with and embrace the ethos of the organization they are in.

(Original article by Tao Zhen陶震)

Alibaba Tech

First hand, detailed, and in-depth information about Alibaba’s latest technology → Facebook: “Alibaba Tech”. Twitter: “AlibabaTech”.