Problem solve Get help with specific problems with your technologies, process and projects.

Scaling Agile software development: Challenges and solutions

Software consultant Nari Kannan describes how agile practices and work can be scaled appropriately for success in large organizations. Using lean thinking, reduction of waste, and appropriately organizing work and people, agile can be successfully adapted, regardless of the size of the organization.

The ability to scale agile software development to large organizations has always had skeptics. Typical arguments are that magile works for small software development groups but not for large ones. Or, that they use outsourcing providers with fixed price contracts for software development and an agile methodology does not provide the discipline for them to fulfill contracts without a great deal of specification and design upfront.

Scaling agile software development to large organizations is still possible if enough attention is paid to:

  • Scaling agile practices – Understanding agile practices and making sure that the rest of the organization also does the same.
  • Scaling agile work – Organizing work and people appropriately for scaling agile properly.

Scaling agile practices to a large organization
Lean thinking guides agile practices significantly. The sources of many ideas in lean thinking are the Toyota production system (TPS) and the House of Quality that many lean companies practicing lean thinking use. The main principle in lean thinking is the idea that people are inherently responsible, capable of self-organization and learning, and do not need active management from supervisors. The other main idea in lean thinking is continuous improvement. Continuous improvement is best practiced by software development people that actually do the work. The Japanese technique of Gembutsu or "Go See" is applicable here. The principle is that each software development in each product or project environment is different and that methodologies and practices need to be tailored by the people who do the work after observing what is happening with the project closely for a while.

Reduction of waste is another strong agile practice that needs to be understood clearly and scaled in a large organization. Duplication of code in two different software projects is pretty common and well-known. Teams waiting for requirements documents to be complete and approved, waiting for design documents for coding to start, waiting for completed code for testing to start are all well-known wastes due to delays. Many processes like the stage-gate and other product management practices introduce their own delays. Software development teams waste time twiddling their thumbs while they are waiting.

For success, misconceptions about scaling with agile in large organizations need to be addressed. Agile does not mean there should be no documentation. Agile does not mean you are not disciplined. Agile does not mean no planning. The Agile Manifesto lays out a continuum of emphasis – individuals and interactions over processes and documentation. It just means that individuals and interactions are more important than any one process or extensive documentation, but not unimportant. Removing misconceptions is very important for agile to scale because such misconceptions have the potential to derail adoption.

Scaling agile work to a large organization
Organizing agile work to a large organization consists of two major areas that need to be addressed. Tackling one without the other is ineffective and counterproductive. These are organizing the work to be done and organizing people.

Organizing work traditionally has been done along some internal divisions like product divisions (personal tax preparation products, corporate tax preparation software products, for example) or functional divisions (user interface group, database management group, middleware group, etc.) or based on platforms (Windows, Windows Mobile, Unix, etc).

All of these ways of organizing work waste enormous amounts of time in unused talents and waste of time in waiting. In practice, there are almost always delays in handoffs and people are waiting for someone to give them something so that they can continue their own work. The UI group may be waiting for the middleware group to finish their designs. There could be enormous duplication of code – the two product divisions could be writing the same code to do the same thing without realizing it. There could be very good programmers that are good in UI design, coding and database design and implementation. The silo method of organizing work leaves a lot of talent untapped and unused.

It is better to organize work around requirements or features. Requirement areas will have their own requirement area owners that report to the product owner. Requirement areas could be IP protocols or performance or device support in the case of a telecommunication software product, for example.

Or, they could be organized around features such as downloading device data or batch download of data in the case of an embedded hardware/software product. In both cases, you will see that teams address entire set of coding functions - coding, UI design and development, and database design and development also.

Organizing people for scaling agile requires a lot of organizational change. It needs to be reflected in the policies and procedures of the company and needs to be adopted and used on a daily basis diligently for agile to be effective. Just adopting the superficial ways or organizing work without addressing these will be ineffective. Organizing people needs to follow the principles of empowerment of people, self-organization and self-management.

Reporting hierarchies need to be flattened first and the reporting spans should be larger. If people are empowered and self-managed, you need fewer managers to oversee their work. Managers need to be coaches or subject matter experts. Multi-skilling and job rotation needs to be built into the system. Software engineers may need to be experts at coding, architecture, design, database design and development and testing. Job titles prevent people from utilizing their full potential and contribute their best to the organization. Since now teamwork needs to be emphasized, reward structures need to be modified and job titles get in the way of team work. Job titles need to become generic and pay is tied to seniority and experience and automatic. These are pretty radical changes but without them re-organizing work alone may not help agile scale. These changes enable employees to be more proactive in taking on responsibilities and self-management and contribution.

Agile scaling, distributed and offshore software development
Agile scaling is really difficult with distributed and offshore software development. Many ideas that work when software development is centralized break down when the teams are distributed or done offshore.

Cultural differences, time zone differences do not pose big problems when software development is centralized. However, they become big problems in scaling agile development when teams are distributed, and some are offshore. The key here is to adapt and modify agile practices appropriately so that they work properly. A Daily Standup is possible if the entire team is in the same building or campus. If they are distributed across the globe only a weekly standup may be more practical and advisable. Clients or product owners may not be available for a daily standup at odd hours (because of time zone differences) and a weekly standup may be the only feasible solution.

Another way to address this is to use the distributed or offshore team as a self-contained requirement area group or a feature group. Communication is the #1 problem with distributed or offshore teams. There are no easy answers there except to use many communication mechanisms as possible – Skype or daily video conference, weekly team meetings, personal visits onsite by offshore teams, and personal visits by the client, onshore teams to the offshore location, at least every quarter or so.

Agile software development works in the small and can also work in the large, if approached carefully and many organizational changes and approaches are diligently made, and followed. Understanding and infusing the principles behind agile practices goes a long way in making scaling agile to large organizations successful. The keys are in not adopting only the superficial rituals but really adapt agile practices to the situation at hand, one organization at a time. Every organization and every software development project has its own unique aspects and a single magic bullet may not work in all cases. The underlying principle in agile is this flexibility and adaptation rather than blindly following a single set of prescriptions!

About the author
Nari Kannan is currently the Chief Delivery Officer of V-Soft Consulting, Inc., a Louisville, Kentucky-based software consulting firm. Nari has 20 years of experience in information technology and started out as a senior software engineer at Digital Equipment Corp. He has since served variously as vice president of engineering or CTO of six Silicon Valley startup companies, working in the areas of business process improvement, IT consulting, automotive claims processing, human resources and logistics applications. He can be reached at [email protected]

Dig Deeper on Topics Archive