“I avoid dogma involving anything that suggests any ‘one and only right way to do Agile,’ but I also acknowledge there are some non-negotiable characteristics of any approaches referred to as ‘Agile,’” answers author Ken Collier when asked if there were any absolutes that were required in order for an organization to claim Agility. In part one of this two-part interview, we heard about some of the differences and challenges with developing data warehousing and business intelligence applications using Agile development practices. We continue the conversation and find out more about what Collier believes are those non-negotiable characteristics of Agile development.
SSQ: One of the characteristics of Agile Analytics development is collaboration. You write, “Frequent collaboration between the technical and user communities is critical to success.” You note the project community includes users, business owners, stakeholders, executive sponsors, technical experts, project managers and others. However, different users have different and often conflicting viewpoints. How is it possible to prioritize features and requirements when listening to so many voices?
Collier: This can be a challenge, and is typically resolved by the “product owner” through collaborative bartering. Conflicting requirements must be resolved, which is best accomplished by stakeholders talking to one another face to face. Regarding conflicting priorities, I’m a big advocate of short horizon planning on larger, more complex DW/BI programs. For example, a 36 month program should be chunked into 3-6 month planning horizons. Each planning horizon has a stated theme or focus. When it’s time to plan for the next horizon we focus down to 2-week iterations and the creation of user stories that deliver on the theme or focus of that horizon. In this way competing priorities are often distributed into different planning horizons. For example, the next three months will focus on delivering BI features needed by the finance department, and the following three months will focus on delivering the HR features, and so on. I also use a technique developed by Luke Hohmann (author of Innovation Games) called Buy a Feature. This is a “serious game” designed to help a diverse customer community prioritize the desired features. I probably should have written more about these approaches in the book. Maybe I’ll do that in the 2nd edition.
SSQ: You note that Agile Analytics is a style, not a framework. It is not synonymous with Scrum or XP, but an “adaption of principles and practices from a variety of [Agile] methods to the complexities of data-intensive, analytics-based systems integration efforts like data warehousing and data mart development.” Do you foresee the possibility of an Agile Analytics framework being developed specifically to address the complexities associated with DW/BI applications?
Collier: I suppose that’s possible, but I would hate to see methodologists turn something flexible and adaptable into something rigid and heavyweight. Alistair Cockburn, one of the original creators of the Agile Manifesto and Agile Alliance, describes each project and organization as a different “swamp.” Having a map to navigate one swamp won’t help you navigate other swamps. I’m wary of any prescriptive process that professes to fit all projects equally well. With this book I’ve tried to compile a collection of effective project management, team leadership, collaboration, design, and technical practices that will be helpful in all Agile DW/BI projects. I suppose that’s a little bit like a framework without being too prescriptive. Each project community will need to tailor the practices I introduce to the specific technologies, organizational structure, skill sets, problems domains, and other nuances of their project environment.
SSQ: Some Agile purists believe that unless you are following a framework such as Scrum, you are not “truly Agile.” Though your book does not make this claim, you also suggest that there are some mandatory practices that must take place in order to claim “Agility.” For example, you say, “every iteration must result in demonstrable working software.” Might there be instances where an iteration is dedicated to re-factoring or planning or other tasks in which demonstrable working software is not an output?
Collier: The people you describe are zealots, not “Agile purists.” I have yet to see a situation where rigid dogma is helpful. Real “Agile purists” are those who focus on the four core values of the Manifesto and the 12 guiding principles that support those values. I start out with the first guiding principle, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” This is the basis for the goal of delivering at least one demonstrable feature each iteration. Once teams get good being Agile, then they may choose to periodically devote an entire iteration to non-business-value activities. In high-performing Agile teams these iterations are the rare exception, whereas new Agile teams are often quick to excuse themselves from a focus on delivering business value, which quickly becomes a slippery slide back into old habits. I avoid dogma involving anything that suggests any “one and only right way to do Agile,” but I also acknowledge there are some non-negotiable characteristics of any approaches referred to as “Agile.”
SSQ: What other absolutes (besides following the Manifesto) are there in order for a team to claim they are truly doing Agile analytics development and do you believe these differ at all from those practicing Agile development on non-analytic applications?
Collier: Absolutes? I’m uncomfortable suggesting that anything is absolute. However, if a team is going to claim that they are truly doing Agile Analytics development I expect to see that new BI features are being delivered to customers every few weeks; these features are fully tested and of production quality; these features are built on top of a data warehouse whose architecture is evolving towards excellence based on customer needs; there is a high degree of collaboration between the delivery team and customer community, as well as within the delivery team; and the business stakeholders are delighted that their needs are being met. Teams that are achieving these outcomes are Agile regardless of what their specific process looks like. It is less important that Agile Analytics teams do precisely what I suggest than that they produce these outcomes. I like what Agile luminary, Mike Cohn says, "Agile is not something you become, it's something you become more of" (during Agile2010 keynote address).
This Q&A is based on the book, 'Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing' by Ken Collier, published by Pearson/Addison-Wesley Professional, ISBN 032150481X, Aug.2011, Copyright 2012 Pearson Education, Inc. For more info please visit the publisher site: www.informit.com/title/032150481X
Ken Collier has worked with Agile methods since 2003, and pioneered the integration of Agile methods with data warehousing, business intelligence, and analytics to create the Agile Analytics style. He continues to refine these ideas as technical lead and project manager on several Agile DW/BI project teams. Collier frequently trains DW/BI teams in Agile Analytics, and has been a keynote speaker on the subject. He is founder and president of KWC Technologies, Inc. and a senior consultant in the Cutter Consortium's Agile Development and Business Intelligence practice areas.