Agile for data warehousing and business intelligence applications

Find out how development of business intelligence and data warehousing applications differ from traditional application development, and how Agile principles and techniques can still be applied.

Agile methodologies are becoming increasingly popular for software development projects of all kinds, but what considerations must be made when developing business intelligence applications? Today we talk to Ken Collier about his book, Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing.

SSQ: Ken, your book talks about Agile practices and techniques that can be used specifically when developing data warehousing / business intelligence (DW/BI) applications. Can you briefly give an example of how this type of development differs from traditional application development?

Ken Collier: Unlike application development, DW/BI is an integration and configuration of commercial tools with customization occurring in the underlying data models and data manipulation “code” (ETL, SQL scripts, stored procedures, etc.) Instead of using object-oriented languages and principles we stitch together off-the-shelf tools so that they work together seamlessly. Although the fundamentals of Agile software still apply, it can be very hard for DW/BI practitioners to easily see how to adapt concepts like feature-slicing, test-driven development, and the continuous delivery of business value to DW/BI development.

SSQ: Agile development has been called “light-weight” in terms of processes and documentation and was originally thought more appropriate for smaller, less-complex projects. Would you say that DW/BI development has been slower to adopt Agile development styles because of the difficulties involved with using Agile for larger, complex projects?

Collier: It’s a misconception that Agile is only appropriate for smaller projects. There are many examples of Agile being used successfully on large scale programs (see Dean Leffingwell’s book, Scaling Software Agility). Instead I would say that the data community has been slower to adopt Agile methods because it’s hard to map the effective Agile software techniques to the nuances of DW/BI. The DW/BI community typically has a data-centric mindset. That is, the delivery of working BI features has always been secondary to the hard work of integrating and preparing the data for analysis. Unfortunately this has led to 18-36 month project cycles with little business value being delivered. Agile DW/BI requires a fundamental shift to a value-centric mindset. That is uncomfortable for data modelers and architects who are under the misconception that data must be thoroughly & properly modeled and integrated up-front before BI applications are built against the warehouse.

SSQ: You compare DW/BI development to alpine-style climbing, saying that “it is essential that we have a sufficient amount of planning, the necessary support to be successful, and an appropriate amount of protocol.” However, it seems rather subjective for teams to determine how much planning is “sufficient.” What guidelines can you give to help teams determine how much planning should be done up front?

Collier: Great question! And, yes it is somewhat subjective. This is one of the harder aspects of learning to be Agile. We need to do just enough up-front planning, modeling, and design so that everyone in the project community is galvanized around a shared understanding of the vision, direction, and approach. The most effective Agile teams know that everything is subject to change, so they seek minimally sufficient up-front planning so that they can start creating value as early as possible and find out if they are building the right thing. How much up-front planning and design varies from project to project. During planning Agile teams continuously ask, “What is keeping us from getting started developing?” They focus on answering the essential questions to get started rather than trying to answer all the unanswered questions before getting started.

SSQ: You also say in your comparison that “the line between siege-style and alpine style mountaineering is not precisely defined, and alpine-style expeditions may include some siege-style practices.” Are you in favor then of hybrid approaches for DW/BI development? If so, are there traditional techniques that you feel are particularly useful or detrimental?

Collier: I’m certainly not advocating that we abandon the good techniques that have been working in traditional DW/BI. Traditional “waterfall” project management (akin to siege-style climbing) calls for requirements analysis, specification, design, develop, integrate, test, and release. This remains a great problem solving process that calls for sound engineering discipline at each step. What doesn’t work is expecting that we can do each step only once and get it right the first time. Instead we need to do a little bit of all the same steps every day or every few days so that we’re continuously designing, building, and testing our ideas and adapting to feedback. I don’t really think of that as a hybrid approach, but instead as a significant elimination of traditional techniques that aren’t helpful such as phase gates for freezing requirements and design; change control processes for limiting change; work-breakdown structures for tracking task completion; Gantt & PERT charts for rigidly following a plan; big design upfront (BDUF); and others. Many of the good traditional technical practices around data modeling, database optimization, ETL development remain important, but these need to be augmented by modern practices like test automation, build automation, deployment automation, version control, and others.

SSQ: You suggest a couple of slight variations in the Agile Manifesto if it were specific for Agile Analytics development. Rather than “Working software over comprehensive documentation” you suggest “Working DW/BI systems over comprehensive documentation” and rather than “Customer collaboration over contract negotiation,” you suggest “End-user and stakeholder collaboration over contract negotiation.” However, isn’t the meaning really the same? DW/BI systems are the working software. End-users and stakeholders are the customers. Are there really any differences between Agile development and Agile Analytics development?

Collier: Yes, I don’t really tinker too much with the Agile Manifesto because it’s very powerful as written. All I did in that section was map some software specific terminology to typical DW/BI terminology to help my DW/BI readers make the connection to their domain. I suspect it wasn’t necessary, but some people get hung-up in thinking that Agile values are only applicable to software. The differences between Agile software development and Agile Analytics are less in the fundamental values and principles and more in the techniques and practices. Almost everything I introduce in the book is an adaptation of the great ideas of very smart people to the unique challenges of DW/BI. I’m not sure I could argue that any of the ideas in the book are purely original. However, the very successful Agile movement has been largely ignored by the DW/BI community. I’m hoping this book will help change that.

The conversation continues in What is Agile development? Agile Analytics author answers.

This Q&A is based on the book, Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Ken Collier, published by Pearson/Addison-Wesley Professional, ISBN 032150481X, Aug.2011, Copyright 2012 Pearson Education, Inc. For more info please visit the publisher site:

Ken Collier has worked with Agile methods since 2003, and pioneered the integration of Agile methods with data warehousing, business intelligence and analytics to create the Agile Analytics style. He continues to refine these ideas as technical lead and project manager on several Agile DW/BI project teams. Collier frequently trains DW/BI teams in Agile Analytics, and has been a keynote speaker on the subject. He is founder and president of KWC Technologies, Inc. and a senior consultant in the Cutter Consortium's Agile Development and Business Intelligence practice areas.

Dig Deeper on Topics Archive