Does moving to Agile development eliminate up-front requirements gathering?
There's a bit of a myth that moving to an Agile development process eliminates the need for gathering project requirements up front, and instead introduces the concept of discovering requirements "as you go." While there is some truth to the statement, it is particularly misleading to think about things this way.
Agile processes, which value delivering working software above delivering comprehensive documentation, do not completely abandon gathering project requirements in advance of writing code. They do, however, abandon the notion that you can completely define the requirements before you start doing any work. Borrowing a metaphor from the human resources domain, what you want to have before you start is a "T-shaped" understanding of the requirements before you begin coding. You want a broad (but shallow) understanding of the space, and a deep (but narrow) set of insights into the most important area in which you will be writing code.
Attempting to completely define the requirements before you begin work would be to try and form a box-shaped understanding -- covering the entire breadth of the domain in depth. There are two significant downsides to this approach -- both of which are at the heart of Agile philosophy. First, you delay building anything while you attempt to build a comprehensive understanding of the domain. This delay incurs costs -- opportunity costs of not being in the market, at a minimum. Second, as the statistics gathered by the Standish Group in their annual CHAOS report support, a high percentage of the requirements you document will be wrong. This box-shaped approach is called Big Up-Front Requirements (BUFR), and Agile processes do studiously avoid the approach, but without abandoning some up-front requirements work.
As a guiding principle, Agile processes encourage doing work at the "last responsible moment." You should only go deep in defining requirements in as narrow a scope as possible -- only enough to define a single iteration's worth of work. However, you should also do the "most important" work first; this accelerates the rate at which the team delivers value, both by delivering the greatest value first (where it continues to generate returns throughout and beyond the life of the project), and by reducing the amount of required rework (because the area of greatest value is likely to be the area that is best understood a priori, and least likely to be widely off the mark).
In order to pick the right place to "go deep," you have to do a shallow survey of the domain, to develop some context for forming insights. Many startups will appear to "jump right in" without having done this work, but I would argue that both the breadth and at least some of the depth of understanding of the market was already in-hand before the founder started bringing the team together. When you're building a product for a new-to-your-team domain, you better start with some T-shaped requirements work before you begin committing resources.
The team will absolutely need to develop additional "spikes" of understanding of the domain in depth -- gradually building out a box-shaped understanding, in parallel with development activities. But that's a topic for another answer.
This was first published in August 2012