Prototype before development, not just a spec: why a 100-page document will not save your project

A company spends several months preparing a specification. The document runs to more than a hundred pages, and sales, operations, finance and IT have all commented on it. Once the contract with the supplier is signed, months of development follow, and at the first demo someone says: “That is not what we meant.”

The scenario is illustrative, but the problem behind it is real. In the international NaPiRE survey, completed by 228 organisations from 10 countries, respondents selected the most critical problems they faced in requirements engineering. The problem cited most often was incomplete or hidden requirements (48% of respondents). Communication flaws between the team and the customer were cited by 41%, and of the top ten problems, this was the one respondents most often identified as a major cause of project failure.

An extensive specification creates a sense of certainty, but it cannot be used to test what really matters: whether people will understand the solution and be able to get their work done with it. A software prototype reveals this within a few days to a few weeks, while fixes are still cheap, rather than after months of development.

Why a signed-off specification does not prevent misunderstanding

A signature confirms agreement with the text; it does not confirm that both parties understand the same thing by it. In his 1987 essay No Silver Bullet, Frederick P. Brooks Jr. wrote that the hardest part of building software is deciding precisely what to build, and that no other part is harder to put right later. He argued that it is in fact impossible for a client to specify the requirements of modern software fully, precisely and correctly before trying out several versions of it.

In our view, a thick document tends to mask this problem. Every additional page adds to the sense of completeness, yet each department mainly reads its own chapter, and nobody writes down what everyone takes for granted. The company gets its first feedback on something tangible only once code has been built on top of the misunderstanding. In the same essay, Brooks argued that software procurement at the time rested on a fundamentally flawed assumption: that a system can be satisfactorily specified in advance, put out to tender, built and installed.

Same sentence, different screens

A document can only be read, not used, so it cannot show whether everyone understands it in the same way. Take a sample requirement: “orders above the limit are approved by the manager”. The finance director pictures a morning overview for bulk approval, the warehouse manager a notification on their mobile, and the developer a status field in a table. All of these interpretations are consistent with the sentence, and each one implies a different amount of work and a different price.

Exceptions only come to light when the task is actually being carried out. Who approves when the manager is on holiday? Does the approval still apply once the quantity has changed?

A quarter of NaPiRE respondents also ranked among the most critical problems the fact that stakeholders struggle to separate requirements from solutions they already know. The UK government’s GOV.UK Service Manual advises that in the discovery phase, when the problem is explored before anything is built, a predetermined solution should be reframed as a problem: not “build an interactive map of contact centres” but “how can we make it easier for people to find their nearest centre”.

What a prototype is and what it is not

A prototype is a simplified version of a future product that lets you try out a solution before it is coded. Kara Pernice of Nielsen Norman Group describes it as a hypothesis: a candidate solution that is most directly tested by watching people work with it. It is not a minimum viable product (MVP), which Eric Ries describes as the version of a new product that brings a team the most validated learning about customers for the least effort. A prototype is a narrower tool: it tests a proposed solution before it is built.

According to Pernice, a prototype’s fidelity is the degree to which it resembles the final system in its interactivity, its visuals, and its content and commands. For decisions made before development, we distinguish three levels in practice:

• Sketch or paper prototype. Rather than a system, it is a person familiar with the design who responds, handing the user one hand-drawn screen after another. In the related Wizard of Oz technique, someone in another room controls the screen, so you can even try out behaviour that nobody has programmed yet. Jakob Nielsen counts paper prototyping among the fastest and cheapest techniques. It is well suited to validating the basic approach and flow.
• Clickable prototype. Linked screens that respond like an application, even though nothing is running behind them. It tests whether people can complete key tasks unaided.
• Working prototype with real data. A limited piece of real code connected to a sample of data or to a high-risk integration. It is useful when the uncertainty lies not in the interface but in the data or the technology.

Our rule for choosing: the lowest fidelity that can still answer the question at hand. According to Nielsen, testing early rather than late is such an advantage that it outweighs even differences in prototype quality. IDEO.org’s Design Kit allows anywhere from a few days to a few weeks for rapid prototyping, depending on what is being tested.

What a software prototype must demonstrate

A prototype is a test, not a presentation. According to the GOV.UK Service Manual, you do not need to prototype the entire user journey; it is often enough to focus on the most challenging parts and test the riskiest assumptions. Before development starts, a prototype should answer these questions:

1. Do people understand the flow? Can someone who was not in the room when it was designed complete the task from start to finish? Every moment of hesitation in a test could become a support call or an order error once the system is live.
2. Have the key decisions been made? Words on a page let you put off a disagreement; a screen does not. Whatever is decided now will not linger as uncertainty in the quote.
3. Does the design hold up with real data? In practice, names tend to run longer than in the sample, fields are often left empty and records duplicated.
4. Does the riskiest integration work? An integration is a connection to another system, such as your accounting software. The UK’s Register to vote service would not have worked without an API (an interface for exchanging data) connected to the registration systems of more than 400 local councils. That is why the team focused on it as early as the alpha phase, which is intended for prototyping.
5. Will users adopt it? A prototype does not guarantee adoption, but it does indicate whether people would prefer the new process to the workarounds they use today. The article The most expensive system is the one your people never use explains why this matters.

Testing with users: observe rather than ask

Usability testing means that a person from the target group works through a real task on the prototype, such as the sample task “raise a credit note against this invoice”, while you observe where they hesitate and what they misinterpret. A questionnaire captures what people think they would do. Observation shows what they actually do.

Jakob Nielsen works from a model in which a first study with five participants uncovers around 85% of usability problems, meaning the points where people get stuck or make mistakes. Instead of one study with 15 people, he recommends three rounds of five, because each subsequent round checks the fixes and uncovers deeper problems. This applies to comparable users: with two markedly different groups, he recommends three to four people from each.

When a late change really is expensive

The cost of change is what it costs to revise a decision at a given stage of a project. In a 2001 article, Barry Boehm and Victor Basili reported that a software problem found and fixed after delivery often costs 100 times more than one dealt with during the requirements and design phase. They included the word “often” deliberately: for small, non-critical systems, they put the ratio closer to 5:1. They also noted that projects at the time spent around 40 to 50% of their effort on avoidable rework, and named hastily specified requirements as one of its two main sources.

Yet the rise in cost over time is not universal. Across 171 projects, Tim Menzies and his co-authors found no evidence that resolving issues at a later stage took consistently or substantially more effort than resolving them shortly after they arose. The projects were predominantly small to medium-sized and used the Team Software Process methodology, and the data did not cover the period after delivery. The authors themselves point out that the effect may occur only in certain types of project.

In our view, it is not the calendar that makes a late change expensive, but everything that has been built on the flawed decision in the meantime: code, integrations, documentation, training and the data model. Kara Pernice points out that throwing away code is very expensive, whereas throwing away a prototype is not. Moreover, once the system is live, the company pays for an uncorrected misunderstanding every day in the form of friction in routine work. The article Good design is not decoration shows how to calculate that friction.

A prototype therefore pays off above all where the most would be built on a flawed decision: in projects with many users, several integrations and a long development cycle. Its scope and cost can be capped in advance; the cost of a mistake cannot.

A prototype as the basis for quotes and estimates

When suppliers price a 100-page document, each of them is pricing its own interpretation and its own contingency for uncertainty. A cautious supplier inflates the price, a less cautious one pitches it too low, and the difference surfaces later in change requests: work outside the original agreement that is invoiced separately.

A clickable prototype with acceptance criteria changes the starting point. According to the GOV.UK Service Manual, acceptance criteria are a list of outcomes used as a checklist to confirm that a service has done its job and met the user’s need. They are often written in the form “it’s done when…”; for the Register to vote service, for example, one criterion is that the user knows how to register online. The prototype shows how things should work, the criteria define when the work is done, and suppliers are therefore all pricing the same thing.

The manual also advises reconsidering why you need a feature if you are struggling to articulate its goal. A feature that is never built costs nothing to develop or maintain.

The risks of prototyping

• Mistaking it for the finished product. A clickable prototype looks finished, but it lacks security, data handling and error handling. Pernice warns that when a design looks polished, it is easy for management to give in to the temptation to take it live straight away.
• Over-polishing the visuals. At low fidelity, according to Pernice, users are more willing to voice negative reactions, designers are less attached to the design and stakeholders can see that the work is not finished. Polish therefore tends to dampen honest feedback.
• Leaving the prototype’s future undecided. GOV.UK advises building only enough in the alpha phase to test ideas, not production-quality code, and expecting to throw the code away at the end of the phase. Whether the prototype will be discarded or developed further needs to be decided before it is built. Otherwise, inertia can turn it into the foundation of the product, complete with technical debt: shortcuts that make every subsequent change more expensive.

When you do not need a prototype

A prototype reduces uncertainty. Where there is none, or where it can be reduced more cheaply, a prototype is an unnecessary cost.

• You are buying off-the-shelf software without customisation. A trial version or a demo in which people try out their real tasks will do the job of a prototype.
• The system is small and non-critical. Boehm and Basili write that such systems can be developed more efficiently in a less formal mode of continuous prototyping. Here, the product itself, developed in short iterations, takes the place of a separate prototype.
• The question is not how, but whether anyone wants it. In that case, a smoke test may be enough: a way of checking for interest before the solution exists. Eric Ries describes how his team spent two weeks building a feature that nobody wanted, when, in his view, a simple smoke test with an AdWords advert would have been enough to reveal this.
• The problem is not in the software. If a process contains an unnecessary step, a software prototype will simply move it onto a screen. The step needs to be removed first. Whether a custom system is worth it at all is explored in the article A good internal system can pay for itself faster than a new hire.

According to GOV.UK, stopping a project at the end of the discovery phase is not a failure if the research shows that this is the best decision: it saves both time and money. So when deciding whether to build a prototype, look at where the greatest uncertainty in the project lies and how much a mistake would cost.

Key takeaways

• Before signing off an extensive specification, use a prototype to test the riskiest part: the most common flow, a contentious exception or an unproven integration.
• Give every prototype a question and a success criterion up front; otherwise the test turns into a presentation.
• Test with five comparable people on real tasks, and run another round once the fixes are in.
• Base both the specification and the choice of supplier on a tested prototype and acceptance criteria, so that everyone is pricing the same thing.
• Start by naming the uncertainty you are addressing. Sometimes a trial version of off-the-shelf software, a smoke test or a change to the process will remove it more cheaply.