How to test everything

They say there’s no free lunch, but at least there’s free breakfast. Last week I attended a “breakfast seminar” with Robert C. Martin (Uncle Bob). There really was free food.

Anyway, Uncle Bob held an extremely entertaining and useful introduction to the FitNesse testing tool. He got me hooked on it, but I’m even more fascinated by something else. I asked him, “what is the nature of the test API [you've been talking about]?” He answered that question, and also another, more general one: What is a good strategy for complete automated integration and acceptance testing of an application? That was the question I really wanted to ask, only I wasn’t quite aware that I wanted to ask it. And, impressively, he answered it anyway.

As I understood it, the strategy he outlined was as follows. The diagram is probably only a mild perversion of the one he drew on the whiteboard.

• Test all features through a test API just underneath the user inteface. These are the main acceptance tests; the ones that are used to decide whether a feature is done or not done yet.
• Test the user interface in isolation, running calls to a fake system underneath.
• Run just a few tests all the way through the application to test the plumbing between the user interface and the rest of the system.

The reason why we don’t want all our acceptance tests to go through the user interface is that the user interface has a tendency to change often, and changes in the user interface tend to breaks lots of tests.

This resonates reasonably well with my experience, although I think testing “everything” through the user interface can work sometimes, if the user interface is a very plain one.

In addition, we want unit tests for each small piece of the application. The tests outlined above are just catch the few bugs that are not detected by the unit tests.