Brandon Savage is writing a series on code improvement using a code example (starting with Peer Review: Taking Code And Making It Better). In other words, it’s about refactoring, which is practically my favorite subject. Although I don’t agree with all of it, it’s mostly good advice. I recommend it.

That said, I do have something important to add. The series is missing the first, most basic rule: Don’t refactor unless you have good automated test coverage (typically with unit tests). And if there are no test, write them before you start refactoring. If you don’t, you’ll make mistakes and get lost in a frustrating bug search, unless you’re very stingy and don’t refactor too much. With test coverage, you have freedom to experiment, to change something and change it back if you don’t like the result. This, above all, is what makes refactoring such a great learning experience.

Some of those who have commented on the post Peer Review: Looking Into Abstraction (Greg Beaver, Jeff Carouth) mention testing and point out the need to inject objects so that they can be replaced with mock objects for testing. This is a wise move, and practically unavoidable in this case. To support refactoring, unit tests need to be run frequently, without accessing outside services (twitter or email in this case) that take time to process.

This means that in this case, it’s actually necessary to refactor a bit before proper unit tests can be implemented. This is what Michael Feathers calls the Legacy Code Dilemma in his book Working Effectively with Legacy Code. Certainly building this dilemma into new code is not a good idea. Make the code unit-testable before it’s too late.

The debate on the “one assertion” principle continues. Pádraic Brady and I agree on the general principle, but since he is so categorical in his second article, I needed to inspect his reasoning more closely. It’s clear that our disagreement is more than superficial. The more I think about it, the more I disagree.

The state of the art

Pádraic maintains that one assertion per test is a rule that should always be followed unless there is a specific good reason to break it. I prefer it as a guideline, as does Robert C. Martin in the book Clean Code. The reference is not intended as an appeal to authority to “prove” that I’m right. I’m just making the point that I think this reflects the current state of the art, which is not necessarily perfect, of course.

How small is a behavior?

So what is the basis for wanting to enforce this principle as a strict rule? Pádraic claims that “1 behavior == 1 assertion”.

How does that make sense? The starting point is Behavior-Driven Development (BDD). I agree with the BDD principle that one test should test one behavior—as a guideline. It’s makes tests more focused, more readable and makes for better error messages when tests fail. Above all, it forces you to think about what is a behavior and what is not. And the idea of one assertion per test drives that thinking process.

From that point of view, “1 behavior == 1 assertion” seems totally logical. Unfortunately, it misses the point. It introduces a mechanical equivalence that negates the very thinking process I was just advocating. It makes the conceptual notion of a single, atomic behavior dependent on syntax and on the implementation specifics of a test framework.

For example, let’s say we want to test that a product object represents a certain model, a Logitech Harmony One remote control.

$this->assertEquals('Logitech', $product->make);
$this->assertEquals('Harmony One', $product->model);

It’s two assertions, but we’re only testing for one thing. We just want to make sure this product is what it should be and not something else. But let’s think some more about it. There are several ways we could make this into one assertion instead of two. This one, for instance:

$this->assertTrue(
'Logitech' == $product->make  &&  'Harmony One' == $product->model);

Not very pretty, is it? I think we can agree that this is not an improvement. More importantly, it doesn’t change the nature of what we are testing. We didn’t change what we’re testing, we just changed the way we express the test. A better way to get one assertion would be to make a custom assertion method such as assertLogitechHarmonyOne(). That would be more readable, but it wouldn’t help our statistics. The test framework would still see two assertions and report this as a 1 to 2 relationship between tests and assertions. And again, what we’re testing remains the same.

It gets worse. We’re testing one thing, but are we testing one behavior? No, we’re just testing the outcome. This is another logical flaw in the “1 behavior == 1 assertion” equivalence. A behavior is not tested by assertions, it’s tested by a complete test method. What happens before the assertions also counts, obviously. You could have multiple behaviors in the test with a single outcome that could be tested by a single assertion.

But I haven’t mentioned the simplest way to get one assertion per test. Just delete all assertions but one. It’s cheating, and problably toxic the quality of your tests, but it satisfies the criterion and generates good-looking statistics. It’s the thing that’s bound to happen if you try to force this on developers.

Better not use the “one assertion” principle as a strait jacket. It works better as exercise equipment to jog your mind.

Pádraic Brady pleads for the principle that a unit test method should have only one assertion. His point is perfectly valid; there are several good reasons why one assertion per method is a smart guideline when writing unit tests. But since he drags up the term “lazy”, I feel the need for soul-searching and confession.

The average for my current unit tests still hovers around three or four assertions per test method. And when I think about it, I see that the reason is that I’m not always taking the time to practice “one assertion per test” as a strict rule. Instead, it’s part of my problem-solving strategy when I start to get confused by my own tests; when the tests get so complex that I can’t quickly understand what’s going on where. Mock objects, in particular, contribute to this, since mock expectations specify what should happen later instead of testing what’s already happened.

Anyway, when I encounter this problem, known unsurprisingly as Complex Test or Obscure test, my first impulse is to make sure the code I’m testing (known as the system under test or SUT) is not too complex. Classes that are too large and methods that are too long are my prime examples of code that’s hard to test in a simple way. After I check the SUT, I look at the tests, and yes, then one of my goals is to move toward the “one assertion” ideal. It really is helpful at that point.

I’m open to the possibility that I should be more consistent in applying this principle. He may be right that there is a danger of “lazy/shallow testing”. In any case, as with all code, bugs and deficiencies are easier to spot the more readable it is.

Anyway, Uncle Bob held an extremely entertaining and useful introduction to the FitNesse testing tool. He got me hooked on it, but I’m even more fascinated by something else. I asked him, “what is the nature of the test API [you've been talking about]?” He answered that question, and also another, more general one: What is a good strategy for complete automated integration and acceptance testing of an application? That was the question I really wanted to ask, only I wasn’t quite aware that I wanted to ask it. And, impressively, he answered it anyway.

As I understood it, the strategy he outlined was as follows. The diagram is probably only a mild perversion of the one he drew on the whiteboard.

• Test all features through a test API just underneath the user inteface. These are the main acceptance tests; the ones that are used to decide whether a feature is done or not done yet.
• Test the user interface in isolation, running calls to a fake system underneath.
• Run just a few tests all the way through the application to test the plumbing between the user interface and the rest of the system.

The reason why we don’t want all our acceptance tests to go through the user interface is that the user interface has a tendency to change often, and changes in the user interface tend to breaks lots of tests.

This resonates reasonably well with my experience, although I think testing “everything” through the user interface can work sometimes, if the user interface is a very plain one.

In addition, we want unit tests for each small piece of the application. The tests outlined above are just catch the few bugs that are not detected by the unit tests.

This might need updating, since the testing capabilities of the Zend Framework have grown substantially since then. In particular, there is now a component called Zend_Test. I haven’t had time to study it closely yet, but I hope to do so soon.

So am I wrong? Did I get carried away? Did my critical faculty go on vacation somewhere nice and sunny? I admit that sometimes I deliberately look at the positive and ignore the negative. (And sometimes I do the opposite; It’s a good exercise if you’re careful.)

I wasn’t drunk, anyway. But let me take a closer look at the particular line of code I was praising:

$this->assertThat($form->hasSelect(withName('statusConfirm'))->hasValues(),
    array('Yes','No'));

My main point is that it’s close to plain English. Not everyone agrees that that’s a good thing, but I argue that we’re built (genetically wired, in fact) to understand natural languages, not program code. Therefore code should be easier to understand when it approximates natural language and expression. And we’re trying to create or approximate a Domain Specific Language (DSL), which should express exactly what’s required for the domain and not the demands of the technical implementation.

So for this experiment, let’s translate this one into a (plain English sentence:

Assert that Form (this particular one) has a select menu with the name “statusConfirm” and values “yes” and “no”

Translating back into code, it might look more like this:

$this->assertThat($form)->hasSelect()->withName('statusConfirm')->andValues('yes','no');

To me, this is even more natural than the other one. I think we’ve gotten rid of some syntax that has to do with implementation details rather than making the API simple to use.

It also seems clear to me how this could be implemented. All of the method calls could be to an assertion object that would take all these various inputs and always return itself at the end of the method so you can chain the calls in what’s known as a fluent interface.

Implementing automated tests by using Seleniums API methods has several drawbacks. Selenium is great for what it does, providing a generic framework for testing a generic application. Using the Testing_SeleniumDSL framework, I will show you how to create your own Domain Specific Language (DSL), which would allow you to write tests in the language of your business rather than in Seleniums language.

I’m quite impressed by the examples he presents, such as:

$this->assertThat($form->hasSelect(withName('statusConfirm'))->hasValues(),
    array('Yes','No'));

This is truly expressive, readable code. I’ve refactored web test code in this direction many times, but I admit I never got quite this far.

The DSL is planned as an open source release. It would be interesting to try something similar for the SimpleTest web tester, which is my favorite for testing web interfaces without too much JavaScript. (Based on the paparrazzi principle.)

The background is the idea that unit test methods, for the sake of readability, should test only one single behavior. This may mean several tests for one method under test, since one method may have several behaviors. One version of this is the notion of one assertion per test.

Web tests are not quite the same thing as unit test. On the Sitepoint forum, in a discussion on web testing, I said:

It’s also very slow to have too many separate test methods. So instead, to make it readable, I tend to use custom assertions.

Marcus Baker asked me for examples. I tried to illustrate the principle with an example that may be a something of a caricature.

If performance and speed were not an issue, I might do something like this, keeping each test method short and sweet: PHP Code:

class ArticleListTest extends WebTestCase {

    function setUp() {
        // Some code to log in and go to the article list page
        //....
    }

    function testHttpStatusOk() {
        //...
    }

    function testHasCorrectTitle() {
        //...
    }

    function testHasNoDebugInfoAccidentallyLeftBehind() {
        //...
    }
}

Unfortunately, it would be nauseatingly slow, since you have to traverse some web pages for every single test method. Instead, I can try to get similar small chunking by using custom assertions: PHP Code:

class VariousPageTests extends WebTestCase {

    function testLoginAndArticleList() {
        // Some code to log in
        //....

        $this->assertStartPage();

        // Some code to go to article list page
        //....

        $this->assertArticleListPage();
    }

    function assertArticleListPage() {
        $this->assertHttpStatusOk();
        $this->assertHasCorrectTitle();
        $this->assertHasNoDebugInfoAccidentallyLeftBehind();
    }

    function assertStartPage() {
        //
    }
}

This way, the code has names for approximately the same details as in the first example.

Several challenges presented themselves. It might seem as if the ZF controller system is not optimally designed for testing action controllers, but there are always ways to get around obstacles.

A ZF action controller (which is really just a group of actions collected in a class) always extends Zend_Controller_Action. (For an introduction, see the official QuickStart).

The one I was playing with was an authentication controller:

class AuthController extends Zend_Controller_Action...

That’s all we need to know about the specific class under test for now.

I always like to set up the basic objects for the test as instance variables in the setUp() method, so that I can write many small test methods exercising them in various ways. To even get started testing an action controller, we need to instantiate it outside its usual environment, the Zend Front Controller. It requires a request object and a response object:

class AuthControllerTest extends UnitTestCase {

  function setUp() {
    $this->request = new Zend_Controller_Request_Http();
    $this->response = new Zend_Controller_Response_Cli();
    $this->controller = new AuthController($this->request,$this->response);
  }

(I’m using SimpleTest here, but the same techniques should apply to PHPUnit with different syntax.)

The challenge increases increase when we want to use View Helpers, such as the flash messenger. When programming the action controller, it looks like this:

$flashMessenger = $this->_helper->FlashMessenger;

To test this meaningfully, we need to replace the flash messenger with a mock object. That means also having to replace $this->_helper, which is a “helper broker” object.

First, we generate the mock classes:

Mock::generate('Zend_Controller_Action_HelperBroker','MockHelperBroker');
Mock::generate('Zend_Controller_Action_Helper_FlashMessenger','MockFlashMessenger');

Unfortunately, the helper broker is a protected instance variable, with no apparent way to change it. But it’s easy to fix by adding a setter method to the action controller class:

class AuthController extends Zend_Controller_Action...

  public function setHelperBroker($helperBroker) {
    $this->_helper = $helperBroker;
  }

This allows us to set up the mock helper broker and let it return the flash messenger:

$this->helperBroker = new MockHelperBroker;
$this->controller->setHelperBroker($this->helperBroker);

$this->flashMessenger = new MockFlashMessenger;
$this->helperBroker->setReturnValue(
  '__get',$this->flashMessenger,array('FlashMessenger'));

Now we can use the mock objects to test an action which is supposed to send a flash message:

function testIdentifyActionSendsFlashMessage() {
    $this->flashMessenger->expectOnce(
        'addMessage',
        array('Please provide a username and password.'));
    $this->controller->identifyAction();
}

That’s as far as I’m going with this now. There may be more later.

I keep saying that sort of thing, too. I also say it raises your IQ. But when I attended a presentation by Uncle Bob (Robert C. Martin) last summer, I was mildly shocked to see that he described TDD as tedious (but a requirement for professionalism). I confronted him politely about it, emphasizing that above all I find TDD relaxing. Why? Because bugs bug me. I get stressed when I have to search for a bug and even more so when it takes much longer than expected (I’m sure that never happens to you). With TDD, bugs are always small and easy to locate.

Uncle Bob seemed to agree with me as far as his own experience was concerned, but he believed that many developers found the non-TDD process exciting rather than stressful.

Sounds weird to me. But I know from long experience that sometimes people are different from me in ways that make reality seem stranger than fiction.