Increasing Cucumber Test Speed using Annotations

Cucumber is a neat framework for writing English language specifications, sometimes called Behaviour Driven Development or Specification By Example.

We are using Cucumber to test an API, and so we have perhaps strayed from the true path and used language which is not necessarily at the business logic level, however it does serve well as documentation for its intended audience.

One of the features of Cucumber is a test template, called a Scenario Outline, which enables one to parameterize a test with values:

  Scenario Outline: Calling by code gives full title
    Given a thing with code <code> and title <title>
    When calling "/rest/thing/<code>"
    Then expect json:
    """
      {"thing":{"code":"<code>", "title": "<title>"}}
    """
  Examples:
    |code|title  |
    |a   |Apples |
    |b   |Bananas|
    |c   |Cucumbers|

This is very powerful and enables one to add a lot of tests very quickly.

If your test setup takes non-negligible time, such as creating a clean new database, then you will quickly create a test suite whose running time is heading towards hours.

We addressed this by only recreating the database when necessary. We know when that is: it is when the test alters the database in such a way as to interfere with it or other tests being run again. Initially I thought of this quality as idempotency however I realised what I was actually talking about was repeatability. If your test creates a record with a field which has a unique constraint then you will need to delete that record before that test can be repeated.

We can use the Scenario Outline functionality to repeat a test, if it is not already parameterized, by simply adding an unused parameter.

  Scenario Outline: Create a thing
    When putting "/rest/thing/b/Banana"
    Then expect json:
    """
      {created:{"thing":{"code":"b", "title": "Banana"}}}
    """
    Then remove thing b
  Examples:
    |go|
    |one|
    |two|

We now know that this test can be run twice, so we are justified in tagging it @repeatable.

Initialisation when needed

If a test is @repeatable then we know that we do not need to create a clean database.

The Java snippet below sets the variable DIRTY if the test is not tagged @repeatable.

  private static boolean DIRTY = true;

  @Before(value = "@reuseCurrentDb", order = 1)
  public void reuseCurrentDb() {
    DIRTY = false;
  }
  @Before(order = 2)
  public void resetDB() {
    if (DIRTY) {
      try {
        if (CONNECTION == null) {
          CONNECTION = init(dataSource_.getConnection());
        }
        importDB(CONNECTION, dataSet_, DatabaseOperation.CLEAN_INSERT);
      }
      catch (Exception e) {
        throw new RuntimeException("Import failed before test run", e);
      }
    }
  }


  @After(order = 2)
  public void afterAll() {
    DIRTY = true;
  }


  @After(value = "@repeatable", order = 1)
  public void afterRepeatable(Scenario scenario) {
    if (!scenario.isFailed()) {
      DIRTY = false;
    }
  }

In retrospect it looks as though we should have inverted this, with a tag @dirties, as now almost every test is tagged @repeatable. However that does not take into account the development sequence: the test is first made to run once and then is made @repeatable.

Anticipated and Unanticipated Wins

This was intended to speed up our tests and did: from 48 minutes to 6 minutes.

The unintended win was that by focussing on cleanup we discovered three database tables which did not have correct deletion cascading.

This approach may surface other hidden problems, either with your tests or with the system under test; this is a good thing.

Update (2014-06-19)

By chipping away at the cascades, adding methods to capture is from returned JSON and adding deletion methods one is able to reduce the database creations to one, and any dirtying of the database throws an exception.

761 Scenarios (761 passed)
10364 Steps (10364 passed)
1m40.157s

  private static boolean DIRTY = true;

  private static HashMap rowCounts_;

  private HashMap rowCounts() {
    if (rowCounts_ == null) {
      rowCounts_ = setupTableRowCounts();
    }
    return rowCounts_;
  }

  @Before(order = 1)
  public void resetDB() {
    if (DIRTY) {
      try {
        if (I_DB_CONNECTION == null) {
          I_DB_CONNECTION = init(dataSource_.getConnection());
        }
        importDB(I_DB_CONNECTION, dataSet_, DatabaseOperation.CLEAN_INSERT);
      }
      catch (Exception e) {
        throw new RuntimeException("Import failed before test run", e);
      }
    }
  }


  private HashMap setupTableRowCounts() {
    HashMap tableRowCounts = new HashMap();
    try {
      String[] normalTables = { "TABLE" };
      DatabaseMetaData m = I_DB_CONNECTION.getConnection().getMetaData();
      ResultSet tableDescs = m.getTables(null, null, null, normalTables);
      while (tableDescs.next()) {
        String tableName = tableDescs.getString("TABLE_NAME");
        tableRowCounts.put(tableName, I_DB_CONNECTION.getRowCount(tableName));
      }
      tableDescs.close();
    }
    catch (SQLException e) {
      throw new RuntimeException(e);
    }
    return tableRowCounts;
  }

  private void checkTableRowCounts(Scenario scenario) {
    try {
      DatabaseMetaData m = I_DB_CONNECTION.getConnection().getMetaData();
      String[] normalTables = { "TABLE" };
      ResultSet tableDescs = m.getTables(null, null, null,
          normalTables);
      String problems = "";
      while (tableDescs.next()) {
        String tableName = tableDescs.getString("TABLE_NAME");
        int old = rowCounts().get(tableName);
        int current = I_DB_CONNECTION.getRowCount(tableName);
        if (old != current) {
         problems += " Table " + tableName + " was " + old 
                     + " but now is " + current + "\n";
        }
      }
      tableDescs.close();
      if (!problems.equals("")) {
        problems = "Scenario " + scenario.getName() + " :\n" + problems;
        throw new RuntimeException(problems);
      }
    }
    catch (SQLException e) {
      throw new RuntimeException(e);
    }
  }


  @After(order = 2)
  public void afterAll(Scenario scenario) {
    if (scenario.isFailed()) {
      DIRTY = true;
    }
    else {
      DIRTY = false;
      try {
        checkTableRowCounts(scenario);
      }
      catch (RuntimeException e) {
        DIRTY = true;
        throw e;
      }
    }
  }