100 percent test coverage is the starting point, not the impossible end point.
For interpreted languages every line must be exercised by the tests or
you will discover a syntax error caused by a typo during a demo. The tests take the place of the compiler in other languages.
For compiled languages the main, or original, purpose of tests is to ensure against regressions: new code breaking old code.
A relatively new purpose for tests has been as the medium of Test Driven Development. The tests are written before the code, as scaffolding, and high test coverage
is a side effect.
Perfectly reasonable unit tests in Java can lead to less than complete coverage.
Tests which assert that the right thing happens under normal circumstances are of great value in ensuring backwards compatibility.
The value of the remaining tests can be questioned: why test exception handling?
This question should perhaps be reversed: if it is not worth testing then why is it in the code?
So lets take for granted that a high percentage test coverage is normal.
I want to argue that it is worthwhile putting in the effort to get from high to total test coverage.
Reasons to achieve total coverage
Martin Fowler in his article on Test Coverage
sets up a straw man that is really a rant against Goodhart's Law : writing tests to hit a percentage is not the reason to write tests.
By achieving 100 per cent coverage you finesse the whole 'acceptable percentage' argument.
In my experience there are bugs in uncovered code. Uncovered code is likely to be code that has never been run. How often do programmers write code which runs as expected first time?
The final push to one hundred per cent can be seen as a separate review stage of the project, highlighting problems with algorithms or the remnants of ideas which were abandoned during development. This is particularly valuable in one person projects, where changing to the review perspective can be difficult.
A practical benefit to the programmer of Total Test Coverage, if you are guided by a test coverage tool such as Cobertura and JaCoCo, is that you do not keep returning to the same nearly-covered class, to remember you have tested it to the current limits of your tools. Removing the need to remember that there is a good reason why a particular class is not fully covered is a significant benefit.
One such limit in Java has been with the language since its creation. It is caused by exception handling forced on the developer by language classes which throw checked exceptions (IOException I'm looking at you) or by checked exceptions in an API (eg SQLException in the JDBC API).
A silly snag is the private constructor pattern. This involves writing a constructor with private access for static classes
to ensure that its methods are only accessed statically. An unwanted side effect of this approach is that the constructor is never called
and so shows up as uncovered in our coverage report. There is a mechanism for instantiating these objects in tests, which is so ugly it would never be used in live code, is shown.
Another limit to complete coverage I have discovered was introduced with Java Generics: inaccessible generated bridging methods.
Here I present techniques for addressing these issues to remove the barriers to complete coverage.
Instantiating static classes with private constructors
We do not care about the non-coverage of constructors which have private access, we just don't want them to appear on the report and we do not want to clutter the code with annotations to instruct the coverage tool to ignore the method.
@Test
public void testPrivateConstructors() throws Exception {
excercisePrivateConstuctor(StaticUtils.class);
}
public static void excercisePrivateConstuctor(final Class<?> clazz)
throws NoSuchMethodException, InvocationTargetException,
InstantiationException, IllegalAccessException {
assertTrue("There must be only one constructor", clazz.getDeclaredConstructors().length == 1);
final Constructor<?> constructor = clazz.getDeclaredConstructor();
assertTrue("The constructor is accessible", constructor.isAccessible() == false);
assertTrue("The constructor is not private", Modifier.isPrivate(constructor.getModifiers()));
constructor.setAccessible(true);
constructor.newInstance();
constructor.setAccessible(false);
for (final Method method : clazz.getMethods()) {
if (method.getDeclaringClass().equals(clazz)) {
assertTrue("There exists a non-static method:" + method,
Modifier.isStatic(method.getModifiers()));
}
}
}
Calling hidden generated methods by introspection
During the development of a CSV Unifier I discovered that the bridging methods generated by Java to enable generics are shadowed and hence not available to the programmer
other than by introspection.
My model of a CSV sheet implements the Java generic interface Map,
ie a sheet is a Map<String, CsvRecord> and a CsvRecord is a
Map<String, CsvField>.
The generic interface method Map.put
V put(K key, V value);
The implementation which, due to generic type erasure, has the same type as the non-generic form.
@Override
public CsvField put(String key, CsvField field) {
if (!field.getColumn().getName().equals(key))
throw new CsvInvalidKeyException(
"Key (" + key + ") not equal to " +
"Field column name (" +
field.getColumn().getName() + ")");
return addField(field);
}
Bridging methods
During the compilation of generic code Java quietly generates what are called bridging methods these are visible to Cobertura but not to you, the coder, so Cobertura tries to tell you that you have not exercised these methods by marking the class definition line as not covered. The following code will print out all methods including
generated bridging methods
for (Method m : CsvTable.class.getMethods()) {
System.out.println(m.toGenericString());
}
public net.pizey.csv.CsvRecord get(java.lang.Object)
public java.lang.Object get(java.lang.Object)
public net.pizey.csv.CsvRecord put(java.lang.String,net.pizey.csv.CsvRecord)
public java.lang.Object put(java.lang.Object,java.lang.Object)
public net.pizey.csv.CsvRecord remove(java.lang.Object)
public java.lang.Object remove(java.lang.Object)
The put(Object key, Object value) method cannot be accessed normally
as it is masked by our generic version.
We can however invoke it using introspection:
public void testBridgingPut() {
CsvTable t = new CsvTable("src/test/resources/sheet2.csv",
UnificationOptions.LOG);
Object o = t.get((Object) "1");
Method method = t.getClass().getMethod("put",
new Class[] { Object.class, Object.class });
method.invoke(t, "jj", o);
}
Another approach to bridging methods is drastic: do not implement the
generic version. The interface Clonable is not a generic interface,
but it can be applied to generic classes. The generic version of clone() for the CsvRecord class would return a generic class CsvRecord
@Override
public CsvRecord clone() {
return this.clone(this.getTable());
}
Unfortunately this would generate a hidden, non-generic, bridging method:
@Override
public Object clone() {
return this.clone(this.getTable());
}
This bridging method could only be accessed by introspection. In this case I recommend writing the non-generic version in your code, avoiding it being generated for you.
JDBC Exception handling sniper
All calls to the JDBC API throw SQLException. Databases are expensive to mock
and best practice is to test against a pure java database such as HSQLDB or Apache Derby.
However this will lead either to untested catch blocks or declaring SQLException in your method calls.
To address this I wrote a JDBC Error Injector which enables the programmer to target a particular exception handling block and exercise it.
How it works
Each interface within the API has a decorator whose constructor takes
an instance. Any of the methods which return another instance of the API
will now return a decorated instance.
The decorated instance can be told to throw an Exception, either
whenever it is called or after being called a number of times.
This enables you to cover cases that would otherwise be impossible to
cover without a custom mock.
Using a decorated driver
You can decorate any JDBC Driver in your tests
such that it returns a ThrowingConnection
instead of a Connection,
then use the ThrowingConnection
as you would the Connection.
package org.melati.poem.dbms.test.sql;
import java.sql.Driver;
import org.hsqldb.jdbcDriver;
import org.melati.poem.dbms.test.sql.ThrowingDriver;
/**
* A decorated Hsqldb jdbcDriver.
*/
public class HsqldbThrowingJdbcDriver
extends ThrowingDriver
implements Driver {
public HsqldbThrowingJdbcDriver() {
super(new jdbcDriver());
}
}
The package provides the following classes.
package org.melati.poem.dbms.test.sql;
import java.sql.Driver;
/**
* A {@link Driver} decorated to throw an SQLException on command.
*
*/
public class ThrowingDriver
extends ThrowingDriverVariant
implements Driver {
/**
* Constructor.
* @param d the driver to decorate
*/
public ThrowingDriver(Driver d) {
it = d;
}
}
The code is complicated by conditional inclusion of JDBC3 or JDBC4 code dependant upon JDK version:
package org.melati.poem.dbms.test.sql;
public abstract class ThrowingDriverVariant extends ThrowingDriverJdbc4 {
}
public abstract class ThrowingDriverJdbc4
extends ThrowingDriverJdbc3
implements Driver {
}
For each method which returns an object from within the API, ie a JDBC object,
the shouldThrow method is called to check whether this method should throw an exception on this invocation.
package org.melati.poem.dbms.test.sql;
import java.sql.Connection;
import java.sql.Driver;
import java.sql.DriverPropertyInfo;
import java.sql.SQLException;
import java.util.Properties;
/**
* The JDBC3 members of a {@link Driver}, decorated to throw
* an SQLException on command.
*
* @author timp
* @since 5 Feb 2008
*
*/
public abstract class ThrowingDriverJdbc3
extends Thrower
implements Driver {
Driver it = null;
/**
* {@inheritDoc}
* @see java.sql.Driver#acceptsURL(java.lang.String)
*/
public boolean acceptsURL(String url) throws SQLException {
if (shouldThrow(this.getClass().getInterfaces()[0], "acceptsURL"))
throw new SQLException("Driver bombed");
return it.acceptsURL(url);
}
/**
* Return the decorated Connection.
* {@inheritDoc}
* @see java.sql.Driver#connect
*/
public Connection connect(String url, Properties info)
throws SQLException {
if (shouldThrow(this.getClass().getInterfaces()[0], "connect"))
throw new SQLException("Driver bombed");
return new ThrowingConnection(it.connect(url, info));
}
/**
* {@inheritDoc}
* @see java.sql.Driver#getMajorVersion()
*/
public int getMajorVersion() {
return it.getMajorVersion();
}
/**
* {@inheritDoc}
* @see java.sql.Driver#getMinorVersion()
*/
public int getMinorVersion() {
return it.getMinorVersion();
}
/**
* {@inheritDoc}
* @see java.sql.Driver#jdbcCompliant()
*/
public boolean jdbcCompliant() {
return it.jdbcCompliant();
}
/**
* {@inheritDoc}
* @see java.sql.Driver#getPropertyInfo
*/
public DriverPropertyInfo[] getPropertyInfo(
String url, Properties info) throws SQLException {
if (shouldThrow(this.getClass().getInterfaces()[0], "getPropertyInfo"))
throw new SQLException("Driver bombed");
return it.getPropertyInfo(url, info);
}
}
Each JDBC Class is decorated by extending the Thrower class:
package org.melati.poem.dbms.test.sql;
import java.util.Hashtable;
/**
* A class which can throw on demand.
*
* @author timp
* @since 10 Feb 2007
*
*/
public abstract class Thrower {
static Hashtable throwers = new Hashtable();
protected Thrower() {}
/**
* Tell named method to start throwing exceptions.
* @param i Interface class object
* @param methodName name in class.methodName format
*/
public static void startThrowing(Class i, String methodName) {
String fullName = i.getName() + "." + methodName;
throwers.put(fullName, new Integer(1));
}
/**
* Tell named method to start throwing exceptions.
* @param i Interface class object
* @param methodName name in class.methodName format
*/
public static void startThrowingAfter(
Class i, String methodName, int goes) {
String fullName = i.getName() + "." + methodName;
throwers.put(fullName, new Integer(1 + goes));
}
/**
* Tell named method to stop throwing exceptions.
* @param i Interface class object
* @param methodName name in class.methodName format
*/
public static void stopThrowing(Class i, String methodName) {
String fullName = i.getName() + "." + methodName;
throwers.put(fullName, new Integer(0));
}
/**
* Check whether method should throw,
* called once for every method invocation.
* @param i Interface class object
* @param methodName name in class.methodName format
* @return whether method named should throw exception
*/
public static boolean shouldThrow(Class i, String methodName) {
String fullName = i.getName() + "." + methodName;
if (throwers.get(fullName) == null)
throwers.put(fullName, new Integer(0));
int toGo = ((Integer)throwers.get(fullName)).intValue();
if (toGo == 0)
return false;
else {
toGo = toGo - 1;
throwers.put(fullName, new Integer(toGo));
return toGo == 0 ? true : false;
}
}
}
Simple Example
You notice that there is uncovered Exception handling associated
with failure of ResultSet.close() during database initialisation, tested here in the method getDb().
public void testConnect() {
ThrowingResultSet.startThrowing(ResultSet.class, "close");
try {
getDb();
fail("Should have blown up");
} catch (SQLSeriousPoemException e) {
assertEquals("ResultSet bombed",
e.innermostException().getMessage());
}
ThrowingResultSet.stopThrowing(ResultSet.class, "close");
}
Sub-classed Test Example
The test (DatabaseTest) is written to test the functionality in the normal way, then subclassed with a ThrowingConnection
to test the exception handling.
public class org.melati.poem.test.throwing.DatabaseTest
extends org.melati.poem.test.DatabaseTest {
private static Database db;
// connect using throwing driver
private static Database getDb() {
db = new PoemDatabase();
db.connect("m2", "org.melati.poem.dbms.test.HsqldbThrower",
"jdbc:hsqldb:mem:m2",
"sa",
"",
4);
assertEquals("org.melati.poem.PoemDatabase",
db.getClass().getName());
assertEquals(4, db.getFreeTransactionsCount());
return db;
}
public void testFirstObject() {
ThrowingResultSet.startThrowing(ResultSet.class, "next");
try {
super.testFirstObject();
fail("Should have bombed");
} catch (SQLSeriousPoemException e) {
assertEquals("ResultSet bombed",
e.innermostException().getMessage());
}
ThrowingResultSet.stopThrowing(ResultSet.class, "next");
}
}
Throw on Third Call Example
The exception handling you want to excercise is actually the third call
to that method in your test's trajectory to the method under test.
public void testGetObjectInt() {
ThrowingConnection.startThrowingAfter(
Connection.class,"prepareStatement", 2);
try {
super.testGetObjectInt();
fail("Should have blown up");
} catch (SimplePrepareFailedPoemException e) {
assertEquals("Connection bombed",
e.innermostException().getMessage());
} finally {
ThrowingConnection.stopThrowing(Connection.class,
"prepareStatement");
}
}
I hope that this is of use, or helps you scratch that itch and that you too can achieve total test coverage.
