Java Code Coverage Tools

Originally authored by Doug Galligan.

Overview

Code coverage provides an insight into areas of the code not exercised through testing or operation of a program. While code coverage is no guarantee of proper test coverage, lack of code coverage is a guarantee of missing test coverage. These tools aid developers and white box testers in prioritizing and constructing tests that exercise the code base.

Java has several products both open and close sourced that provide code coverage. To date two products have been evaluated. EMMA and Atlassian Clover. Other products not investigated where jcoverage and off-shoot Cobertura. Having been largely a solved problem for many years, most of the open source projects are not actively developed. Most differentiating features are focused on easy of use, report formats and tooling support (ant, maven, Eclipse etc.).

OCAP-RI Requirements

• The RI runs the JavaME 1.4 JVM. This JVM provides a sub-set of APIs from the JSE environment. In particular Clover uses String.replaceAll() in it's source code instrumentation. This is not available in the RI's JVM, and eliminates this product from consideration
• The RI currently does not properly shutdown the JVM on exit. Unfortunately this is the most popular hook for writing coverage data to disk. Most products allow other ways to dump coverage data through APIs embedded in the application or IPC through sockets.
• The RI uses a custom class loader. Most products have a process where they replace the class loader and instrument the code on the fly. The product used by the RI must have an offline process for instrumenting the code.
• For CTP testing the RI is restarted prior to the test being run. The product needs to have an ability to merge coverage data from prior runs in order to provide a full coverage report for a suite of CTP tests.

EMMA meets all of the above requirements.

EMMA Implementation

EMMA Website The version used as of this writing is emma-stable-2.1.5320-lib.zip.

The offline instrumentation was used in order to work with the RI. EMMA has an option to instrument a jar file in place. Also if a jar file is already implemented it will leave it alone. This allows the runRI.sh script to blindly instrument at the beginning of a coverage run without regard to previous attempts to instrument.

Installing the JAR File

In order to access the emma.jar file at runtime the RI needs to tell the class loader where to find the jar file. Instead of modifying the mpeenv.ini file, this page follows the method used for junit support.

1. Place the jar file into $OCAPROOT/tools/generic/emma/emma.jar
2. Add emma.jar to the build.test.CLASSPATH variable in $OCAPROOT/java/OCAP-debug.properties (example below)
3. Go to $OCAPROOT/java
4. ant build.test.lib  (rebuilds support-test.jar)

build.test.CLASSPATH= \
    tools/generic/junit3.8.1/junit.jar \
    tools/generic/GroboUtils-1/lib/core/GroboTestingJUnit-1.0.0-core.jar \
    tools/generic/emma/emma.jar

Using EMMA in RI

The runRI.sh script includes the option -coverage that will instrument the code and in the case of CTP tests dump the coverage prior to resetting the RI.

instr_ri()
{
   if [ -f $OCAPROOT/tools/generic/emma/emma.jar ]
   then
      echo "Instrumenting RI"
      java -cp $OCAPROOT/tools/generic/emma/emma.jar emma instr -m overwrite -cp $envDir/sys/ocap-classes.jar
   else
      emma_error
   fi
}
dump_coverage()
{
   if [ -f $OCAPROOT/tools/generic/emma/emma.jar ]
   then
      echo "Dumping Coverage Information"
      java -cp $OCAPROOT/tools/generic/emma/emma.jar emma ctl -c coverage.get,,true,false
   else
      emma_error
   fi
}

There are more options than the ones presented here, for a full listing please refer to the EMMA website. ie. coverage for subset of code.To run these outside of the runRI.sh script swap $envDir for $OCAPROOT/bin/$OCAPTC/env.

Two files will be created.

At this point your ocap-classes.jar file is instrumented. In order to go back to normal. 

1. cd $OCAPROOT/java
2. ant

CTP Test Issue

EMMA does not set the SO_REUSEADDR socket option, and therefore needs to wait till the socket is closed by the operating system. This is problematic with the CTP tests that expect the RI to become available shortly after reset. The current default timeout after reset is 10 seconds (10 1 second tries). The parameter needs to be adjusted up 60 seconds in the atelite/config/main.cfg file.

atelite.helper.resetiut         = ate4ri --host=127.0.0.1 --reconnect=10 boot && test_if4ri --host=127.0.0.1 --reconnect=10 log `_gettpprop tpid`

to

atelite.helper.resetiut         = ate4ri --host=127.0.0.1 --reconnect=10 boot && test_if4ri --host=127.0.0.1 --reconnect=70 log `_gettpprop tpid`

Generating Reports

Reports are generated by combining the files generated by instrumentation and dumping code coverage. From $PLATFORMROOT run the following command.

java -cp $OCAPROOT/tools/generic/emma/emma.jar emma report -r html -in coverage.em,coverage.ec \
-sp $OCAPROOT/java/src/hn,$OCAPROOT/java/src/base,$OCAPROOT/java/src/dvr,$OCAPROOT/java/src/fp,$OCAPROOT/java/src/msm,$OCAPROOT/java/src/svm

There are many options that you can pass to the reporting command, for a full listing please refer to the EMMA website. ie. sorting of columns.

Capturing Coverage when Running Xlets or the MSO Guides

No changes are required to the Xlets. Instrumenting the stack is done as described above.

$ ./runRI.sh  -deletelog -deletestorage -coverage
deleting RILog.txt
deleting all storage
Instrumenting RI
EMMA: processing instrumentation path ...
EMMA: instrumentation path processed in 3984 ms
EMMA: [2280 class(es) instrumented, 854 resource(s) copied]
EMMA: metadata merged into [c:\CablelabsRI2\OCAPRI\trunk\ri\RI_Platform\coverage
.em] {in 266 ms} run_for_win32

Exercise the Xlet or Guide and periodically dump coverage data. In another shell in the $PLATFORMROOT directory enter the java command below.

$  java -cp $OCAPROOT/tools/generic/emma/emma.jar emma ctl -c coverage.get,,true,false
EMMA: processing control command sequence ...
EMMA: executing [coverage.get (coverage.ec,true,false)] ...
EMMA: coverage.get: local copy of coverage data merged into [c:\CablelabsRI2\OCAPRI\trunk\ri\RI_Platfor\coverage.ec] {in 63 ms}
EMMA: coverage.get: command completed in 125 ms
EMMA: control command sequence complete

When testing is complete generate a coverage report using the command described in the previous section.

Note: Pressing CTRL-C to terminate the RI throws an EMMARuntimeException which is benign and may be safely ignored.

emma ctl: coverage.get: RPC failure while executing [coverage.get]
Exception in thread "main" com.vladium.emma.EMMARuntimeException: coverage.get:
RPC failure while executing [coverage.get]

Current Limitations

Currently the last test of a CTP test run will not be dumped, as the dump only occurs on resetting the RI. Ctrl-C on the runRI.sh script will attempt a dump, but is met with mixed results. You can either re-run a CTP test or execute a dump from the command line.

If you want to dump the coverage at any time the RI is running, then you will need to execute the command above used in the dump_coverage() function.