Using the SBST-benchmark Framework

  1. Before you can use the SBST-benchmark Framework to benchmark your own tool X, you first need to implement a protocol that will facilitate interaction between the Framework and your tool X.

  2. Assume that you have implemented the protocol meant above; make a script called runtool that can invoke this protocol. Create a directory, ToolXBenchmarking, where the SBST-benchmark Framework can put its temporary files. Put runtool in this directory.

    1. Your tool X must output the generated test cases in JUnit4 format in ToolXBenchmarking/temp/testcases
    2. Your tool X must store intermediate data in ToolXBenchmarking/temp/data

  3. Copy the script runbenchmark to some conventient place, e.g. you can just put it in ToolXBenchmarking, and adapt it. The script looks like this: 
    #! /bin/sh
# Specify the dir where you put this script:
TOOL_HOME=somewhere/my_tool_X
# Specify the location of the SBST-benchmark Framework's jar file:
CONF=$TOOL_HOME/benchmarks.list
# Specify the locations of jacoco, pittest, and junit:
JAR=$TOOL_HOME/benchmarktool-0.0.2-jar-with-dependencies.jar
JUNIT_JAR=$TOOL_HOME/junit-4.10.jar
PITEST_JAR=$TOOL_HOME/pitest-0.31.jar
JACOCO_JAR=$TOOL_HOME/jacocoagent.jar
# Specify the commands to invoke javac and java:
JAVAC_CMD=javac
JAVA_CMD=java
# This is the command to start the benchmarking:
exec $JAVA_CMD -Dsbst.benchmark.jacoco="$JACOCO_JAR" -Dsbst.benchmark.java="$JAVA_CMD" -Dsbst.benchmark.javac="$JAVAC_CMD" -Dsbst.benchmark.config="$CONF" -Dsbst.benchmark.junit="$JUNIT_JAR" -Dsbst.benchmark.pitest="$PITEST_JAR" -jar "$JAR" $*
    Importantly, you need to specify the location of SBST-benchmark Framework's jar file.

    You can see that we also need the following libraries: jacoco (to measure code coverage), pitest (to do mutation tests), and junit.

  4. Specify which target classes are to be used in your benchmark. These are to be specified in the file benchmarks.list, which are linked from the above script. Here is an example of such a list.

  5. Run the runbenchmark script from there. The command has this syntax: 
    runbenchmark <benchmark> <tooldirectory> <debugoutput>
    benchmark is the name of a benchmark-group, as you have specified in your file benchmarks.list.
    tooldirectory is the directory where you have put your runtool script; so in our example it is somepath/ToolXBenchmarking.
    debugoutput is a boolean flag; if true the framework will also produce debugging outputs.

The Benchmark Automation Protocol

To let the SBST-benchmark Framework controls your tool X, you need to implement the protocol specified below.

Additionally, once the protocol is implemented, you need to provide a script or an executable named runtool that will invoke the protocol. This script will later be automatically invoked; but you need to provide it.

The protocol is a very simple line based protocol over the standard input and output channels. The following table describes the protocol, every step consists of a line of text received by your side of the protocol on STDIN or what it should sent to STDOUT.

STEP MESSAGES STDIN MESSAGES STDOUT DESCRIPTION
1 BENCHMARK Signals the start of a benchmark run; directory $HOME/temp is cleared
2 directory Directory with the source code SUT
3 directory Directory with compiled class files of the SUT
4 number Number of entries in the class path (N)
5 directory/jar file Class path entry (repeated N times)
6 number Number of classes to be covered (M)
7 CLASSPATH Signals that the testing tool required additional classpath entries
8 Number Number of additional class path entries (K)
9 directory/jar file Repeated K times
10 READY Signals that the testing tool is ready to receive challenges
11 class name The name of the class for which unit tests must be generated.
12 READY Signals that the testing tool is ready to receive more challenges; test cases in $HOME/temp/testcases are analyzed; subsequently $HOME/temp/testcases is cleared; goto step 11 until M class names have been processed

Test Case Format

The tests generated by tool X should be in the form of java files containing JUnit4 tests.

The generated test cases will be compiled against