T3 Manual

T3 is a tool to automatically test a Java class.
Author: Wishnu Prasetya
License: GPL v3

T3 is a light weight testing tool to automatically test Java classes. Given a target class CUT, T3 randomly generates a set of test-sequences against CUT. Each sequence starts in principle with the creation of an instance of CUT followed by calls to the object’s methods, or updates to its fields. T3 can generate a large amount of such test sequences to trigger faulty behavior. Pre and post-conditions can be expressed using in-code assertions. A convenient way to specify custom values and objects generators is supported.

T3 is the successor of T2. The main idea is still the same as T2, however the engine has been completely revamped. T3 makes a lot of use of Java 8's closures. It also has the option to use streams, to generate test-sequences in parallel. Another major difference is that T3 offers a nicer way for the user to specify custom value generators.

A simple example.

Here is a simple class implementing a sorted list of integers. The list is sorted in ascending order. It has a method to insert an element, and a method to retrieve the greatest element from the list.

package Examples;

import java.util.LinkedList;

// Sorted list of integers; in ascending order.
public class SimpleIntSortedList {
	
    private LinkedList s;
    private Integer max;

    public SimpleIntSortedList() { s = new LinkedList(); }

    public void insert(Integer x) {
    	assert x!=null : "PRE";
        int i = 0;
        for (Integer y : s) {
            if (y > x) break;
            i++;
        }
        s.add(i, x);
        // bug: should be x > max
        if (max == null || x < max) max = x;
    }

    // Retrieve the greatest element from the list, if it is not empty.
    public Integer get() {
    	assert !s.isEmpty() : "PRE";
        Integer x = max;
        s.remove(max);
        if (s.isEmpty()) max = null ;
        else max = s.getLast() ;
        assert s.isEmpty() || x >= s.getLast() : "POST";
        return x;
    }
    
    // a class invariant
    private boolean classinv__() {
        return s.isEmpty() || s.contains(max);
    }
}

To run T3 to test this class, we can do it like this:

java -ea -cp T3.jar Sequenic.T3.T3Cmd Examples.SimpleIntSortedList

Specify the full name of the target class that we want to test. In the above example it is: Examples.SimpleIntSortedList. Before that we can specify a list of options; for now, we pass no options.
Enable Java assertions with the -ea option.
Assertions that are to be interpreted as pre-conditions should be marked by the string "PRE", as in the above example. Test-sequences that violate pre-conditions are not valid test-sequences, and are dropped. Thrown IllegaArgumentException is treated the same as violating a pre-condition. Violating other assertions count as an error.
T3 also reports other kind of uncaught exceptions.
A class invariant can be specified in a private method named classinv__() that returns a boolean, as in the above example. This method is automatically called by T3 after every test-step in every test-sequence it generates. If the method returns false, or when it throws an exception, it counts as an error.

The run will produce a summary like the one below:

Suite size : 302           -- T3 generated a set of 302 test-sequences
Average length : 9.364239  -- the average length of sequences, in number of steps
Executed   : 2             -- post-executed test-sequences, see explanation below
Violating  : 1             -- error found
Invalid    : 0             -- number of invalid test-sequences
Failing    : 0             -- number of failing/broken test-sequences
Runtime    : 9 ms

T3 peforms its work in two rounds: it first generate the test suite, and then in the second round it reports the finding. To report the finding, it actually re-run the suite. The above statistic says that in the first round T3 generates a suite of 302 test-sequences. In the second round it will re-run every test-sequence it generated. By default, it is configured to stop at the first violation/error it finds. The statistic says that it re-executed two test-sequence, and found one violation (which thus must be in the 2nd sequence). None of the executed test-sequences are invalid (violating pre-conditions) or broken (fails to run, for some reason). Invalid and broken sequences are droppped during the generate-phase. However, if the SUT is non-deterministic, previously valid and unbroken test-sequence can still become bad when re-run.

When T3 finds an error, it will print the thrown exception, and a trace of the test-sequence, as shown below. Every test-step preceeding the error will be shown, by default up to previous 10 steps.

The report below is caused by a mistake in the method insert, which incorrectly updates the internal variable max. As a result, the method get may not return the greatest element, which is detected by the post-condition assertion in the method. The violating execution is essentially:

Create an empty list.
Call insert(0).
Call insert(8), here max was incorrectly updated.
Call get(), its assertion detects the error.

** [step 0] CON Sequenic.T3.Examples.SimpleIntSortedList() -- calling constructor
   -- returned value:
    (Sequenic.T3.Examples.SimpleIntSortedList) @ 0 
    s (LinkedList)  @  1
    max null
** [step 1] CALL insert on REF 0                           -- calling insert(0)
   with
   (0)
   -- state of the receiver object:                        -- the state of the list is shown here:
    (Sequenic.T3.Examples.SimpleIntSortedList) @ 0
    s (LinkedList)  @  1
         [0] (Integer) : 0
    max (Integer) : 0
** [step 2] CALL insert on REF 0                           -- calling insert(8)
   with
   (8)
   -- state of the receiver object:
    (Sequenic.T3.Examples.SimpleIntSortedList) @ 0
    s (LinkedList)  @  1
         [0] (Integer) : 0
         [1] (Integer) : 8
    max (Integer) : 0                                      -- look, max has an incorrect value
** [step 3] CALL get on REF 0                              -- calling get()
   with
   ()
   >> VIOLATION!                                           -- a violation is reported
   -- state of the receiver object:
    (Sequenic.T3.Examples.SimpleIntSortedList) @ 0
    s (LinkedList)  @  1
         [0] (Integer) : 8
    max (Integer) : 8   -- thrown exception:
java.lang.AssertionError: POST
	at Sequenic.T3.Examples.SimpleIntSortedList.get(SimpleIntSortedList.java:62)
    ...

T3's pair-wise algorithm and options

Let CUT be the target class. The class is testable if it is not abstract nor an interface. If it is not, T3 will test some of its testable subclasses instead, if it can find them. The default is 2, but this can be changed.

Now, assume CUT is testable. When testing it, T3 pretends to be a member of a client package, say P. By default, this is the package to which the CUT belongs to.The following members will be put in the testing scope:

All constructors declared by CUT, which are visible from P.
All methods declared and inherited by CUT, which are visible from P. Inherited methods which are overriden in CUT are not included. Methods inherited from Object are excluded.
All fields declared and inherited by CUT, which are visible from P. Inherited fields which are shadowed in CUT are not included. Fields inherited from Object are excluded.

Internally, T3 generates an ADT (Abstract Data Type) test suite, and a non-ADT suite. They are then added together to make the final test suite.

Each test-sequence in an ADT-suite starts with the creation of an instance of the CUT, and every step along the sequence affects or inspects this target instance/object. Obviously this requires that CUT can be instantiated. This can be done by calling a constructor of CUT, if there is one. T3 also looks for creator methods, which are static methods of CUT, with CUT as return type, and does not have a parameter x of type CUT.

Sequences in the non-ADT-suite focus on testing static methods and fields of CUT.

ADT-suite.

An ADT-suite consists of two sub-suites: C-suite and P-suite. A C-suite consists of sequences that start with the creation of an instance of CUT (through a constructor or a creator method), followed by a suffix. The suffix is a sequence of steps; the default length is 3. Each step can be an update to a field of the target object, or a call to a method of the target object. For each constructor or creator method which is within the testing scope, maxSamplesToCollectForEachMethod number of test sequences will be generated; the default is 100.

A P-suite consists of sequences, each has the form of c++σ++[m1,m2]++τ, where c is a step that creates the target object; σ is a prefix, and τ is a suffix. The default length of σ is 8, that of τ is 3. [m1,m2] is a pair methods from the testing scope, and m1 is a mutator. A mutator method is a method that can change the state of the target object. T3 does not currently implement a precise algorithm to identify which methods are mutators. All methods whose names do not start with "get" or "is" are assumed to be mutators. For each such pair [m1,m2], T3 will generate a bunch of P-type test sequences (so, this is basically pair-wise testing). T3 will try to distribute the sequences, so that each method in the testing scope will be covered by at least maxSamplesToCollectForEachMethod number of P-type sequences; the default is 100.

Currently there is a limit of 1000 pairs. If this number would be exceeded, then T3 only generate sequences for these kinds of pairs:

Pairs of the form [m,m], where m is a method in the testing scope.

Pairs of the form [m1,m2], for each mutator m1. However, it will only be paired with randomly chosen k number of m2's. This k is calculated as such, that the total number of pairs would be around the limit 1000.

Since it can be very hard to generate a valid sequence, there is no guarantee that each method will be covered by maxSamplesToCollectForEachMethod number of test sequences. Neither can we guarantee that, for example, each P-sequence will be a full c++σ++[m1,m2]++τ. However, T3 will strive to meet these requirements. If T3 fails to extend the sequence it currently tries to generate with a next step, it will retry a new next step for some K number of times.

Non-ADT-suite.

A non-ADT suite consists of sequences of the form σ++[m1,m2]++&tau. In particular, it has no creation step. All steps are either an update to a static field of CUT, or a call to its static method. For the rest, it is the same idea as in the ADT testing.

T3 options.

General usage:

java -ea -cp <T3jar><:other path>*  Sequenic.T3.T3Cmd  <options>* <targetclass>

Options:

 -co,--collection <int>             Maximum size of collections and arrays that T3 generates. Default 3.
 -core,--core <int>                 To specify the number of cores.
 -cvg,--customvalgen <class-name>   A class specifying a custom values generator.
 -d,--savedir <dir>                 Save the generated suite in this directory.
 -dg,--dropgood                     If set, non-violating traces will be dropped.
 -fup,--fieldupdate <float>         The probability to do a field update. Default is 0.1.
 -help,--help                       Print this message.
 -lf,--logfile <file>               To echo messages to a log-file.
 -ms,--msamples <float>             Multiplier on number of samples to collect per goal. Default 4.0.
 -norc,--nooracle                   If set, T3 will not inject oracles in when ran in the regression mode.
 -obn,--objectsNesting <int>        Maximum object-depth that T3 generates. Default 4.
 -pl,--prefixlength <int>           Maximum prefix length of each sequence. Default 8.
 -reg,--regressionmode              If set, T3 will generate a suite for regression test; the target class is assumed correct.
 -rs,--reportStream <file>          To send report to a file.
 -sd,--scandir <dir>                Directory to scan for classes.
 -sex,--showexc                     When set, will show the first exception throwing execution.
 -sl,--suffixlength <int>           Maximum suffix length of each sequence. Default 3.
 -sqr,--seqretry <int>              Maximum number that each sequence is retried. Default 5.
 -ss,--splitsuite <int>             Split the generated suite to this number of smaller suites. Default 1.
 -str,--stepretry <int>             Maximum number that each step is retried. Default 30.
 -tfop,--tfop                       When set, we assume to test from a different package.
 -to,--timeout <int>                When set, specifies time out in ms.
 -tps,--tracePrintStream <file>     To send sequence-prints to a file.
 -vp,--variprefix                   If set, T3 will generate variable legth prefixes, up to the specified maximum.

Writing custom value generators.

Consider the following class representing persons. The constructor requires two strings representing a person's name and her email, and integer representing her age, and another integer representing some code. Say, that valid codes are -1,0,1. If we just let T3 use its default value generator to randomly generate them, most of the time we will get uninteresting values, or even invalid values.

package Examples;

public class Person {

  private String name ;
  private String email ;
  private int age;
  private int code ;
	
  public Person(String name, String email, int age, int code) {
    this.name = name ;
    this.email  = email ;
    this.age = age ;
    this.code = code ;
  }	
  ...
}

We can write a custom value generator, here is an example:

package Examples;

import java.util.function.Function;
import Sequenic.T3.Generator.Generator;
import Sequenic.T3.Sequence.Datatype.*;
import static Sequenic.T3.Generator.GenCombinators.* ;
import static Sequenic.T3.Generator.Value.ValueMetaGeneratorsCombinators.* ;

public class CustomInputGenerator {
	
  public static Generator myvalgen =
    FirstOf(Integer(OneOf(17,18,60)).If(hasParamName("age")),
            Integer(OneOf(-1,0,1)).If(hasParamName("code")),
            String(OneOf("anna","bob")).If(hasParamName("name")),
            String(OneOf("anna@gmail.com","k1@ppp")).If(hasParamName("email"))
    ) ;                        
}

For any parameter (of any constructor or method) of type int or Integer, whose name is "age", the generator will randomly select from the list (17,18,60). If the parameter is named "code", it will randomly select from (-1,0,1). For other integer-types parameter, T3 will use its default generator.

For the above to work, you should compile the target class with with the -parameters option enabled.

You also need to inform T3 where it can find the custom generator, using the --customvalgen option. T3 assumed the testgenerator is bound to a static field named myvalgen, as in the above example.

java -ea -cp T3.jar Sequenic.T3.T3Cmd --customvalgen Examples.CustomInputGenerator Examples.Person

Controlling the distribution of the generated values.

The function OneOf(x1,x2,...) is exported by some class in T3. It is a selector that randomly chooses one of its parameter. This function is overloaded to range over various primitive-typed and string parameters. It used Java's built-in random generator, which has uniform distribution. There are two ways to influence the distrubution. The easiest way is by duplicating values, as in:

OneOf(-1,0,0,1)

which would have twice more chance to select a 0 over -1 and 1.

We can also define our own random selector; it needs to be a method that returns an instance of Function<Unit,T>, where T is the type of value that we want to produce. For example, to select from a list of integers, based on Gaussian distribution on the indices of the choices, we can define this:

static Random rnd = new java.util.Random(seed) ;

static Function<Unit,Integer> GaussianOneOf(int ...s) {
   return unit -> {
      int k = rnd.nextGaussian() * s.length ;
      return s[k] ;
   } ;      
}

Then we can use it, e.g. as in:

FirstOf(Integer(OneOf(17,18,60)).If(hasParamName("age")),
        Integer(GaussianOneOf(-1,0,1)).If(hasParamName("code")),
        ...
       )

Object generator.

We can also write a generator that produces objects. Suppose the class Person above has another constructor, as shown below:

public Person(String name, Email email, int age, int code) { ...  }

Notice that it now requires an instance of Email as a parameter. T3 generates instances, but sometimes we need to have more control on the kind of instances we want to pass as inputs. A custom object generator must be a static method of this signature: O generator(int k), where O is the class whose instances we want to generate. Alternatively, a generator can also be an instance of Function<Integer,O>. Here is an example, of a custom Email generator:

static Person.Email genEmail(int k) {
  switch (k) { 
   case 0 : return new Email("root") ;
   case 1 : return new Email("annaz") ;
   default : return new Email("guest") ;
  }
}

Now we need to adjust our custom value generator, to also include the above generator:

public static Generator myvalgen =
  FirstOf(Integer(OneOf(17,18,60)).If(hasParamName("age")),
          Integer(OneOf(-1,0,1)).If(hasParamName("code")),
          String(OneOf("anna","bob")).If(hasParamName("name")),
          String(OneOf("anna@gmail.com","k1@ppp")).If(hasParamName("email")),
          Apply(CustomInputGenerator.class,"genEmail",OneOf(0,1,2)).If(hasClass(Email.class).and(hasParamName("email")))     
    ) ;                        
}

The last line says, to use the genEmail generator, and passing to it a randomly chosen integer, taken from the list (0,1,2). The "If"-condition specifies that generator is only applied on parameters of type Email, and whose paratemers' name is "email".

To run T3 using the above custom generator, we call it e.g. as below (assuming the generator is put in the class Examples.CustomInputGenerator):

java -ea -cp T3.jar Sequenic.T3.T3Cmd --customvalgen Examples.CustomInputGenerator Examples.Person

Saving and replaying test suites.

By default T3 only generates the test suite internally. It does not save it. However, with the --savedir dir option we can save the suite, and later we can replay it. The suite will be saved in the specified directory, named Cname_timestamp.tr, where Cname is the fully qualified name of CUT.

Saved test suites can be replayed. For example, suppose T3 has saved a suite in a file named Examples.SimpleIntSortedList__1393751293489.tr. To replay it we do:

java -cp T3.jar -sx Sequenic.T3.ReplayCmd Examples.SimpleIntSortedList__1393751293489.tr

This causes the test suite to be replayed. The replay will stop if a test sequence throws an exception. The -sx option will cause this sequence to be printed. Alternatively, we can use the --runall option to run the whole suite. At the end, statistic will be printed, e.g.:

Suite size : 302
Average length : 9.427153
Executed   : 302
Violating  : 198
Invalid    : 0
Failing    : 0
Runtime    : 189 ms

The general syntax to replay is:

java -cp <T3.jar><:otherpath>*  Sequenic.T3.ReplayCmd  [options]* <file or dir>

Options:

 -b,--bulk <string>               If specified the target is a directory; all suite-files there whose name is prefixed with this string are replayed.
 -help,--help                     Print this message.
 -lf,--logfile <file>             To echo messages to a log-file.
 -ra,--runall                     If set, all sequences will be replayed; exception will not be rethrown.
 -rm,--regressionmode             If set, and runall is false, only thrown OracleError will be rethrown.
 -rs,--reportStream <file>        To send report to a file.
 -sd,--showdepth <int>            If set, when on object is printed, it is printed up to this depth. Defailt=3.
 -sl,--showlength <int>           If set will print each step in the suffix of each replayed sequence, up to this length. Default=10.
 -sx,--showXexec                  If set, then execution that throws an exception will be printed.
 -tps,--tracePrintStream <file>   To send sequence-prints to a file.

Using T3 to generate regression test-suite.

A test suite generated by T3 can always be used to do regression testing. However, the suite that we get by default will lack negative tests. When given invalid inputs, a program should ideally reject those inputs, rather than just consuming them and potentially producing harmful behavior. A negative test is aimed at testing this ability. To make T3 to also generate negative tests, invoke it with --regressionmode option enabled. Violations to pre-conditon now no longer count as error, and neither does throwing an IllegalArgumentException. Consequently, a sequence that leads to such a situation is no longer dropped; it is a negative test.

Calling T3 from APIs.

T3 can also be called from APIs, so that e.g. you can call it from a JUnit test class. The simplest way is to simply call its main-method, and passing options to it as a string. For example:

Sequenic.T3.T3Cmd.main("-d . Examples.SimpleIntSortedList") ;

This will generate test sequences for Examples.SimpleIntSortedList, and save them as suites in the current directory. Typically two suites will be generates: ADT_Examples.SimpleIntSortedList_timestap.tr and nonADT_Examples.SimpleIntSortedList_timestap.tr

To replay them we call:

Sequenic.T3.ReplayCmd.main("-sx -b ADT_Examples.SimpleIntSortedList . ") ;
Sequenic.T3.ReplayCmd.main("-sx -b nonADT_Examples.SimpleIntSortedList . ") ;

Note that T3Cmd is used to generate test suites. If it finds a violation, it does not re-throw it. It saves the violating sequence in a suite, and proceeds with the generation of the next sequence. ReplayCmd can be used to replay a saved suite, and re-throw the first violation it finds. So, if you want to call T3 from a Junit test method, you probably want to arrange it like this:

@Test
public void testC() {
   if(test-cases for C exists)  {
      Sequenic.T3.ReplayCmd.main("-sx -b ADT_C . ") ;
      Sequenic.T3.ReplayCmd.main("-sx -b nonADT_C . ") ;
   }
   else Sequenic.T3.T3Cmd.main("-d . C") ;
}

Sequenic.T3.T3SuiteGenAPI

This class provides the logical API underneath T3Cmd. It allows you to directly control T3's configuration and to collect the generated test suites. Here is an example:

useT3() throws Exception {
  Config config = new Config() ;  
  // setting T3 parameters:
  config.CUT = Person.class ;  // target class to test
  config.setDirsToClasses("./bin") ;
  T3SuiteGenAPI tester = new T3SuiteGenAPI(null,config) ;  // null means we use T3 custom value generator
  // generate suites:
  List SS = tester.suites(true,".") ;
  // now, what do you want to do with SS? As an example:
  tester.reportCoverage(SS.get(0));	 
}

Custom sequence generator.

With the -customvalgen option we can specify a custom value generator to be used by T3. However, it still uses its default sequence generator. If it is needed to specify a custom sequence generator, we can use the class Sequenic.T3.CustomSuiteGenAPI.

customSuiteGen() throws Exception {		
   	 Config config = new Config() ;
   	 config.CUT = Person.class ;
     config.setDirsToClasses("./bin") ;
     // a flexible generator:   
     CustomSuiteGenAPI sg = new CustomSuiteGenAPI(null,config) ;  // null means we use T3 custom value generator
     sg.scope.configureForADTtesting();
     
	 // now generate a suite of 500 sequences, using the specified sequence generator:
     SUITE S = sg.suite(SequenceWhile(
        		r -> ! r.isFail(),
        		sg.create(),
        		sg.mutatorsSegment(10),
        		sg.instrument(),
        		sg.method("friend"),
        		sg.method("unfriend"),
        		sg.nonMutatorsSegment(10)
        		),
        		500
        		) ;

      sg.reportCoverage(S) ; 
	
}

The generator above will generate a suite targetting the class Person. This suite will consist of maximum 500 sequences. Each sequence is generated with the above specified sequence-generator. It says that each sequence is to be like this:

The sequence starts with the creation of a target object (an instance of Person),
followed by a segment of up to 10 calls to mutator-methods or field updates,
followed by a call to friend then unfriend,
followed by a segment of up to 10 calls to non-mutator-methods.