Coco Test Engine Driver

Overview

The cocotestengine program 'drives' a target program to generate test data and to run tests based on this data. If during the test execution or discovery, the target program crashes, cocotestengine restarts it and continues.

The cocotestengine works in three different modes:

• Discovery mode: This mode is enabled with the --discover flag.

  In discovery mode, cocotestengine tries to find new tests. If there exists a data file with test data, it uses them as a starting point.

  If a newly found row of data is better (covers more code) than an existing one, and the existing one is not validated, then the existing row could be thrown out to keep the data set minimal.

• Test mode: This mode is enabled when neither --discover nor --learn are given.

  In test mode, cocotestengine reads the test data from the data file, runs them, and reports how many were successful and which failed.

  In test mode, it is possible to run the tests that are implemented in a target program, or only a single test.

• Learn mode: This mode is enabled with the --learn flag.

  In learn mode, the driver runs all the rows of a test, but instead of checking whether the outputs agree with the values in the data file, it takes the outputs of its test run and stores them in the data file.

  For rows that were previously validated, if the output value is different, then the validated flag is reset to false for that row.

  Learn mode is possible for one test or all tests in a target program.

Summary

cocotestengine -e <target> [-d <directory>] [-t <test>] [--learn | --discover] [options]
cocotestengine -e <target> --list
cocotestengine -h

Return values

In case of a crashed and a failed test together, the crash takes precedence.

Output file

At the end of each test, learn, or discovery run, the target program writes a .csexe file. If the file already exists, the new data is appended to it.

Ending the program.

If the discovery process takes too long, one can press Ctrl-C. The driver then stops discovery and writes the data that it has found to the data file.

Command line arguments

General options

• -h | --help: Print a help message and exit.
• -e <path>: Set the path to the target executable.

  If neither --cs-learn nor --cs-discover is set, the driver runs all rows for all tests that are supported by the target program, then prints a summary of the results.

• -d <path> | --data-dir=<path>: Set the path to the directory that contains the data and configuration files.

  The default value is the working directory of the driver.

• -t <string>: Set the name of the test that should be executed.

  If neither --cs-learn nor --cs-discover is set, the driver runs all rows for this test and prints a summary of the results.

• -l | --list: Print a list of all the tests that are supported by the target program.
• -v | --verbose: Enables verbose mode.
• --learn: Enable learn mode.

  The option -t must be set. The driver then runs all rows for this test and stores for each row the output values of the test in the data file.

• --discover: Enable discovery mode.

  The option -t must be set. The driver then runs test data discovery for this test.

Options for discovery mode

The following options are only meaningful in the discovery mode.

• --max-tries=<int>: Maximal number of rows tried during test generation.

  The default value is 10000.

• --max-time=<int>: Maximal time for discovery.

  A value of 0 means unlimited time. Every other value sets a timeout in seconds, after which discovery stops. The default value is 0.

• --max-coverage=<float>: Coverage at which discovery stops.

  If the value is positive and the coverage of the test that were already reached exceeds that value, discovery is stopped. A negative value disables this check. The default is -1.

• --update=<int>: Delay in seconds between status updates.

  If the value is positive, it specifies the time period at which the driver prints a one-line summary about the results of the discovery that were reached so far. A value of 0 disables the feature.

  The summary contains the number of data rows that were found so far and the coverage they achieve together.

• --save-interval=<int>: Interval in seconds between saving the discovered tests.

  The default value is 60. A value of 0 disables saving.

• --seed=<int>: Seed used for the random number generator used in test data generation.

  By default, the number of seconds since the Epoch will be used.

Options for fine-tuning the driver program

The following options are rarely needed.

• --timeout=<int>: Set a timeout for communication with the target.

  The value is measured in milliseconds, and the default is 300000, which is 5 minutes. If the driver communicates with the target and waits for an answer for longer than this time, then the target finishes with a failure.

• --batch-size Set the number of rows that the driver sends in one batch to the target.

  The default is 100. The number has some effect on the efficiency of the data transfer between driver and target.