SimpleTest v1.0 

 The Simple Development Library regression testing framework 

Project: the Simple Development Library.

Table of contents

1. Package description

Synopsis: the Simple Development Library regression testing framework.

Keywords: regression, test and package.

1.1 Overview

This package provides a complete framework for automated regression testing.

Procedures are provided to initialize, reset and shutdown the test environment, set criteria of which cases to test, run the test cases as well as others to obtain the test results and statistics.

1.2 Usage

The ::Simple::Test::initialize procedure sets up a test environment where test cases can be run via the test-case procedure. By using ::Simple::Test::set-test-criteria, one can set a criteria to specify which test cases to test or ignore.

Upon testing a test case, its prerequisites are first assessed; if any prerequisite is not fulfilled, the test case is skipped. Next, the list of dependencies are checked. If any of the dependencies did not pass, the test case is skipped. Otherwise, the test case is run, and by comparing its output, errors and return values with the corresponding expected values, each test is flagged as passed or failed.

The detailed test results and the flags given to the corresponding call to the test-case procedures are available via the ::Simple::Test::results and ::Simple::Test::flags procedures, respectively. Finally, the overall statistics of all the test cases can be obtained via the ::Simple::Test::statistics procedure.

Use ::Simple::Test::reset to reset the test environment and ::Simple::Test::shutdown to clean up after testing.

1.3 Examples

# Install the package
package require SimplePackage
::Simple::Package::require-and-install SimpleTest

# Initialize the test environment
::Simple::Test::initialize

# Set the test criteria: test cases number 1, 3 and following
::Simple::Test::set-test-criteria 1,3:

# Execute several test cases

# This one passes
test-case test-case-1 {
   First test case
} -setup {

   # Create a variable in the test environment
   set foo 1

} -script {
      puts -nonewline $foo
} -output 1

# This one is ignored
test-case test-case-2 {
   Second test case
} -script {
}

# This one fails
test-case test-case-3 {
   Third test case
} -script {
      return $foo
} -return 2

# This one is skipped
test-case test-case-4 {
   Fourth test case
} -prerequisites {
   {$foo == 2}
} -script {
}

# This one passes also
test-case test-case-5 {
   Fifth test case
} -script {
      error {ERROR MESSAGE}
} -cleanup {

   # Delete the variable in the test environment
   unset foo

} -error {ERROR MESSAGE}

# This one depends on test-case-1 and test-case-2
test-case test-case-6 {
   Sixth test case
} -dependencies {
   test-case-1 test-case-2
} -script {
   puts -nonewline 1
} -output 1
   
# This displays the following:
#   |1 test-case-1     First test case: passed
#   |2 test-case-2    Second test case: ignored
#   |3 test-case-3     Third test case: failed
#   |        -return  = {2}
#   |   Actual return = {1}
#   |4 test-case-4    Fourth test case: skipped
#   |5 test-case-5     Fifth test case: passed
#   |6 test-case-6     Sixth test case: skipped

foreach result [::Simple::Test::results]\ 
   optionPairs [::Simple::Test::flags] {

   # Get the test case information
   foreach {number tag title ignored skipped passed diagnostics}\ 
      $result break
   if {$ignored} {
      set status ignored
   } elseif {$skipped} {
      set status skipped
   } elseif {$passed} {
      set status passed
   } else {
      set status failed
   }

   # Display a short report for each test case
   puts [format {%1d %10s %20s %s} $number $tag $title: $status]

   # For failed tests, display the test case flags and diagnostics
   if {![string compare $status failed]} {
      foreach {option contents} $optionPairs {
         puts [format {%16s = {%s}} $option $contents]
      }
      foreach {reason extra} $diagnostics {
         puts [format {%16s = {%s}} $reason $extra]
      }
   }
}

# Final tests statistics
foreach {nTests passed failed skipped ignored}\ 
   [::Simple::Test::statistics] break

# Display some test cases statistics
# This displays the following:
#   |Total test   cases 6
#   |Test cases  passed 2
#   |Test cases  failed 1
#   |Test cases skipped 2
#   |Test cases ignored 1
puts [format {Total test   cases %d} $nTests]
puts [format {Test cases  passed %d} $passed]
puts [format {Test cases  failed %d} $failed]
puts [format {Test cases skipped %d} $skipped]
puts [format {Test cases ignored %d} $ignored]

# Shutdown the test environment
::Simple::Test::shutdown

1.4 History

1.5 Copyright

Copyright (C) 1999-2005, Juan C. Gil (jgil@gmv.es).

2. Package public API

Paradigm: procedural.

Requisites: SimpleType 1.0.

2.1 Types

2.1.1 testtag

Description: test case tag: public-api-1, ...

2.1.2 testcriteria

Description: criteria for test cases to be tested: :, 3, 3,7, :7, ..

2.2 Options

2.2.1 -store

Type: list.

Value: (empty string).

Description: flags to store for each test case tested, which can later be obtained via the ::Simple::Test::flags procedure.

2.3 Namespaces

1. ::Simple::Test

2. ::Simple::Test::Priv

2.4 Commands

2.4.1 ::Simple::Test::cget ?-store?

Synopsis: gets the package options.

Details: see section 4.1.

2.4.2 ::Simple::Test::configure ?-store list?

Synopsis: configures the package options.

Details: see section 4.2.

2.4.3 ::Simple::Test::flags

Synopsis: returns the test stored flags.

Details: see section 4.3.

2.4.4 ::Simple::Test::initialize

Synopsis: initializes the test environment.

Details: see section 4.4.

2.4.5 ::Simple::Test::reset

Synopsis: resets the test environment.

Details: see section 4.5.

2.4.6 ::Simple::Test::results

Synopsis: returns the test cases results.

Details: see section 4.6.

2.4.7 ::Simple::Test::set-test-criteria testCriteria

Synopsis: sets the criteria to specify which test cases to test.

Details: see section 4.7.

2.4.8 ::Simple::Test::shutdown

Synopsis: shutdowns the test environment.

Details: see section 4.8.

2.4.9 ::Simple::Test::statistics

Synopsis: returns test cases statistics.

Details: see section 4.9.

2.4.10 test-case tag title ?-regexp? ?-prerequisites script-list? ?-dependencies testtag-list? ?-setup script? ?-script script? ?-cleanup script? ?-output string? ?-error string? ?-return string?

Synopsis: runs a test case.

Details: see section 4.10.

2.5 Errors

2.5.1 ::Simple::Test::ENVIRONMENT-NOT-INITIALIZED

Message: test environment not initialized.

Explanation: the attempted operation requires an initialized test environment.

Corrective action: use the ::Simple::Test::initialize procedure to initialize the test environment.

2.5.2 ::Simple::Test::NO-SCRIPT-FLAG

Message: no -script for test "test case tag".

Explanation: test case test case tag lacks the compulsory -script flag.

2.5.3 ::Simple::Test::RETURN-AND-ERROR-FLAGS

Message: both -return and -error given for test case "test case tag".

Explanation: test case test case tag has both -return and -error flags, but these are mutually exclusive.

2.5.4 ::Simple::Test::BAD-SET-OF-RANGES

Message: bad set of ranges "ranges", range "range" is not well formed.

Explanation: the set of ranges ranges ranges has bad format; range is the bad range.

Corrective action: a set of ranges is one or more ranges separated by commas. A range is one unsigned integer or two unsigned integers separated by a colon where the first, the second or even both unsigned integers may not be present, meaning that the range is open by the left, by the right, or by both ends, respectively.

2.5.5 ::Simple::Test::BAD-FLAG-TO-STORE

Message: bad flag to store "flag": must be -cleanup, -dependencies, -error, -output, -prerequisites, -regexp, -return, -script or -setup.

Explanation: flag flag is not allowed.

Corrective action: supply a flag among those in the list.

3. Package further information

3.1 Details

• The test environment is in fact a Tcl slave interpreter in which the puts command has been modified so that the test cases output can be collected.

• The test interpreter name is contained in the ::Simple::Test::TestInterp variable. It is possible to set it to the empty string so as to use the toplevel interpreter. This is useful to run the tests within debugers which may have problems with child interpreters.

3.2 Todo

• Consider whether it is worth it migrating the test infrastructure to use the facilities of the Tcl test harness package, tcltest.

• Add support for criteria to specify which test cases to test based on the test case tag.

• Keep track of the test cases tags. Ensure they are not duplicated. Ensure tags of the form <tag>-<number> are correlative.

• Allow to specify a user defined procedure for handling each test case results as soon as they are run.

4. Package command details

4.1 ::Simple::Test::cget ?-store?

4.1.1 Command description

Synopsis: gets the package options.

Access mode: public.

4.1.2 Command public API

Arguments:

Returns: the requested option value or the whole list of options if none specified.

4.2 ::Simple::Test::configure ?-store list?

4.2.1 Command description

Synopsis: configures the package options.

Access mode: public.

4.2.2 Command public API

Arguments:

Returns: the empty string.

4.3 ::Simple::Test::flags

4.3.1 Command description

Synopsis: returns the test stored flags.

Access mode: public.

4.3.1.1 Overview

This procedure returns a list containing the stored flags and their contents for a test case as given to the corresponding test-case procedure. Each element in the list is a list of two-element lists, being the first a flag and the second its contents; this format is suitable to be used with the array get command. As -regexp is a boolean flag, its contents is set to the empty string.

The stored flag are those of the -store package option. By default, no flags are stored. Nevertheless, if a test case fails the given items among those in the -store package option are stored along with the -regexp flag. If the test case throws an error when no error was expected, this procedure return an -error flag with "NONE" as contents.

4.3.2 Command public API

Arguments: none.

Returns: the test cases flags.

4.4 ::Simple::Test::initialize

4.4.1 Command description

Synopsis: initializes the test environment.

Access mode: public.

4.4.2 Command public API

Arguments: none.

Returns: the empty string.

Effects:

• Creates the test interpreter.

4.4.3 Command further information

4.4.3.1 Remarks

• It is compulsory to call this procedure before testing.

4.5 ::Simple::Test::reset

4.5.1 Command description

Synopsis: resets the test environment.

Access mode: public.

4.5.2 Command public API

Arguments: none.

Returns: the empty string.

4.6 ::Simple::Test::results

4.6.1 Command description

Synopsis: returns the test cases results.

Access mode: public.

4.6.1.1 Overview

This procedure returns a list containing for each test case tested the following elements:

Possible reasons and their corresponding extra information are the following:

The actual output, error or return appears when they did not match the expected output, error or return values, respectively. Notice that the actual output may appear together with the actual error or return, but not both. The error information appears when an error is thrown from the test case script.

4.6.2 Command public API

Arguments: none.

Returns: the test cases results.

4.7 ::Simple::Test::set-test-criteria testCriteria

4.7.1 Command description

Synopsis: sets the criteria to specify which test cases to test.

Access mode: public.

4.7.1.1 Overview

This procedure sets the expression that specifies which test cases to test from the given criteria. Test cases matching the criteria are tested while all others are ignored.

4.7.2 Command public API

Arguments:

Returns: the empty string.

4.7.3 Command further information

4.7.3.1 Todo

• Allow criteria based on test case tags.

4.8 ::Simple::Test::shutdown

4.8.1 Command description

Synopsis: shutdowns the test environment.

Access mode: public.

4.8.2 Command public API

Arguments: none.

Returns: the empty string.

Effects:

• Deletes the test interpreter.

4.9 ::Simple::Test::statistics

4.9.1 Command description

Synopsis: returns test cases statistics.

Access mode: public.

4.9.2 Command public API

Arguments: none.

Returns: a list with the following elements:

4.10 test-case tag title ?-regexp? ?-prerequisites script-list? ?-dependencies testtag-list? ?-setup script? ?-script script? ?-cleanup script? ?-output string? ?-error string? ?-return string?

(Exported from ::Simple::Test).

4.10.1 Command description

Synopsis: runs a test case.

Access mode: exported.

Keywords: test.

4.10.1.1 Overview

This is the actual test case handler. Works by running the test case in the test environment where the output of puts is available at the end. If that output matches the expected output and any error message and return values match the expected ones, the test case is considered passed, otherwise the test fails. The output of this procedure is stored and can be obtained via the ::Simple::Test::results procedure.

If the expected output, return value or error are empty, the test passes only if it does not produce any output, return value or error, respectively. If the expected return value or error is not given, then its actual value is not checked.

By specifying the -regexp flag, the expected output and return value or error are matched as regular expressions so that it is possible to validate non-constant values.

Before running the test case, each prerequisite in the list of prerequisites is evaluated. If any of them fails, the test is skipped. Next, the list of dependencies is checked. If any of the dependencies does not pass, the test is skipped.

Otherwise, the setup script is evaluated afterwards. If there is an error, the test is failed. Otherwise, the actual test is run followed by the cleanup script. An error in the cleanup script also results in the test case to be regarded as failed.

4.10.1.2 Examples

# A test which checks some output
test-case T1 {Test 1} -script {puts -nonewline "foo"} -output foo

# A test which checks an error message
test-case T2 {Test 2} -script {return -code error bar} -error bar

# A test which checks the output using a regular expression
test-case T3 {Test 3} -regexp -script {puts -nonewline foo} -output {oo$}

4.10.2 Command public API

Arguments:

Returns: whether the test passed or not.

Effects:

• Those of the evaluated scripts in the test environment.

4.10.3 Command further information

4.10.3.1 Remarks

• The -script argument is compulsory.

• The -error and -return arguments are mutually exclusive.

• It is compulsory to have called ::Simple::Test::initialize before calling this procedure.

• The order of the flags is irrelevant but their parameters can not start by an hyphen.