Testing Infrastructure

Introduction

This is the testing HOWTO for SST. By following the steps in this procedure, the following can be accomplished:

Testing Framework

Overview

The SST test framework is based upon shunit2. Initially, the testing framework will only support simple testing, where the Software Under Test (SUT) is exercised (possibly with a known input file), and this output is captured in a file. This output file is then compared against a known-good output file which is kept under configuration management. If this comparison succeeds, then the test is marked as a PASS, otherwise, it is marked as a FAIL.

Only the test management functions of the shunit2 test harness will be used initially. The assertion capabilities are not used.

Description

shunit2 allows for creation of tests using a Bash shell script.

After successful compilation and installation of SST by Bamboo, the tests are run and the test results are then processed and reported by Bamboo.

Definitions

Test Construction

Here is a skeleton view of a shunit2 test suite script, testSuite_example.sh:

#!/bin/sh 
...
L_SUITENAME="SST_SuiteNameHere_suite" # Name of this suite of tests
...
# Test functions
#   These exercise a group of related features
test_thisSuite_0001() {
  # If necessary, generate SDL file as input.
...
  # Execute SUT, capture SUT output to file
...
}
test_thisSuite_0002() {
...
}
test_thisSuite_0003() {
...
}
...

# shunit overloadable utility functions, use as needed for test and
# suite setup and teardown
# oneTimeSetUp() {
    # Define any initialization or setup that must occur once
    # for this entire suite of tests
# }
oneTimeTearDown() {
    # Run difftoxml.pl Perl script, which
    #   1. Runs "diff" between SUT outputs and their
    #      known-good reference files
    #      a) If diff result shows no difference, test passes
    #      b) Otherwise, test fails
    #   2. Creates an JUnit-style XML file containing
    #      the diff results for Bamboo.
...
}
# SetUp() {
    # Define any general setup that must occur once
    # for each test function in this test suite.
# }
# TearDown() {
    # Define any general cleanup that must occur once
    # for each test function in this test suite.
# }

# Invoke shunit2. Any function in this file whose name starts with
# "test"  will be automatically executed.
(. ${SHUNIT2_SRC}/shunit2)

Test Suite Construction

In order to create a new test suite:

  1. Copy the supplied template file testSuite_TEMPLATE.sh to a new test suite file in the same directory.
  2. Create a new name for the test suite by defining the L_SUITENAME variable at the top of the new test suite file. This should be a unique suite name that does not contain spaces.
  3. If necessary, set the new test suite file permissions to 755 (rwxr-xr-x) using the chmod command.

Once a new test suite file has been created from the template, it should only be necessary to add test functions; the remainder of the suite should be sufficiently defined.

Test Function Construction

When a new test suite is created from the template, an example test function is included. Follow the general form of the example function, as illustrated in the test function code snippet that follows. Naturally, document and comment your test thoroughly.

#-------------------------------------------------------------------------------
# Test:
#     test_portals_0001
# Purpose:
#     Exercise the portals4_sm_sdlgen executable
# Inputs:
#     None
# Outputs:
#     test_portals_0001.out file
# Expected Results
#     Match of output file against reference file
# Caveats:
#     The output files must match the reference file *exactly*,
#     requiring that the command lines for creating both the output
#     file and the reference file be exactly the same.
#-------------------------------------------------------------------------------
test_portals_0001() {

    # Define a common basename for test output and reference
    # files, and ".out" extension. XML postprocessing requires this.
    testDataFileBase="test_portals_0001"
    outFile="${SST_TEST_OUTPUTS}/${testDataFileBase}.out"
    # This is the expected name and location of the reference file
    #referenceFile="${SST_TEST_REFERENCE}/${testDataFileBase}.out"
    # Add basename to list for XML processing later
    L_TESTFILE+=(${testDataFileBase})

    # Generate an SDL file for SST to load for testing
    sdlGenerator="${SST_TEST_INSTALL_BIN}/portals4_sm_sdlgen"
    sdlFile="${SST_TEST_SDL_FILES}/${testDataFilesBase}.sdl"
    sdlGeneratorArgs="-x 16 -y 16 -z 16 -r 16 --application allreduce.tree_triggered --noise_runs 0 --output ${sdlFile} --ranks 2"
    (${sdlGenerator} ${sdlGeneratorArgs})

    # Define Software Under Test (SUT) and its runtime arguments
    sut="${SST_TEST_INSTALL_BIN}/sst.x"
    sutArgs="${sdlFile}"

    if [ -f ${sut} ] && [ -x ${sut} ]
    then
        # Run SUT and capture its output
        (${sut} ${sutArgs} > ${outFile})
    fi
}

Useful Test Definitions

The following are definitions that may be useful to test script developers.

Variable Definition
SST_TEST_SDL_FILES Location of static SDL input files in GIT (see note 1.)
SST_TEST_INPUTS Location of miscellaneous SUT input files in GIT
SST_TEST_OUTPUTS Directory where SUT output files should be placed
SST_TEST_INSTALL_BIN Directory where SUTs are installed (see note 2.)

  1. Special note on SST_TEST_SDL_FILES There may be occasions when a static, predefined SDL file is preferred. These SDL files are not dynamically generated at test time, but are instead created ahead of time, and stored for use at test time. This variable contains the location of these SDL files.

  2. Special note on SST_TEST_INSTALL_BIN Since it is not possible to have a priori knowledge of the user’s SST installation location, it is possible to override the default value of SST_TEST_INSTALL_BIN by defining the SST_INSTALL_BIN_USER environment variable to a user-appropriate value.

Test Suite Execution

After the test has been created, the entire test suite can be executed to verify that the tests run as expected. It may be necessary to define the SST_INSTALL_BIN_USER environment variable to specify the user’s SST installation location, which may differ from the SST installation location used by Bamboo when it performs the nightly build.

Test Construction Checklist

  1. Create a new test
    • as part of an existing test suite, or
    • create a new test suite
  2. Run the test suite to verify that it performs as expected
    • Review XML output file in $SST_HOME/test/test-reports
  3. Check in the test suite
  4. Check in test reference file
  5. Check in test input file, if necessary
  6. If the test suite should be run automatically by Bamboo after the nightly build, add the test suite to the appropriate place in bamboo.sh, and check in bamboo.sh