Stress Tests

From MantidProject

The Mantid repository contains a Python module, called stresstesting, that is designed to allow a predefined set of tests to run and report the results. The framework is designed in a similar manner to CxxTest but here each separate test should have its own Python class that inherits from stresstesting.MantidStressTest. Setting up a framework object then runs all of the tests within a given location that inherit from this base class.

Writing A New Test

A new stress test should inherit from the stresstesting.MantidStressTest class defined in the stresstesting module and redefine the runTest function, for example, a simple mantid test might be:

 import stresstesting
 from mantidsimple import *
 
 # A simple LoadRaw test
 class SimpleTest(stresstesting.MantidStressTest):
   # Redefine the base class runTest function
   def runTest(self):
     LoadRaw('GEM38370.raw', 'GEM')

Note the self argument that runTest function takes. This must be present so that Python can call it as a member function, see e.g. Core Python Programming By Wesley J. Chun.

Running a test

A test file can be run as part of a set of tests within a specified directory or in isolation as a single file. A python script called runStressTests.py, located in the trunk/Tests/StressTests directory of the repository, can be used to execute the desired tests (This requires a Mantid.properties file within the same location as the script. The test here also assumes that the datasearch.directories variable has been set). It is set up so that by default executing the script will run all of the test files within the trunk/Tests/StressTests/MantidScript directory on the command line and report the results out to the console. Executing the script with the --help argument displays the usage text and describes how to run the tests differently,

Usage: runStressTests [options] [file|directory]

[file|directory]		Run either a set tests in a given file or all 
                		tests in a given directory. The default behaviour
                		runs all tests in ../StressTests/MantidScript
Options: 
	output=		The location to send the output. Supported options are text and db,
         		for example --output=text,db will send results to the console and database
	--gui		Run tests in MantidPlot script environment

Command Line Environment

The default environment is the command line Python environment. To execute our simple test within this environment, first change to the location of the runStressTests script. Assuming that the path to the file containing the SimpleTest class is MyTests/FirstTest.py, executing the command

python runStressTests.py MyTests/FirstTest.py

will run the SimpleTest and give the following output (this will vary between systems)

Tue, 29 Sep 2009 16:41:11: Executing FirstTest.SimpleTest

******************************
	test_name      ->   FirstTest.SimpleTest
	host_name      ->   localhost.localdomain
	environment    ->   debian
	test_date      ->   2009-09-29 16:41:11 
	status         ->   success
	iteration time_taken->   1 2.04
******************************

Given that the framework is designed to stress Mantid, most tests will probably want to run many loops of the given test. This has been accommodated within the MantidTest base class by the addition of a function, maxIterations that can be overridden in the inheriting class. Be default, maxIterations<code> returns 1 so that leaving out the function definition causes a single execution of the test. To change the number simple reimplment the function in the inherited class, e.g.

 import stresstesting
 from mantidsimple import *
 
 # A simple LoadRaw test
 class SimpleTest(stresstesting.MantidStressTest):
   # Redefine the base class runTest function
   def runTest(self):
     LoadRaw('GEM38370.raw', 'GEM')
 
  # Run test 10 times
  def maxIterations(self):
    return 10

Tue, 29 Sep 2009 16:50:49: Executing FirstTest.SimpleTest

******************************
	test_name      ->   FirstTest.SimpleTest
	host_name      ->   localhost.localdomain
	environment    ->   debian
	test_date      ->   2009-09-29 16:50:49
	status         ->   success
	iteration time_taken->   1 2.03
	iteration time_taken->   2 1.86
	iteration time_taken->   3 1.73
	iteration time_taken->   4 1.76
	iteration time_taken->   5 1.73
	iteration time_taken->   6 1.76
	iteration time_taken->   7 1.72
	iteration time_taken->   8 1.73
	iteration time_taken->   9 1.71
	iteration time_taken->   10 1.74
******************************

python runStressTests.py output=text,db MyTests