System Tests
From MantidProject
The Mantid system tests are part of the automated testing that is performed on the latest development code and consist of a suite of tests that provide broader coverage than the unit tests are able.
Contents |
Test schedule and output
They are run nightly on the test machine NDW161. If the tests pass, then the installer(s) relating to the version under test are copied to the download server, where they are marked as the development build, and the subversion tag 'systemtested' is update, which can be used by anyone wanting to build the latest tested version from source.
If the tests have failed
If the tests fail, then an email indicating this, and showing which tests failed, is sent to the mantid-developers mailing list. For further details, it is necessary to log onto NDW161, where the log output from the lastest run can be found in the C:\MantidSystemTests\SystemTests\logs directory. There will also be a nexus file pertaining to the failed test, named testname-mismatch.nxs, in the C:\MantidSystemTests\SystemTests directory which can be compared to the previously 'correct' result in C:\MantidSystemTests\SystemTests\AnalysisTests\ReferenceResults. It should be noted that these tests are not designed to diagnose faults, only to detect when problems may arise.
The appropriate action may be to correct an error in the code that's been inadvertently introduced, but equally it may be to replace the reference result in the repository with the new one if the change is due to a well-understood and expected improvement. Anticipation of this latter situation would be beneficial to keeping the available development download as up-to-date as possible.
Running the tests off-schedule
Finally, the execution of the system tests can be forced by double-clicking on the MantidTests.bat file in C:\MantidSystemTests. This will install the latest successfully-built windows msi, update the system test directories from the repository and then run the tests.
Writing a test
The (python) code for the system tests can be found in the subversion repository at trunk/Test/SystemTests. The bulk of the infrastructure is shared with (pinched from?) the stress testing framework. Like their 'stress' equivalents, system tests inherit from the stresstesting.MantidStressTest class. The methods that need to be overridden are runTest(self), where the python code that runs the test should be placed, and validate(self), which should simply return a pair of strings: the name of the final workspace that results from the runTest method and the name of a nexus file that should be saved in the ReferenceResults sub-directory in the repository. The test code itself is likely to be the output of a Save History command, though it can be any python code.
Any single test can be executed on an individual's computer by using the runIndividualTest.py driver file. Note that this file contains a number of paths that are likely to need customising.
Once satisfied that the test works, it should be uploaded to the AnalysisTests directory, with the template result going in the ReferenceResults sub-folder. It will then be included in the suite of tests from the following night.
