<h1>The deal.II Testsuite</h1>
- <p class="todo">
- The deal.II testsuite consists of two parts, the
- <a href="#build_tests">build tests</a> and the
- <a href="#regression_tests">regression tests</a>. While the build tests
- just check if the
+ <p>
+ The deal.II testsuite consists of two parts:
+ <i>build tests</i> and the
+ <i>regression testsuite</i>. While the build tests
+ are used to check that the
library can be compiled on different systems and with different (versions
of) compilers, the regression tests are actually run and their output
- compared with previously stored. These two testsuites are
+ compared with previously stored output files to verify that what
+ worked yesterday still works today. These two testsuites are
described below.
</p>
- <p class="todo">
- deal.II has a testsuite that, at the time this article is written
- (mid-2013), has some 2,900 small programs (growing by roughly one per
- day) that we run every time we make a change to make sure that no
- existing functionality is broken. The expected output is also stored in
+ <p>
+ deal.II has a testsuite that has, at the time this article is written
+ (mid-2013), some 2,900 small programs (growing by roughly one per day)
+ that we run every time we make a change to make sure that no existing
+ functionality is broken. The expected output for every test is stored in
our subversion archive, and when you run a test you are notified if a
- test fails. These days, every time we add a significant piece of
- functionality, we add at least one new test to the testsuite, and we
- also do so if we fix a bug, in both cases to make sure that future
- changes do not break what we have just checked in. In addition, some
- machines run the tests every night and send the results back home; this
- is then converted into
+ test produces different output. These days, every time we add a
+ significant piece of functionality, we add at least one new test to the
+ testsuite, and we also do so if we fix a bug, in both cases to make sure
+ that future changes do not break what we have just checked in. Machines
+ running tests send results
+ back home and these are then converted into
<a href="http://dealii.mathsim.eu/cgi-bin/regression_quick.pl"
target="body">a webpage showing the status of our regression tests</a>.
</p>
<div class="toc">
<ol>
- <li><a href="#setup">Set up the testsuite</a></li>
+ <li><a href="#quick">Quick instructions</a></li>
+ <li><a href="#setup">Setting up the testsuite</a></li>
<ol>
- <li><a href="#setupdownload">Download the testsuite</a></li>
- <li><a href="#setupconfigure">Prepare the testsuite</a></li>
+ <li><a href="#setupdownload">Downloading the testsuite</a></li>
+ <li><a href="#setupconfigure">Preparing the testsuite</a></li>
</ol>
- <li><a href="#run">Run the testsuite</a></li>
+ <li><a href="#run">Running the testsuite</a></li>
<ol>
<li><a href="#runoutput">How to interpret the output</a></li>
</ol>
<li><a href="#layout">Testsuite development</a></li>
<ol>
<li><a href="#layoutgeneral">General layout</a></li>
- <li><a href="#layoutcomparisonfile">Comparison file</a></li>
+ <li><a href="#restrictbuild">Restricting tests for build configurations</a></li>
+ <li><a href="#restrictfeature">Restricting tests for feature configurations</a></li>
+ <li><a href="#mpi">Running tests with MPI</a></li>
<li><a href="#layoutaddtests">Adding new tests</a></li>
</ol>
- <li><a href="#submit">Submit test results</a></li>
- <li><a href="#build_tests">The build tests</a></li>
+ <li><a href="#submit">Submitting test results</a></li>
+ <li><a href="#build_tests">Build tests</a></li>
+ <ol>
+ <li><a href="#dedicatedbuilds">Dedicated build tests</a></li>
+ </ol>
</ol>
</div>
+ <a name="quick"></a>
+ <h2>Quick instructions</h2>
+
+ <p>
+ If you're impatient, use the following commands:
+ <pre>
+ $ mkdir new_directory
+ $ cd new_directory
+ $ svn checkout https://svn.dealii.org/trunk .
+ $ mkdir build
+ $ cd build
+ $ cmake ../deal.II
+ $ make -j16
+ $ make -j16 setup_test
+ $ ctest -j16
+ </pre>
+ The exact meaning of all of these commands will be explained in much
+ greater detail below.
+ </p>
+
<a name="setup"></a>
- <h2>Set up the testsuite</h2>
+ <h2>Setting up the testsuite</h2>
- <p class="todo"> Here, some text is missing</p>
+ <p> In order to run it, you need to download and set up the testsuite
+ first. The following paragraphs detail how to do that.
+ </p>
<a name="setupdownload"></a>
- <h3>Download the testsuite</h3>
+ <h3>Downloading the testsuite</h3>
<p>
- In order to run the testsuite you have to download it first. The
- easiest way is to directly check out the testsuite along with deal.II
- from the subversion repository. Go to an empty directory where you
+ To download the testsuite, check it out from the subversion repository,
+ along with deal.II. To this end, go to an empty directory where you
want to test deal.II and do this:
<pre>
$ svn checkout https://svn.dealii.org/trunk .
</pre>
- (Do not forget the dot "." at the end.) This should leave you with
+ (The period at the end puts everything from under <code>trunk/</code>
+ into the current directory, rather than creating a
+ new <code>trunk/</code> directory.) You will then have
two folders:
<pre>
</pre>
</p>
- <p>
- <b>Note:</b> If you want to check out the testsuite separately, you
- can do so with
- <pre>
-
- $ svn checkout https://svn.dealii.org/trunk/tests
- </pre>
- </p>
-
<p>
<b>Note:</b> CMake will pick up any testsuite that is located in a
<code>tests</code> folder next to the source directory
</p>
<a name="setupconfigure"></a>
- <h3>Prepare the testsuite</h3>
+ <h3>Preparing the testsuite</h3>
<p>
To enable the testsuite, configure and build deal.II in a build
$ make setup_test
</pre>
- This will set up all tests supported by the current configuration
- (and not otherwise disabled due to <code>TEST_PICKUP_REGEX</code>).
- Now, the testsuite can be run in the _build directory_ via the
- <code>ctest</code> command (as will be explained in the next
- section).
+ This will set up all tests supported by the current configuration.
+ The testsuite can now be run in the current <i>build directory</i> as
+ described below.
</p>
- Additionally there are also the following targets available:
+
+ <p>
+ Setup can be fine-tuned using the following commands:
<pre>
$ make clean_test - runs the 'clean' target in every testsuite subproject
</pre>
<p>
- The testsuite uses the following CMake variables:
+ In addition, when setting up the testsuite, the following environment
+ variables can be used to override default behavior when
+ calling <code>make setup_test</code>:
<pre>
TEST_PICKUP_REGEX
- - A regular expression to filter tests. If this is a nonempty string
- only tests that match the regular expression will be set up. An empty
- string is interpreted as a catchall.
+ - A regular expression to select only a subset of tests during setup.
+ An empty string is interpreted as a catchall (this is the default).
TEST_DIFF
- - the diff tool and command line to use for comparison. If numdiff is
+ - The diff tool and command line to use for comparison. If numdiff is
available it defaults to "numdiff -a 1e-6 -q", otherwise plain diff
is used.
TEST_TIME_LIMIT
- - The time limit (in seconds) a single test is allowed to run. Defaults
+ - The time limit (in seconds) a single test is allowed to take. Defaults
to 180 seconds
</pre>
- These options can be set as environment variables prior to the call to the
- setup_test target:
- <pre>
-
- $ TEST_PICKUP_REGEX="^build_tests/" TEST_TIME_LIMIT="120" make setup_test
- </pre>
</p>
<p>
<b>Note:</b> Specifying these options via environment variables is
- volatile, i.e. if <code>$ make setup_test</code> is invoked a second
+ volatile, i.e. if <code>make setup_test</code> is invoked a second
time without the variables set in environment, the option will be
reset to the default value. If you want to set these options
permanently, set them via cmake as CMake variable in the build
$ cmake -DTEST_PICKUP_REGEX="<regular expression>" .
</pre>
- Please also note that a variable set via cmake always _overrides_ one
- set via environment. If you wish to reset such a variable again,
- undefine it in the cache:
- <pre>
-
- $ cmake -UTEST_PICKUP_REGEX .
- </pre>
+ A variable set via cmake always <i>overrides</i> one
+ set via environment.
</p>
<a name="run"></a>
- <h2>Run the testsuite</h2>
+ <h2>Running the testsuite</h2>
<p>
- Now, the testsuite can be run in the _build directory_ via
- <pre>
-
- $ ctest [-j x]
- </pre>
- where x is the number of concurrent tests that should be run. The
- testsuite is huge (!) and will need around 12h on current computer
- running single threaded. If you only want to run a subset of tests
- matching a regular expression, you can use
+ The testsuite can now be run in the <i>build directory</i> via
<pre>
- $ ctest [-j x] -R '<regular expression>'
+ $ ctest [-j N]
</pre>
+ Here, <code>N</code> is the number of concurrent tests that should be
+ run, in the same way as you can say <code>make -jN</code>. The testsuite
+ is huge and will need around 12h on current computers
+ running single threaded.
</p>
<p>
- <b>Note:</b> You can also invoke <code>ctest</code> under
- <code>BUILD_DIR/tests</code> or any subdirectory under
- <code>BUILD_DIR/tests</code>. This will only invoke the tests that
- are located under the subdirectory.
- </p>
-
- <p>
- To get verbose output of tests (which is otherwise just logged into
- <code>Testing/Temporary/LastTest.log</code>) specify
+ If you only want to run a subset of tests
+ matching a regular expression, or if you want to exclude tests matching
+ a regular expression, you can use
<pre>
- $ ctest -V [...]
+ $ ctest [-j N] -R '<positive regular expression>'
+ $ ctest [-j N] -E '<negative regular expression>'
</pre>
- Alternatively, if you're just interested in verbose output of failing
- tests, <code>--output-on-failure</code>.
</p>
<p>
install the
<a href="http://www.nongnu.org/numdiff/">numdiff</a> tool that compares
stored and newly created output files based on floating point
- tolerances. To use it, simply export it via the <code>PATH</code>
+ tolerances. To use it, simply export where the <code>numdiff</code>
+ executable can be found via the <code>PATH</code>
environment variable so that it can be found during
<code>make setup_test</code>.
</p>
- <p>
- In a similar vain, there is also an option to disable tests matching a
- regular exression:
- <pre>
-
- $ ctest -E '<regular expression>' [...]
- </pre>
- </p>
-
<a name="runoutput"></a>
<h3>How to interpret the output</h3>
Errors while running CTest
</pre>
If a test failed (like <code>base/thread_validity_08.debug</code> in above
- example output), you might want to find out what exactly went wrong.
- So, invoke <code>ctest</code> to just run the above test with verbose
- output:
+ example output), you might want to find out what exactly went wrong. To
+ this end, you can search
+ through <code>Testing/Temporary/LastTest.log</code> for the exact output
+ of the test, or you can rerun this one test, specifying <code>-V</code>
+ to select verbose output of tests:
<pre>
$ ctest -V -R "base/thread_validity_08.debug"
<code>PASSED</code>: the test run successful
</li>
</ul>
-
+ Typically, tests fail because the output has changed, and you will see
+ this in the DIFF phase of the test.
+ </p>
<a name="layout"></a>
- <h3>Testsuite development</h3>
+ <h2>Testsuite development</h2>
- <p class="todo"> Here, some text is missing</p>
+ <p>
+ The following outlines what you need to know if you want to understand
+ how the testsuite actually works, for example because you may want to
+ add tests along with the functionality you are currently developing.
+ </p>
<pre>
category/test.cc
- category/test[...].output
- </pre>
+ category/test.output
+ </pre>
+ <code>category</code> will be one of the existing subdirectory
+ under <code>tests/</code>, e.g., <code>lac/</code>, <code>base/</code>,
+ or <code>mpi/</code>. Historically, we have grouped tests into the
+ directories <code>base/, lac/, deal.II/</code> depending on their
+ functionality, and <code>bits/</code> if they were small unit tests, but
+ in practice we have not always followed this rigidly. There are also
+ more specialized directories <code>trilinos/, petsc/,
+ serialization/, mpi/</code> etc, whose meaning is more obvious.
<code>test.cc</code> must be a regular executable (i.e. having an
<code>int main()</code> routine). It will be compiled, linked and
run. The executable should not output anything to <code>cout</code>
(at least under normal circumstances, i.e. no error condition),
instead the executable should output to a file <code>output</code>
- under the current working directory.
+ in the current working directory. In practice, we rarely write the
+ source files completely from scratch, but we find an existing test that
+ already does something similar and copy/modify it to fit our needs.
</p>
<p>
- In detail, for a regular test the following 3 stages will be run:
+ For a normal test, <code>ctest</code> will typically run the following 3
+ stages:
<ul>
<li>
<code>BUILD</code>: The build stage generates an executable in
<code>BUILD_DIR/tests/<category>/<test></code>.
</li>
<li>
- <code>RUN</code>: The run stages invokes the executable that
- generates an output file
+ <code>RUN</code>: The run stage then invokes the executable in the
+ directory where it is located. By convention, each test puts its
+ output into a file simply called <code>output</code>, which will
+ then be located in
<code>BUILD_DIR/tests/<category>/<test>/output</code>.
If the run fails (e.g. because the program aborts with an error
code) the file <code>output</code> is renamed to
<code>failing_diff</code>.
</li>
</ul>
-
- <p>
</p>
- <a name="layoutcomparisonfile"></a>
- <h3>Comparison file</h3>
+
+ <a name="restrictbuild"></a>
+ <h3>Restricting tests for build configurations</h3>
<p>
- The full file signature for a comparison file is
+ Comparison file can actually be named in a more complex way than
+ just <code>category/test.output</code>:
<pre>
category/test.[with_<feature>=<on|off>.]*[mpirun=<x>.][<debug|release>.]output
</pre>
- which is explained in detail below.
- </p>
-
- <h4>Restrict tests for build configurations</h4>
- <p>
- Normally, a test will be set up for debug and release configuration
- (if deal.II was configured with combined <code>DebugRelease</code>
- build type) or for the available build configuration (if deal.II was
- configured either with <code>Debug</code> or with
- <code>Release</code> only build type).
+ Normally, a test will be set up so that it runs twice, once in debug and
+ once in release configuration.
If a specific test can only be run in debug or release configurations but
not in both it is possible to restrict the setup by prepeding
<code>.debug</code> or <code>.release</code> directly before
category/test.debug.output
</pre>
- This way, test will only be set up to build and run against the debug
- library.
- </p>
-
- <p>
- <b>Note:</b> It is possible to provide both configuration types at the
- same time:
+ This way, the test will only be set up to build and run against the debug
+ library. If a test should run in both configurations but, for some
+ reason, produces different output (e.g., because it triggers an
+ assertion in debug mode), then you can just provide two different output
+ files:
<pre>
- category/test.debug.output
- category/test.release.output
+ category/test.debug.output
+ category/test.release.output
</pre>
- This will set up two seperate tests, one for the debug configuration that
- will be tested against test.debug.output, and similarly one for release.
+ </p>
+
- <h4>Restrict tests for feature configurations</h4>
+ <a name="restrictfeature"></a>
+ <h3>Restricting tests for feature configurations</h3>
<p>
In a similar vain as for build configurations, it is possible to restrict
tests to specific feature configurations, e.g.:
category/test.with_umfpack=on.output, or
category/test.with_zlib=off.output
</pre>
- These tests will only be set up if the specified feature was configured
- accordingly.
- </p>
-
- <p>
- <b>Note:</b> It is possible to provide different output files for disabled/enabled
+ These tests will only be set up if the specified feature was configured.
+ It is possible to provide different output files for disabled/enabled
features, e.g.
<pre>
category/test.with_64bit_indices=on.output
category/test.with_64bit_indices=off.output
</pre>
- </p>
- <p>
- <b>Note:</b> It is possible to declare multiple constraints subsequently, e.g.
+ It is also possible to declare multiple constraints subsequently, e.g.
<pre>
category/test.with_umfpack=on.with_zlib=on.output
</pre>
</p>
<p>
- <b>Note:</b> Quite a number of test categories are already guarded so
- that the contained tests will only be set up if the feature is
- enabled. In this case a feature constraint in the output file name is
- redundant and should be avoided. (Folders with guards are
+ <b>Note:</b> The tests in some subdirectories of <code>tests/</code> are
+ automatically run only if some feature is enabled. In this case a
+ feature constraint encoded in the output file name is
+ redundant and should be avoided. In particular, this holds for
+ subdirectories
<code>distributed_grids</code>, <code>lapack</code>,
<code>metis</code>, <code>petsc</code>, <code>slepc</code>,
- <code>trilinos</code>, <code>umfpack</code>, <code>gla</code>,
- <code>mpi</code>)
+ <code>trilinos</code>, <code>umfpack</code>, <code>gla</code>, and
+ <code>mpi</code>
</p>
- <h4>Run mpi tests with mpirun</h4>
+
+ <a name="mpi"></a>
+ <h3>Running tests with MPI</h3>
<p>
- If a test should be run with mpirun in parallel, specify the number x of
- simultaneous processes in the following way:
+ If a test should be run with MPI in parallel, the number of MPI
+ processes <code>N</code> with which a program needs to be run for
+ comparison with a given output file is specified as follows:
<pre>
- category/test.mpirun=x.output
+ category/test.mpirun=N.output
</pre>
+ It is quite typical for an MPI-enabled test to have multiple output
+ files for different numbers of MPI processes.
</p>
- <p>
- <b>Note:</b> It is possible to provide multiple output files for different mpirun
- values.
<a name="layoutaddtests"></a>
<h3>Adding new tests</h3>
<p>
- As mentioned above, we add a new test every
- time we add new functionality to the library or fix a bug. If you
- want to contribute code to the library, you should do this
- as well. Here's how: you need a testcase,
- a subdirectory with the same name as the test, and a file with the
- expected output.
+ We typically add one or more new tests every
+ time we add new functionality to the library or fix a bug. If you
+ want to contribute code to the library, you should do this
+ as well. Here's how: you need a testcase and a file with the
+ expected output.
</p>
<h4>The testcase</h4>
<p>
- For the testcase, we usually start from a template like this:
- <pre class="cmake"> <!-- TODO -->
+ For the testcase, we usually start from one of the existing tests, copy
+ and modify it to where it does what we'd like to test. Alternatively,
+ you can also start from a template like this:
+ <pre>
// ---------------------------------------------------------------------
// $Id$
//
// a short (a few lines) description of what the program does
#include "../tests.h"
-#include <iostream>
-#include <fstream>
+#include <iostream>
+#include <fstream>
// all include files you need here
}
</pre>
- <p>You open an output file <code>output</code> in the current working
- directory and then write all output you generate to it, through the
+ <p>This code opens an output file <code>output</code> in the current working
+ directory and then writes all output you generate to it, through the
<code>deallog</code> stream. The <code>deallog</code> stream works like
any other <code>std::ostream</code> except that it does a few more
things behind the scenes that are helpful in this context. In above
- case, we only write a zero to the output file. Most tests actually
+ case, we only write a zero to the output file. Most tests of course
write computed data to the output file to make sure that whatever we
compute is what we got when the test was first written.
</p>
have traditionally been into the <code>base/</code>,
<code>lac/</code>, <code>deal.II/</code>, <code>fe/</code>,
<code>hp/</code>, or <code>multigrid/</code> directories, depending on
- where the classes that are tested are located.
- </p>
-
- <p>
- We have started to create more atomic tests which
- are usually very small and test only a single aspect of the
- library, often only a single function. These tests go into the
- <code>bits/</code> directory and often have names that are
- composed of the name of the class being tested and a two-digit
- number, e.g., <code>dof_tools_11</code>. There are
+ where the classes that are tested are located. More atomic tests often go
+ into <code>bits/</code>. There are also
directories for PETSc and Trilinos wrapper functionality.
</p>
<p>
If you run your new test executable this way, the test should compile
- and run successfully but fail in the diff stage (due to the empty
+ and run successfully but fail in the diff stage (because of the empty
comparison file). You will get an output file
- <code>BUILD_DIR/category/my_new_test/output</code> that should be
- used to compare all future runs with. If the test is relatively
- simple, it is often a good idea to look at the output and make sure
- that the output is actually what you had expected. However, if you do
- complex operations, this may sometimes be impossible, and in this
- case we are quite happy with any reasonable output file just to make
- sure that future invokations of the test yield the same results.
+ <code>BUILD_DIR/category/my_new_test/output</code>. Take a look at it to
+ make sure that the output is what you had expected. (For complex tests,
+ it may sometimes be impossible to say whether the output is correct, and
+ in this case we sometimes just take it to make
+ sure that future invokations of the test yield the same results.)
</p>
<p>
</pre>
</p>
+
<h4>Checking in</h4>
<p>
<a name="submit"></a>
- <h2>Submit test results</h2>
+ <h2>Submitting test results</h2>
<p class="todo">
Explain how to use <code>run_testsuite.cmake</code> in all imaginable
<a name="build_tests"></a>
- <h2>The build tests</h2>
-
- <p class="todo">
- Update this section
- </p>
+ <h2>Build tests</h2>
<p>
- With our build tests, we check if deal.II can be compiled on
+ Build tests are used to check that deal.II can be compiled on
different systems and with different compilers as well as
different configuration options. Results are collected in a
database and can be accessed <a
</pre>
</p>
+ <p>
+ What this does is to create a temporary directory, compile and build
+ deal.II in it, and for good measure also build the tutorial
+ programs. The fourth of the commands above then sends the resulting
+ status files to a daemon that presents this information on the website
+ linked to above.
+ </p>
+
<p>
The <code>build_test</code> script supports the following options:
<pre>
</pre>
An example configuration file can be found <a
href="../users/Config.sample">here</a>. Options can be passed either via
- environment
+ environment variables
<pre>
export CONFIGFILE=MyConfiguration.conf
periodically, but not immediately after a mail has been received).
</p>
+
+ <a name="dedicatedbuilds"></a>
<h3>Dedicated build tests</h3>
<p>
- There is a detailed example for dedicated build tests on the <a
+ Build tests work best if they run automatically and periodically.
+ There is a detailed example for such dedicated build tests on the <a
href="https://code.google.com/p/dealii/wiki/BuildTests">wiki</a>.
</p>
projects / dealii.git / commitdiff