<h1>The deal.II Testsuite</h1>
<p>
- The deal.II testsuite consists of two parts:
- <i>build tests</i> and the
- <i>regression testsuite</i>. While the build tests
- are used to check that the
- library can be compiled on different systems and with different (versions
- of) compilers, the regression tests are actually run and their output
- compared with previously stored output files to verify that what
- worked yesterday still works today. These two testsuites are
- described below.
+ deal.II has a testsuite that has, at the time this article was last
+ updated (April 2015), some 3,600 small programs (growing by roughly
+ one per day) that we run every time we make a change to make sure
+ that no existing functionality is broken. The expected output for
+ every test is stored in an <code>*.output</code>, and when you run a
+ test you are notified if a test produces different output.
</p>
<p>
- deal.II has a testsuite that has, at the time this article is written
- (mid-2013), some 2,900 small programs (growing by roughly one per day)
- that we run every time we make a change to make sure that no existing
- functionality is broken. The expected output for every test is stored in
- an <code>*.output</code>, and when you run a test you are notified if a
- test produces different output. These days, every time we add a
- significant piece of functionality, we add at least one new test to the
- testsuite, and we also do so if we fix a bug, in both cases to make sure
- that future changes do not break what we have just checked in. Machines
- running tests send results
- back home and these are then converted into
+ These days, every time we add a significant piece of functionality,
+ we add at least one new test to the testsuite, and we also do so if
+ we fix a bug, in both cases to make sure that future changes do not
+ break this functionality (again). Machines running the testsuite
+ submit the results to
<a href="http://cdash.kyomu.43-1.org/index.php?project=deal.II"
target="body">a webpage showing the status of our regression tests</a>.
</p>
<div class="toc">
<ol>
- <li><a href="#quick">Quick instructions</a></li>
<li><a href="#setup">Setting up the testsuite</a></li>
<ol>
- <li><a href="#setupdownload">Downloading the testsuite</a></li>
- <li><a href="#setupconfigure">Preparing the testsuite</a></li>
+ <li><a href="#setupbuild">For a build directory</a></li>
+ <li><a href="#setupinstalled">For an already installed library</a></li>
</ol>
<li><a href="#run">Running the testsuite</a></li>
<ol>
</ol>
<li><a href="#submit">Submitting test results</a></li>
<li><a href="#build_tests">Build tests</a></li>
- <ol>
- <li><a href="#dedicatedbuilds">Dedicated build tests</a></li>
+ <ol>
+ <li><a href="#dedicatedbuilds">Dedicated build tests</a></li>
</ol>
</ol>
</div>
- <a name="quick"></a>
- <h2>Quick instructions</h2>
-
- <p>
- If you're impatient, use the following commands:
-<pre>
-$ git clone https://github.com/dealii/dealii
-$ mkdir dealii/build
-$ cd dealii/build
-$ cmake ..
-$ make -j16
-$ make -j16 setup_tests
-$ ctest -j16
-</pre>
- The exact meaning of all of these commands will be explained in much
- greater detail below.
- </p>
-
<a name="setup"></a>
<h2>Setting up the testsuite</h2>
- <p> In order to run it, you need to download and set up the testsuite
- first. The following paragraphs detail how to do that.
- </p>
-
- <a name="setupdownload"></a>
- <h3>Downloading the testsuite</h3>
-
<p>
- Just download the current development sources:
+ The testsuite is part of the development sources of deal.II and located
+ under the <code>tests</code> subdirectory. The easiest way to obtain
+ both of them is to check out the current development sources via git:
<pre>
$ git clone https://github.com/dealii/dealii
</pre>
</p>
- <a name="setupconfigure"></a>
- <h3>Preparing the testsuite</h3>
+ <a name="setupbuild"></a>
+ <h3>For a build directory</h3>
<p>
- To enable the testsuite, configure and build deal.II in a build
- directory as normal (installation is not necessary). After that you
- can setup the testsuite via the "setup_tests" target:s
+ To enable the testsuite for a given build directory, ensure that
+ deal.II is successfully configured and build (installation is not
+ necessary). After that you can set up the testsuite via the
+ "setup_tests" target:
<pre>
$ make setup_tests
</pre>
</p>
<p>
- Setup can be fine-tuned using the following commands:
+ The setup can be fine-tuned using the following commands:
<pre>
$ make regen_tests - reruns configure stage in every testsuite subproject
variables can be used to override default behavior when
calling <code>make setup_tests</code>:
<pre>
-TEST_DIFF
- - The diff tool and command line to use for comparison. If numdiff is
- available it defaults to "numdiff -a 1e-6 -q", otherwise plain diff
- is used.
-
TEST_TIME_LIMIT
- The time limit (in seconds) a single test is allowed to take. Defaults
to 180 seconds
- A regular expression to select only a subset of tests during setup.
An empty string is interpreted as a catchall (this is the default).
</pre>
+For example,
+<pre>
+TEST_PICKUP_REGEX="umfpack" make setup_tests
+</pre>
+will only enable tests which match the string "umfpack" in category or
+name.
+ </p>
+
+ <a name="setupinstalled"></a>
+ <h3>For an already installed library</h3>
+
+ <p>
+ The testsuite can also be set up for an already installed library
+ (starting with version 8.3). For this, create a build directory for
+ the testsuite and run cmake pointing to the <code>tests</code>
+ subdirectory, e.g.,
+<pre>
+$ mkdir tests/build
+$ cd tests/build
+$ cmake ..
+</pre>
+ After that the same configuration targets as described <a
+ href="setupbuild">above</a> are available.
</p>
<a name="run"></a>
</li>
</ol>
In order to achieve the first two, configure the library with
- <pre>
- cmake -DCMAKE_BUILD_TYPE=Debug -DDEAL_II_SETUP_COVERAGE=ON <...>
- </pre>
+<pre>
+cmake -DCMAKE_BUILD_TYPE=Debug -DDEAL_II_SETUP_COVERAGE=ON <...>
+</pre>
You can then build the library and run the tests as usual.
</p>
functionality to gather all profiling files and submit them to cdash
where we already gather testsuite results
(see <a href="#submit">below</a>). You can do so by invoking
- <pre>
- ctest -DCOVERAGE=YES <...> -S ../tests/run_testsuite.cmake
- </pre>
+<pre>
+ctest -DCOVERAGE=ON <...> -S ../tests/run_testsuite.cmake
+</pre>
when running the testsuite, or directly by
- <pre>
- ctest <...> -S ../tests/run_coverage.cmake
- </pre>
+<pre>
+ctest <...> -S ../tests/run_coverage.cmake
+</pre>
</p>
<p>
<code>BUILD_DIR/tests/<category>/<test></code>.
</li>
<li>
- <code>RUN</code>: The run stage then invokes the executable in the
- directory where it is located. By convention, each test puts its
- output into a file simply called <code>output</code>, which will
- then be located in
+ <code>RUN</code>: The run stage then invokes the executable in
+ the directory where it is located. By convention, each test
+ either prints its test results directly to <code>stdout</code>,
+ or directly into a file called <code>output</code> (in the
+ current working directory). The latter takes precedence.
+ The output (via stdout or file) will then be located in
<code>BUILD_DIR/tests/<category>/<test>/output</code>.
If the run fails (e.g. because the program aborts with an error
code) the file <code>output</code> is renamed to
</p>
<pre>
CMAKE_MINIMUM_REQUIRED(VERSION 2.8.8)
-INCLUDE(${DEAL_II_SOURCE_DIR}/cmake/setup_testsubproject.cmake)
+INCLUDE(../setup_testsubproject.cmake)
PROJECT(testsuite CXX)
INCLUDE(${DEAL_II_TARGET_CONFIG})
DEAL_II_PICKUP_TESTS()