From 2964657078981b7e4df2725d9acc6e428f1e6e95 Mon Sep 17 00:00:00 2001
From: Matthias Maier
- The deal.II testsuite consists of two parts:
- build tests and the
- regression testsuite. While the build tests
- are used to check that the
- library can be compiled on different systems and with different (versions
- of) compilers, the regression tests are actually run and their output
- compared with previously stored output files to verify that what
- worked yesterday still works today. These two testsuites are
- described below.
+ deal.II has a testsuite that has, at the time this article was last
+ updated (April 2015), some 3,600 small programs (growing by roughly
+ one per day) that we run every time we make a change to make sure
+ that no existing functionality is broken. The expected output for
+ every test is stored in an
- deal.II has a testsuite that has, at the time this article is written
- (mid-2013), some 2,900 small programs (growing by roughly one per day)
- that we run every time we make a change to make sure that no existing
- functionality is broken. The expected output for every test is stored in
- an
- If you're impatient, use the following commands:
-The deal.II Testsuite
*.output
, and when you run a
+ test you are notified if a test produces different output.
*.output
, and when you run a test you are notified if a
- test produces different output. These days, every time we add a
- significant piece of functionality, we add at least one new test to the
- testsuite, and we also do so if we fix a bug, in both cases to make sure
- that future changes do not break what we have just checked in. Machines
- running tests send results
- back home and these are then converted into
+ These days, every time we add a significant piece of functionality,
+ we add at least one new test to the testsuite, and we also do so if
+ we fix a bug, in both cases to make sure that future changes do not
+ break this functionality (again). Machines running the testsuite
+ submit the results to
a webpage showing the status of our regression tests.
Quick instructions
-
-
-$ git clone https://github.com/dealii/dealii
-$ mkdir dealii/build
-$ cd dealii/build
-$ cmake ..
-$ make -j16
-$ make -j16 setup_tests
-$ ctest -j16
-
- The exact meaning of all of these commands will be explained in much
- greater detail below.
-
In order to run it, you need to download and set up the testsuite - first. The following paragraphs detail how to do that. -
- - -
- Just download the current development sources:
+ The testsuite is part of the development sources of deal.II and located
+ under the tests
subdirectory. The easiest way to obtain
+ both of them is to check out the current development sources via git:
$ git clone https://github.com/dealii/dealii- -
- To enable the testsuite, configure and build deal.II in a build - directory as normal (installation is not necessary). After that you - can setup the testsuite via the "setup_tests" target:s + To enable the testsuite for a given build directory, ensure that + deal.II is successfully configured and build (installation is not + necessary). After that you can set up the testsuite via the + "setup_tests" target:
$ make setup_tests@@ -127,7 +95,7 @@ $ make setup_tests
- Setup can be fine-tuned using the following commands: + The setup can be fine-tuned using the following commands:
$ make regen_tests - reruns configure stage in every testsuite subproject @@ -141,11 +109,6 @@ $ make prune_tests - removes all testsuite subprojects variables can be used to override default behavior when callingmake setup_tests
:-TEST_DIFF - - The diff tool and command line to use for comparison. If numdiff is - available it defaults to "numdiff -a 1e-6 -q", otherwise plain diff - is used. - TEST_TIME_LIMIT - The time limit (in seconds) a single test is allowed to take. Defaults to 180 seconds @@ -154,6 +117,29 @@ TEST_PICKUP_REGEX - A regular expression to select only a subset of tests during setup. An empty string is interpreted as a catchall (this is the default).+For example, ++TEST_PICKUP_REGEX="umfpack" make setup_tests ++will only enable tests which match the string "umfpack" in category or +name. + + + +For an already installed library
+ ++ The testsuite can also be set up for an already installed library + (starting with version 8.3). For this, create a build directory for + the testsuite and run cmake pointing to the
tests
+ subdirectory, e.g., ++$ mkdir tests/build +$ cd tests/build +$ cmake .. ++ After that the same configuration targets as described above are available. @@ -360,9 +346,9 @@ xx: =============================== OUTPUT END ============================ In order to achieve the first two, configure the library with -- cmake -DCMAKE_BUILD_TYPE=Debug -DDEAL_II_SETUP_COVERAGE=ON <...> -++cmake -DCMAKE_BUILD_TYPE=Debug -DDEAL_II_SETUP_COVERAGE=ON <...> +You can then build the library and run the tests as usual. @@ -372,13 +358,13 @@ xx: =============================== OUTPUT END ============================ functionality to gather all profiling files and submit them to cdash where we already gather testsuite results (see below). You can do so by invoking -- ctest -DCOVERAGE=YES <...> -S ../tests/run_testsuite.cmake -++ctest -DCOVERAGE=ON <...> -S ../tests/run_testsuite.cmake +when running the testsuite, or directly by -- ctest <...> -S ../tests/run_coverage.cmake -++ctest <...> -S ../tests/run_coverage.cmake +@@ -436,10 +422,12 @@ category/test.output
BUILD_DIR/tests/<category>/<test>
.
RUN
: The run stage then invokes the executable in the
- directory where it is located. By convention, each test puts its
- output into a file simply called output
, which will
- then be located in
+ RUN
: The run stage then invokes the executable in
+ the directory where it is located. By convention, each test
+ either prints its test results directly to stdout
,
+ or directly into a file called output
(in the
+ current working directory). The latter takes precedence.
+ The output (via stdout or file) will then be located in
BUILD_DIR/tests/<category>/<test>/output
.
If the run fails (e.g. because the program aborts with an error
code) the file output
is renamed to
@@ -718,7 +706,7 @@ $ ctest -V -R "category/my_new_test"
CMAKE_MINIMUM_REQUIRED(VERSION 2.8.8) -INCLUDE(${DEAL_II_SOURCE_DIR}/cmake/setup_testsubproject.cmake) +INCLUDE(../setup_testsubproject.cmake) PROJECT(testsuite CXX) INCLUDE(${DEAL_II_TARGET_CONFIG}) DEAL_II_PICKUP_TESTS() -- 2.39.5