From 2964657078981b7e4df2725d9acc6e428f1e6e95 Mon Sep 17 00:00:00 2001 From: Matthias Maier Date: Thu, 2 Apr 2015 16:19:21 +0200 Subject: [PATCH] Documentation: Update documentation on testsuite --- doc/developers/testsuite.html | 140 ++++++++++++++++------------------ 1 file changed, 64 insertions(+), 76 deletions(-) diff --git a/doc/developers/testsuite.html b/doc/developers/testsuite.html index 9ec77e6320..5d293d4aa6 100644 --- a/doc/developers/testsuite.html +++ b/doc/developers/testsuite.html @@ -15,40 +15,30 @@

The deal.II Testsuite

- The deal.II testsuite consists of two parts: - build tests and the - regression testsuite. While the build tests - are used to check that the - library can be compiled on different systems and with different (versions - of) compilers, the regression tests are actually run and their output - compared with previously stored output files to verify that what - worked yesterday still works today. These two testsuites are - described below. + deal.II has a testsuite that has, at the time this article was last + updated (April 2015), some 3,600 small programs (growing by roughly + one per day) that we run every time we make a change to make sure + that no existing functionality is broken. The expected output for + every test is stored in an *.output, and when you run a + test you are notified if a test produces different output.

- deal.II has a testsuite that has, at the time this article is written - (mid-2013), some 2,900 small programs (growing by roughly one per day) - that we run every time we make a change to make sure that no existing - functionality is broken. The expected output for every test is stored in - an *.output, and when you run a test you are notified if a - test produces different output. These days, every time we add a - significant piece of functionality, we add at least one new test to the - testsuite, and we also do so if we fix a bug, in both cases to make sure - that future changes do not break what we have just checked in. Machines - running tests send results - back home and these are then converted into + These days, every time we add a significant piece of functionality, + we add at least one new test to the testsuite, and we also do so if + we fix a bug, in both cases to make sure that future changes do not + break this functionality (again). Machines running the testsuite + submit the results to a webpage showing the status of our regression tests.

    -
  1. Quick instructions
  2. Setting up the testsuite
    1. -
    2. Downloading the testsuite
    3. -
    4. Preparing the testsuite
    5. +
    6. For a build directory
    7. +
    8. For an already installed library
  3. Running the testsuite
    1. @@ -70,54 +60,32 @@
  4. Submitting test results
  5. Build tests
  6. -
      -
    1. Dedicated build tests
    2. +
        +
      1. Dedicated build tests
- -

Quick instructions

- -

- If you're impatient, use the following commands: -

-$ git clone https://github.com/dealii/dealii
-$ mkdir dealii/build
-$ cd dealii/build
-$ cmake ..
-$ make -j16
-$ make -j16 setup_tests
-$ ctest -j16
-
- The exact meaning of all of these commands will be explained in much - greater detail below. -

-

Setting up the testsuite

-

In order to run it, you need to download and set up the testsuite - first. The following paragraphs detail how to do that. -

- - -

Downloading the testsuite

-

- Just download the current development sources: + The testsuite is part of the development sources of deal.II and located + under the tests subdirectory. The easiest way to obtain + both of them is to check out the current development sources via git:

 $ git clone https://github.com/dealii/dealii
 

- -

Preparing the testsuite

+ +

For a build directory

- To enable the testsuite, configure and build deal.II in a build - directory as normal (installation is not necessary). After that you - can setup the testsuite via the "setup_tests" target:s + To enable the testsuite for a given build directory, ensure that + deal.II is successfully configured and build (installation is not + necessary). After that you can set up the testsuite via the + "setup_tests" target:

 $ make setup_tests
 
@@ -127,7 +95,7 @@ $ make setup_tests

- Setup can be fine-tuned using the following commands: + The setup can be fine-tuned using the following commands:

 $ make regen_tests - reruns configure stage in every testsuite subproject
 
@@ -141,11 +109,6 @@ $ make prune_tests - removes all testsuite subprojects
       variables can be used to override default behavior when
       calling make setup_tests:
 
-TEST_DIFF
-  - The diff tool and command line to use for comparison. If numdiff is
-    available it defaults to "numdiff -a 1e-6 -q", otherwise plain diff
-    is used.
-
 TEST_TIME_LIMIT
   - The time limit (in seconds) a single test is allowed to take. Defaults
     to 180 seconds
@@ -154,6 +117,29 @@ TEST_PICKUP_REGEX
   - A regular expression to select only a subset of tests during setup.
     An empty string is interpreted as a catchall (this is the default).
 
+For example, +
+TEST_PICKUP_REGEX="umfpack" make setup_tests
+
+will only enable tests which match the string "umfpack" in category or +name. +

+ + +

For an already installed library

+ +

+ The testsuite can also be set up for an already installed library + (starting with version 8.3). For this, create a build directory for + the testsuite and run cmake pointing to the tests + subdirectory, e.g., +

+$ mkdir tests/build
+$ cd tests/build
+$ cmake ..
+
+ After that the same configuration targets as described above are available.

@@ -360,9 +346,9 @@ xx: =============================== OUTPUT END ============================ In order to achieve the first two, configure the library with -
-  cmake -DCMAKE_BUILD_TYPE=Debug -DDEAL_II_SETUP_COVERAGE=ON <...>
-      
+
+cmake -DCMAKE_BUILD_TYPE=Debug -DDEAL_II_SETUP_COVERAGE=ON <...>
+
You can then build the library and run the tests as usual.

@@ -372,13 +358,13 @@ xx: =============================== OUTPUT END ============================ functionality to gather all profiling files and submit them to cdash where we already gather testsuite results (see below). You can do so by invoking -
-  ctest -DCOVERAGE=YES <...> -S ../tests/run_testsuite.cmake
-      
+
+ctest -DCOVERAGE=ON <...> -S ../tests/run_testsuite.cmake
+
when running the testsuite, or directly by -
-  ctest <...> -S ../tests/run_coverage.cmake
-      
+
+ctest <...> -S ../tests/run_coverage.cmake
+

@@ -436,10 +422,12 @@ category/test.output BUILD_DIR/tests/<category>/<test>.

  • - RUN: The run stage then invokes the executable in the - directory where it is located. By convention, each test puts its - output into a file simply called output, which will - then be located in + RUN: The run stage then invokes the executable in + the directory where it is located. By convention, each test + either prints its test results directly to stdout, + or directly into a file called output (in the + current working directory). The latter takes precedence. + The output (via stdout or file) will then be located in BUILD_DIR/tests/<category>/<test>/output. If the run fails (e.g. because the program aborts with an error code) the file output is renamed to @@ -718,7 +706,7 @@ $ ctest -V -R "category/my_new_test"

     CMAKE_MINIMUM_REQUIRED(VERSION 2.8.8)
    -INCLUDE(${DEAL_II_SOURCE_DIR}/cmake/setup_testsubproject.cmake)
    +INCLUDE(../setup_testsubproject.cmake)
     PROJECT(testsuite CXX)
     INCLUDE(${DEAL_II_TARGET_CONFIG})
     DEAL_II_PICKUP_TESTS()
    -- 
    2.39.5