From 50c7635220d8cb80cb76cd35ecbacf31d25f3609 Mon Sep 17 00:00:00 2001 From: Matthias Maier Date: Thu, 20 Aug 2015 15:28:50 -0500 Subject: [PATCH] Documentation: Document user testsuite feature This adds a bunch of documentation on how to use the testsuite facilities in user projects. Closes #1249 In reference to #1246 --- cmake/macros/macro_deal_ii_pickup_tests.cmake | 4 + doc/documentation.html | 1 + doc/users/testsuite.html | 291 ++++++++++++++++++ 3 files changed, 296 insertions(+) create mode 100644 doc/users/testsuite.html diff --git a/cmake/macros/macro_deal_ii_pickup_tests.cmake b/cmake/macros/macro_deal_ii_pickup_tests.cmake index 2a86367355..5909566eaa 100644 --- a/cmake/macros/macro_deal_ii_pickup_tests.cmake +++ b/cmake/macros/macro_deal_ii_pickup_tests.cmake @@ -29,6 +29,10 @@ # TEST_LIBRARIES_RELEASE # - specifying additional libraries (and targets) to link against. # +# TEST_TARGET or +# TEST_TARGET_DEBUG and TEST_TARGET_RELEASE +# - specifying a test target to be executed for a parameter run. +# # TEST_TIME_LIMIT # - specifying the maximal wall clock time in seconds a test is # allowed to run diff --git a/doc/documentation.html b/doc/documentation.html index 96274862d8..b61fb7f41a 100644 --- a/doc/documentation.html +++ b/doc/documentation.html @@ -30,6 +30,7 @@ installation instructions

diff --git a/doc/users/testsuite.html b/doc/users/testsuite.html new file mode 100644 index 0000000000..544195ebed --- /dev/null +++ b/doc/users/testsuite.html @@ -0,0 +1,291 @@ + + + + + The deal.II Testsuite + + + + + + + + +

Setting up testsuite in user projects

+ +

+ This page provides details about how to set up a testsuite in a user + project similar to the one that is used to test deal.II itself. +

+ +

Overview
Examples

Simple configuration
Advanced configuration

+ + +

Overview

+ +

+ deal.II features an extensive testsuite to ensure + consistent, well-defined behavior of its building blocks during + development and for releases. But, the larger a user program/project + becomes, the more important it is to also check user code for + continued correctness during development. This is mainly done via + unit and + regression testing. +

+ +

+ deal.II provides a mechanism to conveniently set up unit and + regression tests in a user project (very much like they are handled + in the library itself). At its heart a test is a small executable + that is invoked and an output file for comparison. The executable + that should be run can be defined in two different ways: Either as a + source file in conjunction with a comparison file: +

+my_test_1.cc
+my_test_1.output

+ In this case my_test_1.cc contains a full executable (with + a main function) that produces some output. The screen output is then + compared against my_test_1.output. Alternatively, a + parameter file together with a comparison file can be provided: +

+my_test_2.prm
+my_test_2.output

+ In this case an already built executable (that is defined by a CMake + variable) is invoked with the path of my_test_2.prm as + first argument. Again, its screen output is compared against + my_test_2.output +

+ + +

Examples

+ +

+ This section presents two different examples of how to use the + testsuite facilities. Possible directory layouts together with the + necessary CMake configuration are discussed. +

+ + +

Simple configuration

+ +

+ For the purpose of an example, let us pretend that step-1 could read + input files (defined on the command line) and do some computation + based on their contents. Then, we can set up tests for expected + output for given a given configuration file. + + This can be done by creating a subdirectory tests and + adding a test of the second type (i.e., parameter file and comparison + file). In detail the directory and file layout is as follows: +

+CMakeLists.txt
+step-1.cc
+tests/CMakeLists
+tests/my_test.output
+tests/my_test.prm

+ In order to enable testing the top-level CMakeLists.txt + file has to be augmented by a call to ENABLE_TESTING() + and a subsequent descent into the tests/ subdirectory via + ADD_SUBDIRECTORY(tests). For convenience, here is the + full top-level CMakeLists.txt file: +

+SET(TARGET step-1)
+SET(TARGET_SRC step-1.cc)
+
+CMAKE_MINIMUM_REQUIRED(VERSION 2.8.8)
+FIND_PACKAGE(deal.II 8.3 REQUIRED
+  HINTS ${deal.II_DIR} ${DEAL_II_DIR} ../ ../../ $ENV{DEAL_II_DIR}
+  )
+
+DEAL_II_INITIALIZE_CACHED_VARIABLES()
+PROJECT(${TARGET})
+DEAL_II_INVOKE_AUTOPILOT()
+
+# Enable testing and descent into tests/ subdirectory:
+ENABLE_TESTING()
+ADD_SUBDIRECTORY(tests)

+ The corresponding file tests/CMakeLists.txt contains + only two statements: +

+SET(TEST_TARGET ${TARGET})
+DEAL_II_PICKUP_TESTS()

+ The first statement sets the variable TEST_TARGET to the + executable that should be invoked (in our case the contents of the + variable TARGET). The second statement is a call to a + deal.II macro that will go through the directory contents and define + all test targets. +

+ +

Due to the fact that step-1 produces only two lines of output and + parses no parameters, we can set up a somewhat silly test by just + providing the comparison file and an empty parameter file: +

+$ touch tests/my_test.prm
+$ echo "Grid written to grid-1.eps" >  tests/my_test.output
+$ echo "Grid written to grid-2.eps" >> tests/my_test.output

+ After that, reconfigure and call the test driver ctest: +

+$ cmake .
+[...]
+$ ctest
+Test project .../examples/step-1
+    Start 1: tests/my_test.debug
+1/1 Test #1: tests/my_test.debug ..............   Passed    1.72 sec
+
+100% tests passed, 0 tests failed out of 1
+
+Total Test time (real) =   1.72 sec

+ +

+ Remark: The test driver will compare the combined output + stream of stdout and stderr against the comparison file. If the test + creates a file output and writes to it, the comparison + file is compared against this output file instead. In this case + stdout and stderr are discarded. +

+ + +

Advanced configuration

+ +

+ Above setup is too inflexible for larger projects that might consist + of individual libraries and an independent main program. Therefore, + as a second example a project is presented that consists of a support + library "support" and an executable "step". The task shall be to + provide unit tests for the library "support" and simple configuration + type tests for "step". In detail: +

+CMakeLists.txt
+
+src/CMakeLists.txt
+src/step.cc
+src/support.cc
+
+tests/step/CMakeLists
+tests/step/my_test_1.prm
+tests/step/my_test_1.output
+
+tests/support/CMakeLists
+tests/support/my_test_2.cc
+tests/support/my_test_2.output

+ +

+ Again, we want to use the "autopilot" configuration for user projects + (see the cmake documentation for + details). The top-level CMakeLists.txt is now solely + responsible for finding deal.II, enable testing, and descending into + subdirectories: +

+# top-level CMakelists.txt
+
+CMAKE_MINIMUM_REQUIRED(VERSION 2.8.8)
+FIND_PACKAGE(deal.II 8.3 REQUIRED
+  HINTS ${deal.II_DIR} ${DEAL_II_DIR} ../ ../../ $ENV{DEAL_II_DIR}
+  )
+
+DEAL_II_INITIALIZE_CACHED_VARIABLES()
+PROJECT(step)
+ENABLE_TESTING()
+
+ADD_SUBDIRECTORY(src)
+ADD_SUBDIRECTORY(tests/step)
+ADD_SUBDIRECTORY(tests/support)

+ The library and executable are defined in + src/CMakeLists.txt: +

+# src/CMakeLists.txt
+
+# set up shared library by hand:
+ADD_LIBRARY(support SHARED support.cc)
+DEAL_II_SETUP_TARGET(support)
+
+# set up executable with autopilot macro:
+SET(TARGET "step")
+SET(TARGET_SRC step.cc)
+DEAL_II_INVOKE_AUTOPILOT()
+TARGET_LINK_LIBRARIES(${TARGET} support)

+ + Similarly to the first example, setting up tests for the executable + "step" is just a matter of defining a variable and a call to a macro: +

+# tests/step/CMakeLists.txt
+
+SET(TEST_TARGET step)
+DEAL_II_PICKUP_TESTS()

+ In contrast, the tests for the support library consist of a source file + that has a main function. The object file generated from this source + file will be linked against deal.II and every library listed in + TEST_LIBRARIES: +

+# tests/support/CMakeLists.txt
+
+SET(TEST_LIBRARIES support)
+DEAL_II_PICKUP_TESTS()

+ Again, reconfigure and run ctest: +

+$ cmake .
+$ ctest
+Test project .../examples/step
+    Start 1: step/my_test_1.debug
+1/2 Test #1: step/my_test_1.debug .............   Passed    0.21 sec
+    Start 2: support/my_test_2.debug
+2/2 Test #2: support/my_test_2.debug ..........   Passed    0.22 sec
+
+100% tests passed, 0 tests failed out of 2
+
+Total Test time (real) =   0.43 sec

+ +

+ Remark: For further information consult the testsuite documentation for + the library. With the sole exception of the testsuite setup (that + happens unconditionally in user testsuites), this documentation also + applies for user testsuites. +

+ +

+ Remark: The full configuration options for + DEAL_II_PICKUP_TESTS() are: +

+TEST_LIBRARIES
+TEST_LIBRARIES_DEBUG
+  - additionally used for tests with debug configuration
+TEST_LIBRARIES_RELEASE
+  - additionally used for tests with release configuration
+
+TEST_TARGET or
+TEST_TARGET_DEBUG and TEST_TARGET_RELEASE
+  - used instead of TEST_TARGET for debug/release configuration
+
+NUMDIFF_EXECUTABLE, DIFF_EXECUTABLE
+  - pointing to valid diff executables. If NUMDIFF_EXECUTABLE is not
+    "numdiff" it will be ignored and DIFF_EXECUTABLE is used instead
+
+TEST_TIME_LIMIT
+  - specifying the maximal wall clock time in seconds a test is allowed
+    to run

+ +

+ +

+ + + -- 2.39.5