From 21fb23b80e8ec7a08e3bc80bc72719b394135a7e Mon Sep 17 00:00:00 2001
From: Wolfgang Bangerth
- make hierarchical.exe
+ make hierarchical/exe
- to compile and link the executable. (Note that executables in the
- tests directories have the extension .exe
even on
- unix systems, as this made writing the makefiles much simpler.) If
+ to compile and link the executable. (For each test there is a not
+ only a file with suffic .cc
but also a subdirectory with the
+ same name, in which we store among other things the executable for that
+ test, under the name exe
.) If this fails, i.e. if
you can't compile or link, then you probably already know where
the problem is, and how to fix it. If you could compile and link
the test, you will want to make sure that it executes correctly
and produces an output file:
- make hierarchical.output
+ make hierarchical/output
- If this produces errors or triggers assertions, then you will want
- to use a debugger on the executable to figure out what happens. On
+ (As you see, the output file is also stored in the subdirectory with the
+ test's name.) If this produces errors or triggers assertions, then you will
+ want to use a debugger on the executable to figure out what happens. On
the other hand, if you are sure that this also worked, you will
want to compare the output with the stored output from CVS:
- make hierarchical.OK
+ make hierarchical/OK
If the output isn't equal, then you'll get to see something like
this:
- =====Checking====== hierarchical.output
- +++++Error+++++++++ hierarchical.OK. Use make verbose=on for the diffs
+ =====Checking====== hierarchical/output
+ +++++Error+++++++++ hierarchical/OK. Use make verbose=on for the diffs
Because the diffs between the output we get and the output we
expected can sometimes be very large, you don't get to see it by
default. However, following the suggestion printed, if you type
- make hierarchical.OK verbose=on
+ make hierarchical/OK verbose=on
you get to see it all:
- =====Checking====== hierarchical.output
+ =====Checking====== hierarchical/output
12c12
< DEAL::0.333 1.667 0.333 -0.889 0.296 -0.988 0.329 -0.999 0.333 -1.000 0.333 -1.000
---
> DEAL::0.333 0.667 0.333 -0.889 0.296 -0.988 0.329 -0.999 0.333 -1.000 0.333 -1.000
- +++++Error+++++++++ hierarchical.OK
+ +++++Error+++++++++ hierarchical/OK
In this case, the second number on line 12 is off by one. To find
the reason for this, you again should use a debugger or other
@@ -349,7 +351,7 @@
int main ()
{
- std::ofstream logfile("my_new_test.output");
+ std::ofstream logfile("my_new_test/output");
deallog.attach(logfile);
deallog.depth_console(0);
@@ -360,8 +362,9 @@ int main ()
return 0;
}
- The basic idea is that you open an output file with the same base
- name as your test, and then write all output you generate to it,
+ The basic idea is that you open an output file in a directory with the same
+ name as your test (don't forget to create this directory), and then write
+ all output you generate to it,
through the deallog
stream (which works just like any
other std::ostream
except that it does a few more
things behind the scenes that are helpful in this context). In
@@ -386,7 +389,7 @@ int main ()
library, often only a single function. These tests go into the
bits/
directory and often have names that are
composed of the name of the class being tested and a two-digit
- number.
+ number (for example, dof_tools_11
).
If you have done this, you can try to run
- make my_new_test.output + make my_new_test/outputwhich should compile, link, and run your test. Running your test should generate the desired output file. @@ -421,7 +424,8 @@ int main ()
If you run your new test executable, you will get an output file
- that should be used to compare all future runs with. If the test
+ mytestname/output
that should be used to compare all future
+ runs with. If the test
is relatively simple, it is often a good idea to look at the
output and make sure that the output is actually what you had
expected. However, if you do complex operations, this may
@@ -432,42 +436,84 @@ int main ()
The next step is to copy this output file to the place where the
- scripts can find it when they compare with newer runs. For this,
- there are directories
- tests/results/i686-pc-linux-gnu+gcc2.95
,
- tests/results/i686-pc-linux-gnu+icc7.1
,
- tests/results/mips-sgi-irix6.5+MIPSpro7.4
, etc. that
- encode on which platform and with which compiler the output was
- generated. These different directories are necessary since
- floating point computations are often not exactly reproducible
- quantitatively if you use different CPUs or compilers, even though
- they may be qualitatively equivalent. We may therefore have to
- store multiple output files for the same test.
+ scripts can find it when they compare with newer runs. For this, you first
+ have to understand how correct results are verified. It works in the
+ following way: for each test, we have subdirectories
+ testname/cmp
where we store the expected results in a file
+ testname/cmp/generic
. If you create a new test, you should
+ therefore create this directory, and copy the output of your program,
+ testname/output
to testname/cmp/generic
.
+
+ So why generic
? The reason is that sometimes test results
+ differ slightly from platform to platform, for example because numerical
+ roundoff is different due to different floating point implementations on
+ different CPUs. What this means is that sometimes a single stored output is
+ not enough to verify that a test functioned properly: if you happen to be
+ on a platform different from the one on which the generic output was
+ created, your test will always fail even though it produces almost exactly
+ the same output.
+
+ To avoid this, what the makefiles do is to first check whether an output
+ file is stored for this test and your particular configuration (platform
+ and compiler). If this isn't the case, it goes through a hierarchy of files
+ with related configurations, and only if none of them does it take the
+ generic output file. It then compares the output of your test run with the
+ first file it found in this process. To make things a bit clearer, if you
+ are, for example, on a i686-pc-linux-gnu
box and use
+ gcc4.0
as your compiler, then the following files will be
+ searched for (in this order):
+
+testname/cmp/i686-pc-linux-gnu+gcc4.0 +testname/cmp/i686-pc-linux-gnu+gcc3.4 +testname/cmp/i686-pc-linux-gnu+gcc3.3 +testname/cmp/generic ++ (This list is generated by the
tests/hierarchy.pl
script.)
+ Your output will then be compared with the first one that is actually
+ found. The virtue of this is that we don't have to store the output files
+ from all possible platforms (this would amount to gigabytes of data), but
+ that we only have store an output file for gcc4.0 if it differs from that
+ of gcc3.4, and for gcc3.4 if it differs from gcc3.3. If all of them are the
+ same, we would only have the generic output file.
Most of the time, you will be able to generate output files only
for your own platform and compiler, and that's alright: someone
else will create the output files for other platforms
- eventually. You only have to put your file into the correct
- directory, which is actually easy to find: there is a link
- tests/compare
that points to the directory that will
- be used to compare with. If you have put your test
- my_new_test.cc
into tests/bits/
, for
- example, then you chould copy my_new_test.output
into
- tests/compare/bits
.
+ eventually. You therefore only have to copy your output file to
+ testname/cmp/generic
.
At this point you can run
- make my_new_test.OK + make my_new_test/OKwhich should compare the present output with what you have just copied into the compare directory. This should, of course, succeed, since the two files should be identical. +
+ On the other hand, if you realize that an existing test fails on your
+ system, but that the differences (as shown when running with
+ verbose=on
, see above) are only marginal and around the 6th or
+ 8th digit, then you should check in your output file for the platform you
+ work on. For this, you could copy testname/output
to
+ testname/cmp/myplatform+compiler
, but your life can be easier
+ if you simply type
+
+ make my_new_test/ref ++ which takes your output and copies it to the right place automatically. + + +
- cvs add bits/my_new_test.cc compare/bits/my_new_test.output - cvs commit -m "New test" bits/my_new_test.cc \ - compare/bits/my_new_test.output bits/Makefile + cvs add bits/my_new_test.cc + cvs add bits/my_new_test + cvs add bits/my_new_test/cmp + cvs add bits/my_new_test/cmp/generic + cvs commit -m "New test" bits/my_new_test* bits/MakefileIf you don't have CVS write access, talk to us on the mailing - list; writing testcase is a worthy and laudable task, and we would + list; writing testcases is a worthy and laudable task, and we would like to encourage it by giving people the opportunity to contribute! - -
- If you are working on a system or with a compiler for which test
- output files haven't been generated yet, things a slightly more
- complicated because you have to set up a new directory in
- tests/results
so that tests/compare
can
- point to it. There are several ways to do that.
-
- First, there are combinations of system and compiler for which we
- get exactly the same output as for another combination. For
- example, on an x86 linux, gcc 3.3 produces the same output as gcc
- 3.2. There is no need to have two directories under
- tests/results
that contain the many megabytes of
- output files twice. If your system is of this type, then your
- simplest way is to edit tests/results/Makefile
: at
- the bottom of this file is a target .links
that
- allows to create symbolic links from one (existing) directory to a
- new one. For example, you will find there
-
- linkdirs-i686-pc-linux-gnu+gcc3.3-to-i686-pc-linux-gnu+gcc3.2 -- which creates a directory
i686-pc-linux-gnu+gcc3.3
- that really is only a link to
- i686-pc-linux-gnu+gcc3.2
.
-
-
-
- The second, and most frequent possibility is that your combination
- of system and compiler yields output files that are almost always
- equal to another one, and only a few tests yield different
- output. In this case, you would generate all output files (using
- just make
in tests/
will generate them),
- then create a new directory in results/
for your
- combination, and populate it with the output files you
- generated. Then pick the existing directory for which the test
- results are closest to yours, and in your own copy delete all the
- output files that are identical to the ones in the other
- directory. Finally, add a target to the Makefile of the form
-
- linkfiles-mips-sgi-irix6.5+MIPSpro7.4-to-i686-pc-linux-gnu+gcc3.2 -- What happens in this case is that when you call the makefile, it - goes through the
mips-sgi-irix6.5+MIPSpro7.4
(for
- output files generated on SGI/Mips systems with the SGI MIPSpro
- compiler) and for each test for which there is no output file it
- creates a link to the corresponding output file in
- i686-pc-linux-gnu+gcc3.2
. In this particular case,
- only 51 output files are presently stored, whereas the other
- roughly 400 are identical to the ones generated by gcc 3.2 on
- linux.
-
-
- - The third possibility is that you entirely populate your directory - with your output file. However, this is inefficient. In order to - store all output files, it presently takes 28MB; however, - this should be unnecessary since most compilers and platforms - generate identical output for almost all tests. Thus, populating - CVS with large and unnecessary files is not a good idea. It is - also an unnecessary burden when tests are added: if entire - directories or single output files are linked as shown above, then - a new output file has to be added only once to be used by a larger - number of platform/compiler combinations, but it has to be added - for every fully populated directory. We therefore discourage this - option. -
- The deal.II mailing list -- 2.39.5