Documentation: Update testsuite.html to mention multiple comparison files

diff --git a/doc/developers/testsuite.html b/doc/developers/testsuite.html
index ab59777e3fa5042d017362cf8f3da13c8e17aa43..2ef7985f70ac2f5d55e276362f7308b227703895 100644 (file)
--- a/doc/developers/testsuite.html
+++ b/doc/developers/testsuite.html
@@ -53,6 +53,7 @@
           <li><a href="#restrictfeature">Restricting tests to feature configurations</a></li>
           <li><a href="#mpi">Running tests with MPI</a></li>
           <li><a href="#binary">Tests with binary output</a></li>
+          <li><a href="#variants">Tests with multiple comparison files</a></li>
           <li><a href="#expect">Changing condition for success</a></li>
           <li><a href="#layoutaddtests">Adding new tests</a></li>
           <li><a href="#layoutaddcategory">Adding new categories</a></li>
@@ -532,6 +533,7 @@ category/test.mpirun=N.output
       files for different numbers of MPI processes.
     </p>
 
+
     <a name="binary"></a>
     <h3>Tests with binary output</h3>
     <p>
@@ -546,6 +548,43 @@ category/test.binary.output
       files.
     </p>
 
+
+    <a name="variants"></a>
+    <h3>Tests with multiple comparison files</h3>
+    <p>
+      Sometimes it is necessary to provide multiple comparison files for a
+      single test. Additional comparison files have the same path as the
+      main comparison file (in this case <code>test.output</code>) followed
+      by a dot and a variant description:
+<pre>
+category/test.output
+category/test.output.2
+category/test.output.3
+category/test.output.4
+</pre>
+      The testsuite will try to match the output against all variants in
+      alphabetical order starting with the main output file.
+    </p>
+
+    <p>
+      <b>Warning:</b> This mechanism is only meant as a last resort for
+      tests where <i>all</i> no alternative approach is viable. Especially,
+      consider first to
+      <ol>
+        <li> make the test more robust such that differences can be
+          expressed in round-off errors detectable by numdiff.
+        <li> restrict comparison files to specific feature versions
+          <a href="#restrictfeature">versions of an external feature</a>.
+      </ol>
+    </p>
+
+    <p>
+      <b>Note:</b> The main comparison file (i.e., the one ending in
+      <code>output</code> is mandatory. Otherwise, no test will be
+      configured.
+    </p>
+
+
     <a name="expect"></a>
     <h3>Changing condition for success</h3>
     <p>
@@ -561,7 +600,6 @@ category/test.expect=run.output
     </p>
 
 
-
     <a name="layoutaddtests"></a>
     <h3>Adding new tests</h3>