Nektar++ 4.3.4: all the regression tests fail after a successful compilation

The issue

After successfully compiling Nektar 4.3.4, we proceeded to run the “ctest” command to execute all the regression tests. Unluckily for us, every single one failed:

292/399 Test #292: ADRSolver_Helmholtz3D_CubePeriodic_RotateFace ……………………………….***Failed    0.01 sec
Start 293: ADRSolver_CubeAllElements
293/399 Test #293: ADRSolver_CubeAllElements …………………………………………………***Failed    0.01 sec
Start 294: ADRSolver_ImDiffusion_m12

What was going on behind the scenes

We needed to find out why all the tests failed. It had nothing to do with the LD_LIBRARY_PATH environment variable as discussed here, so I decided to run the ctest utility with the first regression test, adding some debug and verbose flags along the way:

ctest -VV –output-log TEST –output-on-failure -R StdRegions_StdProject1D_Seg_Orth_P6_Q7

This time, I got more information about the error:

5: Test command: /nektar/nektar++-4.3.4/build/tests/Tester “//nektar/nektar++-4.3.4/library/Demos/StdRegions/Tests/StdProject1D_Seg_Orth_P6_Q7.tst”

5: Test timeout computed to be: 9.99988e+06
5: Error occurred running test:
5: Command: StdProject1D-g 1 6 7 1>output.out 2>output.err
5: Files left in /nektar/nektar++-4.3.4/build/library/Demos/StdRegions/tmp_StdProject1D_Seg_Orth_P6_Q7
1 Test #5: StdRegions_StdProject1D_Seg_Orth_P6_Q7 …***Failed 0.01 sec
Error occurred running test:
Command: StdProject1D-g 1 6 7 1>output.out 2>output.err
Files left in /nektar/nektar++-4.3.4/build/library/Demos/StdRegions/tmp_StdProject1D_Seg_Orth_P6_Q7

Have a look at the executable’s name: StdProject1D-g. There was no such binary in our Nektar++ dist directory. We had a StdProject1D executable instead. So the Tester binary (compiled withing the Nektar++ distribution), was looking for binaries with a wrong name. The “-g” literal sounded like “debug” versions of the binaries. That did not make any sense, because we had compiled Nektar++ setting the CMAKE_BUILD_TYPE as  “Release” within the cmake build system. In theory, this setting should have been set the proper flags for the Tester component compilation.

Therefore, I looked inside the Tester.cpp file and I found some comments related to this issue:

// Construct test command to run. If in debug mode, append "-g"
// Output from stdout and stderr are directed to the files output.out
// and output.err, respectively.

Following from Tester.cpp, I ended up reading the file TestData.cpp:

#elif !defined(NDEBUG)
          m_executable += "-g";

So that was it; without the NDEBUG directive, the Tester program would be generated with all the “debug” versions of the binaries, thus the “-g” literal would be appended to their names. And yet, the binaries that were generated by  compiling Nektar++ were non-debug versions! No wonder why Tester could not find them and all the regression tests failed!

Fixing the issue

The flag that needed to be added to the build process for the Tester program was: -DNDEBUG. So I edited the build/tests/CMakeFiles/Tester.dir/flags.make file in order to add it:

CXX_FLAGS = -Wno-deprecated -DNDEBUG -isystem /nektar/nektar++-4.3.4/build/ThirdParty/dist/include -isystem /nektar/nektar++-4.3.4/ThirdParty/dist/include -isystem /usr/lib/x86_64-linux-gnu/../include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -I/usr/include/vtk-5.8 -I/nektar/nektar++-4.3.4/build/ThirdParty/libsmvf1.0/lib -I/nektar/nektar++-4.3.4 -I/nektar/nektar++-4.3.4/library -I/nektar/nektar++-4.3.4/solvers -I/nektar/nektar++-4.3.4/utilities -I/nektar/nektar++-4.3.4/tests

After that, I cleaned up and built the Tester program once again:

cd build/tests
make clean

Having now the right names for the binaries, I could run the ctest utility without issues at all. All the regression tests worked like a charm. Job done.