Defining dependent compile and run tests using CMake/CTest

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Defining dependent compile and run tests using CMake/CTest

Neil Carlson
I have a test code (think compiler regression test) that want to compile as a test, and if it compiles and links successfully, to go on to run as a subsequent test.  I found a thread from Sept that pointed to a method used in boost-cmake for the first compile test.  It moves execution of the compile/link from the make step to the ctest step, and works great.  I'm now using the FIXTURES_REQUIRED and FIXTURES_SETUP test properties to make the run test depend on the successful completion of the compile test, but this doesn't seem to be working properly.  It does order the tests so that the compile test is done first, However if it fails, ctest spews a long message for the run test detailing all the places it looked for the executable (not created because the compile/link test failed) but could not find.  It's as if ctest tries to set up the test before checking to see whether it should be run at all.  Am I doing something incorrect here?

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Defining dependent compile and run tests using CMake/CTest

Craig Scott-3
Please post your CMakeLists.txt so we can see how you've got the fixtures and tests defined.

On Mon, Nov 27, 2017 at 2:49 AM, Neil Carlson <[hidden email]> wrote:
I have a test code (think compiler regression test) that want to compile as a test, and if it compiles and links successfully, to go on to run as a subsequent test.  I found a thread from Sept that pointed to a method used in boost-cmake for the first compile test.  It moves execution of the compile/link from the make step to the ctest step, and works great.  I'm now using the FIXTURES_REQUIRED and FIXTURES_SETUP test properties to make the run test depend on the successful completion of the compile test, but this doesn't seem to be working properly.  It does order the tests so that the compile test is done first, However if it fails, ctest spews a long message for the run test detailing all the places it looked for the executable (not created because the compile/link test failed) but could not find.  It's as if ctest tries to set up the test before checking to see whether it should be run at all.  Am I doing something incorrect here?

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake



--
Craig Scott
Melbourne, Australia

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Defining dependent compile and run tests using CMake/CTest

Neil Carlson

On Sun, Nov 26, 2017 at 12:42 PM, Craig Scott <[hidden email]> wrote:
Please post your CMakeLists.txt so we can see how you've got the fixtures and tests defined.

Here's what I've got (I put the run test first to verify that the reordering takes place):

add_test(NAME intel-20171126-run COMMAND intel-20171126)
set_tests_properties(intel-20171126-run PROPERTIES PASS_REGULAR_EXPRESSION "pass")
set_tests_properties(intel-20171126-run PROPERTIES FIXTURES_REQUIRED foo)

add_executable(intel-20171126 EXCLUDE_FROM_ALL intel-20171126.f90)
add_test(NAME intel-20171126
    COMMAND ${CMAKE_COMMAND} --build . --target intel-20171126 --config $<CONFIGURATION>
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
set_tests_properties(intel-20171126 PROPERTIES FIXTURES_SETUP foo)

After posting my question, I manually created the "executable" ('touch intel-20171126`)
and re-ran ctest.  Sure enough, that time I didn't get the long message spew about
being able to find the executable, but instead a 1-line message, which is what I had
expected to see originally

      Start  2: intel-20171126
 1/13 Test  #2: intel-20171126 ...................***Failed    0.15 sec
      Start  1: intel-20171126-run
Failed test dependencies: intel-20171126
 2/13 Test  #1: intel-20171126-run ...............***Not Run   0.00 sec


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Defining dependent compile and run tests using CMake/CTest

Craig Scott-3


On Mon, Nov 27, 2017 at 10:06 AM, Neil Carlson <[hidden email]> wrote:

On Sun, Nov 26, 2017 at 12:42 PM, Craig Scott <[hidden email]> wrote:
Please post your CMakeLists.txt so we can see how you've got the fixtures and tests defined.

Here's what I've got (I put the run test first to verify that the reordering takes place):

add_test(NAME intel-20171126-run COMMAND intel-20171126)
set_tests_properties(intel-20171126-run PROPERTIES PASS_REGULAR_EXPRESSION "pass")
set_tests_properties(intel-20171126-run PROPERTIES FIXTURES_REQUIRED foo)

add_executable(intel-20171126 EXCLUDE_FROM_ALL intel-20171126.f90)
add_test(NAME intel-20171126
    COMMAND ${CMAKE_COMMAND} --build . --target intel-20171126 --config $<CONFIGURATION>
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
set_tests_properties(intel-20171126 PROPERTIES FIXTURES_SETUP foo)

After posting my question, I manually created the "executable" ('touch intel-20171126`)
and re-ran ctest.  Sure enough, that time I didn't get the long message spew about
being able to find the executable, but instead a 1-line message, which is what I had
expected to see originally

      Start  2: intel-20171126
 1/13 Test  #2: intel-20171126 ...................***Failed    0.15 sec
      Start  1: intel-20171126-run
Failed test dependencies: intel-20171126
 2/13 Test  #1: intel-20171126-run ...............***Not Run   0.00 sec


Thanks, that makes the situation clear. I've traced through the CTest source and I think the logic can be improved. I've created an issue for this in the bug tracker here:


I'll look into a fix, but for now that bug report includes a workaround you can use in the meantime.

On a side note, triggering parts of the build when running tests may seem like a convenient solution, but be aware it has some less obvious gotchas. If you are doing this with more than one test, multiple tests might try and build simultaneously. If the things being built share any common dependencies, the two builds may interfere with each other and overwrite each other's files, etc. In the case of parallel builds, multiple build invocations may result in excessive load due to too many tasks running at once. This can cause out of memory issues if compiling or linking particularly big files. If you want to be sure these things can't happen, you may want to force such tests to run serially to guarantee that they are the only build job:

set_tests_properties(intel-20171126 PROPERTIES RUN_SERIAL YES)

If you are confident that you can guarantee that you know all the test cases that must not run simultaneously, you can use the RESOURCE_LOCK test property instead to only make those specific tests run not at the same time as each other. This will allow other harmless tests to run concurrently with a build test case, leading to more efficient test execution when parallel testing is enabled.

--
Craig Scott
Melbourne, Australia

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Defining dependent compile and run tests using CMake/CTest

Craig Scott-3
Merge request 1526 should fix this behavior. Would be great if you could try it out to confirm that it addresses your particular situation.


On Mon, Nov 27, 2017 at 9:19 PM, Craig Scott <[hidden email]> wrote:


On Mon, Nov 27, 2017 at 10:06 AM, Neil Carlson <[hidden email]> wrote:

On Sun, Nov 26, 2017 at 12:42 PM, Craig Scott <[hidden email]> wrote:
Please post your CMakeLists.txt so we can see how you've got the fixtures and tests defined.

Here's what I've got (I put the run test first to verify that the reordering takes place):

add_test(NAME intel-20171126-run COMMAND intel-20171126)
set_tests_properties(intel-20171126-run PROPERTIES PASS_REGULAR_EXPRESSION "pass")
set_tests_properties(intel-20171126-run PROPERTIES FIXTURES_REQUIRED foo)

add_executable(intel-20171126 EXCLUDE_FROM_ALL intel-20171126.f90)
add_test(NAME intel-20171126
    COMMAND ${CMAKE_COMMAND} --build . --target intel-20171126 --config $<CONFIGURATION>
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
set_tests_properties(intel-20171126 PROPERTIES FIXTURES_SETUP foo)

After posting my question, I manually created the "executable" ('touch intel-20171126`)
and re-ran ctest.  Sure enough, that time I didn't get the long message spew about
being able to find the executable, but instead a 1-line message, which is what I had
expected to see originally

      Start  2: intel-20171126
 1/13 Test  #2: intel-20171126 ...................***Failed    0.15 sec
      Start  1: intel-20171126-run
Failed test dependencies: intel-20171126
 2/13 Test  #1: intel-20171126-run ...............***Not Run   0.00 sec


Thanks, that makes the situation clear. I've traced through the CTest source and I think the logic can be improved. I've created an issue for this in the bug tracker here:


I'll look into a fix, but for now that bug report includes a workaround you can use in the meantime.

On a side note, triggering parts of the build when running tests may seem like a convenient solution, but be aware it has some less obvious gotchas. If you are doing this with more than one test, multiple tests might try and build simultaneously. If the things being built share any common dependencies, the two builds may interfere with each other and overwrite each other's files, etc. In the case of parallel builds, multiple build invocations may result in excessive load due to too many tasks running at once. This can cause out of memory issues if compiling or linking particularly big files. If you want to be sure these things can't happen, you may want to force such tests to run serially to guarantee that they are the only build job:

set_tests_properties(intel-20171126 PROPERTIES RUN_SERIAL YES)

If you are confident that you can guarantee that you know all the test cases that must not run simultaneously, you can use the RESOURCE_LOCK test property instead to only make those specific tests run not at the same time as each other. This will allow other harmless tests to run concurrently with a build test case, leading to more efficient test execution when parallel testing is enabled.

--
Craig Scott
Melbourne, Australia



--
Craig Scott
Melbourne, Australia

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake