[CTest] Using rerun-failed when tests pass the first time

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[CTest] Using rerun-failed when tests pass the first time

CMake mailing list
Hey all,

My team is using CMake as our build system, and we're also using CTest for our unit tests.  We wanted to speed up running tests by only running the tests that failed previously or when the binary file had changed.  While trying to use '--rerun-failed', we noticed that all our tests were being run on subsequent ctest invocations, even though they had all passed on the previous step.  After looking into it, we saw that the file 'LastTestsFailed.log' was not being created.  We tried making a simple empty one, but that didn't change anything.  We then changed one of our tests to fail, then tried running 'ctest' with '--rerun-failed' again, and on the subsequent run it only re-ran that failed test.  We reverted that test back to normal, and now '--rerun-failed' only runs that test, which makes sense based on the documentation.

Is this by design?  We expected that if all our tests never failed on the first invocation, that subsequent runs would not cause those tests to re-run unless the binary had changed.  The behaviour also seems inconsistent, since if you add an entry into an empty 'LastTestsFailed.log' for a non-existant test, then subsequent test runs will not run any of your tests.  I've given steps below on how to reproduce the issue with the attached demo.

Version: ctest version 3.7.2

Steps to Reproduce:
 1. Copy over the sources.
 2. Make a build directory, change to that build directory.
 3. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; all tests should run.
 4. Run 'ctest --rerun-failed' again; all tests should run.
 5. Open up the 'CMakeLists.txt' you copied in the earlier step, uncomment the division test.
 6. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; all tests should run.
 7. Run 'ctest --rerun-failed' again; only the 'division' test should run.
 8. Open up the 'CMakeLists.txt' you copied in the earlier step, comment the division test.
 9. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; should output "No tests were found!!!"

Expected Behaviour:
 1. If no 'LastTestsFailed.log' exists, should create one; if no tests failed, then this file should be empty.
 2. On subsequent runs, if a 'LastTestsFailed.log' exists, but is empty, then no tests should be run.

Attached is the example code (example.zip)
MD5 (example.zip) = 786c191c767f3865b1fc307efe022076


Cheers,
Ford


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

example.zip (2K) Download Attachment