[CTest] Using rerun-failed when tests pass the first time
My team is using CMake as our build system, and we're also using CTest for our unit tests. We wanted to speed up running tests by only running the tests that failed previously or when the binary file had changed. While trying to use '--rerun-failed', we noticed that all our tests were being run on subsequent ctest invocations, even though they had all passed on the previous step. After looking into it, we saw that the file 'LastTestsFailed.log' was not being created. We tried making a simple empty one, but that didn't change anything. We then changed one of our tests to fail, then tried running 'ctest' with '--rerun-failed' again, and on the subsequent run it only re-ran that failed test. We reverted that test back to normal, and now '--rerun-failed' only runs that test, which makes sense based on the documentation.
Is this by design? We expected that if all our tests never failed on the first invocation, that subsequent runs would not cause those tests to re-run unless the binary had changed. The behaviour also seems inconsistent, since if you add an entry into an empty 'LastTestsFailed.log' for a non-existant test, then subsequent test runs will not run any of your tests. I've given steps below on how to reproduce the issue with the attached demo.
Version: ctest version 3.7.2
Steps to Reproduce: 1. Copy over the sources. 2. Make a build directory, change to that build directory. 3. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; all tests should run. 4. Run 'ctest --rerun-failed' again; all tests should run. 5. Open up the 'CMakeLists.txt' you copied in the earlier step, uncomment the division test. 6. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; all tests should run. 7. Run 'ctest --rerun-failed' again; only the 'division' test should run. 8. Open up the 'CMakeLists.txt' you copied in the earlier step, comment the division test. 9. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; should output "No tests were found!!!"
Expected Behaviour: 1. If no 'LastTestsFailed.log' exists, should create one; if no tests failed, then this file should be empty. 2. On subsequent runs, if a 'LastTestsFailed.log' exists, but is empty, then no tests should be run.
Attached is the example code (example.zip) MD5 (example.zip) = 786c191c767f3865b1fc307efe022076