ctest // fixtures and --repeat-until-fail <n>

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ctest // fixtures and --repeat-until-fail <n>

Sergei Nikulov
Hello All,

Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with
--repeat-until-fail <n> option?

My expectation, if I requesting a test run for example 10 times, the
sequence should be as follows:

fixture_setup, then test, then fixture_cleanup repeats 10 times

But in practice, I've got following sequence

1. fixture_setup runs 10 times
2. test runs 10 times
3. fixture_cleanup runs 10 times

Is it expected behavior?
I'm using ctest version 3.14.2

Thank you in advance.

--
Best Regards,
Sergei Nikulov
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ctest // fixtures and --repeat-until-fail <n>

Craig Scott-3


On Tue, Apr 16, 2019 at 10:07 PM Sergei Nikulov <[hidden email]> wrote:
Hello All,

Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with
--repeat-until-fail <n> option?

My expectation, if I requesting a test run for example 10 times, the
sequence should be as follows:

fixture_setup, then test, then fixture_cleanup repeats 10 times

But in practice, I've got following sequence

1. fixture_setup runs 10 times
2. test runs 10 times
3. fixture_cleanup runs 10 times

Is it expected behavior?
I'm using ctest version 3.14.2

The documentation for the --repeat-until-fail option says "Require each test to run <n> times without failing in order to pass". A fixture setup or cleanup test is still a test, so they will also be run <n> times. The logic that implements repeating tests sets up a counter on each test and it runs that test <n> times before marking that test as complete. This is why you see fixture_setup run 10 times, then test runs 10 times and lastly fixture_cleanup runs 10 times.

Whenever I've wanted to use --repeat-until-fail (admittedly not all that common for me), I'm generally only interested in repeatedly running one particular test. For that case, I would typically run the test once with fixture cleanups disabled (i.e. use a command line "ctest -FC . -R myFlakeyTest"). I can then run ctest again for just my flaky test with all fixtures disabled (e.g. "ctest -FA . -R myFlakeyTest --repeat-until-fail 10"). This strategy won't work for everyone's situation, but it has been effective for me.
 
--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ctest // fixtures and --repeat-until-fail <n>

Sergei Nikulov
>
> On Tue, Apr 16, 2019 at 10:07 PM Sergei Nikulov <[hidden email]> wrote:
>>
>> Hello All,
>>
>> Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with
>> --repeat-until-fail <n> option?
>>
>> My expectation, if I requesting a test run for example 10 times, the
>> sequence should be as follows:
>>
>> fixture_setup, then test, then fixture_cleanup repeats 10 times
>>
>> But in practice, I've got following sequence
>>
>> 1. fixture_setup runs 10 times
>> 2. test runs 10 times
>> 3. fixture_cleanup runs 10 times
>>
>> Is it expected behavior?
>> I'm using ctest version 3.14.2
>
>
> The documentation for the --repeat-until-fail option says "Require each test to run <n> times without failing in order to pass". A fixture setup or cleanup test is still a test, so they will also be run <n> times. The logic that implements repeating tests sets up a counter on each test and it runs that test <n> times before marking that test as complete. This is why you see fixture_setup run 10 times, then test runs 10 times and lastly fixture_cleanup runs 10 times.
>

I understand that fixture is still a test.
To achieve repeatable behavior -R "test_continues_*" is enough.

But if it calls FIXTURE why not apply the specific behavior as
expected from fixture (init/teardown)?

--
Best Regards,
Sergei Nikulov
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ctest // fixtures and --repeat-until-fail <n>

Craig Scott-3


On Tue, Apr 16, 2019 at 11:36 PM Sergei Nikulov <[hidden email]> wrote:
>
> On Tue, Apr 16, 2019 at 10:07 PM Sergei Nikulov <[hidden email]> wrote:
>>
>> Hello All,
>>
>> Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with
>> --repeat-until-fail <n> option?
>>
>> My expectation, if I requesting a test run for example 10 times, the
>> sequence should be as follows:
>>
>> fixture_setup, then test, then fixture_cleanup repeats 10 times
>>
>> But in practice, I've got following sequence
>>
>> 1. fixture_setup runs 10 times
>> 2. test runs 10 times
>> 3. fixture_cleanup runs 10 times
>>
>> Is it expected behavior?
>> I'm using ctest version 3.14.2
>
>
> The documentation for the --repeat-until-fail option says "Require each test to run <n> times without failing in order to pass". A fixture setup or cleanup test is still a test, so they will also be run <n> times. The logic that implements repeating tests sets up a counter on each test and it runs that test <n> times before marking that test as complete. This is why you see fixture_setup run 10 times, then test runs 10 times and lastly fixture_cleanup runs 10 times.
>

I understand that fixture is still a test.
To achieve repeatable behavior -R "test_continues_*" is enough.

But if it calls FIXTURE why not apply the specific behavior as
expected from fixture (init/teardown)?

It's mostly related to the way it is implemented internally. Because a fixture setup/cleanup is still just another test, it has all the features of a regular test, including honouring the option that tells ctest to run it multiple times. For some projects, it may be desirable for the fixture setup/cleanup to run only once but re-run the fixture-requiring test(s) to run multiple times. For other projects, it might be desirable for the fixture setup/cleanup tests to also run multiple times (a setup test might simply be polling a service to confirm it is up, for example). Since ctest cannot know which of the two scenarios the project wants, it simply treats all tests the same, whether they are fixture setup/cleanup tests or not.

Perhaps one way of dealing with this might be to add a new boolean test property which, when set, has the meaning of "only run once" when the --repeat-until-fail option is used.

--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake