Make Timeout not be a failure

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Make Timeout not be a failure

CMake mailing list
I have a test where I start a program that I know will create some
output that I can test.

But I have no way to kill this program, so I let Ctest kill it with a
TIMEOUT.

The next step greps the output to see if it worked.

So it would be nice if  PASS_REGULAR_EXPRESSION  could override TIMEOUT
and not mark the test as failed.

For example:

     set_tests_properties(${importMaps} PROPERTIES
PASS_REGULAR_EXPRESSION "some string I know I will find")
     #                                                         this
^^^^^^^^^^^^^^^^^^^^^^ makes the test pass even if it times out
     set_tests_properties(${importMaps} PROPERTIES TIMEOUT 60)

--
Donald [|]
A bad day in [] is better than a good day in {}.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Make Timeout not be a failure

Kyle Edwards
On Thu, 2019-06-20 at 15:45 -0400, Donald MacQueen [|] via CMake wrote:

> I have a test where I start a program that I know will create some 
> output that I can test.
>
> But I have no way to kill this program, so I let Ctest kill it with
> a 
> TIMEOUT.
>
> The next step greps the output to see if it worked.
>
> So it would be nice if  PASS_REGULAR_EXPRESSION  could override
> TIMEOUT 
> and not mark the test as failed.
>
> For example:
>
>      set_tests_properties(${importMaps} PROPERTIES 
> PASS_REGULAR_EXPRESSION "some string I know I will find")
>      #                                                         this 
> ^^^^^^^^^^^^^^^^^^^^^^ makes the test pass even if it times out
>      set_tests_properties(${importMaps} PROPERTIES TIMEOUT 60)

You could wrap your test in a CMake script that calls execute_process()
with a TIMEOUT argument, and then greps the output of the command for
the desired expression. For example:

CMakeLists.txt:

add_test(NAME mytest COMMAND ${CMAKE_COMMAND} -DMYEXE=${PATH_TO_MYEXE}
-P ${CMAKE_CURRENT_LIST_DIR}/ExecuteTest.cmake)

ExecuteTest.cmake:

execute_process(COMMAND ${PATH_TO_MYEXE} TIMEOUT 60 OUTPUT_VARIABLE
output)
if(NOT output MATCHES "^my_desired_regex$")
  message(FATAL_ERROR "myexe did not produce desired output")
endif()

Hope that helps

Kyle
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Make Timeout not be a failure

CMake mailing list
HI Kyle,

Where are the -D and the -P switches of add_test documented?

Thanks for the suggestion.

On 6/20/2019 3:52 PM, Kyle Edwards wrote:

> On Thu, 2019-06-20 at 15:45 -0400, Donald MacQueen [|] via CMake wrote:
>> I have a test where I start a program that I know will create some
>> output that I can test.
>>
>> But I have no way to kill this program, so I let Ctest kill it with
>> a
>> TIMEOUT.
>>
>> The next step greps the output to see if it worked.
>>
>> So it would be nice if  PASS_REGULAR_EXPRESSION  could override
>> TIMEOUT
>> and not mark the test as failed.
>>
>> For example:
>>
>>       set_tests_properties(${importMaps} PROPERTIES
>> PASS_REGULAR_EXPRESSION "some string I know I will find")
>>       #                                                         this
>> ^^^^^^^^^^^^^^^^^^^^^^ makes the test pass even if it times out
>>       set_tests_properties(${importMaps} PROPERTIES TIMEOUT 60)
> You could wrap your test in a CMake script that calls execute_process()
> with a TIMEOUT argument, and then greps the output of the command for
> the desired expression. For example:
>
> CMakeLists.txt:
>
> add_test(NAME mytest COMMAND ${CMAKE_COMMAND} -DMYEXE=${PATH_TO_MYEXE}
> -P ${CMAKE_CURRENT_LIST_DIR}/ExecuteTest.cmake)
>
> ExecuteTest.cmake:
>
> execute_process(COMMAND ${PATH_TO_MYEXE} TIMEOUT 60 OUTPUT_VARIABLE
> output)
> if(NOT output MATCHES "^my_desired_regex$")
>    message(FATAL_ERROR "myexe did not produce desired output")
> endif()
>
> Hope that helps
>
> Kyle

--
Donald [|]
A bad day in [] is better than a good day in {}.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Make Timeout not be a failure

Kyle Edwards
On Thu, 2019-06-20 at 16:47 -0400, Donald MacQueen [|] wrote:
> HI Kyle,
>
> Where are the -D and the -P switches of add_test documented?

Those are not part of add_test(), they are arguments that get passed to
the CMake executable. See for details:

https://cmake.org/cmake/help/v3.15/manual/cmake.1.html

Kyle
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake