test depending on code compilation

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

test depending on code compilation

Franck Houssen
I need to design a test (= a bash script) such that :
1) run a dedicated executable for the test (to be compiled)
2) diff the run output with a reference log file

The test is created with : add_test(mytest ./mytest.sh). The bash script would look like:
>> more mytest.sh
/path/to/mytestexe > out.log
diff out.log out.ref

The dedicated executable for the test is created with : add_executable(mytestexe mytestexe.cpp)

Ideally, I would like mytestexe to be compiled when I type "make test" (before to run the test) but not "make".

Seems impossible to add a dependencie with : add_dependencies(mytest mytestexe). I googled this and found https://stackoverflow.com/questions/733475/cmake-ctest-make-test-doesnt-build-tests but this was not really clear to me...I added : add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} DEPENDS mytestexe).

The 2 unexpected problems I have left are:
1) mytestexe is compiled everytime I type "make" which is a solution but is not really what I am looking for (also compiled when I type "make check" which is expected).
    => is there a way for make not to compile mytestexe (note I didn't added ALL in add_custom_target so I am not sure to know why make builds mytestexe)
2) Also the verbosity is broken with check : make test ARGS="-V" is verbose , but, make check ARGS="-V" is not.
    => how to deal with that ?

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: test depending on code compilation

Franck Houssen
I tried to reduce the example:

>> ls
CMakeLists.txt  main.cpp  mytest.cpp  mytest.sh

>> more *
::::::::::::::
CMakeLists.txt
::::::::::::::
cmake_minimum_required(VERSION 3.7)
enable_language(CXX)

project(main)
add_executable(main main.cpp)
add_executable(mytestcpp mytest.cpp)

include(CTest)
enable_testing()
configure_file("${CMAKE_CURRENT_SOURCE_DIR}/mytest.sh" "${CMAKE_CURRENT_BINARY_DIR}/mytest.sh")
add_test(NAME mytestsh COMMAND ./mytest.sh WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}")

add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} DEPENDS mytestcpp)
::::::::::::::
main.cpp
::::::::::::::
int main() {return 0;}
::::::::::::::
mytest.cpp
::::::::::::::
int main() {return 0;}
::::::::::::::
mytest.sh
::::::::::::::
#!/bin/bash
exit 0

I want make to build only main => KO
>> make
[ 50%] Built target main
[100%] Built target mytestcpp

I want make check to build mytestcpp and run tests => OK but verbose is KO.
>> make check ARGS="-V"
[100%] Built target mytestcpp
Test project /tmp/BUILD
    Start 1: mytestsh
1/1 Test #1: mytestsh .........................   Passed    0.00 sec

I want make test to run test only => OK and verbose is OK.


De: "Franck Houssen" <[hidden email]>
À: "CMake Mail List" <[hidden email]>
Envoyé: Mercredi 10 Janvier 2018 10:47:34
Objet: [CMake] test depending on code compilation

I need to design a test (= a bash script) such that :
1) run a dedicated executable for the test (to be compiled)
2) diff the run output with a reference log file

The test is created with : add_test(mytest ./mytest.sh). The bash script would look like:
>> more mytest.sh
/path/to/mytestexe > out.log
diff out.log out.ref

The dedicated executable for the test is created with : add_executable(mytestexe mytestexe.cpp)

Ideally, I would like mytestexe to be compiled when I type "make test" (before to run the test) but not "make".

Seems impossible to add a dependencie with : add_dependencies(mytest mytestexe). I googled this and found https://stackoverflow.com/questions/733475/cmake-ctest-make-test-doesnt-build-tests but this was not really clear to me...I added : add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} DEPENDS mytestexe).

The 2 unexpected problems I have left are:
1) mytestexe is compiled everytime I type "make" which is a solution but is not really what I am looking for (also compiled when I type "make check" which is expected).
    => is there a way for make not to compile mytestexe (note I didn't added ALL in add_custom_target so I am not sure to know why make builds mytestexe)
2) Also the verbosity is broken with check : make test ARGS="-V" is verbose , but, make check ARGS="-V" is not.
    => how to deal with that ?

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: test depending on code compilation

Franck Houssen
OK, so I guess there is no way to do that like with autotools...


De: "Franck Houssen" <[hidden email]>
À: "CMake Mail List" <[hidden email]>
Envoyé: Mercredi 10 Janvier 2018 12:22:39
Objet: Re: [CMake] test depending on code compilation

I tried to reduce the example:

>> ls
CMakeLists.txt  main.cpp  mytest.cpp  mytest.sh

>> more *
::::::::::::::
CMakeLists.txt
::::::::::::::
cmake_minimum_required(VERSION 3.7)
enable_language(CXX)

project(main)
add_executable(main main.cpp)
add_executable(mytestcpp mytest.cpp)

include(CTest)
enable_testing()
configure_file("${CMAKE_CURRENT_SOURCE_DIR}/mytest.sh" "${CMAKE_CURRENT_BINARY_DIR}/mytest.sh")
add_test(NAME mytestsh COMMAND ./mytest.sh WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}")

add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} DEPENDS mytestcpp)
::::::::::::::
main.cpp
::::::::::::::
int main() {return 0;}
::::::::::::::
mytest.cpp
::::::::::::::
int main() {return 0;}
::::::::::::::
mytest.sh
::::::::::::::
#!/bin/bash
exit 0

I want make to build only main => KO
>> make
[ 50%] Built target main
[100%] Built target mytestcpp

I want make check to build mytestcpp and run tests => OK but verbose is KO.
>> make check ARGS="-V"
[100%] Built target mytestcpp
Test project /tmp/BUILD
    Start 1: mytestsh
1/1 Test #1: mytestsh .........................   Passed    0.00 sec

I want make test to run test only => OK and verbose is OK.


De: "Franck Houssen" <[hidden email]>
À: "CMake Mail List" <[hidden email]>
Envoyé: Mercredi 10 Janvier 2018 10:47:34
Objet: [CMake] test depending on code compilation

I need to design a test (= a bash script) such that :
1) run a dedicated executable for the test (to be compiled)
2) diff the run output with a reference log file

The test is created with : add_test(mytest ./mytest.sh). The bash script would look like:
>> more mytest.sh
/path/to/mytestexe > out.log
diff out.log out.ref

The dedicated executable for the test is created with : add_executable(mytestexe mytestexe.cpp)

Ideally, I would like mytestexe to be compiled when I type "make test" (before to run the test) but not "make".

Seems impossible to add a dependencie with : add_dependencies(mytest mytestexe). I googled this and found https://stackoverflow.com/questions/733475/cmake-ctest-make-test-doesnt-build-tests but this was not really clear to me...I added : add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} DEPENDS mytestexe).

The 2 unexpected problems I have left are:
1) mytestexe is compiled everytime I type "make" which is a solution but is not really what I am looking for (also compiled when I type "make check" which is expected).
    => is there a way for make not to compile mytestexe (note I didn't added ALL in add_custom_target so I am not sure to know why make builds mytestexe)
2) Also the verbosity is broken with check : make test ARGS="-V" is verbose , but, make check ARGS="-V" is not.
    => how to deal with that ?

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: test depending on code compilation

Mario Werner
In reply to this post by Franck Houssen
On 2018-01-10 10:47, Franck Houssen wrote:
> [snip]
>
> The 2 unexpected problems I have left are:
> 1) mytestexe is compiled everytime I type "make" which is a solution but
> is not really what I am looking for (also compiled when I type "make
> check" which is expected).
>     => is there a way for make not to compile mytestexe (note I didn't
> added ALL in add_custom_target so I am not sure to know why make builds
> mytestexe)

You have to excluded mytestexe from the ALL target if you do not want
that it gets built by default [1]. (e.g., add EXCLUDE_FROM_ALL to
add_executable [2])

> 2) Also the verbosity is broken with check : make test ARGS="-V" is
> verbose , but, make check ARGS="-V" is not.
>     => how to deal with that ?
>
>

Can't help you there. I usually simply use `make check` to build and run
the full test suite in order to get an overview. Afterwards, I directly
call `ctest` with the desired arguments, for example when triaging a bug.

HTH,
Mario


[1] https://cmake.org/cmake/help/v3.0/prop_tgt/EXCLUDE_FROM_ALL.html
[2]
https://cmake.org/cmake/help/v3.0/command/add_executable.html?highlight=EXCLUDE_FROM_ALL

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: test depending on code compilation

Franck Houssen


----- Mail original -----

> De: "Mario Werner" <[hidden email]>
> À: [hidden email]
> Envoyé: Jeudi 11 Janvier 2018 16:27:18
> Objet: Re: [CMake] test depending on code compilation
>
> On 2018-01-10 10:47, Franck Houssen wrote:
> > [snip]
> >
> > The 2 unexpected problems I have left are:
> > 1) mytestexe is compiled everytime I type "make" which is a solution but
> > is not really what I am looking for (also compiled when I type "make
> > check" which is expected).
> >     => is there a way for make not to compile mytestexe (note I didn't
> > added ALL in add_custom_target so I am not sure to know why make builds
> > mytestexe)
>
> You have to excluded mytestexe from the ALL target if you do not want
> that it gets built by default [1]. (e.g., add EXCLUDE_FROM_ALL to
> add_executable [2])
>

OK !

> > 2) Also the verbosity is broken with check : make test ARGS="-V" is
> > verbose , but, make check ARGS="-V" is not.
> >     => how to deal with that ?
> >
> >
>
> Can't help you there. I usually simply use `make check` to build and run
> the full test suite in order to get an overview. Afterwards, I directly
> call `ctest` with the desired arguments, for example when triaging a bug.
>

In travis a make check ARGS="-V" could be convenient to see output of tests while they run (understand what is wrong when if it fails - you can't get files from travis).
Anyway, that not so blocking.

> HTH,
> Mario
>
>
> [1] https://cmake.org/cmake/help/v3.0/prop_tgt/EXCLUDE_FROM_ALL.html
> [2]
> https://cmake.org/cmake/help/v3.0/command/add_executable.html?highlight=EXCLUDE_FROM_ALL
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
>
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: test depending on code compilation

Mario Werner
On 2018-01-11 18:03, Franck Houssen wrote:

>
>
> ----- Mail original -----
>> De: "Mario Werner" <[hidden email]>
>> À: [hidden email]
>> Envoyé: Jeudi 11 Janvier 2018 16:27:18
>> Objet: Re: [CMake] test depending on code compilation
>>
> [snip]
>
>>> 2) Also the verbosity is broken with check : make test ARGS="-V" is
>>> verbose , but, make check ARGS="-V" is not.
>>>     => how to deal with that ?
>>>
>>>
>>
>> Can't help you there. I usually simply use `make check` to build and run
>> the full test suite in order to get an overview. Afterwards, I directly
>> call `ctest` with the desired arguments, for example when triaging a bug.
>>
>
> In travis a make check ARGS="-V" could be convenient to see output of tests while they run (understand what is wrong when if it fails - you can't get files from travis).
> Anyway, that not so blocking.
>

Right, I usually have two custom targets in my projects to deal with
that in a generic way. Namely, "suite" for building the tests and
"check" for building+running the tests.

Developers can then use "make check" as usual to build and run the tests
without providing any arguments to ctest. On the other hand, the CI
system can build using the "suite" target and execute ctest directly
like in the following example. (Of course developers can do the same if
needed)

```

    - mkdir _build
    - cd _build
    - cmake -DCMAKE_BUILD_TYPE=Release ..
    - cmake --build . --target suite -- -j2
    - ctest -V

```

I use the following helper macro to generate the needed boilerplate
code. Usage is then as simple as calling `add_to_suite` for every target.

```

macro(add_to_suite target)
  # add suite target which is used as meta target for building the tests
  if(NOT TARGET "suite")
    add_custom_target("suite")
  endif()

  # add check command which calls ctest
  # it additionally depends on the suite target to build the test cases
  if(NOT TARGET "check")
    add_custom_target("check" COMMAND ${CMAKE_CTEST_COMMAND}
                              WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
                              COMMENT "Executing test suite.")
    add_dependencies("check" "suite")
  endif()

  add_dependencies("suite" ${target})
endmacro()

# usage example (repeat for any number of tests you have
add_executable(testexe EXCLUDE_FROM_ALL ${test_source_files})
target_link_libraries(testexe ${link_dependencies})
add_to_suite(testexe)

```

Note that, instead of specifying EXCLUDE_FROM_ALL on every
add_executable, I usually have all my tests in one directory and specify
EXCLUDE_FROM_ALL on the add_subdirectory call.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: test depending on code compilation

Franck Houssen


----- Mail original -----

> De: "Mario Werner" <[hidden email]>
> À: [hidden email]
> Envoyé: Vendredi 12 Janvier 2018 09:38:50
> Objet: Re: [CMake] test depending on code compilation
>
> On 2018-01-11 18:03, Franck Houssen wrote:
> >
> >
> > ----- Mail original -----
> >> De: "Mario Werner" <[hidden email]>
> >> À: [hidden email]
> >> Envoyé: Jeudi 11 Janvier 2018 16:27:18
> >> Objet: Re: [CMake] test depending on code compilation
> >>
> > [snip]
> >
> >>> 2) Also the verbosity is broken with check : make test ARGS="-V" is
> >>> verbose , but, make check ARGS="-V" is not.
> >>>     => how to deal with that ?
> >>>
> >>>
> >>
> >> Can't help you there. I usually simply use `make check` to build and run
> >> the full test suite in order to get an overview. Afterwards, I directly
> >> call `ctest` with the desired arguments, for example when triaging a bug.
> >>
> >
> > In travis a make check ARGS="-V" could be convenient to see output of tests
> > while they run (understand what is wrong when if it fails - you can't get
> > files from travis).
> > Anyway, that not so blocking.
> >
>
> Right, I usually have two custom targets in my projects to deal with
> that in a generic way. Namely, "suite" for building the tests and
> "check" for building+running the tests.
>
> Developers can then use "make check" as usual to build and run the tests
> without providing any arguments to ctest. On the other hand, the CI
> system can build using the "suite" target and execute ctest directly
> like in the following example. (Of course developers can do the same if
> needed)
>
> ```
>
>     - mkdir _build
>     - cd _build
>     - cmake -DCMAKE_BUILD_TYPE=Release ..
>     - cmake --build . --target suite -- -j2
>     - ctest -V
>
> ```
>
> I use the following helper macro to generate the needed boilerplate
> code. Usage is then as simple as calling `add_to_suite` for every target.
>
> ```
>
> macro(add_to_suite target)
>   # add suite target which is used as meta target for building the tests
>   if(NOT TARGET "suite")
>     add_custom_target("suite")
>   endif()
>
>   # add check command which calls ctest
>   # it additionally depends on the suite target to build the test cases
>   if(NOT TARGET "check")
>     add_custom_target("check" COMMAND ${CMAKE_CTEST_COMMAND}
>                               WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
>                               COMMENT "Executing test suite.")
>     add_dependencies("check" "suite")
>   endif()
>
>   add_dependencies("suite" ${target})
> endmacro()
>
> # usage example (repeat for any number of tests you have
> add_executable(testexe EXCLUDE_FROM_ALL ${test_source_files})
> target_link_libraries(testexe ${link_dependencies})
> add_to_suite(testexe)
>

OK, I see. I'll try that way. Thanks

> ```
>
> Note that, instead of specifying EXCLUDE_FROM_ALL on every
> add_executable, I usually have all my tests in one directory and specify
> EXCLUDE_FROM_ALL on the add_subdirectory call.
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
>
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake