Problem measuring coverage in CTest scripts

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem measuring coverage in CTest scripts

Maarten Anonymous
Hey,

I have a question about using CMake+CTest+CTest scripts for coverage tests.

If I manually build the project and run "make ExperimantalCoverage" in a build folder,
the coverage succeeds.
If I run the coverage from within a CTest script,
it fails.

Am I forgetting something?

A minimum example that exhibits my problem can be found at https://gist.github.com/madebr/07ebd0de59e789cd819643b58db25df9.

This works:
mkdir build && cd build && cmake .. && make && make test && make ExperimentalCoverage && cd ..
This fails:
ctest -S test.ctest

Source listing:
###################
#CMakeLists.txt:
###################
cmake_minimum_required(VERSION 3.9)
project(cmake_coverage)

file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/main.c" "int main() {return 0;}")

add_executable(main
    main.c
    )

target_compile_options(main
    PUBLIC
        "-O0"
        "-fprofile-arcs"
        "-ftest-coverage"
    )

target_link_libraries(main
    gcov
    )

include(CTest)
enable_testing()

add_test(NAME test
    COMMAND main
    )

##############
# test.ctest:
##############
set(CTEST_SOURCE_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}")
set(CTEST_BINARY_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}/build/ctest")

ctest_empty_binary_directory("${CTEST_BINARY_DIRECTORY}")

set(CTEST_SITE "${CMAKE_SYSTEM_NAME}")
set(CTEST_BUILD_NAME "experimental build")

set(CTEST_CMAKE_GENERATOR "Ninja")
set(CTEST_COVERAGE_COMMAND "gcov")

ctest_start("Experimental")
ctest_configure()
ctest_build()
ctest_test()
ctest_coverage()

##############
# CTestConfig.cmake:
##############
set(CTEST_PROJECT_NAME "Simple ctest_coverage project")



--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Problem measuring coverage in CTest scripts

Edoardo Pasca
Afaik, Ctest should be run in the build directory. From your command it seems you run it in the source. I'm not surprised it doesn't work.

Edo

On Oct 28, 2017 6:59 AM, "Maarten Anonymous" <[hidden email]> wrote:
Hey,

I have a question about using CMake+CTest+CTest scripts for coverage tests.

If I manually build the project and run "make ExperimantalCoverage" in a build folder,
the coverage succeeds.
If I run the coverage from within a CTest script,
it fails.

Am I forgetting something?

A minimum example that exhibits my problem can be found at https://gist.github.com/madebr/07ebd0de59e789cd819643b58db25df9.

This works:
mkdir build && cd build && cmake .. && make && make test && make ExperimentalCoverage && cd ..
This fails:
ctest -S test.ctest

Source listing:
###################
#CMakeLists.txt:
###################
cmake_minimum_required(VERSION 3.9)
project(cmake_coverage)

file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/main.c" "int main() {return 0;}")

add_executable(main
    main.c
    )

target_compile_options(main
    PUBLIC
        "-O0"
        "-fprofile-arcs"
        "-ftest-coverage"
    )

target_link_libraries(main
    gcov
    )

include(CTest)
enable_testing()

add_test(NAME test
    COMMAND main
    )

##############
# test.ctest:
##############
set(CTEST_SOURCE_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}")
set(CTEST_BINARY_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}/build/ctest")

ctest_empty_binary_directory("${CTEST_BINARY_DIRECTORY}")

set(CTEST_SITE "${CMAKE_SYSTEM_NAME}")
set(CTEST_BUILD_NAME "experimental build")

set(CTEST_CMAKE_GENERATOR "Ninja")
set(CTEST_COVERAGE_COMMAND "gcov")

ctest_start("Experimental")
ctest_configure()
ctest_build()
ctest_test()
ctest_coverage()

##############
# CTestConfig.cmake:
##############
set(CTEST_PROJECT_NAME "Simple ctest_coverage project")



--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Problem measuring coverage in CTest scripts

Lectem

No since the point of cmake scripts is to build the project (for CI basically).

I'm not sure what happens but I remember people having issues when using ninja? I'm not sure.

You can have a look at
https://github.com/Lectem/cpp-boilerplate
I use a Ctest script for my Travis coverage build.

The script is the following :
https://github.com/Lectem/cpp-boilerplate/blob/master/cmake/CI.CTestScript.cmake


Le sam. 28 oct. 2017 à 08:30, Edoardo Pasca <[hidden email]> a écrit :
Afaik, Ctest should be run in the build directory. From your command it seems you run it in the source. I'm not surprised it doesn't work.

Edo

On Oct 28, 2017 6:59 AM, "Maarten Anonymous" <[hidden email]> wrote:
Hey,

I have a question about using CMake+CTest+CTest scripts for coverage tests.

If I manually build the project and run "make ExperimantalCoverage" in a build folder,
the coverage succeeds.
If I run the coverage from within a CTest script,
it fails.

Am I forgetting something?

A minimum example that exhibits my problem can be found at https://gist.github.com/madebr/07ebd0de59e789cd819643b58db25df9.

This works:
mkdir build && cd build && cmake .. && make && make test && make ExperimentalCoverage && cd ..
This fails:
ctest -S test.ctest

Source listing:
###################
#CMakeLists.txt:
###################
cmake_minimum_required(VERSION 3.9)
project(cmake_coverage)

file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/main.c" "int main() {return 0;}")

add_executable(main
    main.c
    )

target_compile_options(main
    PUBLIC
        "-O0"
        "-fprofile-arcs"
        "-ftest-coverage"
    )

target_link_libraries(main
    gcov
    )

include(CTest)
enable_testing()

add_test(NAME test
    COMMAND main
    )

##############
# test.ctest:
##############
set(CTEST_SOURCE_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}")
set(CTEST_BINARY_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}/build/ctest")

ctest_empty_binary_directory("${CTEST_BINARY_DIRECTORY}")

set(CTEST_SITE "${CMAKE_SYSTEM_NAME}")
set(CTEST_BUILD_NAME "experimental build")

set(CTEST_CMAKE_GENERATOR "Ninja")
set(CTEST_COVERAGE_COMMAND "gcov")

ctest_start("Experimental")
ctest_configure()
ctest_build()
ctest_test()
ctest_coverage()

##############
# CTestConfig.cmake:
##############
set(CTEST_PROJECT_NAME "Simple ctest_coverage project")



--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Problem measuring coverage in CTest scripts

Maarten Anonymous
In reply to this post by Maarten Anonymous
I've searched in the history of the mailing list and I think it's the
same problem as this one:
https://public.kitware.com/pipermail/cmake/2017-September/066176.html

I cannot use the Ninja generator in my script.
When I change the generator to 'Unix Makefiles', the coverage works.

Thanks for responding!

Maarten

On 28 October 2017 at 09:24, Clément Gregoire <[hidden email]> wrote:

> No since the point of cmake scripts is to build the project (for CI
> basically).
>
> I'm not sure what happens but I remember people having issues when using
> ninja? I'm not sure.
>
> You can have a look at
> https://github.com/Lectem/cpp-boilerplate
> I use a Ctest script for my Travis coverage build.
>
> The script is the following :
> https://github.com/Lectem/cpp-boilerplate/blob/master/cmake/CI.CTestScript.cmake
>
>
> Le sam. 28 oct. 2017 à 08:30, Edoardo Pasca <[hidden email]> a écrit
> :
>>
>> Afaik, Ctest should be run in the build directory. From your command it
>> seems you run it in the source. I'm not surprised it doesn't work.
>>
>> Edo
>>
>> On Oct 28, 2017 6:59 AM, "Maarten Anonymous" <[hidden email]>
>> wrote:
>>>
>>> Hey,
>>>
>>> I have a question about using CMake+CTest+CTest scripts for coverage
>>> tests.
>>>
>>> If I manually build the project and run "make ExperimantalCoverage" in a
>>> build folder,
>>> the coverage succeeds.
>>> If I run the coverage from within a CTest script,
>>> it fails.
>>>
>>> Am I forgetting something?
>>>
>>> A minimum example that exhibits my problem can be found at
>>> https://gist.github.com/madebr/07ebd0de59e789cd819643b58db25df9.
>>>
>>> This works:
>>> mkdir build && cd build && cmake .. && make && make test && make
>>> ExperimentalCoverage && cd ..
>>> This fails:
>>> ctest -S test.ctest
>>>
>>> Source listing:
>>> ###################
>>> #CMakeLists.txt:
>>> ###################
>>> cmake_minimum_required(VERSION 3.9)
>>> project(cmake_coverage)
>>>
>>> file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/main.c" "int main() {return 0;}")
>>>
>>> add_executable(main
>>>     main.c
>>>     )
>>>
>>> target_compile_options(main
>>>     PUBLIC
>>>         "-O0"
>>>         "-fprofile-arcs"
>>>         "-ftest-coverage"
>>>     )
>>>
>>> target_link_libraries(main
>>>     gcov
>>>     )
>>>
>>> include(CTest)
>>> enable_testing()
>>>
>>> add_test(NAME test
>>>     COMMAND main
>>>     )
>>>
>>> ##############
>>> # test.ctest:
>>> ##############
>>> set(CTEST_SOURCE_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}")
>>> set(CTEST_BINARY_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}/build/ctest")
>>>
>>> ctest_empty_binary_directory("${CTEST_BINARY_DIRECTORY}")
>>>
>>> set(CTEST_SITE "${CMAKE_SYSTEM_NAME}")
>>> set(CTEST_BUILD_NAME "experimental build")
>>>
>>> set(CTEST_CMAKE_GENERATOR "Ninja")
>>> set(CTEST_COVERAGE_COMMAND "gcov")
>>>
>>> ctest_start("Experimental")
>>> ctest_configure()
>>> ctest_build()
>>> ctest_test()
>>> ctest_coverage()
>>>
>>> ##############
>>> # CTestConfig.cmake:
>>> ##############
>>> set(CTEST_PROJECT_NAME "Simple ctest_coverage project")
>>>
>>>
>>>
>>> --
>>>
>>> Powered by www.kitware.com
>>>
>>> Please keep messages on-topic and check the CMake FAQ at:
>>> http://www.cmake.org/Wiki/CMake_FAQ
>>>
>>> Kitware offers various services to support the CMake community. For more
>>> information on each offering, please visit:
>>>
>>> CMake Support: http://cmake.org/cmake/help/support.html
>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/cmake
>>
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake