CMake Coverage broken with Ninja

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

CMake Coverage broken with Ninja

Gerhard Gappmeier
hi all,

just recently I discovered that our coverage builds on CDash don't work anymore.
By analyzing the problem I found out that this is related to Ninja generator.
I don't know when the problem was introduced but it still exists with CMake 3.9.1.
(GCC and gcov are version 4.9.4, but this is not important)

I attached a simple test program to reproduce the problem.

$> tar xf demo.tar.gz
$> cd demo
$> mkdir bld
$> cmake -GNinja ..
$> ninja Nightly
...
Configure project
   Each . represents 1024 bytes of output
    . Size of output: 0K
Build project
   Each symbol represents 1024 bytes of output.
   '!' represents an error and '*' a warning.
    . Size of output: 0K
   0 Compiler errors
   0 Compiler warnings
Test project /home/gergap/tmp/cmaketest/bld
    Start 1: mytest
1/1 Test #1: mytest ...........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.00 sec
Performing coverage
   Processing coverage (each . represents one file):
    .
   Accumulating results (each . represents one file):
    .Cannot find file: /home/gergap/tmp/cmaketest/bld/Testing/main.c

        Covered LOC:         0
        Not covered LOC:     0
        Total LOC:           0
        Percentage Coverage: 0.00%
...

When using make instead everything works as expected.
$> tar xf demo.tar.gz
$> cd demo
$> mkdir bld
$> cmake ..
$> make Nightly
Configure project
   Each . represents 1024 bytes of output
    . Size of output: 0K
Build project
   Each symbol represents 1024 bytes of output.
   '!' represents an error and '*' a warning.
    . Size of output: 0K
   0 Compiler errors
   0 Compiler warnings
Test project /home/gergap/tmp/cmaketest/bld
    Start 1: mytest
1/1 Test #1: mytest ...........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.00 sec
Performing coverage
   Processing coverage (each . represents one file):
    .
   Accumulating results (each . represents one file):
    .
        Covered LOC:         3
        Not covered LOC:     0
        Total LOC:           3
        Percentage Coverage: 100.00%

Is this problem known?
Are there any fixes/workaround available?

--
mit freundlichen Grüßen / best regards

Gerhard Gappmeier
ascolab GmbH - automation systems communication laboratory
Tel.: +49 911 569846 203
Fax: +49 911 569846 400
Web: http://www.ascolab.com
GPG-KeyId: 7A78B0A7
GPG-Fingerprint: A964 56A2 32E9 A537 AB7E EBD6 8D09 0D72 7A78 B0A7

--
ascolab GmbH
Geschäftsführer: Gerhard Gappmeier, Matthias Damm, Uwe Steinkrauß
Sitz der Gesellschaft: An der Kaufleite 34 • 90562 Kalchreuth • Germany
Registernummer: HRB 9360
Registergericht: Amtsgericht Fürth


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

demo.tar.gz (710 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CMake Coverage broken with Ninja

Haocheng Liu
Hi Gerhard,

On Fri, Sep 1, 2017 at 7:26 AM, Gerhard Gappmeier <[hidden email]> wrote:
hi all,

just recently I discovered that our coverage builds on CDash don't work anymore.
By analyzing the problem I found out that this is related to Ninja generator.
I don't know when the problem was introduced but it still exists with CMake 3.9.1.
(GCC and gcov are version 4.9.4, but this is not important)

I attached a simple test program to reproduce the problem.

$> tar xf demo.tar.gz
$> cd demo
$> mkdir bld
What if you call `chcp 65001` here before the CMake call? (Force using  UTF-8 console) 

$> cmake -GNinja ..
$> ninja Nightly
...
Configure project
   Each . represents 1024 bytes of output
    . Size of output: 0K
Build project
   Each symbol represents 1024 bytes of output.
   '!' represents an error and '*' a warning.
    . Size of output: 0K
   0 Compiler errors
   0 Compiler warnings
Test project /home/gergap/tmp/cmaketest/bld
    Start 1: mytest
1/1 Test #1: mytest ...........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.00 sec
Performing coverage
   Processing coverage (each . represents one file):
    .
   Accumulating results (each . represents one file):
    .Cannot find file: /home/gergap/tmp/cmaketest/bld/Testing/main.c

        Covered LOC:         0
        Not covered LOC:     0
        Total LOC:           0
        Percentage Coverage: 0.00%
...

When using make instead everything works as expected.
$> tar xf demo.tar.gz
$> cd demo
$> mkdir bld
$> cmake ..
$> make Nightly
Configure project
   Each . represents 1024 bytes of output
    . Size of output: 0K
Build project
   Each symbol represents 1024 bytes of output.
   '!' represents an error and '*' a warning.
    . Size of output: 0K
   0 Compiler errors
   0 Compiler warnings
Test project /home/gergap/tmp/cmaketest/bld
    Start 1: mytest
1/1 Test #1: mytest ...........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.00 sec
Performing coverage
   Processing coverage (each . represents one file):
    .
   Accumulating results (each . represents one file):
    .
        Covered LOC:         3
        Not covered LOC:     0
        Total LOC:           3
        Percentage Coverage: 100.00%

Is this problem known?
I believe it's related to CMake issue 17191 that CMake is not properly converting the -showIncludes prefix (${displayedInSystemLanguage}) to its internal UTF-8 encoding.
Are there any fixes/workaround available?

Thanks to Brad it has been fixed in CMake MR 1179. If you do not want to use CMake from source build then adding `chcp 6500`1 should do the trick.
--
mit freundlichen Grüßen / best regards

Gerhard Gappmeier
ascolab GmbH - automation systems communication laboratory
Tel.: <a href="tel:+49%20911%20569846203" value="+49911569846203" target="_blank">+49 911 569846 203
Fax: <a href="tel:+49%20911%20569846400" value="+49911569846400" target="_blank">+49 911 569846 400
Web: http://www.ascolab.com
GPG-KeyId: 7A78B0A7
GPG-Fingerprint: A964 56A2 32E9 A537 AB7E EBD6 8D09 0D72 7A78 B0A7

--
ascolab GmbH
Geschäftsführer: Gerhard Gappmeier, Matthias Damm, Uwe Steinkrauß
Sitz der Gesellschaft: An der Kaufleite 34 • 90562 Kalchreuth • Germany
Registernummer: HRB 9360
Registergericht: Amtsgericht Fürth


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake



--
Best regards
Haocheng

Haocheng LIU
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: <a href="tel:(518)%20881-4421" value="+15188814443" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">518-881-4421

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Coverage broken with Ninja

Gerhard Gappmeier
Hi Haocheng,

`chcp` seems to be a Windows command.
I'm talking about Linux, which is UTF-8 by default:

$> locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

any ideas for the Linux platform?


mit freundlichen Grüßen / best regards

Gerhard Gappmeier
ascolab GmbH - automation systems communication laboratory
Tel.: +49 911 569846 203
Fax: +49 911 569846 400
Web: http://www.ascolab.com
GPG-KeyId: 7A78B0A7
GPG-Fingerprint: A964 56A2 32E9 A537 AB7E EBD6 8D09 0D72 7A78 B0A7

--
ascolab GmbH
Geschäftsführer: Gerhard Gappmeier, Matthias Damm, Uwe Steinkrauß
Sitz der Gesellschaft: An der Kaufleite 34 • 90562 Kalchreuth • Germany
Registernummer: HRB 9360
Registergericht: Amtsgericht Fürth

On 09/01/2017 02:49 PM, Haocheng Liu wrote:
chcp 65001


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Coverage broken with Ninja

Haocheng Liu


On Fri, Sep 1, 2017 at 9:41 AM, Gerhard Gappmeier <[hidden email]> wrote:
Hi Haocheng,

`chcp` seems to be a Windows command.
I'm talking about Linux, which is UTF-8 by default:

$> locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

any ideas for the Linux platform?

Hum, I tried your example, the problem is that ${CMAKE_BUILD_DIRECTORY}/testing is missing a main.c file. If I create a fake file and the coverage reports 100%. I've Cc Brad King since He is most familiar with Ninja related issues.

mit freundlichen Grüßen / best regards

Gerhard Gappmeier
ascolab GmbH - automation systems communication laboratory
Tel.: <a href="tel:+49%20911%20569846203" value="+49911569846203" target="_blank">+49 911 569846 203
Fax: <a href="tel:+49%20911%20569846400" value="+49911569846400" target="_blank">+49 911 569846 400
Web: http://www.ascolab.com
GPG-KeyId: 7A78B0A7
GPG-Fingerprint: A964 56A2 32E9 A537 AB7E EBD6 8D09 0D72 7A78 B0A7

--
ascolab GmbH
Geschäftsführer: Gerhard Gappmeier, Matthias Damm, Uwe Steinkrauß
Sitz der Gesellschaft: An der Kaufleite 34 • 90562 Kalchreuth • Germany
Registernummer: HRB 9360
Registergericht: Amtsgericht Fürth

On 09/01/2017 02:49 PM, Haocheng Liu wrote:
chcp 65001




--
Best regards
Haocheng

Haocheng LIU
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: <a href="tel:(518)%20881-4421" value="+15188814443" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">518-881-4421

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Coverage broken with Ninja

Gerhard Gappmeier
Hi Haocheng,

thanks for helping.
I guess this has to do with another issue. The Ninja generator generates different paths then the default Unix Makefile generator.

This leads to the following behavior in normal build operations:
I introduced a warning in the printf line to demonstrate this.

With Makefiles: mkdir bld && cd bld && cmake ..
Build inside Vim using ":make"

[ 50%] Building C object CMakeFiles/demo.dir/main.c.o
/home/gergap/tmp/demo/main.c: In function ‘main’:
/home/gergap/tmp/demo/main.c:5:12: warning: format ‘%i’ expects a matching ‘int’ argument [-Wformat=]
     printf("Hello World %i\n");
            ^
[100%] Linking C executable demo
[100%] Built target demo

Press ENTER or type command to continue

as you can see the warning message contains the absolute path, so parsing the errors works for Vim (Emacs, or whatever tool),
and you can jump to the error location.

Now the same with Ninja: mkdir bld && cd bld && cmake -GNinja ..
Build inside Vim using ":make" (use ":set makeprg=ninja" first)

cd into build dir '/home/gergap/tmp/demo/bld/../bld'...
executing ninja...
[1/2] Building C object CMakeFiles/demo.dir/main.c.o
../main.c: In function ‘main’:
../main.c:5:12: warning: format ‘%i’ expects a matching ‘int’ argument [-Wformat=]
     printf("Hello World %i\n");
            ^
[2/2] Linking C executable demo
cd back
/home/gergap/tmp/demo/bld

Press ENTER or type command to continue

As you can see now the paths are relative to the build folder, thus jumping to the error location only works when Vim has the build folder set as working dir,
and this is typically not the case.
This is quite annoying, inconsistent with Makefiles and was already reported once here: https://cmake.org/Bug/print_bug_page.php?bug_id=13894
AFAIK this was never fixed.

Now I suspect this to be also the reason for the coverage issues.
I hope this helps.


On 09/01/2017 04:09 PM, Haocheng Liu wrote:


On Fri, Sep 1, 2017 at 9:41 AM, Gerhard Gappmeier <[hidden email]> wrote:
Hi Haocheng,

`chcp` seems to be a Windows command.
I'm talking about Linux, which is UTF-8 by default:

$> locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

any ideas for the Linux platform?

Hum, I tried your example, the problem is that ${CMAKE_BUILD_DIRECTORY}/testing is missing a main.c file. If I create a fake file and the coverage reports 100%. I've Cc Brad King since He is most familiar with Ninja related issues.

mit freundlichen Grüßen / best regards

Gerhard Gappmeier
ascolab GmbH - automation systems communication laboratory
Tel.: <a href="tel:+49%20911%20569846203" value="+49911569846203" target="_blank" moz-do-not-send="true">+49 911 569846 203
Fax: <a href="tel:+49%20911%20569846400" value="+49911569846400" target="_blank" moz-do-not-send="true">+49 911 569846 400
Web: http://www.ascolab.com
GPG-KeyId: 7A78B0A7
GPG-Fingerprint: A964 56A2 32E9 A537 AB7E EBD6 8D09 0D72 7A78 B0A7

--
ascolab GmbH
Geschäftsführer: Gerhard Gappmeier, Matthias Damm, Uwe Steinkrauß
Sitz der Gesellschaft: An der Kaufleite 34 • 90562 Kalchreuth • Germany
Registernummer: HRB 9360
Registergericht: Amtsgericht Fürth

On 09/01/2017 02:49 PM, Haocheng Liu wrote:
chcp 65001




--
Best regards
Haocheng

Haocheng LIU
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: <a href="tel:%28518%29%20881-4421" value="+15188814443" style="color:rgb(17,85,204);font-size:12.8px" target="_blank" moz-do-not-send="true">518-881-4421


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Coverage broken with Ninja

Brad King
On 09/02/2017 03:14 AM, Gerhard Gappmeier wrote:
> This is quite annoying, inconsistent with Makefiles and was already
> reported once here: https://cmake.org/Bug/print_bug_page.php?bug_id=13894
> AFAIK this was never fixed.

Discussion of that issue is now here:

  https://gitlab.kitware.com/cmake/cmake/issues/13894

We made a serious attempt to fix it, but ran into a major problem
described here:

  https://gitlab.kitware.com/cmake/cmake/issues/13894#note_233789

Fixing it requires a feature in Ninja:

  https://github.com/ninja-build/ninja/issues/1251

I posted a proposal to their list about it but got no response:

  https://groups.google.com/forum/#!topic/ninja-build/yJvs7u0n2iA

---

One may use out-of-source builds to get absolute paths to the files
in the main source tree.

-Brad
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Coverage broken with Ninja

Gerhard Gappmeier
Hi Brad,

thanks for that information.
It is great to see that you are working on this.

Could you already verify if this problem is also the cause for the
coverage issue?

On 09/05/2017 01:31 PM, Brad King wrote:

> On 09/02/2017 03:14 AM, Gerhard Gappmeier wrote:
>> This is quite annoying, inconsistent with Makefiles and was already
>> reported once here: https://cmake.org/Bug/print_bug_page.php?bug_id=13894
>> AFAIK this was never fixed.
> Discussion of that issue is now here:
>
>   https://gitlab.kitware.com/cmake/cmake/issues/13894
>
> We made a serious attempt to fix it, but ran into a major problem
> described here:
>
>   https://gitlab.kitware.com/cmake/cmake/issues/13894#note_233789
>
> Fixing it requires a feature in Ninja:
>
>   https://github.com/ninja-build/ninja/issues/1251
>
> I posted a proposal to their list about it but got no response:
>
>   https://groups.google.com/forum/#!topic/ninja-build/yJvs7u0n2iA
>
> ---
>
> One may use out-of-source builds to get absolute paths to the files
> in the main source tree.
>
> -Brad

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Coverage broken with Ninja

Brad King
On 09/05/2017 07:42 AM, Gerhard Gappmeier wrote:
> Could you already verify if this problem is also the cause for the
> coverage issue?

I tried your example in a fully out-of-source build (where the
build tree is not inside the source tree), and coverage works.

-Brad
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake