build step is not thread-safe for CMake

classic Classic list List threaded Threaded
3 messages Options
Hex
Reply | Threaded
Open this post in threaded view
|

build step is not thread-safe for CMake

Hex

hello,


My build step is not thread-safe (the instruction in the COMMAND part). Every build step depends on one source file:


add_custom_command(
        DEPENDS on_source.file
        OUTPUT
                object_file_for_source_file
        COMMAND not-thread-safe-compiler --build on_source.file
        COMMENT
                "Building Source object"
)


add_custom_target DEPENDS on OUTPUT from all custom command's, but in order to prevent parallel execution when cmake is called:


cmake --build . --target all -j30


I need to build each step in any order, but still not as a dependency as this would trigger unnecessary recompilation. Using MAIN_DEPENDENCY defines hierarchical dependency, but this won't help in this situation.


I cannot figure out how to prevent parallel execution of COMMAND in CMake. The only option I see is a single command


add_custom_target(tgt
        DEPENDS on_all_source.files
        COMMAND not-thread-safe-compiler --build on_source1.file

        COMMAND not-thread-safe-compiler --build on_source2.file

        COMMAND not-thread-safe-compiler --build on_source3.file

        . . .

        COMMENT
                "Building target"
)


this would rebuild every source, though.


How do I set this up?


thank you


--

Powered by kitware.com/cmake

Kitware offers various services to support the CMake community. For more information on each offering, please visit https://cmake.org/services

Visit other Kitware open-source projects at https://www.kitware.com/platforms

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

This mailing list is deprecated in favor of https://discourse.cmake.org
Reply | Threaded
Open this post in threaded view
|

Re: build step is not thread-safe for CMake

CMake mailing list
On Mon, 2020-02-10 at 12:49 +0000, hex wrote:

> hello,
>
> My build step is not thread-safe (the instruction in the COMMAND
> part). Every build step depends on one source file:
>
> add_custom_command(
>         DEPENDS on_source.file
>         OUTPUT
>                 object_file_for_source_file
>         COMMAND not-thread-safe-compiler --build on_source.file
>         COMMENT
>                 "Building Source object"
> )
>
> add_custom_target DEPENDS on OUTPUT from all custom command's, but in
> order to prevent parallel execution when cmake is called:
>
> cmake --build . --target all -j30
>
> I need to build each step in any order, but still not as a dependency
> as this would trigger unnecessary recompilation. Using
> MAIN_DEPENDENCY defines hierarchical dependency, but this won't help
> in this situation.
>
> I cannot figure out how to prevent parallel execution of COMMAND in
> CMake. The only option I see is a single command
>
> add_custom_target(tgt
>         DEPENDS on_all_source.files
>         COMMAND not-thread-safe-compiler --build on_source1.file
>         COMMAND not-thread-safe-compiler --build on_source2.file
>         COMMAND not-thread-safe-compiler --build on_source3.file
>         . . .
>         COMMENT
>                 "Building target"
> )
>
> this would rebuild every source, though.
>
> How do I set this up?
>
> thank you

It sounds like you want order-only dependencies: dependencies the
command waits for before building, but whose changes do not actually
trigger a rebuild. As far as I know, add_custom_command() doesn't have
support for this - perhaps it could be added.

This mailing list is deprecated. Please move further discussion to the
Discourse forum:

https://discourse.cmake.org/c/code/11

Kyle
--

Powered by kitware.com/cmake

Kitware offers various services to support the CMake community. For more information on each offering, please visit https://cmake.org/services

Visit other Kitware open-source projects at https://www.kitware.com/platforms

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

This mailing list is deprecated in favor of https://discourse.cmake.org
Reply | Threaded
Open this post in threaded view
|

Re: build step is not thread-safe for CMake

Eric Doenges
Am 10.02.20 um 14:23 schrieb Kyle Edwards via CMake:
On Mon, 2020-02-10 at 12:49 +0000, hex wrote:
hello,

My build step is not thread-safe (the instruction in the COMMAND
part). Every build step depends on one source file:

add_custom_command(
        DEPENDS on_source.file
        OUTPUT
                object_file_for_source_file
        COMMAND not-thread-safe-compiler --build on_source.file
        COMMENT
                "Building Source object"
)

add_custom_target DEPENDS on OUTPUT from all custom command's, but in
order to prevent parallel execution when cmake is called:

cmake --build . --target all -j30

I need to build each step in any order, but still not as a dependency
as this would trigger unnecessary recompilation. Using
MAIN_DEPENDENCY defines hierarchical dependency, but this won't help
in this situation.

I cannot figure out how to prevent parallel execution of COMMAND in
CMake. The only option I see is a single command

add_custom_target(tgt
        DEPENDS on_all_source.files
        COMMAND not-thread-safe-compiler --build on_source1.file
        COMMAND not-thread-safe-compiler --build on_source2.file
        COMMAND not-thread-safe-compiler --build on_source3.file
        . . .
        COMMENT
                "Building target"
)

this would rebuild every source, though.

How do I set this up?

thank you
It sounds like you want order-only dependencies: dependencies the
command waits for before building, but whose changes do not actually
trigger a rebuild. As far as I know, add_custom_command() doesn't have
support for this - perhaps it could be added.

Actually, I think it kind of does, at least with the Ninja generator. If you do something like

add_custom_output(OUTPUT fileA COMMAND ...)
add_custom_target(fileA_target DEPENDS fileA)

add_custom_output(OUTPUT fileB DEPENDS fileA COMMAND ...)
add_custom_output(OUTPUT fileC DEPENDS fileA_target COMMAND ...)

then both fileB and fileC will depend on fileA existing, but fileC will not be rebuilt if fileA changes, while fileB will be.

Disclaimer - I couldn't find this behavior described in the documentation, but it is what I've observed after spending extensive amounts of time messing about with add_custom_command() and add_custom_target() to build the documentation to our product with CMake 3.16. Also, I've only tested this with the Ninja generator, so it may behave differently with other generators.

With kind regards,
Eric Dönges

Dr. Eric Dönges
Senior Software Engineer

MVTec Software GmbH | Arnulfstr. 205 | 80634 Munich | Germany
[hidden email] | Tel: +49 89 457 695-0 | www.mvtec.com

Find our privacy policy here.

 Sign up for our MVTec Newsletter!

Geschäftsführer: Dr. Olaf Munkelt
Amtsgericht München HRB 114695

 

MVTec Software GmbH Logo

--

Powered by kitware.com/cmake

Kitware offers various services to support the CMake community. For more information on each offering, please visit https://cmake.org/services

Visit other Kitware open-source projects at https://www.kitware.com/platforms

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

This mailing list is deprecated in favor of https://discourse.cmake.org