Parallel builds and auto generated header files

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel builds and auto generated header files

Robert Dailey-2
I have a Version.hpp file that I have a custom command tied to which
basically runs CMake in script mode to perform configure_file() on it
during build time. The reason it does this is because it builds
Version.hpp using dynamic information, such as defining a macro with
the current SHA1 being built, etc.

I notice in parallel builds, this header file gets built too late,
because of parallel builds. What if a file includes the header and its
translation unit is built before Version.hpp is generated by the
custom command? Would it be better/simpler to do configure_file() at
configuration time instead of at build time as a custom command?
Here's the logic (some variables are defined elsewhere, but should
give you the gist):


set( generated_source_dir ${CMAKE_CURRENT_BINARY_DIR}/Source )
set( version_file_in  ${CMAKE_CURRENT_SOURCE_DIR}/Version.cpp.in )
set( version_file_out ${generated_source_dir}/Main/Version.cpp )

add_custom_command(
    OUTPUT ${version_file_out}
    COMMAND ${CMAKE_COMMAND}
        -D IN_FILE=${version_file_in}
        -D OUT_FILE=${version_file_out}
        -D GIT_EXECUTABLE=${GIT_EXECUTABLE}
        -D BUILD_VERSION=${ZIOSK_BUILD_VERSION}
        -P ${CMAKE_CURRENT_SOURCE_DIR}/build_version_header.cmake
    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)

source_group( Generated FILES ${version_file_out} )
list( APPEND source_files ${version_file_out} )
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and auto generated header files

Andrew Fuller-2
Using configure_file() at CMake time will appear to work at first glance, but you'll wind up with stale information when you change revisions without any CMake change.  CMake won't re-run automatically and thus your header file won't get updated.  You'll need to do it at build time to ensure it's always up-to-date.

How do the translation units obtain the generated header?  There needs to be some form of dependency between the generated header and the consumer(s).  If you add the generated header as an input to another target, then CMake should ensure the file is generated before processing the dependant target.

Also note that AFAICT add_custom_command will not re-run if the output file already exists (unless invalidated by a dependency).  Since you're grabbing dynamic information like the commit hash, then you likely want it to run _every_ time.  add_custom_target might be better suited for your needs.

From: CMake <[hidden email]> on behalf of Robert Dailey <[hidden email]>
Sent: July 19, 2018 8:05:06 AM
To: CMake
Subject: [CMake] Parallel builds and auto generated header files
 
I have a Version.hpp file that I have a custom command tied to which
basically runs CMake in script mode to perform configure_file() on it
during build time. The reason it does this is because it builds
Version.hpp using dynamic information, such as defining a macro with
the current SHA1 being built, etc.

I notice in parallel builds, this header file gets built too late,
because of parallel builds. What if a file includes the header and its
translation unit is built before Version.hpp is generated by the
custom command? Would it be better/simpler to do configure_file() at
configuration time instead of at build time as a custom command?
Here's the logic (some variables are defined elsewhere, but should
give you the gist):


set( generated_source_dir ${CMAKE_CURRENT_BINARY_DIR}/Source )
set( version_file_in  ${CMAKE_CURRENT_SOURCE_DIR}/Version.cpp.in )
set( version_file_out ${generated_source_dir}/Main/Version.cpp )

add_custom_command(
    OUTPUT ${version_file_out}
    COMMAND ${CMAKE_COMMAND}
        -D IN_FILE=${version_file_in}
        -D OUT_FILE=${version_file_out}
        -D GIT_EXECUTABLE=${GIT_EXECUTABLE}
        -D BUILD_VERSION=${ZIOSK_BUILD_VERSION}
        -P ${CMAKE_CURRENT_SOURCE_DIR}/build_version_header.cmake
    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)

source_group( Generated FILES ${version_file_out} )
list( APPEND source_files ${version_file_out} )
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and auto generated header files

Isaiah Norton
I use essentially the technique in the answer below, except with `add_custom_target` instead of `add_custom_command`, so that the hash-generation is always run regardless of whether the temp file is successfully deleted:


I also use a dummy `add_custom_command` which depends on GITVER_FILE as in the above. IIRC the reason was that in my use case GITVER_FILE is passed as part of a sources list to a macro, and defining the custom command was the only way I could find to properly enforce an ordering dependency in parallel builds:

# dummy output so we can depend on it in the external macro, because CMake.
add_custom_command(OUTPUT "${GITVER_FILE}"
                                      DEPENDS _GEN_GITVER)

(I think it was fine running `add_executable`/`add_library` directly in the current CMakeList, but ordering didn't work when passing to a macro -- at least in the version of cmake where I developed this)

On Mon, Jul 23, 2018 at 2:50 PM Andrew Fuller <[hidden email]> wrote:
Using configure_file() at CMake time will appear to work at first glance, but you'll wind up with stale information when you change revisions without any CMake change.  CMake won't re-run automatically and thus your header file won't get updated.  You'll need to do it at build time to ensure it's always up-to-date.

How do the translation units obtain the generated header?  There needs to be some form of dependency between the generated header and the consumer(s).  If you add the generated header as an input to another target, then CMake should ensure the file is generated before processing the dependant target.

Also note that AFAICT add_custom_command will not re-run if the output file already exists (unless invalidated by a dependency).  Since you're grabbing dynamic information like the commit hash, then you likely want it to run _every_ time.  add_custom_target might be better suited for your needs.

From: CMake <[hidden email]> on behalf of Robert Dailey <[hidden email]>
Sent: July 19, 2018 8:05:06 AM
To: CMake
Subject: [CMake] Parallel builds and auto generated header files
 
I have a Version.hpp file that I have a custom command tied to which
basically runs CMake in script mode to perform configure_file() on it
during build time. The reason it does this is because it builds
Version.hpp using dynamic information, such as defining a macro with
the current SHA1 being built, etc.

I notice in parallel builds, this header file gets built too late,
because of parallel builds. What if a file includes the header and its
translation unit is built before Version.hpp is generated by the
custom command? Would it be better/simpler to do configure_file() at
configuration time instead of at build time as a custom command?
Here's the logic (some variables are defined elsewhere, but should
give you the gist):


set( generated_source_dir ${CMAKE_CURRENT_BINARY_DIR}/Source )
set( version_file_in  ${CMAKE_CURRENT_SOURCE_DIR}/Version.cpp.in )
set( version_file_out ${generated_source_dir}/Main/Version.cpp )

add_custom_command(
    OUTPUT ${version_file_out}
    COMMAND ${CMAKE_COMMAND}
        -D IN_FILE=${version_file_in}
        -D OUT_FILE=${version_file_out}
        -D GIT_EXECUTABLE=${GIT_EXECUTABLE}
        -D BUILD_VERSION=${ZIOSK_BUILD_VERSION}
        -P ${CMAKE_CURRENT_SOURCE_DIR}/build_version_header.cmake
    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)

source_group( Generated FILES ${version_file_out} )
list( APPEND source_files ${version_file_out} )
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake