Concerning IDE support for superbuilds

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Concerning IDE support for superbuilds

Kai Wolf
Hi all,

a common approach for dealing with larger dependencies in a project is using a superbuild approach, where all dependencies get downloaded in a separate repository using ExternalProject_Add() and the actual project is included as a sub repository (e.g. using git submodules) and also build as an external project [1], [2], [3]. This is necessary since external projects don’t play well as a direct dependency for internal targets, as the external projects won’t be created and available until build time, whereas CMake needs some information already at configure time for creating a proper build system.

As discussed previously on this mailing list this approach has some significant drawbacks: Since the actual project is also build as an external project, CMake treats it as an external source respectively and IDE support is not available. One possible solution to this problem is to let the outer project generate a cache file which subsequently can be used in the actual project to configure all paths to the previously downloaded and built external dependencies:

    +----------------+
    |   Superbuild   |  ----> generates init-cache.cmake
    +----------------+

    +----------------+
    | actual project |  ----> uses: cmake -C init-cache.cmake
    +----------------+

However, this means we now need two steps to configure and build the actual project and also cannot easily maintain the external dependencies within one large project.

After thinking about this for a while I came up with the following hackish solution and would like to ask you, if you see any drawback in using this. Basically all I’m doing is to include the actual project first as a dummy external project and set the external projects as dependencies, thus they’ll get built beforehand and in the BUILD_COMMAND of the actual project I’m calling CMake to the project’s CMakeLists.txt again to trigger a reconfigure, writing a cache file and set a specific variable |project_deps_installed| which is used as an indicator in the CMakeLists. configuration to include the actual project via add_subdirectory(). For instance:

    CMakeLists.txt:
    option(BUILD_MYPROJECT „Build the actual project“ ON)
    if(BUILD_MYPROJECT)
      if(project_deps_installed)
        include(${CMAKE_CURRENT_BINARY_DIR}/init-cache.cmake)
        add_subdirectory(MyProject)
      else()
        include(External_project)
      endif()
    endif()

    External_project.cmake
    ExternalProject_Add(MyProject
      SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/MyProject
      BUILD_COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_CURRENT_BINARY_DIR}
        ${CMAKE_COMMAND} ${CMAKE_CURRENT_SOURCE_DIR}
      INSTALL_COMMAND „"
      DEPENDS ${external_deps})

    write_cmake_cache_file(„${cache_script}“ „${cache_args}“)
    set(project_deps_installed TRUE CACHE BOOL „“)

As a result, all external dependencies get build and installed via ExternalProject_Add() and the actual project is configured properly and included via add_subdirectory(), thus it can continue using find_package() as usual and will be pointed to the previously configured dependencies. So far, I haven’t found any downside to this and consider making this part of CMake itself, in case you don’t see any problems using this approach?


Thanks in advance for your input.


[1] https://github.com/OpenChemistry/openchemistry
[2] https://github.com/MITK/MITK
[3] https://github.com/SuperElastix/SuperElastix


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Concerning IDE support for superbuilds

Craig Scott-3
Please excuse the brevity of this response, but a couple of links you may want to read which may be somewhat related or provide further ideas for you:

  • The new FetchContent module which is on master but not yet in any release. I expect it will be available in 3.11.0 when it comes out. This offers an alternative way of incorporating dependencies in a way which will make them all appear directly in an IDE just like the rest of the project.
  • Invoking a nested build to provide something that can be used immediately in the main CMake run is discussed in this stackoverflow question and answer (by me). Pay special attention to the comments in the Updated section about what can happen if a CMake run is aborted/interrupted, leaving the main build trapped by the sub-build.


On Sun, Jan 21, 2018 at 9:16 PM, Kai Wolf <[hidden email]> wrote:
Hi all,

a common approach for dealing with larger dependencies in a project is using a superbuild approach, where all dependencies get downloaded in a separate repository using ExternalProject_Add() and the actual project is included as a sub repository (e.g. using git submodules) and also build as an external project [1], [2], [3]. This is necessary since external projects don’t play well as a direct dependency for internal targets, as the external projects won’t be created and available until build time, whereas CMake needs some information already at configure time for creating a proper build system.

As discussed previously on this mailing list this approach has some significant drawbacks: Since the actual project is also build as an external project, CMake treats it as an external source respectively and IDE support is not available. One possible solution to this problem is to let the outer project generate a cache file which subsequently can be used in the actual project to configure all paths to the previously downloaded and built external dependencies:

    +----------------+
    |   Superbuild   |  ----> generates init-cache.cmake
    +----------------+

    +----------------+
    | actual project |  ----> uses: cmake -C init-cache.cmake
    +----------------+

However, this means we now need two steps to configure and build the actual project and also cannot easily maintain the external dependencies within one large project.

After thinking about this for a while I came up with the following hackish solution and would like to ask you, if you see any drawback in using this. Basically all I’m doing is to include the actual project first as a dummy external project and set the external projects as dependencies, thus they’ll get built beforehand and in the BUILD_COMMAND of the actual project I’m calling CMake to the project’s CMakeLists.txt again to trigger a reconfigure, writing a cache file and set a specific variable |project_deps_installed| which is used as an indicator in the CMakeLists. configuration to include the actual project via add_subdirectory(). For instance:

    CMakeLists.txt:
    option(BUILD_MYPROJECT „Build the actual project“ ON)
    if(BUILD_MYPROJECT)
      if(project_deps_installed)
        include(${CMAKE_CURRENT_BINARY_DIR}/init-cache.cmake)
        add_subdirectory(MyProject)
      else()
        include(External_project)
      endif()
    endif()

    External_project.cmake
    ExternalProject_Add(MyProject
      SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/MyProject
      BUILD_COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_CURRENT_BINARY_DIR}
        ${CMAKE_COMMAND} ${CMAKE_CURRENT_SOURCE_DIR}
      INSTALL_COMMAND „"
      DEPENDS ${external_deps})

    write_cmake_cache_file(„${cache_script}“ „${cache_args}“)
    set(project_deps_installed TRUE CACHE BOOL „“)

As a result, all external dependencies get build and installed via ExternalProject_Add() and the actual project is configured properly and included via add_subdirectory(), thus it can continue using find_package() as usual and will be pointed to the previously configured dependencies. So far, I haven’t found any downside to this and consider making this part of CMake itself, in case you don’t see any problems using this approach?


Thanks in advance for your input.


[1] https://github.com/OpenChemistry/openchemistry
[2] https://github.com/MITK/MITK
[3] https://github.com/SuperElastix/SuperElastix


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake




--
Craig Scott
Melbourne, Australia

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake