CMake with FetchContent instead of Git Submodules

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

CMake with FetchContent instead of Git Submodules

Dustyn Blasig
Hi All,

I'm attempting to replace our use of git submodules with FetchContent flows instead so we can pull pre-built packages if they already exist instead of buildings locally. 

However, I need to support a flow similar to Git submodules where developers can edit a submodule and then rebuild the enclosing project incrementally. Things seem to work OK if I jump into the fetched source directories and check out branches, etc. However, even if I move the download and source/unpacked folders outside the build (binary) directory and then delete the build directory, rerunning CMake will blast away the unpacked source even if the last extraction was with the same checksummed download. I need to ensure a "clean" won't delete someones work in a submodule, which is one reason why Git makes it hard to uninit and remove submodules.

Is there a best practice for a flow like this that I can replicate?

Thanks!

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake with FetchContent instead of Git Submodules

Craig Scott-3


On Fri, Jun 28, 2019 at 12:18 PM Dustyn Blasig <[hidden email]> wrote:
Hi All,

I'm attempting to replace our use of git submodules with FetchContent flows instead so we can pull pre-built packages if they already exist instead of buildings locally. 

However, I need to support a flow similar to Git submodules where developers can edit a submodule and then rebuild the enclosing project incrementally. Things seem to work OK if I jump into the fetched source directories and check out branches, etc. However, even if I move the download and source/unpacked folders outside the build (binary) directory and then delete the build directory, rerunning CMake will blast away the unpacked source even if the last extraction was with the same checksummed download. I need to ensure a "clean" won't delete someones work in a submodule, which is one reason why Git makes it hard to uninit and remove submodules.

Is there a best practice for a flow like this that I can replicate?

(Background info: I'm the creator of the FetchContent module)

If you want to be making changes to the sources, you should clone that repo manually and point the build at it rather than try to modify the one that the project downloads for you. You do this by setting the FETCHCONTENT_SOURCE_DIR_<someName> cache variable to override where the build should find the sources. Here's the relevant part from the FetchContent documentation:

FETCHCONTENT_SOURCE_DIR_<ucName>
If this is set, no download or update steps are performed for the specified content and the <lcName>_SOURCE_DIR variable returned to the caller is pointed at this location. This gives developers a way to have a separate checkout of the content that they can modify freely without interference from the build. The build simply uses that existing source, but it still defines <lcName>_BINARY_DIR to point inside its own build area. Developers are strongly encouraged to use this mechanism rather than editing the sources populated in the default location, as changes to sources in the default location can be lost when content population details are changed by the project. 

The thinking behind this is that the project will assume it is in control of the sources unless told otherwise. By setting the above cache variable, you are telling FetchContent that "I want to use my own sources instead of whatever you would normally use". Either FetchContent is in control or you are.

I use this feature a LOT. Let's say you're working on a project where you need to do some refactoring that requires changes in the top level project and in some of its dependencies. I will have the top level project cloned. I will also have separate manually cloned repos for those dependencies that I need to modify. When I run CMake on my top level project, I use FETCHCONTENT_SOURCE_DIR_<depName> to point the build at my local cloned repos. That makes the build use my local clones so I can modify the sources, change branches, etc. without fear of things being changed under my feet. When I'm done, I'll commit and push my changes in each of the local cloned repos, then I'll update the GIT_HASH details of those dependencies in my top level project. I delete the FETCHCONTENT_SOURCE_DIR_<...> variables from my CMake cache and re-run CMake (and build) to confirm that I've updated my dependency hashes correctly, then I commit and push my changes to the top level project.

I use this strategy with project hierarchies with 40+ dependencies that can be up to maybe 10 levels deep. I can pull out any dependency used anywhere in the hierarchy and work with my own local cloned repo without having to care about where in the project hierarchy that dependency is usually populated. Being able to easily and selectively switch between the project-controlled FetchContent-provided source or my own local cloned repo(s) is critical to being able to do this efficiently. If you look at the CMake cache variables in ccmake or cmake-gui, you will also see all the source directory overrides grouped together because the variable names all start with FETCHCONTENT_SOURCE_DIR (this is why I named the cache variables FetchContent creates this way instead of putting the dependency name at the front of the cache variable name). So you can quickly see for which dependencies you are currently using a local cloned repo instead of what the project normally uses.


--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake with FetchContent instead of Git Submodules

Dustyn Blasig
Wow, how did I miss that when I was looking through the page yesterday? That's exactly what I need ; )

I may try going one step further and adding a flag <COMPONENT>_USE_CLONE=ON that will do the full checkout to a repos directory, initialize the repo to the target tag, and then set FETCHCONTENT_SOURCE_DIR_<COMPONENT> for the user. The former would only be done iff the repo doesn't already exist.

On Fri, Jun 28, 2019 at 5:25 AM Craig Scott <[hidden email]> wrote:


On Fri, Jun 28, 2019 at 12:18 PM Dustyn Blasig <[hidden email]> wrote:
Hi All,

I'm attempting to replace our use of git submodules with FetchContent flows instead so we can pull pre-built packages if they already exist instead of buildings locally. 

However, I need to support a flow similar to Git submodules where developers can edit a submodule and then rebuild the enclosing project incrementally. Things seem to work OK if I jump into the fetched source directories and check out branches, etc. However, even if I move the download and source/unpacked folders outside the build (binary) directory and then delete the build directory, rerunning CMake will blast away the unpacked source even if the last extraction was with the same checksummed download. I need to ensure a "clean" won't delete someones work in a submodule, which is one reason why Git makes it hard to uninit and remove submodules.

Is there a best practice for a flow like this that I can replicate?

(Background info: I'm the creator of the FetchContent module)

If you want to be making changes to the sources, you should clone that repo manually and point the build at it rather than try to modify the one that the project downloads for you. You do this by setting the FETCHCONTENT_SOURCE_DIR_<someName> cache variable to override where the build should find the sources. Here's the relevant part from the FetchContent documentation:

FETCHCONTENT_SOURCE_DIR_<ucName>
If this is set, no download or update steps are performed for the specified content and the <lcName>_SOURCE_DIR variable returned to the caller is pointed at this location. This gives developers a way to have a separate checkout of the content that they can modify freely without interference from the build. The build simply uses that existing source, but it still defines <lcName>_BINARY_DIR to point inside its own build area. Developers are strongly encouraged to use this mechanism rather than editing the sources populated in the default location, as changes to sources in the default location can be lost when content population details are changed by the project. 

The thinking behind this is that the project will assume it is in control of the sources unless told otherwise. By setting the above cache variable, you are telling FetchContent that "I want to use my own sources instead of whatever you would normally use". Either FetchContent is in control or you are.

I use this feature a LOT. Let's say you're working on a project where you need to do some refactoring that requires changes in the top level project and in some of its dependencies. I will have the top level project cloned. I will also have separate manually cloned repos for those dependencies that I need to modify. When I run CMake on my top level project, I use FETCHCONTENT_SOURCE_DIR_<depName> to point the build at my local cloned repos. That makes the build use my local clones so I can modify the sources, change branches, etc. without fear of things being changed under my feet. When I'm done, I'll commit and push my changes in each of the local cloned repos, then I'll update the GIT_HASH details of those dependencies in my top level project. I delete the FETCHCONTENT_SOURCE_DIR_<...> variables from my CMake cache and re-run CMake (and build) to confirm that I've updated my dependency hashes correctly, then I commit and push my changes to the top level project.

I use this strategy with project hierarchies with 40+ dependencies that can be up to maybe 10 levels deep. I can pull out any dependency used anywhere in the hierarchy and work with my own local cloned repo without having to care about where in the project hierarchy that dependency is usually populated. Being able to easily and selectively switch between the project-controlled FetchContent-provided source or my own local cloned repo(s) is critical to being able to do this efficiently. If you look at the CMake cache variables in ccmake or cmake-gui, you will also see all the source directory overrides grouped together because the variable names all start with FETCHCONTENT_SOURCE_DIR (this is why I named the cache variables FetchContent creates this way instead of putting the dependency name at the front of the cache variable name). So you can quickly see for which dependencies you are currently using a local cloned repo instead of what the project normally uses.


--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake