ExternalProject, BundleUtilities, CPack, and RPath management

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

ExternalProject, BundleUtilities, CPack, and RPath management

Clifford Yapp
I am contemplating attempting to use ExternalProject_Add and
fixup_bundle to improve the long term maintainability of a large
software product, but there are some implications that are potentially
involved and I'd like to ask if anyone in the CMake community can
offer insight.  The design constraints to satisfy (that are satisfied
by our current "build everything wth CMake" approach):

1.  We need to be able to install our bundled dependencies seamlessly
with the rest of the project when doing either a "make install" or
"make package" - from the standpoint of the main software project, the
external projects' outputs are handled exactly as if they were created
by the "main" build (right now that's because they *are* part of the
main build, but it's a property we want to preserve regardless.)

2.  Our "build" directory is laid out such that it mimics the
installed layout (including any external dependencies we will be
bundling) and CMake's RPath management settings are defined such that
the build directory is fully functional without requiring any
installation steps, LD_LIBRARY_PATH fiddling, or other user
intervention.  Except for the presence of the ancillary files from
configuration and building, structurally the build directory could be
an install directory.

3.  At no point in the configure, build, or package creation process
should it be necessary to place anything in the final
CMAKE_INSTALL_PREFIX install location - someone without access to the
CMAKE_INSTALL_PREFIX directory should still be able to create a
proper, working package.

4.  Works seamlessly across Windows+MSVC, Linux, MacOSX and the BSDs.
(In years past this was a challenge with ExternalProject_Add and
Windows - trying to mix an nmake external build with an MSVC parent
build on Windows didn't work, if I recall correctly.  Hopefully things
are better nowadays...)

Right now we are achieving these goals by creating CMake build files
for all projects we need to manage as bundled dependencies, but
writing those build systems when they don't already exist is a lot of
work and doesn't scale long term (for example, not even I am crazy
enough to want to write and maintain a separate CMake build for all of
Qt).  I would like to be able to define an "external dependencies"
repository with a build system which simply encodes the necessary
rules to trigger the native build logic from the projects in question,
builds a "deps_install" directory that the primary project's CMake
build can be pointed to for build purposes, and then use the primary
project's RPath settings from then on to manage the deps_install
executable and library files as if they were outputs from build
targets in the primary project.

It feels like some combination of CPack, ExternalProject and
BundleUtilties should be able to make this work as long as the
deps_install binaries have RPaths defined that are suitable for
manipulation by CMake, but unless I'm missing something (hopefully I
am) this looks like a sufficiently obscure corner of the CMake toolbox
that there isn't a lot to be found documenting the nitty-gritty of how
to do it.  Can anyone direct me to a tutorial or a project that
demonstrates how to make the combination of
ExternalProject_Add+BundleUtilities+CPack work in a cross platform
manner?

Thanks,
CY
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: ExternalProject, BundleUtilities, CPack, and RPath management

Jean-Christophe Fillion-Robin-2
Hi Clifford,

You could have a look at the Slicer build system, see https://github.com/Slicer/Slicer. It is based on ExternalProject.


* It uses BundleUtilities only on macos but the caveat of BundleUtilties is that it is very very slow for large projects (indeed it traverses the full set of dependencies for each plugin, apps, ...). At kitware, we internally discussed implementing a C++ based version of bundle utilities but this is not yet available.

There are many more details.

If you have specific questions, you could join the Slicer developer hangout that take place every Tuesday at 10am EST. It is announced on discourse. See https://discourse.slicer.org/c/community/hangout

Hth
Jc


On Sat, Oct 13, 2018 at 5:05 PM Clifford Yapp <[hidden email]> wrote:
I am contemplating attempting to use ExternalProject_Add and
fixup_bundle to improve the long term maintainability of a large
software product, but there are some implications that are potentially
involved and I'd like to ask if anyone in the CMake community can
offer insight.  The design constraints to satisfy (that are satisfied
by our current "build everything wth CMake" approach):

1.  We need to be able to install our bundled dependencies seamlessly
with the rest of the project when doing either a "make install" or
"make package" - from the standpoint of the main software project, the
external projects' outputs are handled exactly as if they were created
by the "main" build (right now that's because they *are* part of the
main build, but it's a property we want to preserve regardless.)

2.  Our "build" directory is laid out such that it mimics the
installed layout (including any external dependencies we will be
bundling) and CMake's RPath management settings are defined such that
the build directory is fully functional without requiring any
installation steps, LD_LIBRARY_PATH fiddling, or other user
intervention.  Except for the presence of the ancillary files from
configuration and building, structurally the build directory could be
an install directory.

3.  At no point in the configure, build, or package creation process
should it be necessary to place anything in the final
CMAKE_INSTALL_PREFIX install location - someone without access to the
CMAKE_INSTALL_PREFIX directory should still be able to create a
proper, working package.

4.  Works seamlessly across Windows+MSVC, Linux, MacOSX and the BSDs.
(In years past this was a challenge with ExternalProject_Add and
Windows - trying to mix an nmake external build with an MSVC parent
build on Windows didn't work, if I recall correctly.  Hopefully things
are better nowadays...)

Right now we are achieving these goals by creating CMake build files
for all projects we need to manage as bundled dependencies, but
writing those build systems when they don't already exist is a lot of
work and doesn't scale long term (for example, not even I am crazy
enough to want to write and maintain a separate CMake build for all of
Qt).  I would like to be able to define an "external dependencies"
repository with a build system which simply encodes the necessary
rules to trigger the native build logic from the projects in question,
builds a "deps_install" directory that the primary project's CMake
build can be pointed to for build purposes, and then use the primary
project's RPath settings from then on to manage the deps_install
executable and library files as if they were outputs from build
targets in the primary project.

It feels like some combination of CPack, ExternalProject and
BundleUtilties should be able to make this work as long as the
deps_install binaries have RPaths defined that are suitable for
manipulation by CMake, but unless I'm missing something (hopefully I
am) this looks like a sufficiently obscure corner of the CMake toolbox
that there isn't a lot to be found documenting the nitty-gritty of how
to do it.  Can anyone direct me to a tutorial or a project that
demonstrates how to make the combination of
ExternalProject_Add+BundleUtilities+CPack work in a cross platform
manner?

Thanks,
CY
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake