Building a repo with multiple applications and install process

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Building a repo with multiple applications and install process

David Jobet
Hello,

at work, we have a mono-repo with multiple applications/libs (dozens).
The build phase is ok, but I'm not sure about the release process.

When we release, we release one application at a time.
(CMAKE_SKIP_INSTALL_ALL_DEPENDENCY is true)
In order to speed up releases, we always perform an incremental build.

Unfortunately, we don't have one unique release process :
process 1 :
- a Jenkins pipeline executes some automatic tests then release the
binary to production. This Jenkins pipeline only builds this single
application, then executes the install step, then packages the binary
with some auxiliary files for deployment in prod.
process 2 :
- the whole source tree is built regularly through Jenkins, then, from
another Jenkins pipeline, an install step will be performed in the
last built directory to deploy only the required application

Both process 1 and process 2 are built in our CMakeLists.txt.

Process 1 just uses regular install directives + ninja install
Pros : simple
Cons : install step can be costly

Install step can be costly because, as the build directory is not
emptied, the install step will install every single binaries left over
from a previous build that have an install rule.
Also, we have install directives for non binary files (python files
for example) which will be installed unconditionally every time.

Process 2 is not triggered through the install step but as a regular
build target. Under the hood, the build step will add a POST_BUILD
step attached to the target that will invoke "cmake -P
${CMAKE_BINARY_DIR}/cmake_install.cmake -DCOMPONENT=${component}"
Pros : more "chirurgical", only install what's required
Cons : - if an application depends on several components, we need to
describe this in cmake (dependencies are described twice : once for
the build, once for the install)
       - need to maintain an extra "non standard" layer (albeit a small layer)

At this point, I'd like to ask if you see simple steps we could take
to stay as simple and standard as possible without paying the cons
(lenghty install step, double description of dependencies, extra layer
to maintain).

I have a proposal of my own, I'm just not sure this is technically
feasible. I will definitively run a POC sometime, but I thought I
would run it by you first to get your advice/experience.

So maybe the problem is we have one monolithic CMake system where all
apps are described.
What if every single application had its own independent CMake system
(while still being able to depend on common libs, that needs to be
built only once) ?
One app would describe its dependencies, the install step would then
stay simple, and only the reachable install directives would be
triggered in a per-app build.

Is it something which is feasible ? How would you implement it ? (I
thought about ExternalProject_Add but have no experience with it and
I'm not sure it would work in that case ?)

Also, we would still need to be able to build all applications in one
single command.
Do you think creating a "meta" cmake, equivalent to what we have right
now, but on top of those independent, per-app cmake, is feasible ?
(Again, I guess using ExternalProject_Add) ?

Thanks very much for your help

David
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Building a repo with multiple applications and install process

David Jobet
Hello,

so I didn't get a lot of answers, here's a quick POC which only uses
add_subdirectory() / include_guard() for more questions.

/CMakeLists.txt
/app1/CMakeLists.txt
/app2/CMakeLists.txt
/common/CMakeLists.txt

We have app1 depending on common, and app2 depending on common.
All 3 projects are living in the same mono-repo.

With those CMakeLists.txt, I'm able to build/install app1, app2, and
common as standalone projects with the following command line
e.g for app1 :
mkdir build_app1 && cd build_app1 && cmake -DCMAKE_INSTALL_PREFIX=
/path/to/app1 && make && DESTDIR=dist make install

I'm also able to build all apps at once using the following
mkdir build_all && cd build_all && cmake -DCMAKE_INSTALL_PREFIX=
/path/to/root && make && DESTDIR=dist make install

See below for the content of the files.

Now the question :
Do you think it would be possible to use the "build all" approach to
populate all .o, libs and executables, then to reconfigure the build
dir, say for app1 so I can then issue "make install" only for app1 ?
This would involve the "build all" project to have the same layout as
the "app1" project which currently is not the case.
In the "build all" project, app1 is a directory
In the "app1" project, app1 is a binary

Thanks for your help

David

---------------------------------------------------
/CMakeLists.txt
cmake_minimum_required(VERSION 3.10 FATAL_ERROR)

project(main-klib CXX)

add_subdirectory(app1)
add_subdirectory(app2)
add_subdirectory(common)

---------------------------------------------------
/app1/CMakeLists.txt
cmake_minimum_required(VERSION 3.10 FATAL_ERROR)
include_guard(GLOBAL)

project(app1 CXX)

add_subdirectory(../common common)

add_executable(app1 main.cc)
target_link_libraries(app1 common::lib)

install(
   TARGETS app1
   RUNTIME DESTINATION bin
)

---------------------------------------------------
/app2/CMakeLists.txt
cmake_minimum_required(VERSION 3.10 FATAL_ERROR)
include_guard(GLOBAL)

project(app2 CXX)

add_subdirectory(../common common)

add_executable(app2 main.cc)
target_link_libraries(app2 common::lib)

install(
   TARGETS app2
   RUNTIME DESTINATION bin
)

---------------------------------------------------
/common/CMakeLists.txt
cmake_minimum_required(VERSION 3.10 FATAL_ERROR)
include_guard(GLOBAL)

project(common CXX)

add_library(common common.cc)
add_library(common::lib ALIAS common)
target_include_directories(common PUBLIC ${PROJECT_SOURCE_DIR})

install(
   FILES common.py
   DESTINATION site-packages
)

On Tue, Feb 5, 2019 at 10:05 AM David Jobet <[hidden email]> wrote:

>
> Hello,
>
> at work, we have a mono-repo with multiple applications/libs (dozens).
> The build phase is ok, but I'm not sure about the release process.
>
> When we release, we release one application at a time.
> (CMAKE_SKIP_INSTALL_ALL_DEPENDENCY is true)
> In order to speed up releases, we always perform an incremental build.
>
> Unfortunately, we don't have one unique release process :
> process 1 :
> - a Jenkins pipeline executes some automatic tests then release the
> binary to production. This Jenkins pipeline only builds this single
> application, then executes the install step, then packages the binary
> with some auxiliary files for deployment in prod.
> process 2 :
> - the whole source tree is built regularly through Jenkins, then, from
> another Jenkins pipeline, an install step will be performed in the
> last built directory to deploy only the required application
>
> Both process 1 and process 2 are built in our CMakeLists.txt.
>
> Process 1 just uses regular install directives + ninja install
> Pros : simple
> Cons : install step can be costly
>
> Install step can be costly because, as the build directory is not
> emptied, the install step will install every single binaries left over
> from a previous build that have an install rule.
> Also, we have install directives for non binary files (python files
> for example) which will be installed unconditionally every time.
>
> Process 2 is not triggered through the install step but as a regular
> build target. Under the hood, the build step will add a POST_BUILD
> step attached to the target that will invoke "cmake -P
> ${CMAKE_BINARY_DIR}/cmake_install.cmake -DCOMPONENT=${component}"
> Pros : more "chirurgical", only install what's required
> Cons : - if an application depends on several components, we need to
> describe this in cmake (dependencies are described twice : once for
> the build, once for the install)
>        - need to maintain an extra "non standard" layer (albeit a small layer)
>
> At this point, I'd like to ask if you see simple steps we could take
> to stay as simple and standard as possible without paying the cons
> (lenghty install step, double description of dependencies, extra layer
> to maintain).
>
> I have a proposal of my own, I'm just not sure this is technically
> feasible. I will definitively run a POC sometime, but I thought I
> would run it by you first to get your advice/experience.
>
> So maybe the problem is we have one monolithic CMake system where all
> apps are described.
> What if every single application had its own independent CMake system
> (while still being able to depend on common libs, that needs to be
> built only once) ?
> One app would describe its dependencies, the install step would then
> stay simple, and only the reachable install directives would be
> triggered in a per-app build.
>
> Is it something which is feasible ? How would you implement it ? (I
> thought about ExternalProject_Add but have no experience with it and
> I'm not sure it would work in that case ?)
>
> Also, we would still need to be able to build all applications in one
> single command.
> Do you think creating a "meta" cmake, equivalent to what we have right
> now, but on top of those independent, per-app cmake, is feasible ?
> (Again, I guess using ExternalProject_Add) ?
>
> Thanks very much for your help
>
> David
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake