requiring parallel building but sequential linking

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

requiring parallel building but sequential linking

Domen Vrankar
Hi,

I'm building llvm project with CMake.
While build process is compiling the code I prefer running "make -j3" on my 4 core pc (bumping processor to 100% on 3 of 4 cores and using up ~5 GB of ram - part of it is system and not build related).
While build process is linking I must run "make" with a single job because otherwise I run out of memory (cores do little work but ram usage goes past 23 GB).

Currently I'm handling this so that I first run "make -j3" and once I get to about 90% (first larger linking) I hit ctrl+c and run "make".

Is there a feature in CMake that would support handling this transition of parallel builds, sequential linking out of the box?
(while I'm mostly interested in Linux/make support, having the same feature for Windows/nmake/ninja through the same command line cmake command would be even better)

Thanks,
Domen

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: requiring parallel building but sequential linking

Craig Scott-3


On Mon, Feb 18, 2019 at 9:35 AM Domen Vrankar <[hidden email]> wrote:
Hi,

I'm building llvm project with CMake.
While build process is compiling the code I prefer running "make -j3" on my 4 core pc (bumping processor to 100% on 3 of 4 cores and using up ~5 GB of ram - part of it is system and not build related).
While build process is linking I must run "make" with a single job because otherwise I run out of memory (cores do little work but ram usage goes past 23 GB).

Currently I'm handling this so that I first run "make -j3" and once I get to about 90% (first larger linking) I hit ctrl+c and run "make".

Is there a feature in CMake that would support handling this transition of parallel builds, sequential linking out of the box?
(while I'm mostly interested in Linux/make support, having the same feature for Windows/nmake/ninja through the same command line cmake command would be even better) 

This is available for Ninja with the JOB_POOL_LINK target property, the default for which can be set project-wide with the CMAKE_JOB_POOL_LINK variable. You can control the number of parallel jobs for a given pool with the JOB_POOLS global property.


--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: requiring parallel building but sequential linking

Eric Noulard


Le lun. 18 févr. 2019 à 00:01, Craig Scott <[hidden email]> a écrit :


On Mon, Feb 18, 2019 at 9:35 AM Domen Vrankar <[hidden email]> wrote:
Hi,

I'm building llvm project with CMake.
While build process is compiling the code I prefer running "make -j3" on my 4 core pc (bumping processor to 100% on 3 of 4 cores and using up ~5 GB of ram - part of it is system and not build related).
While build process is linking I must run "make" with a single job because otherwise I run out of memory (cores do little work but ram usage goes past 23 GB).

Currently I'm handling this so that I first run "make -j3" and once I get to about 90% (first larger linking) I hit ctrl+c and run "make".

Is there a feature in CMake that would support handling this transition of parallel builds, sequential linking out of the box?
(while I'm mostly interested in Linux/make support, having the same feature for Windows/nmake/ninja through the same command line cmake command would be even better) 

This is available for Ninja with the JOB_POOL_LINK target property, the default for which can be set project-wide with the CMAKE_JOB_POOL_LINK variable. You can control the number of parallel jobs for a given pool with the JOB_POOLS global property.

I use that a lot and I shall add that, on my side, link jobs memory consumption heavily depends on the type of build (debug, release, profiling, etc...), moreover 
the available memory amount vary a lot as well whether if the build occurs on a "local" developer desktop or on some CI runner.
So we end up doing some very basic CMake computation in order to automatically adapt to the local resource.
Nothing fancy but it has been proven very useful for us:

if (CMAKE_BUILD_TYPE STREQUAL "Debug")
   cmake_host_system_information(RESULT MYMEM QUERY TOTAL_PHYSICAL_MEMORY)
   # Compute the number of authorized number of link jobs by:
   #   - "saving" 4 GiB of memory
   #   - assume each debug link job may consume 2GiB
   math(EXPR NLJ "(${MYMEM}-4096)/2048")
   set_property(GLOBAL PROPERTY JOB_POOLS link_jobs=${NLJ})
   set(CMAKE_JOB_POOL_LINK link_jobs)
elseif (CMAKE_BUILD_TYPE STREQUAL "Release")
...
endif()

--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake