Parallel build setup question

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel build setup question

Cornelis Bockemühl
Hello,

Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target!

My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration.

I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good!

But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message.

Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here.

And now I am stuck - if not some friendly and knowing person can help me out of this!

Thanks for any helpful hint!
Cornelis

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build setup question

Michael Ellery
I don’t know anything about QTCreator, but the question really boils down to how to get your IDE to pass extra args to the build tool - it’s not really a cmake issue except to the extent that CMake can be used to invoke the build step (after it has been used to generate the build).  It sounds like QT uses “cmake --build” to do the build phase (which makes sense), so you just need to add your “-j N” argument to the build step invocation. If you believe this:  https://doc.qt.io/qtcreator/creator-build-settings.html, then it sounds like maybe “Tool Arguments” is what you want (see “CMake Build Steps”) ?  BTW, the way I do this myself (since I don’t use an IDE) is something like “cmake --build . -- -j 8”  where the double dash stops shell argument processing and effectively passes everything else on to the build tool that cmake ultimately invokes (e.g. make or ninja or msbuild, etc.)

HTH,
Mike

> On Feb 16, 2018, at 8:31 AM, Cornelis Bockemühl <[hidden email]> wrote:
>
> Hello,
>
> Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target!
>
> My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration.
>
> I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good!
>
> But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message.
>
> Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here.
>
> And now I am stuck - if not some friendly and knowing person can help me out of this!
>
> Thanks for any helpful hint!
> Cornelis
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build setup question

Cornelis Bockemühl
Thanks for your hints! And you are right: it is still interesting to know how QtCreator is actually invoking and using cmake.

This is the beginning of a log that the QtCreator was generating during a build. Actually with two modules where nothing had to be done, but still showing how cmake and gmake are being invoked - and all a bit above my understanding of the two tools:

===================================
/usr/bin/cmake -H/Sources/GitLinux/RMT -B/BinariesLinux/RMT/Debug --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /BinariesLinux/RMT/Debug/CMakeFiles /BinariesLinux/RMT/Debug/CMakeFiles/progress.marks
/usr/bin/gmake -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/depend
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/FirstModule.dir/DependInfo.cmake --color=
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/build
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
gmake[2]: Nothing to be done for 'CMakeFiles/FirstModule.dir/build'.
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
[  3%] Built target FirstModule
/usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/depend
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/SecondModule.dir/DependInfo.cmake --color=
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/build
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
gmake[2]: Nothing to be done for 'CMakeFiles/SecondModule.dir/build'.
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
[  6%] Built target SecondModule
...
===========================================

For example I do not really understand all the parameters that are passed to cmake - and I do not find them in the documentation either (like -H, -B or --check-build-system...

Regards,
Cornelis

Am Freitag, den 16.02.2018, 08:59 -0800 schrieb Michael Ellery:
I don’t know anything about QTCreator, but the question really boils down to how to get your IDE to pass extra args to the build tool - it’s not really a cmake issue except to the extent that CMake can be used to invoke the build step (after it has been used to generate the build).  It sounds like QT uses “cmake --build” to do the build phase (which makes sense), so you just need to add your “-j N” argument to the build step invocation. If you believe this:  https://doc.qt.io/qtcreator/creator-build-settings.html, then it sounds like maybe “Tool Arguments” is what you want (see “CMake Build Steps”) ?  BTW, the way I do this myself (since I don’t use an IDE) is something like “cmake --build . -- -j 8”  where the double dash stops shell argument processing and effectively passes everything else on to the build tool that cmake ultimately invokes (e.g. make or ninja or msbuild, etc.)

HTH,
Mike

On Feb 16, 2018, at 8:31 AM, Cornelis Bockemühl <[hidden email]> wrote: Hello, Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target! My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration. I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good! But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message. Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here. And now I am stuck - if not some friendly and knowing person can help me out of this! Thanks for any helpful hint! Cornelis -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build setup question

Konstantin Tokarev
 
 
16.02.2018, 20:20, "Cornelis Bockemühl" <[hidden email]>:
Thanks for your hints! And you are right: it is still interesting to know how QtCreator is actually invoking and using cmake.

You should go to project settings and change cmake --build arguments there
 
 
This is the beginning of a log that the QtCreator was generating during a build. Actually with two modules where nothing had to be done, but still showing how cmake and gmake are being invoked - and all a bit above my understanding of the two tools:
 
===================================
/usr/bin/cmake -H/Sources/GitLinux/RMT -B/BinariesLinux/RMT/Debug --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /BinariesLinux/RMT/Debug/CMakeFiles /BinariesLinux/RMT/Debug/CMakeFiles/progress.marks
/usr/bin/gmake -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/depend
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/FirstModule.dir/DependInfo.cmake --color=
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/build
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
gmake[2]: Nothing to be done for 'CMakeFiles/FirstModule.dir/build'.
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
[  3%] Built target FirstModule
/usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/depend
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/SecondModule.dir/DependInfo.cmake --color=
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/build
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
gmake[2]: Nothing to be done for 'CMakeFiles/SecondModule.dir/build'.
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
[  6%] Built target SecondModule
...
===========================================
 
For example I do not really understand all the parameters that are passed to cmake - and I do not find them in the documentation either (like -H, -B or --check-build-system...
 
Regards,
Cornelis
 
Am Freitag, den 16.02.2018, 08:59 -0800 schrieb Michael Ellery:
I don’t know anything about QTCreator, but the question really boils down to how to get your IDE to pass extra args to the build tool - it’s not really a cmake issue except to the extent that CMake can be used to invoke the build step (after it has been used to generate the build).  It sounds like QT uses “cmake --build” to do the build phase (which makes sense), so you just need to add your “-j N” argument to the build step invocation. If you believe this:  https://doc.qt.io/qtcreator/creator-build-settings.html, then it sounds like maybe “Tool Arguments” is what you want (see “CMake Build Steps”) ?  BTW, the way I do this myself (since I don’t use an IDE) is something like “cmake --build . -- -j 8”  where the double dash stops shell argument processing and effectively passes everything else on to the build tool that cmake ultimately invokes (e.g. make or ninja or msbuild, etc.)

HTH,
Mike


On Feb 16, 2018, at 8:31 AM, Cornelis Bockemühl <[hidden email]> wrote:

Hello,

Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target!

My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration.

I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good!

But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message.

Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here.

And now I am stuck - if not some friendly and knowing person can help me out of this!

Thanks for any helpful hint!
Cornelis
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


,--


Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

 
 
-- 
Regards,
Konstantin
 

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build setup question

Cornelis Bockemühl
Thanks - that did the trick!

And indeed by entering -j4 in the "Tool arguments" field, the build command changes "magically" into "cmake --build . --target all -- -j4" - with the "--" to pass the argument further to gmake.

Actually compile time of my project after a "clean" went down from 23 to 14 min, thus reduction factor 1.6 with 4 processors: I tend to attribute the diffference to "almost 4" to the nature of my project where gmake seems not to see too many opportunities for parallel processing. Maybe it works always module by module, or whatever.

Anyway, my question is answered - thanks again!

Regards,
Cornelis

Am Freitag, den 16.02.2018, 20:25 +0300 schrieb Konstantin Tokarev:
 
 
16.02.2018, 20:20, "Cornelis Bockemühl" <[hidden email]>:
Thanks for your hints! And you are right: it is still interesting to know how QtCreator is actually invoking and using cmake.

You should go to project settings and change cmake --build arguments there
 
 
This is the beginning of a log that the QtCreator was generating during a build. Actually with two modules where nothing had to be done, but still showing how cmake and gmake are being invoked - and all a bit above my understanding of the two tools:
 
===================================
/usr/bin/cmake -H/Sources/GitLinux/RMT -B/BinariesLinux/RMT/Debug --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /BinariesLinux/RMT/Debug/CMakeFiles /BinariesLinux/RMT/Debug/CMakeFiles/progress.marks
/usr/bin/gmake -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/depend
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/FirstModule.dir/DependInfo.cmake --color=
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/build
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
gmake[2]: Nothing to be done for 'CMakeFiles/FirstModule.dir/build'.
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
[  3%] Built target FirstModule
/usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/depend
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/SecondModule.dir/DependInfo.cmake --color=
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
/usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/build
gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
gmake[2]: Nothing to be done for 'CMakeFiles/SecondModule.dir/build'.
gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
[  6%] Built target SecondModule
...
===========================================
 
For example I do not really understand all the parameters that are passed to cmake - and I do not find them in the documentation either (like -H, -B or --check-build-system...
 
Regards,
Cornelis
 
Am Freitag, den 16.02.2018, 08:59 -0800 schrieb Michael Ellery:
I don’t know anything about QTCreator, but the question really boils down to how to get your IDE to pass extra args to the build tool - it’s not really a cmake issue except to the extent that CMake can be used to invoke the build step (after it has been used to generate the build).  It sounds like QT uses “cmake --build” to do the build phase (which makes sense), so you just need to add your “-j N” argument to the build step invocation. If you believe this:  https://doc.qt.io/qtcreator/creator-build-settings.html, then it sounds like maybe “Tool Arguments” is what you want (see “CMake Build Steps”) ?  BTW, the way I do this myself (since I don’t use an IDE) is something like “cmake --build . -- -j 8”  where the double dash stops shell argument processing and effectively passes everything else on to the build tool that cmake ultimately invokes (e.g. make or ninja or msbuild, etc.)

HTH,
Mike


On Feb 16, 2018, at 8:31 AM, Cornelis Bockemühl <[hidden email]> wrote:

Hello,

Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target!

My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration.

I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good!

But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message.

Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here.

And now I am stuck - if not some friendly and knowing person can help me out of this!

Thanks for any helpful hint!
Cornelis

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build setup question

Konstantin Tokarev


16.02.2018, 21:32, "Cornelis Bockemühl" <[hidden email]>:
> Thanks - that did the trick!
>
> And indeed by entering -j4 in the "Tool arguments" field, the build command changes "magically" into "cmake --build . --target all -- -j4" - with the "--" to pass the argument further to gmake.
>
> Actually compile time of my project after a "clean" went down from 23 to 14 min, thus reduction factor 1.6 with 4 processors: I tend to attribute the diffference to "almost 4" to the nature of my project where gmake seems not to see too many opportunities for parallel processing. Maybe it works always module by module, or whatever.
>
> Anyway, my question is answered - thanks again!

Note that if you used Ninja generator, it would use all cores by default.

>
> Regards,
> Cornelis
>
> Am Freitag, den 16.02.2018, 20:25 +0300 schrieb Konstantin Tokarev:
>> 16.02.2018, 20:20, "Cornelis Bockemühl" <[hidden email]>:
>>> Thanks for your hints! And you are right: it is still interesting to know how QtCreator is actually invoking and using cmake.
>>
>> You should go to project settings and change cmake --build arguments there
>>
>>> This is the beginning of a log that the QtCreator was generating during a build. Actually with two modules where nothing had to be done, but still showing how cmake and gmake are being invoked - and all a bit above my understanding of the two tools:
>>>
>>> ===================================
>>> /usr/bin/cmake -H/Sources/GitLinux/RMT -B/BinariesLinux/RMT/Debug --check-build-system CMakeFiles/Makefile.cmake 0
>>> /usr/bin/cmake -E cmake_progress_start /BinariesLinux/RMT/Debug/CMakeFiles /BinariesLinux/RMT/Debug/CMakeFiles/progress.marks
>>> /usr/bin/gmake -f CMakeFiles/Makefile2 all
>>> gmake[1]: Entering directory '/BinariesLinux/RMT/Debug'
>>> /usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/depend
>>> gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
>>> cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/FirstModule.dir/DependInfo.cmake --color=
>>> gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
>>> /usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/build
>>> gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
>>> gmake[2]: Nothing to be done for 'CMakeFiles/FirstModule.dir/build'.
>>> gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
>>> [  3%] Built target FirstModule
>>> /usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/depend
>>> gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
>>> cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/SecondModule.dir/DependInfo.cmake --color=
>>> gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
>>> /usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/build
>>> gmake[2]: Entering directory '/BinariesLinux/RMT/Debug'
>>> gmake[2]: Nothing to be done for 'CMakeFiles/SecondModule.dir/build'.
>>> gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug'
>>> [  6%] Built target SecondModule
>>> ...
>>> ===========================================
>>>
>>> For example I do not really understand all the parameters that are passed to cmake - and I do not find them in the documentation either (like -H, -B or --check-build-system...
>>>
>>> Regards,
>>> Cornelis
>>>
>>> Am Freitag, den 16.02.2018, 08:59 -0800 schrieb Michael Ellery:
>>>> I don’t know anything about QTCreator, but the question really boils down to how to get your IDE to pass extra args to the build tool - it’s not really a cmake issue except to the extent that CMake can be used to invoke the build step (after it has been used to generate the build). It sounds like QT uses “cmake --build” to do the build phase (which makes sense), so you just need to add your “-j N” argument to the build step invocation. If you believe this: https://doc.qt.io/qtcreator/creator-build-settings.html, then it sounds like maybe “Tool Arguments” is what you want (see “CMake Build Steps”) ? BTW, the way I do this myself (since I don’t use an IDE) is something like “cmake --build . -- -j 8” where the double dash stops shell argument processing and effectively passes everything else on to the build tool that cmake ultimately invokes (e.g. make or ninja or msbuild, etc.) HTH, Mike On Feb 16, 2018, at 8:31 AM, Cornelis Bockemühl <[hidden email]> wrote: Hello, Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target! My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration. I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good! But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message. Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here. And now I am stuck - if not some friendly and knowing person can help me out of this! Thanks for any helpful hint! Cornelis


-- 
Regards,
Konstantin
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build setup question

Cornelis Bockemühl
Thanks also for this hint: I will give it a try!

Regards, Cornelis

Am Freitag, den 16.02.2018, 21:35 +0300 schrieb Konstantin Tokarev:
16.02.2018, 21:32, "Cornelis Bockemühl" <[hidden email]>:
Thanks - that did the trick! And indeed by entering -j4 in the "Tool arguments" field, the build command changes "magically" into "cmake --build . --target all -- -j4" - with the "--" to pass the argument further to gmake. Actually compile time of my project after a "clean" went down from 23 to 14 min, thus reduction factor 1.6 with 4 processors: I tend to attribute the diffference to "almost 4" to the nature of my project where gmake seems not to see too many opportunities for parallel processing. Maybe it works always module by module, or whatever. Anyway, my question is answered - thanks again!
Note that if you used Ninja generator, it would use all cores by default.
Regards, Cornelis Am Freitag, den 16.02.2018, 20:25 +0300 schrieb Konstantin Tokarev:
16.02.2018, 20:20, "Cornelis Bockemühl" <[hidden email]>:
Thanks for your hints! And you are right: it is still interesting to know how QtCreator is actually invoking and using cmake.
You should go to project settings and change cmake --build arguments there
This is the beginning of a log that the QtCreator was generating during a build. Actually with two modules where nothing had to be done, but still showing how cmake and gmake are being invoked - and all a bit above my understanding of the two tools: =================================== /usr/bin/cmake -H/Sources/GitLinux/RMT -B/BinariesLinux/RMT/Debug --check-build-system CMakeFiles/Makefile.cmake 0 /usr/bin/cmake -E cmake_progress_start /BinariesLinux/RMT/Debug/CMakeFiles /BinariesLinux/RMT/Debug/CMakeFiles/progress.marks /usr/bin/gmake -f CMakeFiles/Makefile2 all gmake[1]: Entering directory '/BinariesLinux/RMT/Debug' /usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/depend gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/FirstModule.dir/DependInfo.cmake --color= gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' /usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/build gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' gmake[2]: Nothing to be done for 'CMakeFiles/FirstModule.dir/build'. gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' [  3%] Built target FirstModule /usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/depend gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/SecondModule.dir/DependInfo.cmake --color= gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' /usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/build gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' gmake[2]: Nothing to be done for 'CMakeFiles/SecondModule.dir/build'. gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' [  6%] Built target SecondModule ... =========================================== For example I do not really understand all the parameters that are passed to cmake - and I do not find them in the documentation either (like -H, -B or --check-build-system... Regards, Cornelis Am Freitag, den 16.02.2018, 08:59 -0800 schrieb Michael Ellery:
I don’t know anything about QTCreator, but the question really boils down to how to get your IDE to pass extra args to the build tool - it’s not really a cmake issue except to the extent that CMake can be used to invoke the build step (after it has been used to generate the build). It sounds like QT uses “cmake --build” to do the build phase (which makes sense), so you just need to add your “-j N” argument to the build step invocation. If you believe this: https://doc.qt.io/qtcreator/creator-build-settings.html, then it sounds like maybe “Tool Arguments” is what you want (see “CMake Build Steps”) ? BTW, the way I do this myself (since I don’t use an IDE) is something like “cmake --build . -- -j 8” where the double dash stops shell argument processing and effectively passes everything else on to the build tool that cmake ultimately invokes (e.g. make or ninja or msbuild, etc.) HTH, Mike On Feb 16, 2018, at 8:31 AM, Cornelis Bockemühl <[hidden email]> wrote: Hello, Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target! My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration. I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good! But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message. Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here. And now I am stuck - if not some friendly and knowing person can help me out of this! Thanks for any helpful hint! Cornelis
--  Regards, Konstantin

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build setup question

Cornelis Bockemühl
Just to close this thread: Also the ninja tipp turned out to be a very good one! I did the same full built (clear - rebuild) with ninja instead of make -j4 and times were more or less comparable: from initially 23 minutes down to 13-14 with make, and 12-13 with ninja. However, what is much more helpful in practical programming work is the fact that if I change only one or two files, ninja "knows" this instantaneously, i.e. no perceivable overhead to find out about outdated files, while make still takes maybe half a minute or such for checking.

And since this is a very common case during development I decided to stay with the QtCreator - CMake - ninja "team of tools"!

Regards, Cornelis

Am Samstag, den 17.02.2018, 08:06 +0100 schrieb Cornelis Bockemühl:
Thanks also for this hint: I will give it a try!

Regards, Cornelis

Am Freitag, den 16.02.2018, 21:35 +0300 schrieb Konstantin Tokarev:
16.02.2018, 21:32, "Cornelis Bockemühl" <[hidden email]>:
Thanks - that did the trick! And indeed by entering -j4 in the "Tool arguments" field, the build command changes "magically" into "cmake --build . --target all -- -j4" - with the "--" to pass the argument further to gmake. Actually compile time of my project after a "clean" went down from 23 to 14 min, thus reduction factor 1.6 with 4 processors: I tend to attribute the diffference to "almost 4" to the nature of my project where gmake seems not to see too many opportunities for parallel processing. Maybe it works always module by module, or whatever. Anyway, my question is answered - thanks again!
Note that if you used Ninja generator, it would use all cores by default.
Regards, Cornelis Am Freitag, den 16.02.2018, 20:25 +0300 schrieb Konstantin Tokarev:
16.02.2018, 20:20, "Cornelis Bockemühl" <[hidden email]>:
Thanks for your hints! And you are right: it is still interesting to know how QtCreator is actually invoking and using cmake.
You should go to project settings and change cmake --build arguments there
This is the beginning of a log that the QtCreator was generating during a build. Actually with two modules where nothing had to be done, but still showing how cmake and gmake are being invoked - and all a bit above my understanding of the two tools: =================================== /usr/bin/cmake -H/Sources/GitLinux/RMT -B/BinariesLinux/RMT/Debug --check-build-system CMakeFiles/Makefile.cmake 0 /usr/bin/cmake -E cmake_progress_start /BinariesLinux/RMT/Debug/CMakeFiles /BinariesLinux/RMT/Debug/CMakeFiles/progress.marks /usr/bin/gmake -f CMakeFiles/Makefile2 all gmake[1]: Entering directory '/BinariesLinux/RMT/Debug' /usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/depend gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/FirstModule.dir/DependInfo.cmake --color= gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' /usr/bin/gmake -f CMakeFiles/FirstModule.dir/build.make CMakeFiles/FirstModule.dir/build gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' gmake[2]: Nothing to be done for 'CMakeFiles/FirstModule.dir/build'. gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' [  3%] Built target FirstModule /usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/depend gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' cd /BinariesLinux/RMT/Debug && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /Sources/GitLinux/RMT /Sources/GitLinux/RMT /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug /BinariesLinux/RMT/Debug/CMakeFiles/SecondModule.dir/DependInfo.cmake --color= gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' /usr/bin/gmake -f CMakeFiles/SecondModule.dir/build.make CMakeFiles/SecondModule.dir/build gmake[2]: Entering directory '/BinariesLinux/RMT/Debug' gmake[2]: Nothing to be done for 'CMakeFiles/SecondModule.dir/build'. gmake[2]: Leaving directory '/BinariesLinux/RMT/Debug' [  6%] Built target SecondModule ... =========================================== For example I do not really understand all the parameters that are passed to cmake - and I do not find them in the documentation either (like -H, -B or --check-build-system... Regards, Cornelis Am Freitag, den 16.02.2018, 08:59 -0800 schrieb Michael Ellery:
I don’t know anything about QTCreator, but the question really boils down to how to get your IDE to pass extra args to the build tool - it’s not really a cmake issue except to the extent that CMake can be used to invoke the build step (after it has been used to generate the build). It sounds like QT uses “cmake --build” to do the build phase (which makes sense), so you just need to add your “-j N” argument to the build step invocation. If you believe this: https://doc.qt.io/qtcreator/creator-build-settings.html, then it sounds like maybe “Tool Arguments” is what you want (see “CMake Build Steps”) ? BTW, the way I do this myself (since I don’t use an IDE) is something like “cmake --build . -- -j 8” where the double dash stops shell argument processing and effectively passes everything else on to the build tool that cmake ultimately invokes (e.g. make or ninja or msbuild, etc.) HTH, Mike On Feb 16, 2018, at 8:31 AM, Cornelis Bockemühl <[hidden email]> wrote: Hello, Somehow I seem to miss some crucial point regarding setup for parallel build with CMake, so I would be happy if somebody can push me the last few millimeters to hit my target! My configuration is on OpenSuse Linux (Leap - 64-bit), working with QtCreator and CMake, using the "make" configuration. I learned from "the internet" that actually I would have to pass a -jN option to make or gmake (with N being the max number of processors I want to use). Or even more elegant: go for a -lN option to somehow balance the load. Sounds good! But what I do not see is how to pass these nice options to make through CMake and QtCreator! One attempt was to specify CMAKE_CXX_FLAGS=-j4 or the like because some comment "in the internet" seemed to suggest this. However, like I already feared: this passes the option to the compiler call - and that is the wrong address for it, so I get an error message. Basically I think that it is 99% a CMake question and maximum 1% a QtCreator problem, so I am asking this question here. And now I am stuck - if not some friendly and knowing person can help me out of this! Thanks for any helpful hint! Cornelis
--  Regards, Konstantin

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake