cmake + ninja how to use several CPU cores?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

cmake + ninja how to use several CPU cores?

Dave Milter
Hi,

I heard that ninja has great feature it allows build continue without
wainting full link.

So if you have library Lib and executable App,
source code of App may build in parallel with source code of Lib,
and sync only link stage. While other build systems force build of Lib,
and only then start build of any object files of App.

But is it possible to use this feature with cmake?

Here my cmake code to emulate bottleneck of our build:

```
cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(imported)

set(EXTERN_LIB_FILE ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/libextern.so)
set(EXTERN_LIB_INCLUDES ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/gen)
add_custom_command(OUTPUT  ${EXTERN_LIB_FILE}
                   COMMAND sh ./lib_generator.sh
                   WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib)
add_custom_target(extern_lib_target DEPENDS ${EXTERN_LIB_FILE})
add_library(extern_lib SHARED IMPORTED GLOBAL)
add_dependencies(extern_lib extern_lib_target)

set_target_properties(extern_lib
    PROPERTIES
    IMPORTED_LOCATION ${EXTERN_LIB_FILE}
    INTERFACE_INCLUDE_DIRECTORIES ${EXTERN_LIB_INCLUDES})
add_executable(app main.cpp)
target_link_libraries(app extern_lib)
```

we use not C/C++ library as part of our source tree,
and it builds some time,
so I expect that ninja can run "main.cpp" -> main.o compilation during
"extern_lib" build,
but this is not happens.

Is cmake has problem with add_custom_command and ninja,
or the whole idea is wrong, and cmake + ninja can not work in this way,
and I should use some other project generator instead of cmake for this?
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Alan W. Irwin-2
On 2019-07-28 19:03+0300 Dave Milter wrote:

> Hi,
>
> I heard that ninja has great feature it allows build continue without
> wainting full link.
>
> So if you have library Lib and executable App,
> source code of App may build in parallel with source code of Lib,
> and sync only link stage. While other build systems force build of Lib,
> and only then start build of any object files of App.
>
> But is it possible to use this feature with cmake?
>
> Here my cmake code to emulate bottleneck of our build:
>
> ```
> cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
> project(imported)
>
> set(EXTERN_LIB_FILE ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/libextern.so)
> set(EXTERN_LIB_INCLUDES ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib/gen)
> add_custom_command(OUTPUT  ${EXTERN_LIB_FILE}
>                   COMMAND sh ./lib_generator.sh
>                   WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/extern_lib)
> add_custom_target(extern_lib_target DEPENDS ${EXTERN_LIB_FILE})
> add_library(extern_lib SHARED IMPORTED GLOBAL)
> add_dependencies(extern_lib extern_lib_target)
>
> set_target_properties(extern_lib
>    PROPERTIES
>    IMPORTED_LOCATION ${EXTERN_LIB_FILE}
>    INTERFACE_INCLUDE_DIRECTORIES ${EXTERN_LIB_INCLUDES})
> add_executable(app main.cpp)
> target_link_libraries(app extern_lib)
> ```
>
> we use not C/C++ library as part of our source tree,
> and it builds some time,
> so I expect that ninja can run "main.cpp" -> main.o compilation during
> "extern_lib" build,
> but this is not happens.
>
> Is cmake has problem with add_custom_command and ninja,
> or the whole idea is wrong, and cmake + ninja can not work in this way,
> and I should use some other project generator instead of cmake for this?

Hi Dave:

To help answer your question, I see no fundamental issues with your code above.

The

add_dependencies(extern_lib extern_lib_target)

command in your above code means that the external project that is
created by building the extern_lib_target target must be *entirely*
completed before the extern_lib target
is built.  Furthermore, the

target_link_libraries(app extern_lib)

command means extern_lib must be built before the app is linked which, of
course, is a complete necessity.

However, I don't see anything in your CMake code that demands that the
compilation of main.cpp must be delayed until extern_lib is built.  So
can you double-check that the bottleneck is not actually the
(necessary until external_lib is built) delay of linking of app rather
than some unneccessary delay in the compilation of main.cpp?

If the latter is really true, that appears to me to be a CMake bug
(although I cannot make the final judgement call on that since I am an
experienced CMake user rather than a CMake developer).  And if so, does that
same problem (unnecessary delay in compilation rather than linking)
occur for the following simplified case?

cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(test_internal_library)

# Delay 10 seconds so that it is obvious if main.cpp compilation is
# being unnecessarily delayed.

# You should introduce boolean CMake logic here to
# use the equivalent (see
# <https://stackoverflow.com/questions/1672338/how-to-sleep-for-five-seconds-in-a-batch-file-cmd>)
# "timeout 10" command for the Windows case.
add_custom_target(delay
COMMAND sleep 10
)

add_library(intern_lib SHARED ${<list of source code files for that library>)

# Add a 10 second delay until intern_lib is built
add_dependency(intern_lib delay)
add_executable(app main.cpp)
target_link_libraries(app intern_lib)

Also, please organize both test cases as tarballs containing simple
examples (including a simplified lib_generator.sh for the first case,
a 10 second delay inserted in that case as well, and simplified source
code for library (e.g., containing one function to return the "hello,
world" string and a simplified app (a main routine to print out that
string) for both cases).  Such self-contained test projects make it
much more convenient for others to confirm the issue you have
discovered on the platforms they have access to.  For example, if you
supply such self-contained test cases, I would be happy to run both
test cases here on my Linux platform for both ninja and make.  (I have
had a lot of success with parallel builds for the make case on Linux
which is why I would like to also test make just in case it turns out
you have found a ninja-only CMake bug.)

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.org); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Dave Milter
Hi,

On Mon, Jul 29, 2019 at 12:01 AM Alan W. Irwin
<[hidden email]> wrote:

>
> On 2019-07-28 19:03+0300 Dave Milter wrote:
>
> To help answer your question, I see no fundamental issues with your code above.
>
> If the latter is really true, that appears to me to be a CMake bug
> (although I cannot make the final judgement call on that since I am an
> experienced CMake user rather than a CMake developer).  And if so, does that
> same problem (unnecessary delay in compilation rather than linking)
> occur for the following simplified case?
>
In fact my code in previous email is demo code. It already contains
"sleep" inside lib_generator.sh to emulate
real project.
I am just not sure is this good idea to send attachment into this mailing list.
See attachment, it is possible only to build on Linux/Mac, because of
usage bash/gcc for lib_geneator.sh

And I am pretty sure that problem in cmake (or my usage of cmake),
because of build.ninja generated by cmake contains such rules:
```
build cmake_object_order_depends_target_app: phony || extern_lib_target

build CMakeFiles/app.dir/main.cpp.o: CXX_COMPILER__app ../main.cpp ||
cmake_object_order_depends_target_app
```
"||" is "order-only dependencies" according to
https://ninja-build.org/manual.html#_syntax_example .
So cmake instruct ninja do not build main.cpp.o untill extern_lib_target build.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

add_library_imported.tar.gz (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Alan W. Irwin-2
Hi to Dave and Brad:

@Dave: As you are probably already aware Brad is one of the primary
CMake developers so I think he should be in on this discussion.

@Brad:

Using Dave's simple test project
I confirmed the compilation delay issue below in detail for both the
-G"Ninja" case AND the -G"Unix Makefiles" case on Debian Buster.
Would you please take a look at this simple test case to see if you
can find the explanation for this issue?

<Aside>
After years of ignoring CMake + Ninja because of the Fortran issues
for that combination, I have just realized you have a fork of Ninja
(not yet merged upstream) which apparently works well with CMake for the Fortran
case.  So I am in the process of trying that Ninja fork out for all components of PLplot
(including our Fortran binding and examples), and I plan to report to you those
results in a separate thread.  Which explains my renewed interest in Ninja
(in this case not your fork since no Fortran is involved) for this current thread.
</Aside>

@Both: I also plan to look at whether this issue exists for
the internal library case so more later when I get that
corresponding simple test project implemented.

More details below concerning the external library case ....

On 2019-07-29 03:36+0300 Dave Milter wrote:

> Hi,
>
> On Mon, Jul 29, 2019 at 12:01 AM Alan W. Irwin
> <[hidden email]> wrote:
>>
>> On 2019-07-28 19:03+0300 Dave Milter wrote:
>>
>> To help answer your question, I see no fundamental issues with your code above.
>>
>> If the latter is really true, that appears to me to be a CMake bug
>> (although I cannot make the final judgement call on that since I am an
>> experienced CMake user rather than a CMake developer).  And if so, does that
>> same problem (unnecessary delay in compilation rather than linking)
>> occur for the following simplified case?
>>
>
> In fact my code in previous email is demo code. It already contains
> "sleep" inside lib_generator.sh to emulate
> real project.

@Dave:

Great minds think alike concerning clearly demonstrating this issue with the
help of the sleep command.  :-)

> I am just not sure is this good idea to send attachment into this mailing list.

Actually, for obvious reasons of convenience, gzipped tarball
attachments should never be an issue on mailing lists (like the CMake
ones) which are relevant to discussion of software.  (There are
typically mailing-list limits with regard to size, but they are
normally quite generous so I have never run into this issue on the
CMake mailing lists.  Therefore, as expected, your attachment got
through to me without issues.  Also, I double-checked your description
of that attachment which is critical for the input mail chains of some
users.  In your case that description (which you can configure with
your mail software) was "application/GZIP" which is obviously fine for
my input mail chain case and presumably most others on this list who
want to look at that attachment.

@Brad: have you had any trouble seeing that attachment without your
input mail chain stripping it out?

> See attachment, it is possible only to build on Linux/Mac, because of
> usage bash/gcc for lib_geneator.sh
>
> And I am pretty sure that problem in cmake (or my usage of cmake),
> because of build.ninja generated by cmake contains such rules:
> ```
> build cmake_object_order_depends_target_app: phony || extern_lib_target
>
> build CMakeFiles/app.dir/main.cpp.o: CXX_COMPILER__app ../main.cpp ||
> cmake_object_order_depends_target_app
> ```
> "||" is "order-only dependencies" according to
> https://ninja-build.org/manual.html#_syntax_example .
> So cmake instruct ninja do not build main.cpp.o untill extern_lib_target build.

@Dave: Thanks very much for that simple test project.

@Both:

I configured and built it using the Debian Buster = Stable system
versions of cmake (version 3.13.4) and ninja (version 1.8.2) using the
following commands

I. The -G"Ninja" case....

# Fresh start:
# IMPORTANT: first cd to the top-level directory of the source tree. Then

# Get rid of the previously generated external library (if any) to
# insure the 5 second pause associated with that generation always
# occurs.
rm -f extern_lib/libextern.so

# Start with an empty build tree

rm -rf build_dir
mkdir build_dir
cd build_dir

# Configure
time cmake -G"Ninja" .. >& cmake.out
==>

real    0m1.084s
user    0m0.845s
sys     0m0.232s

# Just to be specific (although ninja normally figures this out for itself for parallel builds)
# build with -j16 since I have 16-thread hardware.  Also use -v option to create (slightly more)
# verbose output.
time ninja -j16 -v all
==>

[1/3] cd /home/irwin/David.Milter/20190728/test_ninja/add_library_imported/extern_lib && sh ./lib_generator.sh
Generating...
Generating...DONE
[2/3] /usr/bin/c++   -isystem ../extern_lib/gen  -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c ../main.cpp
[3/3] : && /usr/bin/c++     CMakeFiles/app.dir/main.cpp.o  -o app  -Wl,-rpath,/home/irwin/David.Milter/20190728/test_ninja/add_library_imported/extern_lib ../extern_lib/libextern.so && :

real    0m5.354s
user    0m0.281s
sys     0m0.070s

Most of that real time was due to the 5 second pause (generated by
the sleep command) before the "Generating...DONE" message.  And
clearly these results show that main.cpp was generated after that 5
second pause, i.e., the delayed compile issue is completely and unambiguously
confirmed by me and should be equally easy to confirm by Brad.

I also obtained the following result:

irwin@merlin> grep cmake_object_order_depends_target_app build.ninja
build cmake_object_order_depends_target_app: phony || extern_lib_target
build CMakeFiles/app.dir/main.cpp.o: CXX_COMPILER__app ../main.cpp || cmake_object_order_depends_target_app

which I agree appears to be the reason for the delayed compilation issue in the ninja case.

II. The -G"Unix Makefiles" case....

N.B. It turns out the issue also occurs for the -G"Unix Makefiles"
case as well, i.e.,

# Fresh start:
# IMPORTANT: first cd to the top-level directory of the source tree. Then

# Get rid of the previously generated external library (if any) to
# insure the 5 second pause associated with that generation always
# occurs.
rm -f extern_lib/libextern.so

# Start with an empty build tree
rm -rf build_dir
mkdir build_dir
cd build_dir

# Configure
time cmake -G"Unix Makefiles" .. >& cmake.out
==>

real    0m1.470s
user    0m1.152s
sys     0m0.309s

That's substantially slower (by ~0.4 seconds!) than the Ninja case
which is part of the reason there is so much interest in Ninja these
days compared to (GNU-)make.

# (GNU-make requires a -j option to use parallel builds so make it
# the same -j option as in the Ninja case)
# build with -j16 since I have 16-thread hardware
# Also to keep output simple, do not use VERBOSE=1 option.
time make -j16 all
==>
Scanning dependencies of target extern_lib_target
[ 33%] Generating ../extern_lib/libextern.so
Generating...
Generating...DONE
[ 33%] Built target extern_lib_target
Scanning dependencies of target app
[ 66%] Building CXX object CMakeFiles/app.dir/main.cpp.o
[100%] Linking CXX executable app
[100%] Built target app

real    0m5.482s
user    0m0.397s
sys     0m0.082s

Note, the 5 seconds part of this is due to the pause.  Therefore, the
fact this is ~0.1 seconds slower than the Ninja case is pretty
significant, i.e., this confirms CMake -G"Ninja" provides more nimble
builds than CMake -G"Unix Makefiles" (as expected from the Ninja simplistic
design principles).

More importantly these -G"Unix Makefiles" results confirms that the (compilation delay) issue also
appears for the make case.

I have looked at the various Makefile rules generated by CMake in this
case.  However, they are so complex I am having trouble figuring out
why the compilation of main.cpp is delayed until after the external
library is built.

@Brad: I hope you will be able to satisfy my curiosity about that.

More later for the internal library case.

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.org); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Alan W. Irwin-2
On 2019-07-28 23:39-0700 Alan W. Irwin wrote:

> @Both: I also plan to look at whether this issue exists for
> the internal library case so more later when I get that
> corresponding simple test project implemented.

@Brad and David:

I have now implemented a simple test project for the internal library
case.  See the attached tarball (and please let me know if your input
mail software chain allows you to receive that tarball).  In general I
think this test case improves somewhat on Dave's test project by
making the sleep more explicit at the CMake level, generating the
headers explicitly rather than including them in the tarball,
generating all header and source code files and the library (internal
in this case) within the build tree rather than source tree (which
considerably simplifies making a clean start), and by allowing the
user to specify the -DBUILD_SHARED_LIBS=ON (or OFF) cmake option to
choose whether the library is built shared on static.  (Of course,
none of these improvements should affect the delayed compilation
issue.)

For this new simple test project, here are the results for the -G"Ninja" case (when -DBUILD_SHARED_LIBS=ON is specified):

irwin@merlin> ninja -j16 -v app
[1/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib && sh /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/include_generator.sh
[2/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib && sh /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/lib_generator.sh
[3/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir && sleep 5
[4/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib1.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -c lib/lib1.cpp
[5/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib3.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -c lib/lib3.cpp
[6/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib4.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -c lib/lib4.cpp
[7/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib2.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -c lib/lib2.cpp
[8/10] : && /usr/bin/c++ -fPIC    -shared -Wl,-soname,libinternal_lib.so -o libinternal_lib.so CMakeFiles/internal_lib.dir/lib/lib1.cpp.o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o   && :
[9/10] /usr/bin/c++   -Ilib  -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c ../main.cpp
[10/10] : && /usr/bin/c++     CMakeFiles/app.dir/main.cpp.o  -o app  -Wl,-rpath,/home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir libinternal_lib.so && :

Note that the build of main.cpp.o is (unreasonably in my opinion) delayed to step 9.

And that build is similarly delayed in the -G"Unix Makefiles" case.
That is this newly implemented test case for the internal library
build shows exactly the same unexplained compilation delays for
main.cpp for both -G"Ninja" and -G"Unix Makefiles" as David's test
case for the external library build.

@Brad and other CMake developers who might be lurking here:

Can anybody explain why CMake has this delayed compilation "feature"
for apps that depend on internal or external libraries, and if there
is no compelling reason for it would it be straightforward to remove
it?

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.org); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

add_internal_library.tar.gz (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Bruce Stephens
You might find it interesting to look at build.ninja. By the looks of
it there's a (phony) target guving the dependencies of the library.
(That's called cmake_object_order_depends_target_internal_lib) and
CMakeFiles/app.dir/main.cpp.o is depending on that.

Presumably the idea is that there's no point trying to compile
main.cpp before we could build the library, since (as in this case)
that might require generated headers which main.cpp might need.

If I get rid of the wait5 thing, then main.cpp gets compiled (using
"ninja -v CMakeFiles/app.dir/main.cpp.o") without the library. Just
the headers (and lib/lib1.cpp) get generated.

I think it's reasonable for CMake/Ninja to require the headers be
generated, especially since main.cpp does include one of them (though
CMake/Ninja doesn't know that until later). lib/lib1.cpp is more
arguable, but I imagine there's no real distinction made: main.cpp.o
depends on all the dependencies of the libraries.

On Mon, 29 Jul 2019 at 10:34, Alan W. Irwin <[hidden email]> wrote:

>
> On 2019-07-28 23:39-0700 Alan W. Irwin wrote:
>
> > @Both: I also plan to look at whether this issue exists for
> > the internal library case so more later when I get that
> > corresponding simple test project implemented.
>
> @Brad and David:
>
> I have now implemented a simple test project for the internal library
> case.  See the attached tarball (and please let me know if your input
> mail software chain allows you to receive that tarball).  In general I
> think this test case improves somewhat on Dave's test project by
> making the sleep more explicit at the CMake level, generating the
> headers explicitly rather than including them in the tarball,
> generating all header and source code files and the library (internal
> in this case) within the build tree rather than source tree (which
> considerably simplifies making a clean start), and by allowing the
> user to specify the -DBUILD_SHARED_LIBS=ON (or OFF) cmake option to
> choose whether the library is built shared on static.  (Of course,
> none of these improvements should affect the delayed compilation
> issue.)
>
> For this new simple test project, here are the results for the -G"Ninja" case (when -DBUILD_SHARED_LIBS=ON is specified):
>
> irwin@merlin> ninja -j16 -v app
> [1/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib && sh /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/include_generator.sh
> [2/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir/lib && sh /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/lib/lib_generator.sh
> [3/10] cd /home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir && sleep 5
> [4/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib1.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -c lib/lib1.cpp
> [5/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib3.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -c lib/lib3.cpp
> [6/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib4.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -c lib/lib4.cpp
> [7/10] /usr/bin/c++  -Dinternal_lib_EXPORTS -Ilib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib2.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -c lib/lib2.cpp
> [8/10] : && /usr/bin/c++ -fPIC    -shared -Wl,-soname,libinternal_lib.so -o libinternal_lib.so CMakeFiles/internal_lib.dir/lib/lib1.cpp.o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o   && :
> [9/10] /usr/bin/c++   -Ilib  -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c ../main.cpp
> [10/10] : && /usr/bin/c++     CMakeFiles/app.dir/main.cpp.o  -o app  -Wl,-rpath,/home/irwin/David.Milter/20190728/test_ninja/add_internal_library/build_dir libinternal_lib.so && :
>
> Note that the build of main.cpp.o is (unreasonably in my opinion) delayed to step 9.
>
> And that build is similarly delayed in the -G"Unix Makefiles" case.
> That is this newly implemented test case for the internal library
> build shows exactly the same unexplained compilation delays for
> main.cpp for both -G"Ninja" and -G"Unix Makefiles" as David's test
> case for the external library build.
>
> @Brad and other CMake developers who might be lurking here:
>
> Can anybody explain why CMake has this delayed compilation "feature"
> for apps that depend on internal or external libraries, and if there
> is no compelling reason for it would it be straightforward to remove
> it?
>
> Alan
> __________________________
> Alan W. Irwin
>
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.org); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __________________________
>
> Linux-powered Science
> __________________________--
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Dave Milter
On Mon, Jul 29, 2019 at 1:48 PM Bruce Stephens
<[hidden email]> wrote:
>
> I think it's reasonable for CMake/Ninja to require the headers be
> generated, especially since main.cpp does include one of them (though
> CMake/Ninja doesn't know that until later). lib/lib1.cpp is more
> arguable, but I imagine there's no real distinction made: main.cpp.o
> depends on all the dependencies of the libraries.
>

In my initial example headers files are not generated in cmake scripts.
Only source code are generated, so main.cpp -> main.cpp.o doesn't
depend on anything.
But generated by cmake build.ninja still require link of extern_lib
before starting  main.cpp -> main.cpp.o
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Brad King
On 7/29/19 11:50 AM, Dave Milter wrote:
> Only source code are generated, so main.cpp -> main.cpp.o doesn't
> depend on anything.
> But generated by cmake build.ninja still require link of extern_lib
> before starting  main.cpp -> main.cpp.o

If there are any custom commands in any targets on which main.cpp's
target depends then CMake must pessimistically assume that the
custom commands may generate a header needed to compile `main.cpp`.
Object libraries can be used to break the dependencies that lead
to the assumption.

CMake 3.9 made this much better than it used to be:

  https://gitlab.kitware.com/cmake/cmake/merge_requests/430

Since then all objects in a target can start compiling independently
so long as that target does not depend on any targets with custom
commands.

-Brad
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Alan W. Irwin-2
In reply to this post by Dave Milter
On 2019-07-29 18:50+0300 Dave Milter wrote:

> On Mon, Jul 29, 2019 at 1:48 PM Bruce Stephens
> <[hidden email]> wrote:
>>
>> I think it's reasonable for CMake/Ninja to require the headers be
>> generated, especially since main.cpp does include one of them (though
>> CMake/Ninja doesn't know that until later). lib/lib1.cpp is more
>> arguable, but I imagine there's no real distinction made: main.cpp.o
>> depends on all the dependencies of the libraries.
>>
>
> In my initial example headers files are not generated in cmake scripts.
> Only source code are generated, so main.cpp -> main.cpp.o doesn't
> depend on anything.
> But generated by cmake build.ninja still require link of extern_lib
> before starting  main.cpp -> main.cpp.o
Thanks, Bruce, for spotting that issue with the first version of my
simple example which I oversimplified.  I have
now corrected it (see attached tarball) to follow the use case of
pre-existing headers and an app that links to a library (internal for
my corrected test case, and external for David's test case).

The issue (compilation of main.cpp is unreasonably delayed for
parallel builds until after the library (internal or external) is
built) still shows up in my corrected test case for both -G"Ninja" and
-G"Unix Makefiles" just like happens for David's test case for an
external library.

So the question remains why does CMake constrain parallel builds this
way (to the point that it is a bottlneck for David's real project with
external libraries) for this general use case?

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.org); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

corrected_add_internal_library.tar.gz (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Dave Milter
In reply to this post by Brad King
On Mon, Jul 29, 2019 at 8:24 PM Brad King <[hidden email]> wrote:
>
> CMake 3.9 made this much better than it used to be:
>
>   https://gitlab.kitware.com/cmake/cmake/merge_requests/430
>
> Since then all objects in a target can start compiling independently
> so long as that target does not depend on any targets with custom
> commands.

Is any way to tell cmake that add_custom_command is have no side effect,
and OUTPUT is exactly what it produces?

I see that there is
Tests/RunCMake/Ninja/LooseObjectDepends.cmake

in https://gitlab.kitware.com/cmake/cmake/merge_requests/430

is it negative test, or there is chance for fast compilation?
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Brad King
On 7/29/19 3:22 PM, Dave Milter wrote:
>> Since then all objects in a target can start compiling independently
>> so long as that target does not depend on any targets with custom
>> commands.

See also https://gitlab.kitware.com/cmake/cmake/issues/17097

> Is any way to tell cmake that add_custom_command is have no side effect,
> and OUTPUT is exactly what it produces?

Even if we had that information we don't know what `main.cpp` includes
until after compiling it, by which point it is too late.  It could have
`#include "anything.txt"` for example.  CMake must add these pessimistic
dependencies to ensure a correct build.

The only way to avoid this is to tell CMake that there are no dependencies
for compiling an object file by moving it to an object library that does
not depend on any of the targets with custom commands.

-Brad
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Alan W. Irwin-2
In reply to this post by Brad King
On 2019-07-29 13:24-0400 Brad King wrote:

> On 7/29/19 11:50 AM, Dave Milter wrote:
>> Only source code are generated, so main.cpp -> main.cpp.o doesn't
>> depend on anything.
>> But generated by cmake build.ninja still require link of extern_lib
>> before starting  main.cpp -> main.cpp.o
>
> If there are any custom commands in any targets on which main.cpp's
> target depends then CMake must pessimistically assume that the
> custom commands may generate a header needed to compile `main.cpp`.

Hi Brad:

Thanks for that clarification.  Clearly David's test case for an external library and my own
recently corrected test case for an internal library are not consistent with that assumption which
explains the bad results we get for them.

However, I think there is still a problem when a project is consistent
with that assumption.  See the attached no_custom_add_internal_library.tar.gz
which contains an extremely simple test project with no custom commands at all
(i.e., no wait and headers and source are pre-existing in all cases).

Yet for this project I get the following results with ninja.

irwin@merlin> ninja -j16 -v app
[1/7] /usr/bin/c++  -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib4.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o -c ../lib/lib4.cpp
[2/7] /usr/bin/c++  -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib1.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib1.cpp.o -c ../lib/lib1.cpp
[3/7] /usr/bin/c++  -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib2.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o -c ../lib/lib2.cpp
[4/7] /usr/bin/c++  -Dinternal_lib_EXPORTS -I../lib -fPIC -MD -MT CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -MF CMakeFiles/internal_lib.dir/lib/lib3.cpp.o.d -o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o -c ../lib/lib3.cpp
[5/7] : && /usr/bin/c++ -fPIC    -shared -Wl,-soname,libinternal_lib.so -o libinternal_lib.so CMakeFiles/internal_lib.dir/lib/lib1.cpp.o CMakeFiles/internal_lib.dir/lib/lib2.cpp.o CMakeFiles/internal_lib.dir/lib/lib3.cpp.o CMakeFiles/internal_lib.dir/lib/lib4.cpp.o   && :
[6/7] /usr/bin/c++   -I../lib  -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c ../main.cpp
[7/7] : && /usr/bin/c++     CMakeFiles/app.dir/main.cpp.o  -o app  -Wl,-rpath,/home/irwin/David.Milter/20190728/test_ninja/no_gen_add_internal_library/build_dir libinternal_lib.so && :

irwin@merlin> grep cmake_object_order_depends_target_app build.ninja
build cmake_object_order_depends_target_app: phony || cmake_object_order_depends_target_internal_lib
build CMakeFiles/app.dir/main.cpp.o: CXX_COMPILER__app ../main.cpp || cmake_object_order_depends_target_app

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.org); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Dave Milter
In reply to this post by Brad King
On Mon, Jul 29, 2019 at 10:32 PM Brad King <[hidden email]> wrote:

>
> Even if we had that information we don't know what `main.cpp` includes
> until after compiling it, by which point it is too late.  It could have
> `#include "anything.txt"` for example.  CMake must add these pessimistic
> dependencies to ensure a correct build.
>
> The only way to avoid this is to tell CMake that there are no dependencies
> for compiling an object file by moving it to an object library that does
> not depend on any of the targets with custom commands.
>

But my custom command OUTPUT is shared library,
and in my case it would be strange to '#include' it into some source file.
So what about EXPLICIT_DEPENDS_ONLY mentioned
https://gitlab.kitware.com/cmake/cmake/issues/17097
to mark something that no way to be included into source file?
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: cmake + ninja how to use several CPU cores?

Alan W. Irwin-2
In reply to this post by Brad King
On 2019-07-29 15:32-0400 Brad King wrote:

> [...]We don't know what `main.cpp` includes
> until after compiling it, by which point it is too late.  It could have
> `#include "anything.txt"` for example.  CMake must add these pessimistic
> dependencies to ensure a correct build.

Is this the real reason why even the simplest
no_custom_add_internal_library project (with nothing custom at all)
still demonstrates the issue?

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.org); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake