Tips on reducing configure time

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Tips on reducing configure time

Poughon Victor
Hi all

I am working on a large C++ project using CMake (https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a lot of time. This can be annoying for developers working on features where the configure step needs to be re-run, or for CI in general.

We have looked into it a bit and it seems like one issue is that the configure/generate steps do a lot of small files I/O. Specifically, using:

    inotofy-wait -mr <build_dir>

shows that "cmake ." causes about 150K file access/modify, etc. There are some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp' that appear thousands of times in most of the log file.

I have an SSD which somewhat helps, but for colleagues still on mechanical drives this is especially painful. Is there a way to disable writing all those files to the disk?

Any other tips on optimizing the performance of 'cmake .'?

Thanks,

Victor Poughon


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Tips on reducing configure time

Eric Noulard
Hi Victor,

Le mer. 14 nov. 2018 à 10:20, Poughon Victor <[hidden email]> a écrit :
Hi all

I am working on a large C++ project using CMake (https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a lot of time. This can be annoying for developers working on features where the configure step needs to be re-run, or for CI in general.

Could you give us a clue on "takes quite a lot of time"?
Are we speaking of tens of seconds, minutes etc... ?
 

We have looked into it a bit and it seems like one issue is that the configure/generate steps do a lot of small files I/O. Specifically, using:

    inotofy-wait -mr <build_dir>

shows that "cmake ." causes about 150K file access/modify, etc. There are some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp' that appear thousands of times in most of the log file.

I have an SSD which somewhat helps, but for colleagues still on mechanical drives this is especially painful. Is there a way to disable writing all those files to the disk?

Any other tips on optimizing the performance of 'cmake .'?

Did you try to test drive a switch to ninja generator ?
Ninja generator generates only two files (build.ninja and rules.ninja) you won't have the "mny small files issue".
You can have long configuration times as well but not coming from file access.


 

Thanks,

Victor Poughon


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Tips on reducing configure time

Cristian Adam
In reply to this post by Poughon Victor
On Wed, Nov 14, 2018 at 10:20 AM Poughon Victor <[hidden email]> wrote:
Hi all

I am working on a large C++ project using CMake (https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a lot of time. This can be annoying for developers working on features where the configure step needs to be re-run, or for CI in general.

We have looked into it a bit and it seems like one issue is that the configure/generate steps do a lot of small files I/O. Specifically, using:

    inotofy-wait -mr <build_dir>

shows that "cmake ." causes about 150K file access/modify, etc. There are some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp' that appear thousands of times in most of the log file.

I have an SSD which somewhat helps, but for colleagues still on mechanical drives this is especially painful. Is there a way to disable writing all those files to the disk?

Any other tips on optimizing the performance of 'cmake .'?

Thanks,

Victor Poughon


Hi,

if a subsequent CMake  run is faster than the initial one, you can have a look at

If you cache the results of the configure checks you can have the initial run as fast as subsequent ones!

Cheers,
Cristian.

 

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Tips on reducing configure time

Poughon Victor
In reply to this post by Eric Noulard

Thanks for the replies

 

You can see configure times on our dashboard (switch to advanced view): https://dash.orfeo-toolbox.org/index.php?project=OTB

It ranges from 1-3min. There's even one windows machine that's at 4m30s.

 

Thanks for the tip about Ninja generator, it's indeed much faster. But switching the entire dev team + CI servers from make to ninja is not really an option for us right now because of a bug in ninja that affects our SuperBuild configuration (see here if you're interested in the details: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625)

 

 

Victor Poughon

 

De : Eric Noulard <[hidden email]>
Envoyé : mercredi 14 novembre 2018 11:12
À : Poughon Victor <[hidden email]>
Cc : CMake Mailinglist <[hidden email]>
Objet : Re: [CMake] Tips on reducing configure time

 

Hi Victor,

Le mer. 14 nov. 2018 à 10:20, Poughon Victor <[hidden email]> a écrit :

Hi all

I am working on a large C++ project using CMake (https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a lot of time. This can be annoying for developers working on features where the configure step needs to be re-run, or for CI in general.

 

Could you give us a clue on "takes quite a lot of time"?

Are we speaking of tens of seconds, minutes etc... ?

 


We have looked into it a bit and it seems like one issue is that the configure/generate steps do a lot of small files I/O. Specifically, using:

    inotofy-wait -mr <build_dir>

shows that "cmake ." causes about 150K file access/modify, etc. There are some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp' that appear thousands of times in most of the log file.

I have an SSD which somewhat helps, but for colleagues still on mechanical drives this is especially painful. Is there a way to disable writing all those files to the disk?

Any other tips on optimizing the performance of 'cmake .'?

 

Did you try to test drive a switch to ninja generator ?

Ninja generator generates only two files (build.ninja and rules.ninja) you won't have the "mny small files issue".

You can have long configuration times as well but not coming from file access.

 

 

 


Thanks,

Victor Poughon


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


 

--

Eric


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Tips on reducing configure time

Eric Noulard


Le mer. 14 nov. 2018 à 12:06, Poughon Victor <[hidden email]> a écrit :

Thanks for the replies

 

You can see configure times on our dashboard (switch to advanced view): https://dash.orfeo-toolbox.org/index.php?project=OTB

It ranges from 1-3min. There's even one windows machine that's at 4m30s.


Somes windows system are known to be slow on many files accesses.
However on this particular machine (https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342008 I guess) you already seem
to be using NInja + MSVC. so many small files access to ?makefiles? should not be the culprit. 

That said whatever the system your build and test time seems to dominate the overall time (in the CI).

I have had great speed-up experience using CMake+ninja+ccache  for our build in CI. We did go from 1 hour down to less than 10min.
We only have various linux build (no Windows) it is really a big time savior combo.

We even share the compil' cache between the CI and the developers. So everybody is feeding compile cache and takes benefits from others including CI.

You seem to be using that as well here:
and you get less than 5min build time which corroborate my suggestion.

The same machine build with clang: https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342033 is lasting 30min+
and is not using ccache.
  

Thanks for the tip about Ninja generator, it's indeed much faster. But switching the entire dev team + CI servers from make to ninja is not really an option for us right now because of a bug in ninja that affects our SuperBuild configuration (see here if you're interested in the details: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625)


I see.
Not using ninja in the Superbuild case does not mean you cannot use it in other cases, right?
May be the one-size fits all is a little too much to ask at this point? My opinion though.
Did you report the issue upstream (ninja or cmake)?

I was impressed in the past by the amount of tests orfeo is doing, I'm still are !!
Are you running those tests in parallel (with ctest --parallel) ? It can be a time savior as well is the running machine has a fair amount of cores and memory.


--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Tips on reducing configure time

Poughon Victor

Yes you are correct on all your observations. We already use ninja and ccache wherever we can. In fact we have an issue about the whole end-to-end build performance where we track all effort on this throughout the project (if you're interested: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1649)

 

There are indeed quick and easy wins to be had with those tools (and we are working on it). But my original question is not about that. It's about speeding up the configure step with makefile generator. I still don't think it's normal that it does hundreds of thousands of I/O on files that are a few bytes or even empty. However it's possible that it's because we do something incorrect in our CMakeLists.txt and not CMake's fault.

 

It's true that a few minutes of configure is not much when doing a full build, but consider incremental builds where all you do is change one cxx file (and cmake is triggered because you changed git branch or something). Then the generate step is a significant part of the time you wait.

 

Victor Poughon

 

De : Eric Noulard <[hidden email]>
Envoyé : me
rcredi 14 novembre 2018 12:34
À : Poughon Victor <[hidden email]>
Cc : CMake Mailinglist <[hidden email]>
Objet : Re: [CMake] Tips on reducing configure time

 

 

Le mer. 14 nov. 2018 à 12:06, Poughon Victor <[hidden email]> a écrit :

Thanks for the replies

 

You can see configure times on our dashboard (switch to advanced view): https://dash.orfeo-toolbox.org/index.php?project=OTB

It ranges from 1-3min. There's even one windows machine that's at 4m30s.

 

Somes windows system are known to be slow on many files accesses.

However on this particular machine (https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342008 I guess) you already seem

to be using NInja + MSVC. so many small files access to ?makefiles? should not be the culprit. 

 

That said whatever the system your build and test time seems to dominate the overall time (in the CI).

 

I have had great speed-up experience using CMake+ninja+ccache  for our build in CI. We did go from 1 hour down to less than 10min.

We only have various linux build (no Windows) it is really a big time savior combo.

 

We even share the compil' cache between the CI and the developers. So everybody is feeding compile cache and takes benefits from others including CI.

 

You seem to be using that as well here:

and you get less than 5min build time which corroborate my suggestion.

 

The same machine build with clang: https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342033 is lasting 30min+

and is not using ccache.

  

Thanks for the tip about Ninja generator, it's indeed much faster. But switching the entire dev team + CI servers from make to ninja is not really an option for us right now because of a bug in ninja that affects our SuperBuild configuration (see here if you're interested in the details: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625)

 

I see.

Not using ninja in the Superbuild case does not mean you cannot use it in other cases, right?

May be the one-size fits all is a little too much to ask at this point? My opinion though.

Did you report the issue upstream (ninja or cmake)?

 

I was impressed in the past by the amount of tests orfeo is doing, I'm still are !!

Are you running those tests in parallel (with ctest --parallel) ? It can be a time savior as well is the running machine has a fair amount of cores and memory.

 

 

--

Eric


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Tips on reducing configure time

Eric Noulard


Le mer. 14 nov. 2018 à 13:25, Poughon Victor <[hidden email]> a écrit :

Yes you are correct on all your observations. We already use ninja and ccache wherever we can. In fact we have an issue about the whole end-to-end build performance where we track all effort on this throughout the project (if you're interested: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1649)


End-to-end build performance is a subject of interest to me, I'll have a look. 
 

There are indeed quick and easy wins to be had with those tools (and we are working on it). But my original question is not about that. It's about speeding up the configure step with makefile generator. I still don't think it's normal that it does hundreds of thousands of I/O on files that are a few bytes or even empty. However it's possible that it's because we do something incorrect in our CMakeLists.txt and not CMake's fault.


I don't really know makefile-generator internals so I cannot tell.
For sur the current OTB build dir constructed with Makefile generator is spitting out around 180 Makefile.
May be you can profile the cmake execution by building a debug version of CMake and collect more precise insight on where the bottlenecks may reside.

You may track try_compile which could obviously be slow and may be configure_file as well.
Otherwise I don't know, I guess you'll have to profile the cmake run. 

It's true that a few minutes of configure is not much when doing a full build, but consider incremental builds where all you do is change one cxx file (and cmake is triggered because you changed git branch or something). Then the generate step is a significant part of the time you wait.


Yes that right.
Note however that changing git branch should nor per-se trigger a cmake run.
Only changing a CMakeLists.txt or configured file or explicitely specified by
CMAKE_CONFIGURE_DEPENDS.

Otherwise changing branch with git that only modifies source shouldn't trigger cmake rebuild.
I do that all the time in a classical git branch dev model.



--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake