CMake Project Generation Speedup

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

CMake Project Generation Speedup

Damian
Hi all,

We are still in the process of switching our large Make-based build to CMake. One of the issues we're running into is the time it takes to reparse and regenerate the CMake project (whether ninja, VS, or make) after touching any CMake file. To give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.

Are there any plans to speed this up? Maybe parallelize it in some way or do a better job regenerating only what needs regenerating? Is there anything we can do on our side to reduce our regeneration times?

For example, if using a VS generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.

Thanks for any help.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Project Generation Speedup

J. Caleb Wherry
Did anything ever come of this?

I am in a similar boat: we have >800 targets on our full build (native C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the configure/generate steps takes place in the generate step (>70%).

I understand there is a lot of IO since the generate step has to write the project files and filters for each C++ project (the majority of our projects) for VS generators (what we use). I'm just looking to see if there is anything to look at or potentially speedup up the generate step besides "get a faster drive".

Thanks!
-Caleb

On Thu, Nov 17, 2016 at 11:15 AM Damian <[hidden email]> wrote:
Hi all,

We are still in the process of switching our large Make-based build to CMake. One of the issues we're running into is the time it takes to reparse and regenerate the CMake project (whether ninja, VS, or make) after touching any CMake file. To give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.

Are there any plans to speed this up? Maybe parallelize it in some way or do a better job regenerating only what needs regenerating? Is there anything we can do on our side to reduce our regeneration times?

For example, if using a VS generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.

Thanks for any help.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake


--
J. Caleb Wherry
Scientific Software Engineer

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Project Generation Speedup

J. Caleb Wherry
I was also surprised when "cmake --trace" gave 0 information related to the generate step. I assume this is expected behavior?

Oh and:

CMake: 3.13.4
Visual Studio 2017 15.9.9
Win7

-Caleb

On Wed, Mar 20, 2019 at 4:53 PM J. Caleb Wherry <[hidden email]> wrote:
Did anything ever come of this?

I am in a similar boat: we have >800 targets on our full build (native C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the configure/generate steps takes place in the generate step (>70%).

I understand there is a lot of IO since the generate step has to write the project files and filters for each C++ project (the majority of our projects) for VS generators (what we use). I'm just looking to see if there is anything to look at or potentially speedup up the generate step besides "get a faster drive".

Thanks!
-Caleb

On Thu, Nov 17, 2016 at 11:15 AM Damian <[hidden email]> wrote:
Hi all,

We are still in the process of switching our large Make-based build to CMake. One of the issues we're running into is the time it takes to reparse and regenerate the CMake project (whether ninja, VS, or make) after touching any CMake file. To give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.

Are there any plans to speed this up? Maybe parallelize it in some way or do a better job regenerating only what needs regenerating? Is there anything we can do on our side to reduce our regeneration times?

For example, if using a VS generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.

Thanks for any help.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake


--
J. Caleb Wherry
Scientific Software Engineer


--
J. Caleb Wherry
Scientific Software Engineer

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Project Generation Speedup

CMake mailing list
On Wed, 2019-03-20 at 16:57 -0400, J. Caleb Wherry wrote:
I was also surprised when "cmake --trace" gave 0 information related to the generate step. I assume this is expected behavior?

The purpose of --trace is to debug CMake scripts. No scripts get run during the generate step, so yes, this is expected.

Kyle

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Project Generation Speedup

Kuba Ober
In reply to this post by J. Caleb Wherry
The best advice I had for myself: profile the code and tweak it. The last I looked, and that was a while ago, the generators could use a threaded write queue so that the file output would overlap with file generation. Those probably could be memory mapped files, too, to further minimize the overhead: writing to mapped pages that are dumped to disk via DMA (that’s how page flushes work) is cheaper than copying all the data in little chunks into a small buffer in the file output code, and then once again into the page that ends up flushed to disk. Never mind that the write latencies are on the hot path, where they don’t belong.

If generators could be const-correct (not writing anywhere into the Makefile data structures), they could all be parallelized as well – but I’m not sure whether that’s a minor hack or a major rework. 

Cheers, Kuba

20 mars 2019 kl. 16:53 skrev J. Caleb Wherry <[hidden email]>:

Did anything ever come of this?

I am in a similar boat: we have >800 targets on our full build (native C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the configure/generate steps takes place in the generate step (>70%).

I understand there is a lot of IO since the generate step has to write the project files and filters for each C++ project (the majority of our projects) for VS generators (what we use). I'm just looking to see if there is anything to look at or potentially speedup up the generate step besides "get a faster drive".

Thanks!
-Caleb

On Thu, Nov 17, 2016 at 11:15 AM Damian <[hidden email]> wrote:
Hi all,

We are still in the process of switching our large Make-based build to CMake. One of the issues we're running into is the time it takes to reparse and regenerate the CMake project (whether ninja, VS, or make) after touching any CMake file. To give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.

Are there any plans to speed this up? Maybe parallelize it in some way or do a better job regenerating only what needs regenerating? Is there anything we can do on our side to reduce our regeneration times?

For example, if using a VS generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.

Thanks for any help.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake


--
J. Caleb Wherry
Scientific Software Engineer
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Project Generation Speedup

CMake mailing list
In reply to this post by Damian
Hello,

On 17.11.16 17:15, Damian wrote:
> We are still in the process of switching our large Make-based build to
> CMake. One of the issues we're running into is the time it takes to
> reparse and regenerate the CMake project (whether ninja, VS, or make)
> after touching any CMake file. To give you an idea, we have about 1000
> targets and that takes a good 2 min for CMake to rerun.
>
> Are there any plans to speed this up?

CMake VisualStudio and Xcode generator generates one solution /
xcodeproj per project() commmand. Due to the shared vcxproj files this
is not too expensive for Visual Studio. But for Xcode it is. You can
disable that behavior by specifying

   -DCMAKE_XCODE_GENERATE_TOP_LEVEL_PROJECT_ONLY=ON

on the command line.

Thanks,
-Gregor
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Project Generation Speedup

Cristian Adam
In reply to this post by J. Caleb Wherry
Hi,

CMake has for Visual Studio is a multi configuration generator. (Debug, Release, RelMinSize, RelWithDbgInformation).

If you specify only one configuration you should cut the generation time to 1/4th.


Cheers,
Cristian

On Wed, Mar 20, 2019, 21:53 J. Caleb Wherry <[hidden email]> wrote:
Did anything ever come of this?

I am in a similar boat: we have >800 targets on our full build (native C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the configure/generate steps takes place in the generate step (>70%).

I understand there is a lot of IO since the generate step has to write the project files and filters for each C++ project (the majority of our projects) for VS generators (what we use). I'm just looking to see if there is anything to look at or potentially speedup up the generate step besides "get a faster drive".

Thanks!
-Caleb

On Thu, Nov 17, 2016 at 11:15 AM Damian <[hidden email]> wrote:
Hi all,

We are still in the process of switching our large Make-based build to CMake. One of the issues we're running into is the time it takes to reparse and regenerate the CMake project (whether ninja, VS, or make) after touching any CMake file. To give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.

Are there any plans to speed this up? Maybe parallelize it in some way or do a better job regenerating only what needs regenerating? Is there anything we can do on our side to reduce our regeneration times?

For example, if using a VS generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.

Thanks for any help.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake


--
J. Caleb Wherry
Scientific Software Engineer
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: CMake Project Generation Speedup

CMake mailing list
In reply to this post by J. Caleb Wherry
A round of performance improvements to generate time was done as part
of CMake 3.11 and that significantly helped. What would be helpful is
a performance analysis run of CMake itself to determine if the issue
is that we are IO bound ( and need to do multi-threaded writes ) or
compute bound.

While this information will be project dependent, it would be great to
start getting samples from the community.

On Wed, Mar 20, 2019 at 1:53 PM J. Caleb Wherry <[hidden email]> wrote:

>
> Did anything ever come of this?
>
> I am in a similar boat: we have >800 targets on our full build (native C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the configure/generate steps takes place in the generate step (>70%).
>
> I understand there is a lot of IO since the generate step has to write the project files and filters for each C++ project (the majority of our projects) for VS generators (what we use). I'm just looking to see if there is anything to look at or potentially speedup up the generate step besides "get a faster drive".
>
> Thanks!
> -Caleb
>
> On Thu, Nov 17, 2016 at 11:15 AM Damian <[hidden email]> wrote:
>>
>> Hi all,
>>
>> We are still in the process of switching our large Make-based build to CMake. One of the issues we're running into is the time it takes to reparse and regenerate the CMake project (whether ninja, VS, or make) after touching any CMake file. To give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.
>>
>> Are there any plans to speed this up? Maybe parallelize it in some way or do a better job regenerating only what needs regenerating? Is there anything we can do on our side to reduce our regeneration times?
>>
>> For example, if using a VS generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
>>
>> Thanks for any help.
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
>
>
>
> --
> J. Caleb Wherry
> Scientific Software Engineer
>
> http://www.calebwherry.com
> +1 (615) 708-5651
> [hidden email]
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake