Redundant linking when modifying shared libraries

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Redundant linking when modifying shared libraries

Itay Chamiel
Hi,

I've asked this question on Stack Overflow almost a year ago with no useful responses (with the same topic if you wish to search for it), so I'm trying my luck here.

I work on a large commercial C++ project comprised of a couple dozen dynamically linked shared libraries, each of which has many associated unit tests. Many libs are also dependent on other libs because a lib for some specific functionality will use the code from one of the more common libs. And finally of course there are the production executables which depend on the libs.

There is no question that a change in the API (a header file) of some core common lib should trigger a major recompilation of nearly the entire system. But typically there is only a change in some function's implementation, and the only file compiled is the modified .cpp file, and in theory just the modified lib would need to be linked - thanks to dynamic linking there should be no need to relink anything else. But CMake goes ahead and does it anyway: after relinking the lib it relinks all the unit tests associated with that lib. Then it relinks all the libs in that lib's dependency tree and all their unit tests. Finally it relinks the production executables. Due to the scale of the project this takes a lot of precious time.

I have reproduced this behavior with a simple project based on the example at cmake.org/examples (comments removed for brevity and lib type changed to shared). My system is Ubuntu 16 on an Intel PC and my CMake version is 3.5.1.

Start with an empty directory. Create subdirectories Demo and Hello and then create these files:

* CMakeLists.txt

cmake_minimum_required (VERSION 2.8.11)
project (HELLO)
add_subdirectory (Hello)
add_subdirectory (Demo)

* Demo/CMakeLists.txt

add_executable (helloDemo demo.cpp)
target_link_libraries (helloDemo LINK_PUBLIC Hello)

* Demo/demo.cpp

#include "hello.h"
int main() { hello(); }

* Hello/CMakeLists.txt

add_library (Hello SHARED hello.cpp)
target_include_directories (Hello PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

* Hello/hello.h

void hello();

* Hello/hello.cpp

#include <stdio.h>
void hello() { printf("hello!\n"); }

* now run the commands:

mkdir build
cd build
cmake ../
make

* You may now execute Demo/helloDemo and see hello!.

Now, touch Hello/hello.cpp and make again. You will see that libHello is built and linked as expected, but then the helloDemo executable is also relinked ("Linking CXX executable helloDemo"). The latter step is completely redundant; even if hello.cpp is modified to print a different string the relinked executable remains binary identical.

Is there a way to prevent these redundant build actions? This would be a huge time saver for us.

Thank you,

itay


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Redundant linking when modifying shared libraries

Craig Scott-3


On Thu, Feb 14, 2019 at 9:24 PM Itay Chamiel <[hidden email]> wrote:
Hi,

I've asked this question on Stack Overflow almost a year ago with no useful responses (with the same topic if you wish to search for it), so I'm trying my luck here.

I work on a large commercial C++ project comprised of a couple dozen dynamically linked shared libraries, each of which has many associated unit tests. Many libs are also dependent on other libs because a lib for some specific functionality will use the code from one of the more common libs. And finally of course there are the production executables which depend on the libs.

There is no question that a change in the API (a header file) of some core common lib should trigger a major recompilation of nearly the entire system. But typically there is only a change in some function's implementation, and the only file compiled is the modified .cpp file, and in theory just the modified lib would need to be linked - thanks to dynamic linking there should be no need to relink anything else. But CMake goes ahead and does it anyway: after relinking the lib it relinks all the unit tests associated with that lib. Then it relinks all the libs in that lib's dependency tree and all their unit tests. Finally it relinks the production executables. Due to the scale of the project this takes a lot of precious time.

I have reproduced this behavior with a simple project based on the example at cmake.org/examples (comments removed for brevity and lib type changed to shared). My system is Ubuntu 16 on an Intel PC and my CMake version is 3.5.1.

Start with an empty directory. Create subdirectories Demo and Hello and then create these files:

* CMakeLists.txt

cmake_minimum_required (VERSION 2.8.11)
project (HELLO)
add_subdirectory (Hello)
add_subdirectory (Demo)

* Demo/CMakeLists.txt

add_executable (helloDemo demo.cpp)
target_link_libraries (helloDemo LINK_PUBLIC Hello)

* Demo/demo.cpp

#include "hello.h"
int main() { hello(); }

* Hello/CMakeLists.txt

add_library (Hello SHARED hello.cpp)
target_include_directories (Hello PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

* Hello/hello.h

void hello();

* Hello/hello.cpp

#include <stdio.h>
void hello() { printf("hello!\n"); }

* now run the commands:

mkdir build
cd build
cmake ../
make

* You may now execute Demo/helloDemo and see hello!.

Now, touch Hello/hello.cpp and make again. You will see that libHello is built and linked as expected, but then the helloDemo executable is also relinked ("Linking CXX executable helloDemo"). The latter step is completely redundant; even if hello.cpp is modified to print a different string the relinked executable remains binary identical.

Is there a way to prevent these redundant build actions? This would be a huge time saver for us.


I think you might be looking for the LINK_DEPENDS_NO_SHARED target property (or more likely its associated CMAKE_LINK_DEPENDS_NO_SHARED variable).


--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Redundant linking when modifying shared libraries

Itay Chamiel

On 2/14/19 12:38 PM, Craig Scott wrote:
I think you might be looking for the LINK_DEPENDS_NO_SHARED target property (or more likely its associated CMAKE_LINK_DEPENDS_NO_SHARED variable).


Following with my example, I've added the following line to the file Hello/CMakeLists.txt:


set_target_properties(Hello PROPERTIES LINK_DEPENDS_NO_SHARED true)


However it has no effect. Touching the file Hello/hello.cpp will still cause the "Linking CXX executable helloDemo" step to be performed.

I've also tried adding the same line to the CMakeLists.txt file in the root directory, it also had no effect.


Thank you,


itay


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Redundant linking when modifying shared libraries

Itay Chamiel
In reply to this post by Craig Scott-3
On Thu, Feb 14, 2019 at 12:39 PM Craig Scott <[hidden email]> wrote:
> I think you might be looking for the LINK_DEPENDS_NO_SHARED target property (or more likely its associated CMAKE_LINK_DEPENDS_NO_SHARED variable).

After my previous response I experimented a little more, and I got it
to work. My mistake was that I needed to add the configuration line to
the executable target, not the lib. So in my example, add the
following line to the Demo/CMakeLists.txt file:

set_target_properties(helloDemo PROPERTIES LINK_DEPENDS_NO_SHARED true)

..and it works like a charm. I wonder why this isn't the default
behavior; can you think of any scenario where this would produce
invalid results?

Thanks for your help!

itay
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Redundant linking when modifying shared libraries

CMake mailing list
> I wonder why this isn't the default behavior

By default CMake wants to get a correct build 100% of the time. There
is nothing to stop people from having functions defined in a .cxx file
with no corresponding header, and using manual forward deceleration is
used in a consuming library/executable. By setting
`LINK_DEPENDS_NO_SHARED` to `True` you will convert what was a link
time error ( renaming the function in the .cxx ) to a run time error.

On Thu, Feb 14, 2019 at 12:11 PM Itay Chamiel <[hidden email]> wrote:

>
> On Thu, Feb 14, 2019 at 12:39 PM Craig Scott <[hidden email]> wrote:
> > I think you might be looking for the LINK_DEPENDS_NO_SHARED target property (or more likely its associated CMAKE_LINK_DEPENDS_NO_SHARED variable).
>
> After my previous response I experimented a little more, and I got it
> to work. My mistake was that I needed to add the configuration line to
> the executable target, not the lib. So in my example, add the
> following line to the Demo/CMakeLists.txt file:
>
> set_target_properties(helloDemo PROPERTIES LINK_DEPENDS_NO_SHARED true)
>
> ..and it works like a charm. I wonder why this isn't the default
> behavior; can you think of any scenario where this would produce
> invalid results?
>
> Thanks for your help!
>
> itay
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Redundant linking when modifying shared libraries

Itay Chamiel
On Thu, Feb 14, 2019 at 8:08 PM Robert Maynard
<[hidden email]> wrote:
> By default CMake wants to get a correct build 100% of the time. There
> is nothing to stop people from having functions defined in a .cxx file
> with no corresponding header, and using manual forward deceleration is
> used in a consuming library/executable. By setting
> `LINK_DEPENDS_NO_SHARED` to `True` you will convert what was a link
> time error ( renaming the function in the .cxx ) to a run time error.

First of all, thanks for the answer - it's good to understand the
rationale and to know that in our use case we can safely use this
setting.

On the question of whether to set this as the default, I understand
your position and I personally disagree - but do not wish to start a
debate. However I will say that this simple change has, in a common
case, reduced my build time from 5 minutes to 15 seconds. At the very
least could you consider steps to make this setting more known and
accessible? (No one on Stack Overflow knew about it, and I think
that's saying something.) Perhaps put it in a FAQ, and/or more
prominently in the documentation.

Kind regards,

itay
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Redundant linking when modifying shared libraries

CMake mailing list
I agree that we should document this property better.
I recommend looking at the CMake wiki (
https://gitlab.kitware.com/cmake/community/wikis/home ) and thinking
maybe adding a new recipe for `optimizing redundant linking`.

On Thu, Feb 14, 2019 at 3:11 PM Itay Chamiel <[hidden email]> wrote:

>
> On Thu, Feb 14, 2019 at 8:08 PM Robert Maynard
> <[hidden email]> wrote:
> > By default CMake wants to get a correct build 100% of the time. There
> > is nothing to stop people from having functions defined in a .cxx file
> > with no corresponding header, and using manual forward deceleration is
> > used in a consuming library/executable. By setting
> > `LINK_DEPENDS_NO_SHARED` to `True` you will convert what was a link
> > time error ( renaming the function in the .cxx ) to a run time error.
>
> First of all, thanks for the answer - it's good to understand the
> rationale and to know that in our use case we can safely use this
> setting.
>
> On the question of whether to set this as the default, I understand
> your position and I personally disagree - but do not wish to start a
> debate. However I will say that this simple change has, in a common
> case, reduced my build time from 5 minutes to 15 seconds. At the very
> least could you consider steps to make this setting more known and
> accessible? (No one on Stack Overflow knew about it, and I think
> that's saying something.) Perhaps put it in a FAQ, and/or more
> prominently in the documentation.
>
> Kind regards,
>
> itay
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Redundant linking when modifying shared libraries

Itay Chamiel
Sounds good. The FAQ has a question "Is there a way to skip checking
of dependent libraries when compiling?" - perhaps you can add one just
before that, something like "Is there a way to reduce the amount of
linking when building a large project?".or similar.

Thanks again,

itay

On Thu, Feb 14, 2019 at 11:26 PM Robert Maynard
<[hidden email]> wrote:

>
> I agree that we should document this property better.
> I recommend looking at the CMake wiki (
> https://gitlab.kitware.com/cmake/community/wikis/home ) and thinking
> maybe adding a new recipe for `optimizing redundant linking`.
>
> On Thu, Feb 14, 2019 at 3:11 PM Itay Chamiel <[hidden email]> wrote:
> >
> > On Thu, Feb 14, 2019 at 8:08 PM Robert Maynard
> > <[hidden email]> wrote:
> > > By default CMake wants to get a correct build 100% of the time. There
> > > is nothing to stop people from having functions defined in a .cxx file
> > > with no corresponding header, and using manual forward deceleration is
> > > used in a consuming library/executable. By setting
> > > `LINK_DEPENDS_NO_SHARED` to `True` you will convert what was a link
> > > time error ( renaming the function in the .cxx ) to a run time error.
> >
> > First of all, thanks for the answer - it's good to understand the
> > rationale and to know that in our use case we can safely use this
> > setting.
> >
> > On the question of whether to set this as the default, I understand
> > your position and I personally disagree - but do not wish to start a
> > debate. However I will say that this simple change has, in a common
> > case, reduced my build time from 5 minutes to 15 seconds. At the very
> > least could you consider steps to make this setting more known and
> > accessible? (No one on Stack Overflow knew about it, and I think
> > that's saying something.) Perhaps put it in a FAQ, and/or more
> > prominently in the documentation.
> >
> > Kind regards,
> >
> > itay
> > --
> >
> > Powered by www.kitware.com
> >
> > Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> >
> > Kitware offers various services to support the CMake community. For more information on each offering, please visit:
> >
> > CMake Support: http://cmake.org/cmake/help/support.html
> > CMake Consulting: http://cmake.org/cmake/help/consulting.html
> > CMake Training Courses: http://cmake.org/cmake/help/training.html
> >
> > Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> >
> > Follow this link to subscribe/unsubscribe:
> > https://cmake.org/mailman/listinfo/cmake



--
Itay Chamiel, Software Engineer
[hidden email] | Office: +972-2-591-7800, ext. 7814
OrCam Technologies  www.orcam.com
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake