Fake dependencies of executables to static libs

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Fake dependencies of executables to static libs

Stephan Menzel
Hello List,

In our projects we are using static linkage on Windows (MSVC) and Linux. A number of static libs are created and then linked into executables, such as unit tests, tools and the actual production artifacts. Very normal stuff.

add_library(mylib STATIC mysources)
add_excecutable(mytest mytest.cpp)
target_link_libraries(mytest mylib)

Now although I don't understand the exact reasons behind that, over the years came to accept that depedencies between the lib and the resulting artifcats are not working. Like, when I change a source file to mylib, the lib will get rebuilt but the resulting test tool won't be relinked.
For test driven development or busy change cycles that can put quite a strain on effort, especially on windows. My normal workflow is that I always click in mytest.cpp, add and remove a space to mark it dirty as quickly as possible and thereby cause the rebuild. But sometimes I forget this and this caused me quite a bit of extra work.
My question is, can I fake that dependency somehow? Perhaps by some hack or using some workaround?

What I want to achieve is that when I change the lib, the unit test (or any other test tool using it) that depends on it will be relinked, even if it has not changed.

Is that possible somehow?

Cheers,
Stephan

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Fake dependencies of executables to static libs

Stephan Menzel
Hello again,

no ideas so far? Perhaps I can elaborate. I have tried multiple approaches over the years. Most recently adding the source files of the lib explicitly as dependencies to the created executable.

Like this:

set (SRC
MySource1.cpp
MySource2.cpp
)

set (HDR
MySource1.hpp
MySource2.hpp
)

add_library(mylib STATIC ${SRC} ${HDR})

add_executable(myexecutable main.cpp)
target_link_libraries(myexecutable 
mylib 
)

add_dependencies(myexecutable ${SRC})

This leads to errors though:

CMake Error at CMakeLists.txt:79 (add_dependencies):
  The dependency target "MySource1.cpp" of target "myexecutable" does not exist.

Obviously, I cannot simply state a filename and treat it as a target. I have however seen something similar been done when it comes to code generation. I have also seen explicit subtargets for files being created and then depend on those subtargets. But I don't really know how to do this the right way.

Alternatively I would also appreciate input about what exactly the problem with depending on static libs is. The explanations I remember from my last efforts to tackle this weren't really satisfying. If I understood in more detail why this isn't possible I may be able to better craft a workaround on my own.

Really, anything helps. I am losing time and nerve every day because of this and have been paying this price for so many years that I think it really is time to come up with a solution.

Cheers,
Stephan






On Tue, Apr 2, 2019 at 5:54 AM Stephan Menzel <[hidden email]> wrote:
Hello List,

In our projects we are using static linkage on Windows (MSVC) and Linux. A number of static libs are created and then linked into executables, such as unit tests, tools and the actual production artifacts. Very normal stuff.

add_library(mylib STATIC mysources)
add_excecutable(mytest mytest.cpp)
target_link_libraries(mytest mylib)

Now although I don't understand the exact reasons behind that, over the years came to accept that depedencies between the lib and the resulting artifcats are not working. Like, when I change a source file to mylib, the lib will get rebuilt but the resulting test tool won't be relinked.
For test driven development or busy change cycles that can put quite a strain on effort, especially on windows. My normal workflow is that I always click in mytest.cpp, add and remove a space to mark it dirty as quickly as possible and thereby cause the rebuild. But sometimes I forget this and this caused me quite a bit of extra work.
My question is, can I fake that dependency somehow? Perhaps by some hack or using some workaround?

What I want to achieve is that when I change the lib, the unit test (or any other test tool using it) that depends on it will be relinked, even if it has not changed.

Is that possible somehow?

Cheers,
Stephan

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Fake dependencies of executables to static libs

Paul Smith
On Thu, 2019-04-04 at 06:56 +0100, Stephan Menzel wrote:
add_library(mylib STATIC mysources)
add_excecutable(mytest mytest.cpp)
target_link_libraries(mytest mylib)

Now although I don't understand the exact reasons behind that, over the years came to accept that depedencies between the lib and the resulting artifcats are not working. Like, when I change a source file to mylib, the lib will get rebuilt but the resulting test tool won't be relinked.

This is definitely not true, as stated. I feel like I must be missing something about your stated environment because this is really basic, fundamental CMake behavior, which has always worked.

If in your environment you are not seeing executables rebuilt when libraries they depend on are updated, then there is something more complex about it than the straightforward case you provide above.

Rather than try to find a workaround it would be better to find the root cause. If you can provide a sample setup that behaves as you describe perhaps we could explain what's happening.

Also please let us know which version of CMake you're using.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Fake dependencies of executables to static libs

Shoaib Meenai
In reply to this post by Stephan Menzel

Could you share more details about your CMake version, the platform you're on, and the generator you're using?

 

I haven't run into this issue (and it's pretty counter to how I'd expect CMake to behave). The only similar setting I'm aware of is https://cmake.org/cmake/help/latest/prop_tgt/LINK_DEPENDS_NO_SHARED.html, but that's for shared libraries, not static libraries.

 

As an experiment, I used the following CMakeLists.txt:

 

cmake_minimum_required(VERSION 3.4.3)

project(cmtest C)

add_library(mylib STATIC source.c)

add_executable(mytest test.c)

target_link_libraries(mytest mylib)

 

I'm running CMake 3.12.3 on macOS 10.14.1, and I tried both the Makefile and Ninja generators. With both, any modification to source.c causes mytest to be rebuilt, as I'd expect (and unlike what you're seeing).

 

If you're able to provide a small self-contained CMakeLists.txt that reproduces the issue, that'd also be helpful for debugging.

 

Thanks,

Shoaib Meenai

 

From: CMake <[hidden email]> on behalf of Stephan Menzel <[hidden email]>
Date: Wednesday, April 3, 2019 at 10:56 PM
To: cmake <[hidden email]>
Subject: Re: [CMake] Fake dependencies of executables to static libs

 

Hello again,

 

no ideas so far? Perhaps I can elaborate. I have tried multiple approaches over the years. Most recently adding the source files of the lib explicitly as dependencies to the created executable.

 

Like this:

 

set (SRC

MySource1.cpp

MySource2.cpp

)

 

set (HDR

MySource1.hpp

MySource2.hpp

)

 

add_library(mylib STATIC ${SRC} ${HDR})

 

add_executable(myexecutable main.cpp)

target_link_libraries(myexecutable 

mylib 

)

 

add_dependencies(myexecutable ${SRC})

 

This leads to errors though:

 

CMake Error at CMakeLists.txt:79 (add_dependencies):

  The dependency target "MySource1.cpp" of target "myexecutable" does not exist.

 

Obviously, I cannot simply state a filename and treat it as a target. I have however seen something similar been done when it comes to code generation. I have also seen explicit subtargets for files being created and then depend on those subtargets. But I don't really know how to do this the right way.

 

Alternatively I would also appreciate input about what exactly the problem with depending on static libs is. The explanations I remember from my last efforts to tackle this weren't really satisfying. If I understood in more detail why this isn't possible I may be able to better craft a workaround on my own.

 

Really, anything helps. I am losing time and nerve every day because of this and have been paying this price for so many years that I think it really is time to come up with a solution.

 

Cheers,

Stephan

 

 

 

 

 

 

On Tue, Apr 2, 2019 at 5:54 AM Stephan Menzel <[hidden email]> wrote:

Hello List,

 

In our projects we are using static linkage on Windows (MSVC) and Linux. A number of static libs are created and then linked into executables, such as unit tests, tools and the actual production artifacts. Very normal stuff.

 

add_library(mylib STATIC mysources)

add_excecutable(mytest mytest.cpp)

target_link_libraries(mytest mylib)

 

Now although I don't understand the exact reasons behind that, over the years came to accept that depedencies between the lib and the resulting artifcats are not working. Like, when I change a source file to mylib, the lib will get rebuilt but the resulting test tool won't be relinked.

For test driven development or busy change cycles that can put quite a strain on effort, especially on windows. My normal workflow is that I always click in mytest.cpp, add and remove a space to mark it dirty as quickly as possible and thereby cause the rebuild. But sometimes I forget this and this caused me quite a bit of extra work.

My question is, can I fake that dependency somehow? Perhaps by some hack or using some workaround?

 

What I want to achieve is that when I change the lib, the unit test (or any other test tool using it) that depends on it will be relinked, even if it has not changed.

 

Is that possible somehow?

 

Cheers,

Stephan


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Fake dependencies of executables to static libs

Stephan Menzel
In reply to this post by Paul Smith
Hello Paul and Shoaib,

I'm answering to the both of you.

On Thu, Apr 4, 2019 at 7:13 AM Paul Smith <[hidden email]> wrote:
Now although I don't understand the exact reasons behind that, over the years came to accept that depedencies between the lib and the resulting artifcats are not working. Like, when I change a source file to mylib, the lib will get rebuilt but the resulting test tool won't be relinked.

This is definitely not true, as stated. I feel like I must be missing something about your stated environment because this is really basic, fundamental CMake behavior, which has always worked.

First of all, thanks for the information. I was not aware that this is supposed to work. As I said, I've been struggling with this issue for years now and I recall posting this to this list too but the only explanation I ever found was something along the lines of "dependencies would exist to objects inside the lib, not the lib as it is a static archive". But I'm pulling this out of my 'memory' now, nothing to rely on. All I remembered is that this is not supposed to work. But now you say the opposite, which gives me a little hope.
 
If in your environment you are not seeing executables rebuilt when libraries they depend on are updated, then there is something more complex about it than the straightforward case you provide above.

I have seen this in two major projects I have been working on in the last years. Both were Linux and Windows, MSVC 14 and 15 on Windows. And I do believe the issue is on Windows. I am not sure which CMake version I have seen this first on. Let's make it 3.10.x cause this was when I posted this to the list last time but by then I was already having the problem for a longer time.

Today I am using CMake version 3.13.2 but I have been using many different versions up to this one.
 
Both projects were large but not huge, having about 10 - 15 targets plus additional 50 unit tests plus another 5 test tools. Unfortunately I cannot publish the projects themselves but one that I currently see the issue with is mredis (https://github.com/MrMoose/mredis) which, among others, is pulled in as a submodule. C++ only, some protobuf code generation is involved but not in the targets that I'm talking about.

Since the issue has been persisting for so long now, the projects and CMakeLists have substantially changed many times, so it's really hard to come up with something I would consider 'strange' or 'different' as a possible explanation.


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Fake dependencies of executables to static libs

Francis Giraldeau
Maybe the problem is related to Cotire? I see that it is used in
mredis. I tried to reproduce the issue, and it behaves correctly. You
might want to try to reproduce using the following minimal repo:

  https://gitlab.com/fgiraldeau/cotire-demo

Francis

Le jeu. 4 avr. 2019 à 03:17, Stephan Menzel <[hidden email]> a écrit :

>
> Hello Paul and Shoaib,
>
> I'm answering to the both of you.
>
> On Thu, Apr 4, 2019 at 7:13 AM Paul Smith <[hidden email]> wrote:
>>
>> Now although I don't understand the exact reasons behind that, over the years came to accept that depedencies between the lib and the resulting artifcats are not working. Like, when I change a source file to mylib, the lib will get rebuilt but the resulting test tool won't be relinked.
>>
>>
>> This is definitely not true, as stated. I feel like I must be missing something about your stated environment because this is really basic, fundamental CMake behavior, which has always worked.
>
>
> First of all, thanks for the information. I was not aware that this is supposed to work. As I said, I've been struggling with this issue for years now and I recall posting this to this list too but the only explanation I ever found was something along the lines of "dependencies would exist to objects inside the lib, not the lib as it is a static archive". But I'm pulling this out of my 'memory' now, nothing to rely on. All I remembered is that this is not supposed to work. But now you say the opposite, which gives me a little hope.
>
>>
>> If in your environment you are not seeing executables rebuilt when libraries they depend on are updated, then there is something more complex about it than the straightforward case you provide above.
>
>
> I have seen this in two major projects I have been working on in the last years. Both were Linux and Windows, MSVC 14 and 15 on Windows. And I do believe the issue is on Windows. I am not sure which CMake version I have seen this first on. Let's make it 3.10.x cause this was when I posted this to the list last time but by then I was already having the problem for a longer time.
>
> Today I am using CMake version 3.13.2 but I have been using many different versions up to this one.
>
> Both projects were large but not huge, having about 10 - 15 targets plus additional 50 unit tests plus another 5 test tools. Unfortunately I cannot publish the projects themselves but one that I currently see the issue with is mredis (https://github.com/MrMoose/mredis) which, among others, is pulled in as a submodule. C++ only, some protobuf code generation is involved but not in the targets that I'm talking about.
>
> Since the issue has been persisting for so long now, the projects and CMakeLists have substantially changed many times, so it's really hard to come up with something I would consider 'strange' or 'different' as a possible explanation.
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Fake dependencies of executables to static libs

Stephan Menzel
Hello Francis,

On Thu, Apr 4, 2019 at 6:49 PM Francis Giraldeau <[hidden email]> wrote:
Maybe the problem is related to Cotire? I see that it is used in
mredis. I tried to reproduce the issue, and it behaves correctly. You
might want to try to reproduce using the following minimal repo:

Alas, I can safely rule out cotire. I only included it a few months ago and the issue was already ages old by then. Also, it is optional and some use it, some don't.

What comes to mind now as a possible explanation, as I just a minute ago noticed is what button to press. Perhaps it is a Visual Studio thing. When I, for example, work on the mredis lib I do that test driven using the various testing tools. 

Now in order to do this, I generally set the testing suite as "StartUp Project", as VC calls the currently active target. Then I press 'build target' (as opposed to 'build solution') or simply just "Debug", the green play button to run a test.

When the source to the testing suite is changed, both operations cause build and link of the tool and then run it as they should. When I change a source in the lib they do not.

However, when I press 'build solution', all dependent binaries do seem to be relinked. I just tested this. It's simply that I normally don't do this for low level testing builds because I don't need to build the whole solution just for one test. Interesting. I never noticed that.
May this shed some light on the issue?

Cheers,
Stephan




--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake