How can I compile and link modular code into a single DLL with MSVC?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

How can I compile and link modular code into a single DLL with MSVC?

Jason Heeris
This is a follow-on from "Linking object libraries into a Windows DLL in 3.12rc1"[1] - I figured since I'm reframing that question I should start a new thread.

I have a project that has some common code and about a hundred modules. The modules can have dependencies on each other (only ever as a directed acyclic graph). What I want to do is, for any given module, compile it into a single Windows DLL using MSVC (currently v19 under Visual Studio 2017).

This is already possible with existing build scripts, but these are becoming unmaintainable. The existing scripts compile each individual module into an .obj file and finally link them all into a DLL, as in:

cl.exe [args] common\common.c (creates common.obj)
cl.exe [args] modules\module1\module1.c (creates module1.obj)
...
cl.exe [args] modules\moduleN\ moduleN.c (creates moduleN.obj)

link.exe [args] common.obj module1.obj [...] moduleN.obj (creates moduleN.dll)

(Not shown: the macros to create the declspec declaration that MSVC needs to know what to export in the DLL. It's a type in moduleN and some functions in common.)

I'm trying to figure out a way to do this with CMake that doesn't require me to flatten out the entire dependency graph by hand. Each module has, alongside it, a list of the other modules it *directly* depends on. These are easy to put into a eg. target_link_libraries(...) call. But I can't figure out how to get CMake to walk the whole dependency graph and pull everything into the final link command.

Note that it doesn't have to be done with steps I describe above, if there's another way to produce a single DLL I don't care how it happens. 

What I've tried:

1. Making all the modules shared libs. This just produces a bunch of separate DLLs for each module and common.

2. Chuck Atkins' post[2] suggesting separating usage requirements and objects into different libraries. Since target objects don't propagate using this method, it didn't help.

3. Now that 3.12rc1 is out, I tried using target_link_libraries() on the object libraries themselves[1]. But as I found out from my previous post, target objects don't
propagate this way either.

I'd appreciate any pointers. It seems like this is something that CMake is designed to do, but I just can't figure out how. I have a very minimal example (2 modules and common code) at https://gitlab.com/detly/cmake-dll-ex which shows my attempt using object libraries, if that helps.


Thanks,
Jason


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Innokentiy Alaytsev
Hello!

First of all, a side note:


Jason Heeris wrote
> (Not shown: the macros to create the declspec declaration that MSVC needs
> to know what to export in the DLL. It's a type in moduleN and some
> functions in common.)

Do you create declspec declaration by hand? It may worth looking at the
GenerateExportHeader
<https://cmake.org/cmake/help/v3.7/module/GenerateExportHeader.html>   CMake
module that provides the function generate_export_header(). If you need to
generate some additional macros you can use CUSTOM_CONTENT_FROM_VARIABLE
argument that is supported since CMake 3.7.

Now, let me clarify something:



Jason Heeris wrote

> This is already possible with existing build scripts, but these are
> becoming unmaintainable. The existing scripts compile each individual
> module into an .obj file and finally link them all into a DLL, as in:
>
> cl.exe [args] common\common.c (creates common.obj)
> cl.exe [args] modules\module1\module1.c (creates module1.obj)
> ...
> cl.exe [args] modules\moduleN\ moduleN.c (creates moduleN.obj)
>
> link.exe [args] common.obj module1.obj [...] moduleN.obj (creates
> moduleN.dll)

Is it the code that you use in your CMake build script or just a
representation of the general idea?

Best regards,
Innokentiy Alaytsev




--
Sent from: http://cmake.3232098.n2.nabble.com/
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Jason Heeris
On Wed, 27 Jun 2018 at 15:00, Innokentiy Alaytsev <[hidden email]> wrote:
Do you create declspec declaration by hand? It may worth looking at the
GenerateExportHeader
<https://cmake.org/cmake/help/v3.7/module/GenerateExportHeader.html

Thanks, this will probably be very useful!
  
Is it the code that you use in your CMake build script or just a
representation of the general idea?

It's a representation, it's not in CMake at all currently, so anything's on the table. However, it's not a far stretch from the real build steps. Currently the build system is in another scripting language, which traverses the dependencies from the top-level module, listing all dependants recursively (no duplicates). It then compiles each one into an .obj file and finally links all the .obj files together. Rather than subject you to that, I've added a (very fragile) "build_manual.bat" file to that Gitlab repo. But it contains those cl.exe and link.exe commands, and produces a DLL with a single visible function, "module3()".

The fundamental problem here is simply one of scale. For my sample project, adding common, module1, module2 to module3's dependencies isn't hard. But when it's a couple of dozen dependencies, some of which may change from time to time, with different degrees of indirectness, flattening them all out would be very time consuming. As would finding and updating all parents when sub-sub-module deps change. Tracking direct dependencies only is far more manageable. Hope that makes sense.

Cheers,
Jason

 

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Innokentiy Alaytsev
Hello! I've looked through the sources in your repo. I don't have CMake 3.12 right now, so I could not build it (Object library target "module1" may not link to anything). But that's not important. I don't quite understand why are you using object libraries all over the project. If those modules are not intended to be used outside of the final shader library, then why not to make them static libraries? (Sorry me, please, for just asking without trying) Also, I have an important note about the structure of the project: It may be better to have all the dependencies defined before they are used, i.e. you should not be able to use the target_link_libraries() with an undefined target as a linked library. Unfortunately, I don't have any example for you. The closest to what you describe could be a horrible abomination the project that optionally fetches dependencies from the remote repo and adds them as subdirectories to build and link to them (just like in this example project, but uglier). This seems rather close to what you want to me because dependency of the main project is built into a separate library that is linked to the final artefact. Best regards, Innokentiy Alaytsev

Sent from the CMake mailing list archive at Nabble.com.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Jason Heeris
On Thu, 28 Jun 2018 at 04:16, Innokentiy Alaytsev <[hidden email]> wrote:
> I don't quite understand why are you using object libraries all over the project.

Mainly because I tried a lot of things, and with the release of 3.12 I figured I'd try object libs again.
 
> If those modules are not intended to be used outside of the final shader library, then why not to make them static libraries?

I thought I'd previously tried that with no success, but I tried it just now and it seemed to work! (Maybe I did try it, but on an older version where things worked a bit differently. Or maybe I got confused because the build info never shows module3 being linked to the other sources or libraries. They ARE listed as references in the generated VS project of course.)

I'll test it on the real thing and see how that goes. The only problem I encountered was that the final shared library complains about having "NO SOURCES given to target: module3_dll". I can add "module3.c" as a source, even though technically the static lib should already contain everything from there. I'm not sure why that's an issue though.

In the real project *any* module can be a top level module, so each module will need multiple targets: the static library (so other dependant modules can link against it) and the shared library (for building it as a top level thing); there's also Mex (Matlab) and other Linux-specific targets. But I think that's easily managed by storing common info (ie. direct dependencies) in a variable and maybe writing a function to produce the targets in a structured way.

> Also, I have an important note about the structure of the project: It may be better to have all the dependencies defined before they are used, i.e. you should not be able to use the target_link_libraries() with an undefined target as a linked library.

I have no idea how I'd achieve this though, with so many subdirectories. I think CMake is perfectly happy to resolve these links at generate time rather than as the configuration is parsed though?

Cheers,
Jason

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Innokentiy Alaytsev
Hello!


Jason Heeris wrote
> The only problem I encountered was that the final shared library complains
> about having "NO SOURCES given to target: module3_dll". I can add
> "module3.c" as a source, even though technically the static lib should
> already contain everything from there. I'm not sure why that's an issue
> though.

The reason is that SHARED_LIBRARY targets must have sources to be built. The
fact that everything is already in the static library does not matter. In
this case, it may worth using OBJECT_LIBRARY targets, but in a different
way:  add an object library and provide it as sources for your target
<https://cmake.org/cmake/help/v3.5/command/add_library.html#object-libraries>
. However, I did not succeed when I tried it on my own project (most likely
due to my own incompetence).


Jason Heeris wrote
> On Thu, 28 Jun 2018 at 04:16, Innokentiy Alaytsev &lt;

> alaitsev@

> &gt;
> wrote:
>
>> Also, I have an important note about the structure of the project: It may
> be better to have all the dependencies defined before they are used, i.e.
> you should not be able to use the target_link_libraries() with an
> undefined
> target as a linked library.
>
> I have no idea how I'd achieve this though, with so many subdirectories. I
> think CMake is perfectly happy to resolve these links at generate time
> rather than as the configuration is parsed though?

You may be right about CMake being happy to resolve everything by itself. As
I wrote "It may be better", so I may as well be plain wrong.




--
Sent from: http://cmake.3232098.n2.nabble.com/
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Innokentiy Alaytsev
In reply to this post by Innokentiy Alaytsev
Innokentiy Alaytsev wrote
> The closest to what you describe could be a horrible abomination
> the project that optionally fetches dependencies from the remote repo and
> adds them as subdirectories to build and link to them (just like in  this
> example project &lt;https://github.com/Crascit/DownloadProject&gt;  , but
> uglier).

I think I should clarify something: I don't say and think that  the example
project <https://github.com/Crascit/DownloadProject>   is bad and ugly. I
wanted to say, that my own project that uses the same idea and CMake module
is quite horrible. I am sorry for not making it clear enough.

Best regards,
Innokentiy Alaytsev.



--
Sent from: http://cmake.3232098.n2.nabble.com/
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Jason Heeris
In reply to this post by Jason Heeris
On Thu, 28 Jun 2018 at 08:18, Jason Heeris <[hidden email]> wrote:
> > If those modules are not intended to be used outside of the final shader library, then why not to make them static libraries?
> I'll test it on the real thing and see how that goes. The only problem I encountered was that the final shared library complains about having "NO SOURCES given to target: module3_dll". I can add "module3.c" as a source, even though technically the static lib should already contain everything from there. I'm not sure why that's an issue though.

So I went and tried this on the real project, and it seems like compiling the static libs into a DLL doesn't give the same results as using the obj files directly, as per my initial list of commands. Namely, some symbols seem to not be getting exported, and some flags are mismatched between compiler and linker. It *could* be due to certain compiler flags, which I've raised in another thread. But I need to either add some test code to my sample project that loads and uses the DLL to demonstrate what's wrong, or identify what's different in the real project that's causing this.

It'd be nice if there were a way to get CMake to traverse the dependency graph and just apply specific build commands to each module, even just to debug exactly where this is going wrong, but I don't think I'm up for that challenge just yet.

Cheers,
Jason


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: How can I compile and link modular code into a single DLL with MSVC?

Jason Heeris
On Thu, 28 Jun 2018 at 20:14, Jason Heeris <[hidden email]> wrote:
> So I went and tried this on the real project, and it seems like compiling the static libs into a DLL doesn't give the same results as using the obj files directly, as per my initial list of commands. Namely, some symbols seem to not be getting exported, and some flags are mismatched between compiler and linker.

Finally got to the bottom of this. The linking was fine, but some compilation flags were affecting the binary interface. I've made some hacks in the code to work around this just to verify that it *can* work, and hopefully solving my issues in the other thread (controlling compiler flags) will give me a longer term solution.

Thanks for your help.

- Jason

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake