Building third party libraries along with normal targets

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Building third party libraries along with normal targets

Robert Dailey-2
Interested in hearing everyone's thoughts on this idea of mine.

Right now I have several third party libraries: openssl, boost,
libpng, zlib, etc. List goes on. I need to support these libraries on
at least 3 different platforms: ARM android, x86 linux, x86 windows.
It's a real pain in the rear to build each of these libraries a total
of 3 times (once per platform). It makes upgrades painful.

So I was thinking: If I set up each third party library to build from
source as a normal target with the rest of my targets, this would get
me the libraries for each third party lib "for free". Basically, they
would all build the same way no matter what platform I'm on. I can
move to other platforms or architectures in the future with no extra
effort.

The downside, of course, is that they build along with my normal
targets. Which means doing "ninja clean" will clean them, when I
really don't need to, and it will make build times get longer.

Any thoughts on this idea? Are there any other downsides? Is this the
best way to have library support across platforms?
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Florent Castelli
On 30/03/2017 02:10, Robert Dailey wrote:

> Interested in hearing everyone's thoughts on this idea of mine.
>
> Right now I have several third party libraries: openssl, boost,
> libpng, zlib, etc. List goes on. I need to support these libraries on
> at least 3 different platforms: ARM android, x86 linux, x86 windows.
> It's a real pain in the rear to build each of these libraries a total
> of 3 times (once per platform). It makes upgrades painful.
>
> So I was thinking: If I set up each third party library to build from
> source as a normal target with the rest of my targets, this would get
> me the libraries for each third party lib "for free". Basically, they
> would all build the same way no matter what platform I'm on. I can
> move to other platforms or architectures in the future with no extra
> effort.
This is known as "super build".
Yes, this is exactly why I made my Boost CMake build scripts, which you
use unless you changed your mind today :)
It can be done for other projects as well. Sometimes, they even provide
CMake build scripts you can use directly with "add_subdirectory()" so
you don't have to write CMake scripts or use "ExternalProject_Add()"
(which isn't all great since it doesn't propagate all your current
project settings).

> The downside, of course, is that they build along with my normal
> targets. Which means doing "ninja clean" will clean them, when I
> really don't need to, and it will make build times get longer.
You can use ccache or its Windows variants to make it faster. But you
also shouldn't need "ninja clean" most of the time. Possibly, you could
just clean a specific target "ninja -t clean foo".
If you declare all your dependencies properly, then you could just
always run "ninja" and the build will just be correct.
If your purpose is to see the compiler output again (to fix warnings),
it is acceptable to clean and rebuild (with ccache it should be fast
enough).
Personally, I just have a very long history in my terminal and scroll
back or pipe the build content to a file to look at it later. Some IDEs
will also record all the compilation output and make it available later
(Xcode does it), then it's less of an issue.

> Any thoughts on this idea? Are there any other downsides? Is this the
> best way to have library support across platforms?

The downside is that you have to maintain lots of 3rd party build
scripts. Depending on the side of the project, it might be a big
overhead for the team. But it might also be totally worth it in order to
support a lot of platforms, it just makes it easier than handling a huge
matrix of binaries for each platform, arch and compiler and compiler
options.

"Hey, you want to try LTO? Just add the flag and it's done!".

/Florent

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Robert Dailey-2
On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
<[hidden email]> wrote:
> This is known as "super build".
> Yes, this is exactly why I made my Boost CMake build scripts, which you use
> unless you changed your mind today :)

You mean this?
https://github.com/Orphis/boost-cmake

It's on the drawing board, for sure. It's the best solution I've
found. It's also part of the reason I'm asking this question to begin
with.

Although, this specific method of building boost doesn't use
superbuild, it uses normal add_subdirectory() with target.

> It can be done for other projects as well. Sometimes, they even provide
> CMake build scripts you can use directly with "add_subdirectory()" so you
> don't have to write CMake scripts or use "ExternalProject_Add()" (which
> isn't all great since it doesn't propagate all your current project
> settings).

Long term, I'm thinking a project like hunter[1] would be the best
choice. The problem with hunter is that it doesn't separate concerns
between host machine and target host. Specifically, when cross
compiling.

All the superbuild commands assume linux command syntax, simply
because I specified a toolchain utilizing android NDK. For this reason
it requires a lot of TLC to get working.

[1]: https://github.com/ruslo/hunter

> You can use ccache or its Windows variants to make it faster. But you also
> shouldn't need "ninja clean" most of the time. Possibly, you could just
> clean a specific target "ninja -t clean foo".
> If you declare all your dependencies properly, then you could just always
> run "ninja" and the build will just be correct.
> If your purpose is to see the compiler output again (to fix warnings), it is
> acceptable to clean and rebuild (with ccache it should be fast enough).
> Personally, I just have a very long history in my terminal and scroll back
> or pipe the build content to a file to look at it later. Some IDEs will also
> record all the compilation output and make it available later (Xcode does
> it), then it's less of an issue.

Never used ccache before, what does it do? Also if even if you ninja
-t to clean, it will also clean dependencies. So if I specify those
in-project dependencies that actually build third party libs, those
will be cleaned too.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Florent Castelli
On 30/03/2017 03:54, Robert Dailey wrote:

> On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
> <[hidden email]> wrote:
>> This is known as "super build".
>> Yes, this is exactly why I made my Boost CMake build scripts, which you use
>> unless you changed your mind today :)
> You mean this?
> https://github.com/Orphis/boost-cmake
>
> It's on the drawing board, for sure. It's the best solution I've
> found. It's also part of the reason I'm asking this question to begin
> with.
>
> Although, this specific method of building boost doesn't use
> superbuild, it uses normal add_subdirectory() with target.
It doesn't "super build" its dependencies, you have to provide them
externally, but it is a possible addition.
But it is designed to be part of a super build environment and is used
as such at some large companies shipping to millions of people.

>
>> It can be done for other projects as well. Sometimes, they even provide
>> CMake build scripts you can use directly with "add_subdirectory()" so you
>> don't have to write CMake scripts or use "ExternalProject_Add()" (which
>> isn't all great since it doesn't propagate all your current project
>> settings).
> Long term, I'm thinking a project like hunter[1] would be the best
> choice. The problem with hunter is that it doesn't separate concerns
> between host machine and target host. Specifically, when cross
> compiling.
>
> All the superbuild commands assume linux command syntax, simply
> because I specified a toolchain utilizing android NDK. For this reason
> it requires a lot of TLC to get working.
>
> [1]: https://github.com/ruslo/hunter
There are those issues and also ABI changing flags that can become an
issue as they aren't always propagated from what I can see.

>> You can use ccache or its Windows variants to make it faster. But you also
>> shouldn't need "ninja clean" most of the time. Possibly, you could just
>> clean a specific target "ninja -t clean foo".
>> If you declare all your dependencies properly, then you could just always
>> run "ninja" and the build will just be correct.
>> If your purpose is to see the compiler output again (to fix warnings), it is
>> acceptable to clean and rebuild (with ccache it should be fast enough).
>> Personally, I just have a very long history in my terminal and scroll back
>> or pipe the build content to a file to look at it later. Some IDEs will also
>> record all the compilation output and make it available later (Xcode does
>> it), then it's less of an issue.
> Never used ccache before, what does it do? Also if even if you ninja
> -t to clean, it will also clean dependencies. So if I specify those
> in-project dependencies that actually build third party libs, those
> will be cleaned too.

It's a program you use to record compilation outputs and save them to a
disk cache. If you build a file again, instead of compiling it, it will
just fetch the outputs from the cache and return the same warnings. It's
usually a safe optimization in a build pipeline and is quite easy to use
with CMake and Ninja generator.

It works well with GCC or Clang and there are some alternative versions
for the Microsoft compiler too.

/Florent

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Robert Dailey-2
On Wed, Mar 29, 2017 at 9:32 PM, Florent Castelli
<[hidden email]> wrote:

> On 30/03/2017 03:54, Robert Dailey wrote:
>>
>> On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
>> <[hidden email]> wrote:
>>>
>>> This is known as "super build".
>>> Yes, this is exactly why I made my Boost CMake build scripts, which you
>>> use
>>> unless you changed your mind today :)
>>
>> You mean this?
>> https://github.com/Orphis/boost-cmake
>>
>> It's on the drawing board, for sure. It's the best solution I've
>> found. It's also part of the reason I'm asking this question to begin
>> with.
>>
>> Although, this specific method of building boost doesn't use
>> superbuild, it uses normal add_subdirectory() with target.
>
> It doesn't "super build" its dependencies, you have to provide them
> externally, but it is a possible addition.
> But it is designed to be part of a super build environment and is used as
> such at some large companies shipping to millions of people.
>
>>
>>> It can be done for other projects as well. Sometimes, they even provide
>>> CMake build scripts you can use directly with "add_subdirectory()" so you
>>> don't have to write CMake scripts or use "ExternalProject_Add()" (which
>>> isn't all great since it doesn't propagate all your current project
>>> settings).
>>
>> Long term, I'm thinking a project like hunter[1] would be the best
>> choice. The problem with hunter is that it doesn't separate concerns
>> between host machine and target host. Specifically, when cross
>> compiling.
>>
>> All the superbuild commands assume linux command syntax, simply
>> because I specified a toolchain utilizing android NDK. For this reason
>> it requires a lot of TLC to get working.
>>
>> [1]: https://github.com/ruslo/hunter
>
> There are those issues and also ABI changing flags that can become an issue
> as they aren't always propagated from what I can see.
>
>>> You can use ccache or its Windows variants to make it faster. But you
>>> also
>>> shouldn't need "ninja clean" most of the time. Possibly, you could just
>>> clean a specific target "ninja -t clean foo".
>>> If you declare all your dependencies properly, then you could just always
>>> run "ninja" and the build will just be correct.
>>> If your purpose is to see the compiler output again (to fix warnings), it
>>> is
>>> acceptable to clean and rebuild (with ccache it should be fast enough).
>>> Personally, I just have a very long history in my terminal and scroll
>>> back
>>> or pipe the build content to a file to look at it later. Some IDEs will
>>> also
>>> record all the compilation output and make it available later (Xcode does
>>> it), then it's less of an issue.
>>
>> Never used ccache before, what does it do? Also if even if you ninja
>> -t to clean, it will also clean dependencies. So if I specify those
>> in-project dependencies that actually build third party libs, those
>> will be cleaned too.
>
>
> It's a program you use to record compilation outputs and save them to a disk
> cache. If you build a file again, instead of compiling it, it will just
> fetch the outputs from the cache and return the same warnings. It's usually
> a safe optimization in a build pipeline and is quite easy to use with CMake
> and Ninja generator.
>
> It works well with GCC or Clang and there are some alternative versions for
> the Microsoft compiler too.

At the end of the day, I've solved the Boost problem but I have many
other libraries I still manually build: openssl, libpng, zlib, etc.
It's as you said, maintaining build scripts for all of these will be
challenging but I think that's the proper way to go in my specific
situation (multi platform). It'll be less maintenance than rebuilding
binaries 3x per each.

Do you know how to set up the boost cmake project I linked on github
to work with superbuild? And do you prefer your superbuild execution
to happen during cmake generate (basically cmake invokes itself for
the superbuild) or do you prefer a separate external run, independent
of the normal one (basically 2x cmake invocations)?

Do you know of similar projects for other libraries I mentioned? Could
hand write my own, but seems better to piggyback on someone else's
efforts.

Thanks for your feedback!
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Florent Castelli
On 30/03/2017 04:38, Robert Dailey wrote:

> At the end of the day, I've solved the Boost problem but I have many
> other libraries I still manually build: openssl, libpng, zlib, etc.
> It's as you said, maintaining build scripts for all of these will be
> challenging but I think that's the proper way to go in my specific
> situation (multi platform). It'll be less maintenance than rebuilding
> binaries 3x per each.
>
> Do you know how to set up the boost cmake project I linked on github
> to work with superbuild? And do you prefer your superbuild execution
> to happen during cmake generate (basically cmake invokes itself for
> the superbuild) or do you prefer a separate external run, independent
> of the normal one (basically 2x cmake invocations)?
>
> Do you know of similar projects for other libraries I mentioned? Could
> hand write my own, but seems better to piggyback on someone else's
> efforts.
>
> Thanks for your feedback!

Only a few libraries are tricky to build, libpng or zlib are trivial for
example.

A super-build is just having one big project and defining all the
targets directly in there, using external CMake build scripts through
add_subdirectory() for example. There is just one CMake invocation.
This is how my Boost CMake project (yes, I'm the one being it) works. As
for the Boost dependencies, are you really using them? If you're not,
then you don't need to build the parts of Boost using them, most are
optional.
If you do, then you can provide Find*.cmake scripts returning targets
you've built yourself currently. Feel free to create issues on the bug
tracker if you need assistance with that.

I don't know on the top of my head similar projects for all libraries,
but I know there are many people writing CMake build scripts for all
kind of libraries, so just search on the web :)

/Florent

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Tamás Kenéz
An alternative to the CMake superbuild: leave your actual project intact. Simply create a separate shell script which builds all the dependencies (by running cmake commands). Or, if you do like the ExternalProject stuff, create a superbuild which builds only the dependencies.

Tamas

On Thu, Mar 30, 2017 at 5:01 AM, Florent Castelli <[hidden email]> wrote:
On 30/03/2017 04:38, Robert Dailey wrote:
At the end of the day, I've solved the Boost problem but I have many
other libraries I still manually build: openssl, libpng, zlib, etc.
It's as you said, maintaining build scripts for all of these will be
challenging but I think that's the proper way to go in my specific
situation (multi platform). It'll be less maintenance than rebuilding
binaries 3x per each.

Do you know how to set up the boost cmake project I linked on github
to work with superbuild? And do you prefer your superbuild execution
to happen during cmake generate (basically cmake invokes itself for
the superbuild) or do you prefer a separate external run, independent
of the normal one (basically 2x cmake invocations)?

Do you know of similar projects for other libraries I mentioned? Could
hand write my own, but seems better to piggyback on someone else's
efforts.

Thanks for your feedback!

Only a few libraries are tricky to build, libpng or zlib are trivial for example.

A super-build is just having one big project and defining all the targets directly in there, using external CMake build scripts through add_subdirectory() for example. There is just one CMake invocation.
This is how my Boost CMake project (yes, I'm the one being it) works. As for the Boost dependencies, are you really using them? If you're not, then you don't need to build the parts of Boost using them, most are optional.
If you do, then you can provide Find*.cmake scripts returning targets you've built yourself currently. Feel free to create issues on the bug tracker if you need assistance with that.

I don't know on the top of my head similar projects for all libraries, but I know there are many people writing CMake build scripts for all kind of libraries, so just search on the web :)

/Florent


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Robert Dailey-2
On Thu, Mar 30, 2017 at 3:42 AM, Tamás Kenéz <[hidden email]> wrote:
> An alternative to the CMake superbuild: leave your actual project intact.
> Simply create a separate shell script which builds all the dependencies (by
> running cmake commands). Or, if you do like the ExternalProject stuff,
> create a superbuild which builds only the dependencies.

Problem with that is that it's not a portable solution. The #1 benefit
of using CMake superbuilds (supplied with CMake build solution for the
target library) is that it will function correctly on all platforms.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Elizabeth A. Fischer

I do this using the spack autobuilder.  Only problem is it doesn't run on windows.  Maybe Conda?

On Mar 30, 2017 9:45 AM, "Robert Dailey" <[hidden email]> wrote:
On Thu, Mar 30, 2017 at 3:42 AM, Tamás Kenéz <[hidden email]> wrote:
> An alternative to the CMake superbuild: leave your actual project intact.
> Simply create a separate shell script which builds all the dependencies (by
> running cmake commands). Or, if you do like the ExternalProject stuff,
> create a superbuild which builds only the dependencies.

Problem with that is that it's not a portable solution. The #1 benefit
of using CMake superbuilds (supplied with CMake build solution for the
target library) is that it will function correctly on all platforms.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Miller Henry
In reply to this post by Robert Dailey-2
I have found the following projects which all seem to do some variation of a meta build so that you can build multiple projects that depend on each other and manage dependencies.  (there are a couple others that seem to not be maintained as well)

https://gradle.org/
https://bazel.build/ 
https://github.com/LLNL/spack 
https://github.com/ruslo/hunter 
http://www.biicode.com 
https://conan.io/
https://conda.io/

Unfortunately I have never found anyone who has actually compared even two of these. None of the projects have good marketing: it appears they somehow solve similar problems, but none actually have defined the problem or their solution.  It is like everyone assumes that everyone in the world has their exact same problem and the solution is obvious so the only thing left is the details of implementing it.  This of course tells me nothing about if they handle cross compiling (not a common use case but it is yours and mine), what packages they create, what compromises they make, what they expect of my environment...  These are important questions: I'm pretty sure that I could eliminate several just by comparing my needs to their features.

I'm currently using an in house system that builds everything in a Docker which lets me ensure nobody is accidentally using the wrong compiler. (we cross compile for a x86 target - 90% of the time if you build with gcc for the local system everything will work just fine, the other 10% of the time our system has an imcompatible version of some library and things blow up when you try to use some uncommon feature). I'm thinking about moving to one of the above, but I haven't actually evaluated anything.

If you do evaluate any of the above please document your experience and in particular what is good/bad about the things you look at.

This is getting off-topic for the cmake mailing list, but I don't know where else to move the conversation.


-----Original Message-----
From: CMake [mailto:[hidden email]] On Behalf Of Robert Dailey
Sent: Wednesday, March 29, 2017 9:39 PM
To: Florent Castelli <[hidden email]>
Cc: CMake <[hidden email]>
Subject: Re: [CMake] Building third party libraries along with normal targets

On Wed, Mar 29, 2017 at 9:32 PM, Florent Castelli <[hidden email]> wrote:

> On 30/03/2017 03:54, Robert Dailey wrote:
>>
>> On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
>> <[hidden email]> wrote:
>>>
>>> This is known as "super build".
>>> Yes, this is exactly why I made my Boost CMake build scripts, which
>>> you use unless you changed your mind today :)
>>
>> You mean this?
>> https://github.com/Orphis/boost-cmake
>>
>> It's on the drawing board, for sure. It's the best solution I've
>> found. It's also part of the reason I'm asking this question to begin
>> with.
>>
>> Although, this specific method of building boost doesn't use
>> superbuild, it uses normal add_subdirectory() with target.
>
> It doesn't "super build" its dependencies, you have to provide them
> externally, but it is a possible addition.
> But it is designed to be part of a super build environment and is used
> as such at some large companies shipping to millions of people.
>
>>
>>> It can be done for other projects as well. Sometimes, they even
>>> provide CMake build scripts you can use directly with
>>> "add_subdirectory()" so you don't have to write CMake scripts or use
>>> "ExternalProject_Add()" (which isn't all great since it doesn't
>>> propagate all your current project settings).
>>
>> Long term, I'm thinking a project like hunter[1] would be the best
>> choice. The problem with hunter is that it doesn't separate concerns
>> between host machine and target host. Specifically, when cross
>> compiling.
>>
>> All the superbuild commands assume linux command syntax, simply
>> because I specified a toolchain utilizing android NDK. For this
>> reason it requires a lot of TLC to get working.
>>
>> [1]: https://github.com/ruslo/hunter
>
> There are those issues and also ABI changing flags that can become an
> issue as they aren't always propagated from what I can see.
>
>>> You can use ccache or its Windows variants to make it faster. But
>>> you also shouldn't need "ninja clean" most of the time. Possibly,
>>> you could just clean a specific target "ninja -t clean foo".
>>> If you declare all your dependencies properly, then you could just
>>> always run "ninja" and the build will just be correct.
>>> If your purpose is to see the compiler output again (to fix
>>> warnings), it is acceptable to clean and rebuild (with ccache it
>>> should be fast enough).
>>> Personally, I just have a very long history in my terminal and
>>> scroll back or pipe the build content to a file to look at it later.
>>> Some IDEs will also record all the compilation output and make it
>>> available later (Xcode does it), then it's less of an issue.
>>
>> Never used ccache before, what does it do? Also if even if you ninja
>> -t to clean, it will also clean dependencies. So if I specify those
>> in-project dependencies that actually build third party libs, those
>> will be cleaned too.
>
>
> It's a program you use to record compilation outputs and save them to
> a disk cache. If you build a file again, instead of compiling it, it
> will just fetch the outputs from the cache and return the same
> warnings. It's usually a safe optimization in a build pipeline and is
> quite easy to use with CMake and Ninja generator.
>
> It works well with GCC or Clang and there are some alternative
> versions for the Microsoft compiler too.

At the end of the day, I've solved the Boost problem but I have many other libraries I still manually build: openssl, libpng, zlib, etc.
It's as you said, maintaining build scripts for all of these will be challenging but I think that's the proper way to go in my specific situation (multi platform). It'll be less maintenance than rebuilding binaries 3x per each.

Do you know how to set up the boost cmake project I linked on github to work with superbuild? And do you prefer your superbuild execution to happen during cmake generate (basically cmake invokes itself for the superbuild) or do you prefer a separate external run, independent of the normal one (basically 2x cmake invocations)?

Do you know of similar projects for other libraries I mentioned? Could hand write my own, but seems better to piggyback on someone else's efforts.

Thanks for your feedback!
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Konstantin Tokarev


30.03.2017, 18:15, "Miller Henry" <[hidden email]>:
> I have found the following projects which all seem to do some variation of a meta build so that you can build multiple projects that depend on each other and manage dependencies. (there are a couple others that seem to not be maintained as well)
>
> https://gradle.org/
> https://bazel.build/
> https://github.com/LLNL/spack
> https://github.com/ruslo/hunter
> http://www.biicode.com
> https://conan.io/
> https://conda.io/

Also, https://github.com/GNOME/jhbuild (notably used by GNOME, WebKit, and X.org communities)

>
> Unfortunately I have never found anyone who has actually compared even two of these. None of the projects have good marketing: it appears they somehow solve similar problems, but none actually have defined the problem or their solution. It is like everyone assumes that everyone in the world has their exact same problem and the solution is obvious so the only thing left is the details of implementing it. This of course tells me nothing about if they handle cross compiling (not a common use case but it is yours and mine), what packages they create, what compromises they make, what they expect of my environment... These are important questions: I'm pretty sure that I could eliminate several just by comparing my needs to their features.
>
> I'm currently using an in house system that builds everything in a Docker which lets me ensure nobody is accidentally using the wrong compiler. (we cross compile for a x86 target - 90% of the time if you build with gcc for the local system everything will work just fine, the other 10% of the time our system has an imcompatible version of some library and things blow up when you try to use some uncommon feature). I'm thinking about moving to one of the above, but I haven't actually evaluated anything.
>
> If you do evaluate any of the above please document your experience and in particular what is good/bad about the things you look at.
>
> This is getting off-topic for the cmake mailing list, but I don't know where else to move the conversation.
>
> -----Original Message-----
> From: CMake [mailto:[hidden email]] On Behalf Of Robert Dailey
> Sent: Wednesday, March 29, 2017 9:39 PM
> To: Florent Castelli <[hidden email]>
> Cc: CMake <[hidden email]>
> Subject: Re: [CMake] Building third party libraries along with normal targets
>
> On Wed, Mar 29, 2017 at 9:32 PM, Florent Castelli <[hidden email]> wrote:
>>  On 30/03/2017 03:54, Robert Dailey wrote:
>>>  On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
>>>  <[hidden email]> wrote:
>>>>  This is known as "super build".
>>>>  Yes, this is exactly why I made my Boost CMake build scripts, which
>>>>  you use unless you changed your mind today :)
>>>
>>>  You mean this?
>>>  https://github.com/Orphis/boost-cmake
>>>
>>>  It's on the drawing board, for sure. It's the best solution I've
>>>  found. It's also part of the reason I'm asking this question to begin
>>>  with.
>>>
>>>  Although, this specific method of building boost doesn't use
>>>  superbuild, it uses normal add_subdirectory() with target.
>>
>>  It doesn't "super build" its dependencies, you have to provide them
>>  externally, but it is a possible addition.
>>  But it is designed to be part of a super build environment and is used
>>  as such at some large companies shipping to millions of people.
>>
>>>>  It can be done for other projects as well. Sometimes, they even
>>>>  provide CMake build scripts you can use directly with
>>>>  "add_subdirectory()" so you don't have to write CMake scripts or use
>>>>  "ExternalProject_Add()" (which isn't all great since it doesn't
>>>>  propagate all your current project settings).
>>>
>>>  Long term, I'm thinking a project like hunter[1] would be the best
>>>  choice. The problem with hunter is that it doesn't separate concerns
>>>  between host machine and target host. Specifically, when cross
>>>  compiling.
>>>
>>>  All the superbuild commands assume linux command syntax, simply
>>>  because I specified a toolchain utilizing android NDK. For this
>>>  reason it requires a lot of TLC to get working.
>>>
>>>  [1]: https://github.com/ruslo/hunter
>>
>>  There are those issues and also ABI changing flags that can become an
>>  issue as they aren't always propagated from what I can see.
>>
>>>>  You can use ccache or its Windows variants to make it faster. But
>>>>  you also shouldn't need "ninja clean" most of the time. Possibly,
>>>>  you could just clean a specific target "ninja -t clean foo".
>>>>  If you declare all your dependencies properly, then you could just
>>>>  always run "ninja" and the build will just be correct.
>>>>  If your purpose is to see the compiler output again (to fix
>>>>  warnings), it is acceptable to clean and rebuild (with ccache it
>>>>  should be fast enough).
>>>>  Personally, I just have a very long history in my terminal and
>>>>  scroll back or pipe the build content to a file to look at it later.
>>>>  Some IDEs will also record all the compilation output and make it
>>>>  available later (Xcode does it), then it's less of an issue.
>>>
>>>  Never used ccache before, what does it do? Also if even if you ninja
>>>  -t to clean, it will also clean dependencies. So if I specify those
>>>  in-project dependencies that actually build third party libs, those
>>>  will be cleaned too.
>>
>>  It's a program you use to record compilation outputs and save them to
>>  a disk cache. If you build a file again, instead of compiling it, it
>>  will just fetch the outputs from the cache and return the same
>>  warnings. It's usually a safe optimization in a build pipeline and is
>>  quite easy to use with CMake and Ninja generator.
>>
>>  It works well with GCC or Clang and there are some alternative
>>  versions for the Microsoft compiler too.
>
> At the end of the day, I've solved the Boost problem but I have many other libraries I still manually build: openssl, libpng, zlib, etc.
> It's as you said, maintaining build scripts for all of these will be challenging but I think that's the proper way to go in my specific situation (multi platform). It'll be less maintenance than rebuilding binaries 3x per each.
>
> Do you know how to set up the boost cmake project I linked on github to work with superbuild? And do you prefer your superbuild execution to happen during cmake generate (basically cmake invokes itself for the superbuild) or do you prefer a separate external run, independent of the normal one (basically 2x cmake invocations)?
>
> Do you know of similar projects for other libraries I mentioned? Could hand write my own, but seems better to piggyback on someone else's efforts.
>
> Thanks for your feedback!
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake

--
Regards,
Konstantin
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Elizabeth A. Fischer
In reply to this post by Miller Henry
Miller,

Thank you for your input; I'm hoping we can use it to improve our description of Spack at http://spack.io  .  I'm cross-posting to the Spack list, maybe someone there can add to this.


There has been talk of a comparison; but I'm not familiar enough with the other systems to talk about them.  I can/will say a bit about Spack.

 
but none actually have defined the problem or their solution.

Spack is aimed at people involved in scientific computing.  This is more a social distinction than technical: there is nothing particularly special about scientific software environments, and I know of corporate environments that are considering Spack as well.  However, this intended audience does drive the design in some ways.

1. As with Macports, Homebrew, Gentoo, etc. Spack is focused on building things from source.  There has been some work on packing up Spack builds as RPMs so they can be quickly installed across multiple machines.

2. Spack supports cross-compilers.  Cross compilation is an intrinsic part of some supercomputers, in which the login nodes are a different architecture from the nodes the software runs on.  Maybe a Spacker who has done cross-compiling can speak up here...

3. Spack supports multiple versions of installed packages --- where the "version" is actually a hash of the package version, plus the version of all its dependencies.  Central to Spack is the "concretization" algorithm, which chooses which version of each package to build for you, based on the constraints you give it.  For example, you could build 24 versions of NetCDF, based on different compilers and MPI options you choose.  This kind of flexibility is incredibly useful in a scientific environment, where different top-level applications require different versions of various underlying dependencies.
        AFAIK, this concretization algorithm unique among auto-builders; and it deserves some time to think about (look at the Powerpoint presentations on the Spack page).  More common are builders (eg, those used for Linux distros) that produce a single version of each desired package --- and you have to manually specify the version of each package you want.  This is the step that Spack does automatically for you.

 
what packages they create,

Spack now has over 1000 packages it knows how to build:


NOTE: Be careful when comparing how many packages an auto-builder can create.  Some package managers, for example, have a different "package" for each version that might be installed.  For example, netcdf-1.0, netcdf-1.1, netcdf-1.2, etc.  In Spack, each "package" corresponds to an upstream piece of software (i.e. just netcdf), and Spack knows how to install potentially many versions and variants of it.
 
what compromises they make,
 
what they expect of my environment...

Just Python 2.6 or 2.7: do a "git clone", then run, starting up is very easy.  Spack needs other basic tools as well (git, curl, etc.).  On newer systems, they "just work"; on older systems, Spack can be used to install them.


limitations

1. Spack currently runs on Linux + MacOS, not Windows.

2. Spack doesn't really know how to work with sub-package managers: eg, using Spack to install `pip`, and then using `pip` to install things.  Same thing for `luarocks`, `conda`, `node-js`, etc.  Spack can install these things, just don't expect great integration.  We've thought a lot about how to integrate well, and haven't come up with any home run ideas.  The issue is most significant with node-js when a Spack package wants to depend on a bunch of Javascript packages.

3. Spack's concretization algorithm, while still much faster than a human, is too slow to conveniently use for software stacks of greater than ~100 packages.  This also makes it less than ideal for the Javascript world, where packages are small and numerous.
 
the other 10% of the time our system has an imcompatible version of some library and things blow up when you try to use some uncommon feature).

Spack goes to great lengths to build all dependencies itself, and not "accidentally" depend on system-supplied software.

This is getting off-topic for the cmake mailing list, but I don't know where else to move the conversation.

Feel free to join the Spack mailing list (CC'd above).

-- Elizabeth


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Building third party libraries along with normal targets

Tamás Kenéz
In reply to this post by Robert Dailey-2
[about shell script portability]

We do use shell-scripts on Windows since git-bash comes with git for free. Our workflow is centered around the command-line (git-bash for Windows) and you can run shell scripts in git-bash, that's why compatibility is not a problem. All our scripts and workflows are the same on all our platforms (windows, linux, mac).

This is an issue only if you're not using git on Windows so adding git-bash to the workflow would be an extra hassle.


On Thu, Mar 30, 2017 at 3:45 PM, Robert Dailey <[hidden email]> wrote:
On Thu, Mar 30, 2017 at 3:42 AM, Tamás Kenéz <[hidden email]> wrote:
> An alternative to the CMake superbuild: leave your actual project intact.
> Simply create a separate shell script which builds all the dependencies (by
> running cmake commands). Or, if you do like the ExternalProject stuff,
> create a superbuild which builds only the dependencies.

Problem with that is that it's not a portable solution. The #1 benefit
of using CMake superbuilds (supplied with CMake build solution for the
target library) is that it will function correctly on all platforms.


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Loading...