Using CMake as a package manager vs using a dedicated package management tool (like Conan)

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Using CMake as a package manager vs using a dedicated package management tool (like Conan)

Timothy Wrona
I have been working on a new C++ project and I am trying to decide whether I should use CMake as my package management system or if I should use a dedicated package management tool such as Conan.

For more information on Conan see: https://conan.io/

I am trying to understand the main difference between using Conan to manage dependencies vs using CMakes "FetchContent" module. Are there any compelling reasons to prefer something like Conan?

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: [cmake-developers] Using CMake as a package manager vs using a dedicated package management tool (like Conan)

Craig Scott-3


On Tue, Feb 19, 2019 at 12:46 PM Timothy Wrona <[hidden email]> wrote:
I have been working on a new C++ project and I am trying to decide whether I should use CMake as my package management system or if I should use a dedicated package management tool such as Conan.

For more information on Conan see: https://conan.io/

I am trying to understand the main difference between using Conan to manage dependencies vs using CMakes "FetchContent" module. Are there any compelling reasons to prefer something like Conan?

Excellent question, one that I think deserves a more detailed answer than I can provide here, but I'll try to hit the main points as I see them.

Personally, I think there is no "right answer" or "one size fits all" when it comes to package management for a CMake project. What works well for one situation, person or project may not be as convenient or suitable for another. There are competing needs and views, some of which are personal preferences, others are hard requirements from things like OS distribution policies for packaging. Even just the maturity of a project can have a big influence on how developers may prefer to handle its dependencies and handle it as a dependency of other projects.

The key thing for me is that the developer should ideally have choices when it comes to this area. If a project hard-codes that its dependencies must come from a particular provider (whether that be Conan, Hunter, vcpkg or some other system), this might not be compatible with what the developer's situation allows. You would need to weigh up whether it makes sense to lock the developer into a particular package manager if they want to use your project or not. An inappropriate choice here can mean lower adoption of the project as some may reject it for consideration based on this point alone.

If instead a project relies only on find_package() to find its dependencies, then it is up to the developer to ensure they are all available. This could be done using whatever package manager the developer finds convenient, or they could build the dependency projects from source individually or they might set up a superbuild parent project that builds the dependencies in the required order and makes dependees available to dependers. This gives good flexibility at the cost of more responsibility on the developer than perhaps some would want (again, it will be highly situation-dependent).

A drawback with find_package() is that it assumes you actually have a packageable project. For a variety of reasons, this may not be the case. Consider a large, complex project in its early stages and where multiple teams are working on different subprojects which all get combined into some larger whole. Each of the subprojects may need to be able to build on their own with their own smaller subset of dependencies, but they also need to be able to be incorporated into a larger build (think of different teams working on core toolkits, rendering engines, different algorithm strategies, multiple GUI applications, backend components, etc). No-one may know yet how it should get packaged up and everyone might be focused on just getting a minimal viable prototype up and running as a technical demonstrator. For a case like that, neither find_package() nor a package manager really fits the workflow. In this situation though, FetchContent is a perfect fit, since it doesn't require any packaging to already be in place, it needs no external tools other than CMake and it gives each project precise control over its dependencies down to the individual commit to bring in for each one.

With the above in mind, perhaps the following few questions may be helpful in clarifying what your constraints are and maybe steering you more toward one way or the other:
  • Will the project be incorporated into a Linux distribution at some point (not just be installed on Linux, but be part of the actual Linux distribution as provided by its own native package manager)? If so, I would expect this would pretty much eliminate using any package manager and instead require that you use find_package() to find all dependencies.
  • Are any of the dependencies of the project using a build system other than CMake? If so, they tend to take a bit more work to incorporate into a build via ExternalProject or FetchContent. If you can assume the developer provides those somehow, bringing them in by find_package() shifts responsibility for building them from the project to the developer (or whatever package manager they choose to use). This might be anywhere from entirely appropriate to entirely problematic depending on who your intended audience is.
  • How many dependencies does the project have and what is the maturity of each one? Will the project need to update any of those dependencies often (are they also being actively developed, do you need to follow recent work in them)? What will be the impact on developers working on the project when these dependencies need to be updated (how easy is it for them to update, what assumptions are you making about their development environment and tools)?
  • What breadth of platforms, compilers and CMake generators do you want to support? The bigger this set, the more it may drive you towards building dependencies from source rather than relying on having pre-built packages available to you.
  • Do you want developers to be able to use tools like sanitizers, perform code refactoring across projects or have source code visibility into dependencies within their IDE tools? FetchContent supports all of these requirements quite naturally, but doing so for the other methods it may be more difficult.
  • How will you manage repeatability of building past releases? Will you be able to build a year old release again if you update a build slave? In particular, what assumptions are you making about a CI system's build slaves and what dependencies they have installed (can you have multiple versions of any given dependency available at once, whether that be at a known path or via whatever package manager(s) you choose to use)?

On a side note, this is an area I'm increasingly thinking about these days (I'm the author of the FetchContent module). I'm interested in hearing peoples' views on dependency management and building an understanding of what works for people and what doesn't. If you want to send feedback my way, do feel free to get in touch.

--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: [cmake-developers] Using CMake as a package manager vs using a dedicated package management tool (like Conan)

Kai Wolf-2
Well, as of right now, it currently isn’t possible to use Conan in a non-intrusive way. That is, using it for fetching dependencies without adapting the build configuration to it.
I’ve opened up a pull request (as well as a fix) for this issue here [1].

I’ve implemented pure CMake-based package management solutions based on ExternalProject_Add() (using a Superbuild approach) in the past as well as using Conan for managing dependencies currently for a client of mine.
Based on my experience, I’m confident that really the only reason to use Conan is that it solves the persistence of binary artifacts for you. Since Conan uses a client/server architecture you typically have something like an Artifactory server where Conan is able to store and fetch precompiled packages for you. Currently, CMake doesn’t offer something like this, but I think it should be trivially to implement (think of CDash but for storing binary artifacts).
One issue I have with Conan is that you are forced to duplicate some of your build logic that in my point of view really belongs to the build system and not to your package management solution.


[1] https://github.com/conan-io/conan/issues/4467

--

Kai Wolf

Kai Wolf - SW Consulting

www.kai-wolf.me

XING · LinkedIn · GitHub


Am 19.02.2019 um 11:56 schrieb Craig Scott <[hidden email]>:



On Tue, Feb 19, 2019 at 12:46 PM Timothy Wrona <[hidden email]> wrote:
I have been working on a new C++ project and I am trying to decide whether I should use CMake as my package management system or if I should use a dedicated package management tool such as Conan.

For more information on Conan see: https://conan.io/

I am trying to understand the main difference between using Conan to manage dependencies vs using CMakes "FetchContent" module. Are there any compelling reasons to prefer something like Conan?

Excellent question, one that I think deserves a more detailed answer than I can provide here, but I'll try to hit the main points as I see them.

Personally, I think there is no "right answer" or "one size fits all" when it comes to package management for a CMake project. What works well for one situation, person or project may not be as convenient or suitable for another. There are competing needs and views, some of which are personal preferences, others are hard requirements from things like OS distribution policies for packaging. Even just the maturity of a project can have a big influence on how developers may prefer to handle its dependencies and handle it as a dependency of other projects.

The key thing for me is that the developer should ideally have choices when it comes to this area. If a project hard-codes that its dependencies must come from a particular provider (whether that be Conan, Hunter, vcpkg or some other system), this might not be compatible with what the developer's situation allows. You would need to weigh up whether it makes sense to lock the developer into a particular package manager if they want to use your project or not. An inappropriate choice here can mean lower adoption of the project as some may reject it for consideration based on this point alone.

If instead a project relies only on find_package() to find its dependencies, then it is up to the developer to ensure they are all available. This could be done using whatever package manager the developer finds convenient, or they could build the dependency projects from source individually or they might set up a superbuild parent project that builds the dependencies in the required order and makes dependees available to dependers. This gives good flexibility at the cost of more responsibility on the developer than perhaps some would want (again, it will be highly situation-dependent).

A drawback with find_package() is that it assumes you actually have a packageable project. For a variety of reasons, this may not be the case. Consider a large, complex project in its early stages and where multiple teams are working on different subprojects which all get combined into some larger whole. Each of the subprojects may need to be able to build on their own with their own smaller subset of dependencies, but they also need to be able to be incorporated into a larger build (think of different teams working on core toolkits, rendering engines, different algorithm strategies, multiple GUI applications, backend components, etc). No-one may know yet how it should get packaged up and everyone might be focused on just getting a minimal viable prototype up and running as a technical demonstrator. For a case like that, neither find_package() nor a package manager really fits the workflow. In this situation though, FetchContent is a perfect fit, since it doesn't require any packaging to already be in place, it needs no external tools other than CMake and it gives each project precise control over its dependencies down to the individual commit to bring in for each one.

With the above in mind, perhaps the following few questions may be helpful in clarifying what your constraints are and maybe steering you more toward one way or the other:
  • Will the project be incorporated into a Linux distribution at some point (not just be installed on Linux, but be part of the actual Linux distribution as provided by its own native package manager)? If so, I would expect this would pretty much eliminate using any package manager and instead require that you use find_package() to find all dependencies.
  • Are any of the dependencies of the project using a build system other than CMake? If so, they tend to take a bit more work to incorporate into a build via ExternalProject or FetchContent. If you can assume the developer provides those somehow, bringing them in by find_package() shifts responsibility for building them from the project to the developer (or whatever package manager they choose to use). This might be anywhere from entirely appropriate to entirely problematic depending on who your intended audience is.
  • How many dependencies does the project have and what is the maturity of each one? Will the project need to update any of those dependencies often (are they also being actively developed, do you need to follow recent work in them)? What will be the impact on developers working on the project when these dependencies need to be updated (how easy is it for them to update, what assumptions are you making about their development environment and tools)?
  • What breadth of platforms, compilers and CMake generators do you want to support? The bigger this set, the more it may drive you towards building dependencies from source rather than relying on having pre-built packages available to you.
  • Do you want developers to be able to use tools like sanitizers, perform code refactoring across projects or have source code visibility into dependencies within their IDE tools? FetchContent supports all of these requirements quite naturally, but doing so for the other methods it may be more difficult.
  • How will you manage repeatability of building past releases? Will you be able to build a year old release again if you update a build slave? In particular, what assumptions are you making about a CI system's build slaves and what dependencies they have installed (can you have multiple versions of any given dependency available at once, whether that be at a known path or via whatever package manager(s) you choose to use)?

On a side note, this is an area I'm increasingly thinking about these days (I'm the author of the FetchContent module). I'm interested in hearing peoples' views on dependency management and building an understanding of what works for people and what doesn't. If you want to send feedback my way, do feel free to get in touch.

--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [cmake-developers] Using CMake as a package manager vs using a dedicated package management tool (like Conan)

Kai Wolf
In reply to this post by Craig Scott-3
Well, as of right now, it currently isn’t possible to use Conan in a non-intrusive way. That is, using it for fetching dependencies without adapting the build configuration to it.
I’ve opened up a pull request (as well as a fix) for this issue here [1].

I’ve implemented pure CMake-based package management solutions based on ExternalProject_Add() (using a Superbuild approach) in the past as well as using Conan for managing dependencies currently for a client of mine.
Based on my experience, I’m confident that really the only reason to use Conan is that it solves the persistence of binary artifacts for you. Since Conan uses a client/server architecture you typically have something like an Artifactory server where Conan is able to store and fetch precompiled packages for you. Currently, CMake doesn’t offer something like this, but I think it should be trivially to implement (think of CDash but for storing binary artifacts).
One issue I have with Conan is that you are forced to duplicate some of your build logic that in my point of view really belongs to the build system and not to your package management solution.


--

Kai Wolf

Kai Wolf - SW Consulting

www.kai-wolf.me

XING · LinkedIn · GitHub


Am 19.02.2019 um 11:56 schrieb Craig Scott <[hidden email]>:



On Tue, Feb 19, 2019 at 12:46 PM Timothy Wrona <[hidden email]> wrote:
I have been working on a new C++ project and I am trying to decide whether I should use CMake as my package management system or if I should use a dedicated package management tool such as Conan.

For more information on Conan see: https://conan.io/

I am trying to understand the main difference between using Conan to manage dependencies vs using CMakes "FetchContent" module. Are there any compelling reasons to prefer something like Conan?

Excellent question, one that I think deserves a more detailed answer than I can provide here, but I'll try to hit the main points as I see them.

Personally, I think there is no "right answer" or "one size fits all" when it comes to package management for a CMake project. What works well for one situation, person or project may not be as convenient or suitable for another. There are competing needs and views, some of which are personal preferences, others are hard requirements from things like OS distribution policies for packaging. Even just the maturity of a project can have a big influence on how developers may prefer to handle its dependencies and handle it as a dependency of other projects.

The key thing for me is that the developer should ideally have choices when it comes to this area. If a project hard-codes that its dependencies must come from a particular provider (whether that be Conan, Hunter, vcpkg or some other system), this might not be compatible with what the developer's situation allows. You would need to weigh up whether it makes sense to lock the developer into a particular package manager if they want to use your project or not. An inappropriate choice here can mean lower adoption of the project as some may reject it for consideration based on this point alone.

If instead a project relies only on find_package() to find its dependencies, then it is up to the developer to ensure they are all available. This could be done using whatever package manager the developer finds convenient, or they could build the dependency projects from source individually or they might set up a superbuild parent project that builds the dependencies in the required order and makes dependees available to dependers. This gives good flexibility at the cost of more responsibility on the developer than perhaps some would want (again, it will be highly situation-dependent).

A drawback with find_package() is that it assumes you actually have a packageable project. For a variety of reasons, this may not be the case. Consider a large, complex project in its early stages and where multiple teams are working on different subprojects which all get combined into some larger whole. Each of the subprojects may need to be able to build on their own with their own smaller subset of dependencies, but they also need to be able to be incorporated into a larger build (think of different teams working on core toolkits, rendering engines, different algorithm strategies, multiple GUI applications, backend components, etc). No-one may know yet how it should get packaged up and everyone might be focused on just getting a minimal viable prototype up and running as a technical demonstrator. For a case like that, neither find_package() nor a package manager really fits the workflow. In this situation though, FetchContent is a perfect fit, since it doesn't require any packaging to already be in place, it needs no external tools other than CMake and it gives each project precise control over its dependencies down to the individual commit to bring in for each one.

With the above in mind, perhaps the following few questions may be helpful in clarifying what your constraints are and maybe steering you more toward one way or the other:
  • Will the project be incorporated into a Linux distribution at some point (not just be installed on Linux, but be part of the actual Linux distribution as provided by its own native package manager)? If so, I would expect this would pretty much eliminate using any package manager and instead require that you use find_package() to find all dependencies.
  • Are any of the dependencies of the project using a build system other than CMake? If so, they tend to take a bit more work to incorporate into a build via ExternalProject or FetchContent. If you can assume the developer provides those somehow, bringing them in by find_package() shifts responsibility for building them from the project to the developer (or whatever package manager they choose to use). This might be anywhere from entirely appropriate to entirely problematic depending on who your intended audience is.
  • How many dependencies does the project have and what is the maturity of each one? Will the project need to update any of those dependencies often (are they also being actively developed, do you need to follow recent work in them)? What will be the impact on developers working on the project when these dependencies need to be updated (how easy is it for them to update, what assumptions are you making about their development environment and tools)?
  • What breadth of platforms, compilers and CMake generators do you want to support? The bigger this set, the more it may drive you towards building dependencies from source rather than relying on having pre-built packages available to you.
  • Do you want developers to be able to use tools like sanitizers, perform code refactoring across projects or have source code visibility into dependencies within their IDE tools? FetchContent supports all of these requirements quite naturally, but doing so for the other methods it may be more difficult.
  • How will you manage repeatability of building past releases? Will you be able to build a year old release again if you update a build slave? In particular, what assumptions are you making about a CI system's build slaves and what dependencies they have installed (can you have multiple versions of any given dependency available at once, whether that be at a known path or via whatever package manager(s) you choose to use)?

On a side note, this is an area I'm increasingly thinking about these days (I'm the author of the FetchContent module). I'm interested in hearing peoples' views on dependency management and building an understanding of what works for people and what doesn't. If you want to send feedback my way, do feel free to get in touch.

--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [cmake-developers] Using CMake as a package manager vs using a dedicated package management tool (like Conan)

Timothy Wrona
In reply to this post by Craig Scott-3
Hi Craig,

Thank you for the detailed description!

To answer some of your questions:
  • This project will not be incorporated into a Linux distribution, however I would like to keep it cross platform and it should work on Windows, Mac, and Linux.
  • All of the pieces of the project that I am writing myself are using CMake, but I do have some dependencies on external libs such as "googletest" and "boost". Conan does seem to make using these external dependencies very simple - especially since I am using MinGW to do my compilation and "googletest" doesn't seem to compile out of the box on MinGW without passing specific flags to the compiler. With Conan I get a pre-compiled binary so it just works out of the box.
  • At the current time, this project does not have many dependencies, although it will likely grow quite large. I don't believe the external 3rd party dependencies will need to update frequently, but all of the libraries I am writing for the project will likely change quite a bit throughout working on the project. I would also like to be able to compile these libraries independently for reuse in other projects.
  • I intend to support as many platforms/compilers as possible, I am currently using Qt as my dev environment which allows me to install/configure multiple compilers and try builds with each one - this way I can at least do a build with both MinGW and msvc very quickly.
  • I would like to support tools like sanitizers and code refactoring tools at least within my own libraries (not necessarily with any of the 3rd party libs)
  • I would like past releases to be repeatable/rebuildable as much as possible, although when the compiler is upgraded it is understandable that past versions may have issues. I don't have much understanding of CI systems at this point and it is something I have been trying to become more familiar with. I'd like to avoid having multiple versions of the same dependency, although I don't think having two versions of "googletest" for two separate sub-projects that don't depend on each other would cause any issues.
    • I imagine Conan would make past releases more repeatable since you can fetch binary packages instead of needing to rebuild from source and package versions are always explicit.
I am currently the only developer working on the project, but I would still like to find the most efficient method of managing packages for my particular situation since it will likely save me a lot of pain down the road if I get it right up-front. This project is in the situation where I have a set of different sub-projects (libraries) that all need to be able to be compiled independently for re-use, but I have a main project that will use them all. All of these projects have their own dependencies (some of them 3rd party, some of them written by me). The 3rd party libs will likely rarely change, but the ones written by me may change quite frequently.

This is an approach I was thinking about taking:
  • *So far I have tried this and it is working well* - Manage 3rd party libs with Conan, since I don't need to see the source and it's quite convenient to not need to recompile them this seems to work pretty well. Using "FetchContent" on 3rd party libs and then attempting to compile them yourself can sometimes be tricky (for example "googletest" requires special compiler flags to be compiled with MinGW.) It also ensures that if two projects ask for the same version of a dependency I get only one copy even if the projects are built entirely independent of each other.
  • *I haven't tried this yet, but I am hoping it will work well* - Pull my sub-projects (my own custom libraries) into their own independent git repos and pull them into my main project using "FetchContent". Then when I run "FetchContent" it will checkout the sub-projects and I will have all of the source code available. I am hoping if I do this any changes I make to the sub-projects can easily be committed and pushed back to their own independent repositories.
    • The alternative to this would be to put my own sub-projects into their own Conan packages and use Conan to get them from the main project. But I am thinking since they will change frequently, "FetchContent" may be a better fit for this scenario.
    • Maybe when the sub-project libraries reach a mature and stable release they should be packaged into Conan and fetched in the main project from Conan at that point?
Let me know what you think! :)


On Tue, Feb 19, 2019 at 5:56 AM Craig Scott <[hidden email]> wrote:


On Tue, Feb 19, 2019 at 12:46 PM Timothy Wrona <[hidden email]> wrote:
I have been working on a new C++ project and I am trying to decide whether I should use CMake as my package management system or if I should use a dedicated package management tool such as Conan.

For more information on Conan see: https://conan.io/

I am trying to understand the main difference between using Conan to manage dependencies vs using CMakes "FetchContent" module. Are there any compelling reasons to prefer something like Conan?

Excellent question, one that I think deserves a more detailed answer than I can provide here, but I'll try to hit the main points as I see them.

Personally, I think there is no "right answer" or "one size fits all" when it comes to package management for a CMake project. What works well for one situation, person or project may not be as convenient or suitable for another. There are competing needs and views, some of which are personal preferences, others are hard requirements from things like OS distribution policies for packaging. Even just the maturity of a project can have a big influence on how developers may prefer to handle its dependencies and handle it as a dependency of other projects.

The key thing for me is that the developer should ideally have choices when it comes to this area. If a project hard-codes that its dependencies must come from a particular provider (whether that be Conan, Hunter, vcpkg or some other system), this might not be compatible with what the developer's situation allows. You would need to weigh up whether it makes sense to lock the developer into a particular package manager if they want to use your project or not. An inappropriate choice here can mean lower adoption of the project as some may reject it for consideration based on this point alone.

If instead a project relies only on find_package() to find its dependencies, then it is up to the developer to ensure they are all available. This could be done using whatever package manager the developer finds convenient, or they could build the dependency projects from source individually or they might set up a superbuild parent project that builds the dependencies in the required order and makes dependees available to dependers. This gives good flexibility at the cost of more responsibility on the developer than perhaps some would want (again, it will be highly situation-dependent).

A drawback with find_package() is that it assumes you actually have a packageable project. For a variety of reasons, this may not be the case. Consider a large, complex project in its early stages and where multiple teams are working on different subprojects which all get combined into some larger whole. Each of the subprojects may need to be able to build on their own with their own smaller subset of dependencies, but they also need to be able to be incorporated into a larger build (think of different teams working on core toolkits, rendering engines, different algorithm strategies, multiple GUI applications, backend components, etc). No-one may know yet how it should get packaged up and everyone might be focused on just getting a minimal viable prototype up and running as a technical demonstrator. For a case like that, neither find_package() nor a package manager really fits the workflow. In this situation though, FetchContent is a perfect fit, since it doesn't require any packaging to already be in place, it needs no external tools other than CMake and it gives each project precise control over its dependencies down to the individual commit to bring in for each one.

With the above in mind, perhaps the following few questions may be helpful in clarifying what your constraints are and maybe steering you more toward one way or the other:
  • Will the project be incorporated into a Linux distribution at some point (not just be installed on Linux, but be part of the actual Linux distribution as provided by its own native package manager)? If so, I would expect this would pretty much eliminate using any package manager and instead require that you use find_package() to find all dependencies.
  • Are any of the dependencies of the project using a build system other than CMake? If so, they tend to take a bit more work to incorporate into a build via ExternalProject or FetchContent. If you can assume the developer provides those somehow, bringing them in by find_package() shifts responsibility for building them from the project to the developer (or whatever package manager they choose to use). This might be anywhere from entirely appropriate to entirely problematic depending on who your intended audience is.
  • How many dependencies does the project have and what is the maturity of each one? Will the project need to update any of those dependencies often (are they also being actively developed, do you need to follow recent work in them)? What will be the impact on developers working on the project when these dependencies need to be updated (how easy is it for them to update, what assumptions are you making about their development environment and tools)?
  • What breadth of platforms, compilers and CMake generators do you want to support? The bigger this set, the more it may drive you towards building dependencies from source rather than relying on having pre-built packages available to you.
  • Do you want developers to be able to use tools like sanitizers, perform code refactoring across projects or have source code visibility into dependencies within their IDE tools? FetchContent supports all of these requirements quite naturally, but doing so for the other methods it may be more difficult.
  • How will you manage repeatability of building past releases? Will you be able to build a year old release again if you update a build slave? In particular, what assumptions are you making about a CI system's build slaves and what dependencies they have installed (can you have multiple versions of any given dependency available at once, whether that be at a known path or via whatever package manager(s) you choose to use)?

On a side note, this is an area I'm increasingly thinking about these days (I'm the author of the FetchContent module). I'm interested in hearing peoples' views on dependency management and building an understanding of what works for people and what doesn't. If you want to send feedback my way, do feel free to get in touch.

--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: [cmake-developers] Using CMake as a package manager vs using a dedicated package management tool (like Conan)

Timothy Wrona
Correction:

*I haven't tried this yet, but I am hoping it will work well* - Pull Put my sub-projects (my own custom libraries) into their own independent git repos and pull them into my main project using "FetchContent". Then when I run "FetchContent" it will checkout the sub-projects and I will have all of the source code available. I am hoping if I do this any changes I make to the sub-projects can easily be committed and pushed back to their own independent repositories.  

On Tue, Feb 19, 2019 at 10:33 AM Timothy Wrona <[hidden email]> wrote:
Hi Craig,

Thank you for the detailed description!

To answer some of your questions:
  • This project will not be incorporated into a Linux distribution, however I would like to keep it cross platform and it should work on Windows, Mac, and Linux.
  • All of the pieces of the project that I am writing myself are using CMake, but I do have some dependencies on external libs such as "googletest" and "boost". Conan does seem to make using these external dependencies very simple - especially since I am using MinGW to do my compilation and "googletest" doesn't seem to compile out of the box on MinGW without passing specific flags to the compiler. With Conan I get a pre-compiled binary so it just works out of the box.
  • At the current time, this project does not have many dependencies, although it will likely grow quite large. I don't believe the external 3rd party dependencies will need to update frequently, but all of the libraries I am writing for the project will likely change quite a bit throughout working on the project. I would also like to be able to compile these libraries independently for reuse in other projects.
  • I intend to support as many platforms/compilers as possible, I am currently using Qt as my dev environment which allows me to install/configure multiple compilers and try builds with each one - this way I can at least do a build with both MinGW and msvc very quickly.
  • I would like to support tools like sanitizers and code refactoring tools at least within my own libraries (not necessarily with any of the 3rd party libs)
  • I would like past releases to be repeatable/rebuildable as much as possible, although when the compiler is upgraded it is understandable that past versions may have issues. I don't have much understanding of CI systems at this point and it is something I have been trying to become more familiar with. I'd like to avoid having multiple versions of the same dependency, although I don't think having two versions of "googletest" for two separate sub-projects that don't depend on each other would cause any issues.
    • I imagine Conan would make past releases more repeatable since you can fetch binary packages instead of needing to rebuild from source and package versions are always explicit.
I am currently the only developer working on the project, but I would still like to find the most efficient method of managing packages for my particular situation since it will likely save me a lot of pain down the road if I get it right up-front. This project is in the situation where I have a set of different sub-projects (libraries) that all need to be able to be compiled independently for re-use, but I have a main project that will use them all. All of these projects have their own dependencies (some of them 3rd party, some of them written by me). The 3rd party libs will likely rarely change, but the ones written by me may change quite frequently.

This is an approach I was thinking about taking:
  • *So far I have tried this and it is working well* - Manage 3rd party libs with Conan, since I don't need to see the source and it's quite convenient to not need to recompile them this seems to work pretty well. Using "FetchContent" on 3rd party libs and then attempting to compile them yourself can sometimes be tricky (for example "googletest" requires special compiler flags to be compiled with MinGW.) It also ensures that if two projects ask for the same version of a dependency I get only one copy even if the projects are built entirely independent of each other.
  • *I haven't tried this yet, but I am hoping it will work well* - Pull my sub-projects (my own custom libraries) into their own independent git repos and pull them into my main project using "FetchContent". Then when I run "FetchContent" it will checkout the sub-projects and I will have all of the source code available. I am hoping if I do this any changes I make to the sub-projects can easily be committed and pushed back to their own independent repositories.
    • The alternative to this would be to put my own sub-projects into their own Conan packages and use Conan to get them from the main project. But I am thinking since they will change frequently, "FetchContent" may be a better fit for this scenario.
    • Maybe when the sub-project libraries reach a mature and stable release they should be packaged into Conan and fetched in the main project from Conan at that point?
Let me know what you think! :)


On Tue, Feb 19, 2019 at 5:56 AM Craig Scott <[hidden email]> wrote:


On Tue, Feb 19, 2019 at 12:46 PM Timothy Wrona <[hidden email]> wrote:
I have been working on a new C++ project and I am trying to decide whether I should use CMake as my package management system or if I should use a dedicated package management tool such as Conan.

For more information on Conan see: https://conan.io/

I am trying to understand the main difference between using Conan to manage dependencies vs using CMakes "FetchContent" module. Are there any compelling reasons to prefer something like Conan?

Excellent question, one that I think deserves a more detailed answer than I can provide here, but I'll try to hit the main points as I see them.

Personally, I think there is no "right answer" or "one size fits all" when it comes to package management for a CMake project. What works well for one situation, person or project may not be as convenient or suitable for another. There are competing needs and views, some of which are personal preferences, others are hard requirements from things like OS distribution policies for packaging. Even just the maturity of a project can have a big influence on how developers may prefer to handle its dependencies and handle it as a dependency of other projects.

The key thing for me is that the developer should ideally have choices when it comes to this area. If a project hard-codes that its dependencies must come from a particular provider (whether that be Conan, Hunter, vcpkg or some other system), this might not be compatible with what the developer's situation allows. You would need to weigh up whether it makes sense to lock the developer into a particular package manager if they want to use your project or not. An inappropriate choice here can mean lower adoption of the project as some may reject it for consideration based on this point alone.

If instead a project relies only on find_package() to find its dependencies, then it is up to the developer to ensure they are all available. This could be done using whatever package manager the developer finds convenient, or they could build the dependency projects from source individually or they might set up a superbuild parent project that builds the dependencies in the required order and makes dependees available to dependers. This gives good flexibility at the cost of more responsibility on the developer than perhaps some would want (again, it will be highly situation-dependent).

A drawback with find_package() is that it assumes you actually have a packageable project. For a variety of reasons, this may not be the case. Consider a large, complex project in its early stages and where multiple teams are working on different subprojects which all get combined into some larger whole. Each of the subprojects may need to be able to build on their own with their own smaller subset of dependencies, but they also need to be able to be incorporated into a larger build (think of different teams working on core toolkits, rendering engines, different algorithm strategies, multiple GUI applications, backend components, etc). No-one may know yet how it should get packaged up and everyone might be focused on just getting a minimal viable prototype up and running as a technical demonstrator. For a case like that, neither find_package() nor a package manager really fits the workflow. In this situation though, FetchContent is a perfect fit, since it doesn't require any packaging to already be in place, it needs no external tools other than CMake and it gives each project precise control over its dependencies down to the individual commit to bring in for each one.

With the above in mind, perhaps the following few questions may be helpful in clarifying what your constraints are and maybe steering you more toward one way or the other:
  • Will the project be incorporated into a Linux distribution at some point (not just be installed on Linux, but be part of the actual Linux distribution as provided by its own native package manager)? If so, I would expect this would pretty much eliminate using any package manager and instead require that you use find_package() to find all dependencies.
  • Are any of the dependencies of the project using a build system other than CMake? If so, they tend to take a bit more work to incorporate into a build via ExternalProject or FetchContent. If you can assume the developer provides those somehow, bringing them in by find_package() shifts responsibility for building them from the project to the developer (or whatever package manager they choose to use). This might be anywhere from entirely appropriate to entirely problematic depending on who your intended audience is.
  • How many dependencies does the project have and what is the maturity of each one? Will the project need to update any of those dependencies often (are they also being actively developed, do you need to follow recent work in them)? What will be the impact on developers working on the project when these dependencies need to be updated (how easy is it for them to update, what assumptions are you making about their development environment and tools)?
  • What breadth of platforms, compilers and CMake generators do you want to support? The bigger this set, the more it may drive you towards building dependencies from source rather than relying on having pre-built packages available to you.
  • Do you want developers to be able to use tools like sanitizers, perform code refactoring across projects or have source code visibility into dependencies within their IDE tools? FetchContent supports all of these requirements quite naturally, but doing so for the other methods it may be more difficult.
  • How will you manage repeatability of building past releases? Will you be able to build a year old release again if you update a build slave? In particular, what assumptions are you making about a CI system's build slaves and what dependencies they have installed (can you have multiple versions of any given dependency available at once, whether that be at a known path or via whatever package manager(s) you choose to use)?

On a side note, this is an area I'm increasingly thinking about these days (I'm the author of the FetchContent module). I'm interested in hearing peoples' views on dependency management and building an understanding of what works for people and what doesn't. If you want to send feedback my way, do feel free to get in touch.

--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: [cmake-developers] Using CMake as a package manager vs using a dedicated package management tool (like Conan)

Craig Scott-3
In reply to this post by Timothy Wrona
(dropping the cmake-developers list since this is really a user issue)


On Wed, Feb 20, 2019 at 2:34 AM Timothy Wrona <[hidden email]> wrote:
Hi Craig,

Thank you for the detailed description!

To answer some of your questions:
  • This project will not be incorporated into a Linux distribution, however I would like to keep it cross platform and it should work on Windows, Mac, and Linux.
  • All of the pieces of the project that I am writing myself are using CMake, but I do have some dependencies on external libs such as "googletest" and "boost". Conan does seem to make using these external dependencies very simple - especially since I am using MinGW to do my compilation and "googletest" doesn't seem to compile out of the box on MinGW without passing specific flags to the compiler. With Conan I get a pre-compiled binary so it just works out of the box.
  • At the current time, this project does not have many dependencies, although it will likely grow quite large. I don't believe the external 3rd party dependencies will need to update frequently, but all of the libraries I am writing for the project will likely change quite a bit throughout working on the project. I would also like to be able to compile these libraries independently for reuse in other projects.
  • I intend to support as many platforms/compilers as possible, I am currently using Qt as my dev environment which allows me to install/configure multiple compilers and try builds with each one - this way I can at least do a build with both MinGW and msvc very quickly.
  • I would like to support tools like sanitizers and code refactoring tools at least within my own libraries (not necessarily with any of the 3rd party libs)
  • I would like past releases to be repeatable/rebuildable as much as possible, although when the compiler is upgraded it is understandable that past versions may have issues. I don't have much understanding of CI systems at this point and it is something I have been trying to become more familiar with. I'd like to avoid having multiple versions of the same dependency, although I don't think having two versions of "googletest" for two separate sub-projects that don't depend on each other would cause any issues.
    • I imagine Conan would make past releases more repeatable since you can fetch binary packages instead of needing to rebuild from source and package versions are always explicit.
I am currently the only developer working on the project, but I would still like to find the most efficient method of managing packages for my particular situation since it will likely save me a lot of pain down the road if I get it right up-front. This project is in the situation where I have a set of different sub-projects (libraries) that all need to be able to be compiled independently for re-use, but I have a main project that will use them all. All of these projects have their own dependencies (some of them 3rd party, some of them written by me). The 3rd party libs will likely rarely change, but the ones written by me may change quite frequently.

This is an approach I was thinking about taking:
  • *So far I have tried this and it is working well* - Manage 3rd party libs with Conan, since I don't need to see the source and it's quite convenient to not need to recompile them this seems to work pretty well. Using "FetchContent" on 3rd party libs and then attempting to compile them yourself can sometimes be tricky (for example "googletest" requires special compiler flags to be compiled with MinGW.) It also ensures that if two projects ask for the same version of a dependency I get only one copy even if the projects are built entirely independent of each other.
  • *I haven't tried this yet, but I am hoping it will work well* - Put my sub-projects (my own custom libraries) into their own independent git repos and pull them into my main project using "FetchContent". Then when I run "FetchContent" it will checkout the sub-projects and I will have all of the source code available. I am hoping if I do this any changes I make to the sub-projects can easily be committed and pushed back to their own independent repositories.
    • The alternative to this would be to put my own sub-projects into their own Conan packages and use Conan to get them from the main project. But I am thinking since they will change frequently, "FetchContent" may be a better fit for this scenario.
    • Maybe when the sub-project libraries reach a mature and stable release they should be packaged into Conan and fetched in the main project from Conan at that point?
Let me know what you think! :)

Sounds like you are making progress exploring things already, you'll need to make your own evaluations about which approach will work best for you.

As a data point, in my day job, most of the time I'm working with a project hierarchy that spans about 40+ dependencies, most of which are internal projects that can all be built standalone or as part of a larger whole. Some are small and relatively trivial, others are much larger and have varying age with the associated quirks that come with that. Most come from git but some come from svn and a small number from release tarballs. Some have code generators that also produce CMakeLists.txt files that other parts of the build need to consume (see this stack overflow Q&A for more on that). We use both GoogleTest and Boost, plus various other third party projects (OpenSSL is probably the one that gives us the most pain). We build for Linux, Windows, Mac, Solaris, iOS and Android. We support our devs building with Make, Ninja, Xcode or Visual Studio (we don't use MinGW). Many of the projects are actively undergoing change across multiple teams and people, most of whom don't have an interest in learning about CMake and build systems and just want to focus on coding. The flexibility and robustness that FetchContent has given us has been a critical factor in being able to work effectively and efficiently. That and ccache. ;) We don't use a package manager other than letting the OS provide some common things on a few platforms. For some larger dependencies (e.g. NDK, Qt) we rely on having pre-built packages installed at predictable locations and accept that we will have to maintain these on our build slaves, although this is an area that is still evolving.

I recently posted how I'm currently working on bringing Boost into our build via a FetchContent-based mechanism. That is still somewhat of a work-in-progress and has holes for multi-config generators (i.e. Visual Studio and Xcode), but they don't look insurmountable if those are important to you. It's by no means the only approach, but it's a starting point if you want to explore it (once Boost completes its move to CMake, I hope to be able to drop this and just bring Boost into the build via add_subdirectory() like most of our other dependencies).

 


On Tue, Feb 19, 2019 at 5:56 AM Craig Scott <[hidden email]> wrote:


On Tue, Feb 19, 2019 at 12:46 PM Timothy Wrona <[hidden email]> wrote:
I have been working on a new C++ project and I am trying to decide whether I should use CMake as my package management system or if I should use a dedicated package management tool such as Conan.

For more information on Conan see: https://conan.io/

I am trying to understand the main difference between using Conan to manage dependencies vs using CMakes "FetchContent" module. Are there any compelling reasons to prefer something like Conan?

Excellent question, one that I think deserves a more detailed answer than I can provide here, but I'll try to hit the main points as I see them.

Personally, I think there is no "right answer" or "one size fits all" when it comes to package management for a CMake project. What works well for one situation, person or project may not be as convenient or suitable for another. There are competing needs and views, some of which are personal preferences, others are hard requirements from things like OS distribution policies for packaging. Even just the maturity of a project can have a big influence on how developers may prefer to handle its dependencies and handle it as a dependency of other projects.

The key thing for me is that the developer should ideally have choices when it comes to this area. If a project hard-codes that its dependencies must come from a particular provider (whether that be Conan, Hunter, vcpkg or some other system), this might not be compatible with what the developer's situation allows. You would need to weigh up whether it makes sense to lock the developer into a particular package manager if they want to use your project or not. An inappropriate choice here can mean lower adoption of the project as some may reject it for consideration based on this point alone.

If instead a project relies only on find_package() to find its dependencies, then it is up to the developer to ensure they are all available. This could be done using whatever package manager the developer finds convenient, or they could build the dependency projects from source individually or they might set up a superbuild parent project that builds the dependencies in the required order and makes dependees available to dependers. This gives good flexibility at the cost of more responsibility on the developer than perhaps some would want (again, it will be highly situation-dependent).

A drawback with find_package() is that it assumes you actually have a packageable project. For a variety of reasons, this may not be the case. Consider a large, complex project in its early stages and where multiple teams are working on different subprojects which all get combined into some larger whole. Each of the subprojects may need to be able to build on their own with their own smaller subset of dependencies, but they also need to be able to be incorporated into a larger build (think of different teams working on core toolkits, rendering engines, different algorithm strategies, multiple GUI applications, backend components, etc). No-one may know yet how it should get packaged up and everyone might be focused on just getting a minimal viable prototype up and running as a technical demonstrator. For a case like that, neither find_package() nor a package manager really fits the workflow. In this situation though, FetchContent is a perfect fit, since it doesn't require any packaging to already be in place, it needs no external tools other than CMake and it gives each project precise control over its dependencies down to the individual commit to bring in for each one.

With the above in mind, perhaps the following few questions may be helpful in clarifying what your constraints are and maybe steering you more toward one way or the other:
  • Will the project be incorporated into a Linux distribution at some point (not just be installed on Linux, but be part of the actual Linux distribution as provided by its own native package manager)? If so, I would expect this would pretty much eliminate using any package manager and instead require that you use find_package() to find all dependencies.
  • Are any of the dependencies of the project using a build system other than CMake? If so, they tend to take a bit more work to incorporate into a build via ExternalProject or FetchContent. If you can assume the developer provides those somehow, bringing them in by find_package() shifts responsibility for building them from the project to the developer (or whatever package manager they choose to use). This might be anywhere from entirely appropriate to entirely problematic depending on who your intended audience is.
  • How many dependencies does the project have and what is the maturity of each one? Will the project need to update any of those dependencies often (are they also being actively developed, do you need to follow recent work in them)? What will be the impact on developers working on the project when these dependencies need to be updated (how easy is it for them to update, what assumptions are you making about their development environment and tools)?
  • What breadth of platforms, compilers and CMake generators do you want to support? The bigger this set, the more it may drive you towards building dependencies from source rather than relying on having pre-built packages available to you.
  • Do you want developers to be able to use tools like sanitizers, perform code refactoring across projects or have source code visibility into dependencies within their IDE tools? FetchContent supports all of these requirements quite naturally, but doing so for the other methods it may be more difficult.
  • How will you manage repeatability of building past releases? Will you be able to build a year old release again if you update a build slave? In particular, what assumptions are you making about a CI system's build slaves and what dependencies they have installed (can you have multiple versions of any given dependency available at once, whether that be at a known path or via whatever package manager(s) you choose to use)?

On a side note, this is an area I'm increasingly thinking about these days (I'm the author of the FetchContent module). I'm interested in hearing peoples' views on dependency management and building an understanding of what works for people and what doesn't. If you want to send feedback my way, do feel free to get in touch.

--
Craig Scott
Melbourne, Australia

Get the hand-book for every CMake user: Professional CMake: A Practical Guide

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake