dependencies of cross compiliations

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

dependencies of cross compiliations

Torsten Robitzki
Hi,
I hope this question was not asked before. I work in the embedded field and there it is usually to have at least two different build platforms. The Host platform, where unit tests are build (and where CMake is running) and an embedded Target platform, where targets are build with a cross compiler. Sometimes such a system comes with self-written tools that are build and run on the Host platform to build a target for the embedded Target platform (adding meta data to a binary to be used by a bootloader for example).

Usually I have two different build folders, one for the Host platform and one for the Target platform, using different calls to cmake to choose from a set of tools and targets. But when using this approach, it is necessary that the Host platform build ran before the Target platform build, so that tools that are required for the Target platform are build during the Host target build.

One solution I’ve came up with, is to build the required tools during the Target platform build, using an add_custom_target() to invoke the Target compiler directly. This works fine, as long as the tools are basically build just out of a couple of files.

What would be the „CMake-Way“ to add the tools (that have to be build on the Target platform) as dependency to targets that have to be build for the Target (cross compile) platform?

Kind regards and thanks in advance,

Torsten
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Miller Henry
There are two options. Each with pros and cons.

The first what you are doing now, except you use external project https://cmake.org/cmake/help/v3.12/module/ExternalProject.html to build the host tools instead of add_custom_command. It otherwise has all the cons you mention, but are they significant (this is a question that is different for each project)?  Going back to the manual build for host before the target isn't bad, though it is harder to teach.

The second is to use a package/dependency manager to setup your build environment with all tools. In this approach the host tools would be in a separate source control repository and the package manager will download the latest binaries. Conan.io comes to mind as a tool to do this, but there are at least a dozen others that you can choose from (or write your own if you have a year to spend) . This approach is much more complex, but these systems offer many other features that are often compelling.

I would recommend you spend some time researching the package/dependency manager tools available. It is likely that you (or your team) will look at some feature completely unrelated to your question and say "This is a good solution to our problem". At that point your question changes from "how do I best make cmake do this" to "how do I do this in my new package manager". If after a review nothing stands out, your approach is perfectly fine, the only question is it worth tweaking.

-----Original Message-----
From: CMake [mailto:[hidden email]] On Behalf Of Torsten Robitzki
Sent: Friday, November 9, 2018 3:04 AM
To: [hidden email]
Subject: [CMake] dependencies of cross compiliations

Hi,
I hope this question was not asked before. I work in the embedded field and there it is usually to have at least two different build platforms. The Host platform, where unit tests are build (and where CMake is running) and an embedded Target platform, where targets are build with a cross compiler. Sometimes such a system comes with self-written tools that are build and run on the Host platform to build a target for the embedded Target platform (adding meta data to a binary to be used by a bootloader for example).

Usually I have two different build folders, one for the Host platform and one for the Target platform, using different calls to cmake to choose from a set of tools and targets. But when using this approach, it is necessary that the Host platform build ran before the Target platform build, so that tools that are required for the Target platform are build during the Host target build.

One solution I’ve came up with, is to build the required tools during the Target platform build, using an add_custom_target() to invoke the Target compiler directly. This works fine, as long as the tools are basically build just out of a couple of files.

What would be the „CMake-Way“ to add the tools (that have to be build on the Target platform) as dependency to targets that have to be build for the Target (cross compile) platform?

Kind regards and thanks in advance,

Torsten
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Torsten Robitzki


> Am 09.11.2018 um 15:55 schrieb Miller Henry <[hidden email]>:
>
> There are two options. Each with pros and cons.
>
> The first what you are doing now, except you use external project https://cmake.org/cmake/help/v3.12/module/ExternalProject.html to build the host tools instead of add_custom_command. It otherwise has all the cons you mention, but are they significant (this is a question that is different for each project)?  Going back to the manual build for host before the target isn’t bad, though it is harder to teach.

I think for the example use case, this will work best. Especially, if there are source files (for example constants) are shared among both platforms.

Thanks,

Torsten
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Rolf Eike Beer
In reply to this post by Torsten Robitzki
Am 2018-11-09 10:04, schrieb Torsten Robitzki:

> Hi,
> I hope this question was not asked before. I work in the embedded
> field and there it is usually to have at least two different build
> platforms. The Host platform, where unit tests are build (and where
> CMake is running) and an embedded Target platform, where targets are
> build with a cross compiler. Sometimes such a system comes with
> self-written tools that are build and run on the Host platform to
> build a target for the embedded Target platform (adding meta data to a
> binary to be used by a bootloader for example).
>
> Usually I have two different build folders, one for the Host platform
> and one for the Target platform, using different calls to cmake to
> choose from a set of tools and targets. But when using this approach,
> it is necessary that the Host platform build ran before the Target
> platform build, so that tools that are required for the Target
> platform are build during the Host target build.
>
> One solution I’ve came up with, is to build the required tools during
> the Target platform build, using an add_custom_target() to invoke the
> Target compiler directly. This works fine, as long as the tools are
> basically build just out of a couple of files.
>
> What would be the „CMake-Way“ to add the tools (that have to be build
> on the Target platform) as dependency to targets that have to be build
> for the Target (cross compile) platform?

TL;DR: there is not "good" way yet. But there should be one.

I'm hijacking this and move it to the developers list, because that is
something "big", and we need to think about how to do that. I find it
important to correctly solve this as it would simplify a lot of things.
Especially given that Qt is thinking to use CMake to build Qt itself,
which I bet all of us would love to see. But they will be after us if we
don't offer a solution for this. And given the increasing amount of
cross-setups these days I'm sure that a lot of other people would
benefit.

My first idea was to have something like add_host_executable(), which
would only be called when this is not CMAKE_CROSSCOMPILING, but at the
end I think this clutters things too much.

Then I came up with:

   add_host_build("relative source dir" "build dir" [VARS])

This would create an entirely new CMake scope (with it's own
CMakeCache.txt and the like) in "${CMAKE_CURRENT_BUILD_DIR}/build dir",
and would not take the CMAKE_TOOLCHAIN_FILE into account. People are
free to pass "." as relative source dir in case they want to start at
top level, but they may as well pass "tools", "generators" or whatever
they call it. This is not intended to be called multiple times from the
same project as it would scan for compiler and environment once for
every call, but doing so does no harm beyond being slow(er) and the
targets of one such sub-build not being visible to the others.

My idea would be that things added by add_executable() inside such a
sub-build are visible as targets from the outer build. Other things like
libraries and the like need not to be, they can't be linked in the outer
world. The user is free to build with shared libs inside, and running
the things from the build tree would result in the correct freshly build
libs being picked up because of RPATH or whatever. There is no install
of those targets possible from the outer build, this can entirely be
managed from the host build. Of course one must be possible to set
variables on the host build, that's where VARS comes into play. This
holds a list of variable names that will be passed to the hostbuild. No
values, to avoid all sorts of quoting issues. Helpful would be a special
variable for CMAKE_INSTALL_PREFIX as this needs a bit of attention (and
a non-sysroot thing prefix in the toolchain file). Confused? Granted,
here is an example:

if (CMAKE_CROSSCOMPILING)
     set(HOST_INSTALL_DIR "/some/where")
     add_host_build(. host HOST_INSTALL_DIR)
endif ()
add_executable(magic magic.cpp)
install(TARGETS magic DESTINATION bin) # installs both the host and the
target tool!
add_custom_command(OUTPUT ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp COMMAND
magic) # will call the host build
if (NOT CMAKE_HOST_BUILD)
     add_executable(foo ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp)
     install(TARGETS foo DESTINATION bin)
endif ()

This should end up in a layout like this:

/tmp/install/prefix/tools/bin/magic.exe # Windows host
/tmp/install/prefix/sysroot/usr/bin/magic # Unix guest
/tmp/install/prefix/sysroot/usr/bin/foo

The toolchain file would look somehow like this:

set(CMAKE_HOST_PREFIX prefix
set(CMAKE_SYSROOT ${CMAKE_HOST_PREFIX}/sysroot)

and the CMake command would look like this:

cmake -D CMAKE_TOOLCHAIN_FILE=tc.cmake -D CMAKE_INSTALL_PREFIX=/usr -D
CMAKE_HOST_INSTALL_PREFIX=tools ...

The wish-season is coming up, so that's sort of what I would like to
have. Now it's your turn. No bikeshedding please, only deliveries ;)

Eike
--
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Ray Donnelly
which I bet all of us would love to see.

This is not correct. I would strongly prefer they continue with QBS instead. Cmake is defacto, but very suboptional.

On Tue, Nov 27, 2018, 10:28 AM Rolf Eike Beer <[hidden email] wrote:
Am 2018-11-09 10:04, schrieb Torsten Robitzki:
> Hi,
> I hope this question was not asked before. I work in the embedded
> field and there it is usually to have at least two different build
> platforms. The Host platform, where unit tests are build (and where
> CMake is running) and an embedded Target platform, where targets are
> build with a cross compiler. Sometimes such a system comes with
> self-written tools that are build and run on the Host platform to
> build a target for the embedded Target platform (adding meta data to a
> binary to be used by a bootloader for example).
>
> Usually I have two different build folders, one for the Host platform
> and one for the Target platform, using different calls to cmake to
> choose from a set of tools and targets. But when using this approach,
> it is necessary that the Host platform build ran before the Target
> platform build, so that tools that are required for the Target
> platform are build during the Host target build.
>
> One solution I’ve came up with, is to build the required tools during
> the Target platform build, using an add_custom_target() to invoke the
> Target compiler directly. This works fine, as long as the tools are
> basically build just out of a couple of files.
>
> What would be the „CMake-Way“ to add the tools (that have to be build
> on the Target platform) as dependency to targets that have to be build
> for the Target (cross compile) platform?

TL;DR: there is not "good" way yet. But there should be one.

I'm hijacking this and move it to the developers list, because that is
something "big", and we need to think about how to do that. I find it
important to correctly solve this as it would simplify a lot of things.
Especially given that Qt is thinking to use CMake to build Qt itself,
which I bet all of us would love to see. But they will be after us if we
don't offer a solution for this. And given the increasing amount of
cross-setups these days I'm sure that a lot of other people would
benefit.

My first idea was to have something like add_host_executable(), which
would only be called when this is not CMAKE_CROSSCOMPILING, but at the
end I think this clutters things too much.

Then I came up with:

   add_host_build("relative source dir" "build dir" [VARS])

This would create an entirely new CMake scope (with it's own
CMakeCache.txt and the like) in "${CMAKE_CURRENT_BUILD_DIR}/build dir",
and would not take the CMAKE_TOOLCHAIN_FILE into account. People are
free to pass "." as relative source dir in case they want to start at
top level, but they may as well pass "tools", "generators" or whatever
they call it. This is not intended to be called multiple times from the
same project as it would scan for compiler and environment once for
every call, but doing so does no harm beyond being slow(er) and the
targets of one such sub-build not being visible to the others.

My idea would be that things added by add_executable() inside such a
sub-build are visible as targets from the outer build. Other things like
libraries and the like need not to be, they can't be linked in the outer
world. The user is free to build with shared libs inside, and running
the things from the build tree would result in the correct freshly build
libs being picked up because of RPATH or whatever. There is no install
of those targets possible from the outer build, this can entirely be
managed from the host build. Of course one must be possible to set
variables on the host build, that's where VARS comes into play. This
holds a list of variable names that will be passed to the hostbuild. No
values, to avoid all sorts of quoting issues. Helpful would be a special
variable for CMAKE_INSTALL_PREFIX as this needs a bit of attention (and
a non-sysroot thing prefix in the toolchain file). Confused? Granted,
here is an example:

if (CMAKE_CROSSCOMPILING)
     set(HOST_INSTALL_DIR "/some/where")
     add_host_build(. host HOST_INSTALL_DIR)
endif ()
add_executable(magic magic.cpp)
install(TARGETS magic DESTINATION bin) # installs both the host and the
target tool!
add_custom_command(OUTPUT ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp COMMAND
magic) # will call the host build
if (NOT CMAKE_HOST_BUILD)
     add_executable(foo ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp)
     install(TARGETS foo DESTINATION bin)
endif ()

This should end up in a layout like this:

/tmp/install/prefix/tools/bin/magic.exe # Windows host
/tmp/install/prefix/sysroot/usr/bin/magic # Unix guest
/tmp/install/prefix/sysroot/usr/bin/foo

The toolchain file would look somehow like this:

set(CMAKE_HOST_PREFIX prefix
set(CMAKE_SYSROOT ${CMAKE_HOST_PREFIX}/sysroot)

and the CMake command would look like this:

cmake -D CMAKE_TOOLCHAIN_FILE=tc.cmake -D CMAKE_INSTALL_PREFIX=/usr -D
CMAKE_HOST_INSTALL_PREFIX=tools ...

The wish-season is coming up, so that's sort of what I would like to
have. Now it's your turn. No bikeshedding please, only deliveries ;)

Eike
--
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: building host tools during cross compiliations (was: dependencies of cross compiliations)

Brad King
In reply to this post by Rolf Eike Beer
On 11/27/18 5:28 AM, Rolf Eike Beer wrote:
> Then I came up with:
>
>    add_host_build("relative source dir" "build dir" [VARS])
>
> This would create an entirely new CMake scope (with it's own
> CMakeCache.txt and the like) in "${CMAKE_CURRENT_BUILD_DIR}/build dir",
> and would not take the CMAKE_TOOLCHAIN_FILE into account.
[snip]
> My idea would be that things added by add_executable() inside such a
> sub-build are visible as targets from the outer build.

For reference, some projects are already using the ExternalProject
module to approximate that approach and bring in the host tools as
imported executable targets.  The actual host build is viewed as
a custom target from the cross-compiled project.

-Brad
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Miller Henry
In reply to this post by Rolf Eike Beer
I like the idea, some tweaks that I think you need

add_host_build - executables built here need to be available not as targets, but as variables as if FIND_EXECUTABLE was called on them. This is probably what you meant, but it didn't come across that way.

In the case where you are cross compiling for more than one target it would be useful if you could build the host tools once and then use that build in multiple target builds. There are two different workflows for this: building locally where the host build directory just needs to be shared (how?); building on a CI system where you want to build/package the host tools and then send the package to downstream jobs.  In my opinion this second workflow is the more important one to have work.

I would suggest that it work something like this: A new option -DCMAKE_HOST_TOOLS_BUILD_DIR which if set will specify the directory to find host tools in - if this directory already has a built use, the tools already built there (of course re-run that build as required on source changes); otherwise configure and build host tools in that directory.  If this options is NOT specified first run find_package to see if a cmake config file can be found for the correct version of host tools - if so use them (I think we only want to support cmake config files not the other options find_package looks for), it is the users responsibility to update the installed tools if the source changes.  If all of the above fails fall back to some directory under the build directory (as you proposed?).

The above also solves the problem of what if you want to pass a ton of options to the host build.  Adding overrides to cmake options for host vs target (ie -DHOST_CMAKE_BUILD_TYPE) gets out of hand quickly, both from the developer that has to write all the cmake options and also the user who has to maintain long command lines.  

Keep in mind there are some projects where the above cannot work.  I know of one program where the host tool creates data packed for the specific CPU (endian and bits in an int) and so you cannot share then host build. This project doesn't support cross compiling the host tool so I run it under QEMU, but you can imagine it being properly supported in the future and so add_host_build may need to have something to indicate the target to build the host tool for.

Just some thoughts. Since I'm not doing the work I don't get to tell you how to do it.

-----Original Message-----
From: CMake [mailto:[hidden email]] On Behalf Of Rolf Eike Beer
Sent: Tuesday, November 27, 2018 4:28 AM
To: [hidden email]
Cc: [hidden email]
Subject: Re: [CMake] dependencies of cross compiliations

Am 2018-11-09 10:04, schrieb Torsten Robitzki:

> Hi,
> I hope this question was not asked before. I work in the embedded
> field and there it is usually to have at least two different build
> platforms. The Host platform, where unit tests are build (and where
> CMake is running) and an embedded Target platform, where targets are
> build with a cross compiler. Sometimes such a system comes with
> self-written tools that are build and run on the Host platform to
> build a target for the embedded Target platform (adding meta data to a
> binary to be used by a bootloader for example).
>
> Usually I have two different build folders, one for the Host platform
> and one for the Target platform, using different calls to cmake to
> choose from a set of tools and targets. But when using this approach,
> it is necessary that the Host platform build ran before the Target
> platform build, so that tools that are required for the Target
> platform are build during the Host target build.
>
> One solution I’ve came up with, is to build the required tools during
> the Target platform build, using an add_custom_target() to invoke the
> Target compiler directly. This works fine, as long as the tools are
> basically build just out of a couple of files.
>
> What would be the „CMake-Way“ to add the tools (that have to be build
> on the Target platform) as dependency to targets that have to be build
> for the Target (cross compile) platform?

TL;DR: there is not "good" way yet. But there should be one.

I'm hijacking this and move it to the developers list, because that is something "big", and we need to think about how to do that. I find it important to correctly solve this as it would simplify a lot of things.
Especially given that Qt is thinking to use CMake to build Qt itself, which I bet all of us would love to see. But they will be after us if we don't offer a solution for this. And given the increasing amount of cross-setups these days I'm sure that a lot of other people would benefit.

My first idea was to have something like add_host_executable(), which would only be called when this is not CMAKE_CROSSCOMPILING, but at the end I think this clutters things too much.

Then I came up with:

   add_host_build("relative source dir" "build dir" [VARS])

This would create an entirely new CMake scope (with it's own CMakeCache.txt and the like) in "${CMAKE_CURRENT_BUILD_DIR}/build dir", and would not take the CMAKE_TOOLCHAIN_FILE into account. People are free to pass "." as relative source dir in case they want to start at top level, but they may as well pass "tools", "generators" or whatever they call it. This is not intended to be called multiple times from the same project as it would scan for compiler and environment once for every call, but doing so does no harm beyond being slow(er) and the targets of one such sub-build not being visible to the others.

My idea would be that things added by add_executable() inside such a sub-build are visible as targets from the outer build. Other things like libraries and the like need not to be, they can't be linked in the outer world. The user is free to build with shared libs inside, and running the things from the build tree would result in the correct freshly build libs being picked up because of RPATH or whatever. There is no install of those targets possible from the outer build, this can entirely be managed from the host build. Of course one must be possible to set variables on the host build, that's where VARS comes into play. This holds a list of variable names that will be passed to the hostbuild. No values, to avoid all sorts of quoting issues. Helpful would be a special variable for CMAKE_INSTALL_PREFIX as this needs a bit of attention (and a non-sysroot thing prefix in the toolchain file). Confused? Granted, here is an example:

if (CMAKE_CROSSCOMPILING)
     set(HOST_INSTALL_DIR "/some/where")
     add_host_build(. host HOST_INSTALL_DIR) endif () add_executable(magic magic.cpp) install(TARGETS magic DESTINATION bin) # installs both the host and the target tool!
add_custom_command(OUTPUT ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp COMMAND
magic) # will call the host build
if (NOT CMAKE_HOST_BUILD)
     add_executable(foo ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp)
     install(TARGETS foo DESTINATION bin) endif ()

This should end up in a layout like this:

/tmp/install/prefix/tools/bin/magic.exe # Windows host /tmp/install/prefix/sysroot/usr/bin/magic # Unix guest /tmp/install/prefix/sysroot/usr/bin/foo

The toolchain file would look somehow like this:

set(CMAKE_HOST_PREFIX prefix
set(CMAKE_SYSROOT ${CMAKE_HOST_PREFIX}/sysroot)

and the CMake command would look like this:

cmake -D CMAKE_TOOLCHAIN_FILE=tc.cmake -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_HOST_INSTALL_PREFIX=tools ...

The wish-season is coming up, so that's sort of what I would like to have. Now it's your turn. No bikeshedding please, only deliveries ;)

Eike
--
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Eric Noulard
In reply to this post by Rolf Eike Beer


Le mar. 27 nov. 2018 à 11:28, Rolf Eike Beer <[hidden email]> a écrit :
Am 2018-11-09 10:04, schrieb Torsten Robitzki:
> Hi,
> I hope this question was not asked before. I work in the embedded
> field and there it is usually to have at least two different build
> platforms. The Host platform, where unit tests are build (and where
> CMake is running) and an embedded Target platform, where targets are
> build with a cross compiler. Sometimes such a system comes with
> self-written tools that are build and run on the Host platform to
> build a target for the embedded Target platform (adding meta data to a
> binary to be used by a bootloader for example).
>
> Usually I have two different build folders, one for the Host platform
> and one for the Target platform, using different calls to cmake to
> choose from a set of tools and targets. But when using this approach,
> it is necessary that the Host platform build ran before the Target
> platform build, so that tools that are required for the Target
> platform are build during the Host target build.
>
> One solution I’ve came up with, is to build the required tools during
> the Target platform build, using an add_custom_target() to invoke the
> Target compiler directly. This works fine, as long as the tools are
> basically build just out of a couple of files.
>
> What would be the „CMake-Way“ to add the tools (that have to be build
> on the Target platform) as dependency to targets that have to be build
> for the Target (cross compile) platform?

TL;DR: there is not "good" way yet. But there should be one.


I do agree with that quote I was quite surprised (a long time ago) that CMake did not support cross-compiling.
Back then I was using recursive hand-written makefiles for cross-compiling. When I wanted to build
the whole thing I only had to hit "make" and wait.

Moreover I think CMake cross-compiling support was biased by the fact CMake wasn't designed for that initially.
Please don't take my remark as bare criticism I am using CMake for a long time now, I do like CMake very much
and I was pleased to see the cross-compiling support coming.

However from my point of view and my cross-compiling experience when you cross-compile you have:

1) the host compiler which is used to compile "host tools" 
2) the target compiler (may be several of them) to "cross-compile" 

My assumption are:
 a) when you cross-compile your build is a "whole" and you shouldn't have to setup some superbuild
   structure for building host tools ht_exe and another for target1 tool t1t_exe and another one for target2 tool t2t_exe.
 
 b) what you want is to build:
     ht_exe for the host
     possibly use ht_exe during the build to generate some [source] file
     t1t_exe for the [cross]target1
     t2t_exe for the [cross]target2

 c)  you seldomly compile the same source for the host AND the target, but it may happen.
  
And you want to build all that stuff with a single configure+build command AND take advantage
of fast and efficient parallel build for the **whole build**. I don't want to 

cd /build/for/host
ninja
cd /build/for/target1
ninja
etc...

 
 Helpful would be a special
variable for CMAKE_INSTALL_PREFIX as this needs a bit of attention (and
a non-sysroot thing prefix in the toolchain file). Confused? Granted,
here is an example:

if (CMAKE_CROSSCOMPILING)
     set(HOST_INSTALL_DIR "/some/where")
     add_host_build(. host HOST_INSTALL_DIR)
endif ()
add_executable(magic magic.cpp)
install(TARGETS magic DESTINATION bin) # installs both the host and the
target tool!
add_custom_command(OUTPUT ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp COMMAND
magic) # will call the host build
if (NOT CMAKE_HOST_BUILD)
     add_executable(foo ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp)
     install(TARGETS foo DESTINATION bin)
endif ()

I get your point but I think we may try a more declarative way.

add_executable(magic magic.cpp)
install(TARGETS magic DESTINATION bin)
add_custom_command(OUTPUT ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp COMMAND magic)
add_executable(foo ${CMAKE_CURRENT_BUILD_DIR}/foo.cpp)
install(TARGETS foo DESTINATION bin)

set_target_properties(magic PROPERTIES BUILD_TARGET "host;cross_target1")
set_target_properties(foo PROPERTIES BUILD_TARGET "cross_target1")

after that we know that `magic` is to be built both for "host" and "cross_target1" whereas
`foo` is only for "cross_target1". 

before that we may have to "declaratively" define what is cross_target1 (and may be cross_target2) with something like:

enable_cross_target(NAME cross_target1 TOOLCHAIN ${CMAKE_CURRENT_SOURCE _DIR}/cmake/target1-toolchain.cmake)
enable_cross_target(NAME cross_target2 TOOLCHAIN ${CMAKE_CURRENT_SOURCE _DIR}/cmake/target2-toolchain.cmake)
and assume "host" builtin target is the one coming from the command line.

each define_cross_target(..) will create a separate subdir in the build tree (much like CMAKE_CFG_INTDIR is working for multi-config generators)
may something like ${CMAKE_CURRENT_BINARY_DIR}/${CROSS_TARGET_NAME} if we assume cross target name are unique.

all cmake command (install, add_executable, add_library, etc...) shall know that a target is to be build for the host and/or cross target1
and/or cross target2 etc...

add_custom_command COMMAND argument refering to a built target 
always refer to "host target" because there is usually no need (or mean) to execute cross target during the build anyway.

in the end you will have a project which specify all the target (exe, lib, custom) the usual declarative "modern CMake" way
and you won't have your CMakeLists.txt filled with 

IF(CMAKE_CROSSCOMPILING) 

IF(NOT CMAKE_HOST_BUILD)

controls.

if you need cross_target specific CMAKE_XXXX variables, they should either go in the target toolchain file 
or should be put in with 
cross_target_set(cross_target1 VARIABLE CMAKE_INSTALL_PREFIX "/opt/target1")

You may even decide that if a "build target" is not enabled because you have something like:

if (CROSS_BUILD_TARGET1)
   enable_cross_target(NAME cross_target1 TOOLCHAIN ${CMAKE_CURRENT_SOURCE _DIR}/cmake/target1-toolchain.cmake)
endif()

if (CROSS_BUILD_TARGET2)
   enable_cross_target(NAME cross_target2 TOOLCHAIN ${CMAKE_CURRENT_SOURCE _DIR}/cmake/target2-toolchain.cmake)
endif()

The build could succeed and emit some warning telling that some "cross target" were not built.

This should end up in a layout like this:

/tmp/install/prefix/tools/bin/magic.exe # Windows host
/tmp/install/prefix/sysroot/usr/bin/magic # Unix guest
/tmp/install/prefix/sysroot/usr/bin/foo

The toolchain file would look somehow like this:

set(CMAKE_HOST_PREFIX prefix
set(CMAKE_SYSROOT ${CMAKE_HOST_PREFIX}/sysroot)

and the CMake command would look like this:

cmake -D CMAKE_TOOLCHAIN_FILE=tc.cmake -D CMAKE_INSTALL_PREFIX=/usr -D
CMAKE_HOST_INSTALL_PREFIX=tools ...

I think that most of the time specifying the toolchain on the command line drives you to some superbuild structure.
 

The wish-season is coming up, so that's sort of what I would like to
have. Now it's your turn. No bikeshedding please, only deliveries ;)

I wish an integrated multi-target cross building support in CMake with little or no flow-control scripting command in the CMakeLists.txt. 

--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Torsten Robitzki
> Am 27.11.2018 um 19:55 schrieb Eric Noulard <[hidden email]>:

>
> However from my point of view and my cross-compiling experience when you cross-compile you have:
>
> 1) the host compiler which is used to compile "host tools"
> 2) the target compiler (may be several of them) to "cross-compile"
>
> My assumption are:
>  a) when you cross-compile your build is a "whole" and you shouldn't have to setup some superbuild
>    structure for building host tools ht_exe and another for target1 tool t1t_exe and another one for target2 tool t2t_exe.
>
>  b) what you want is to build:
>      ht_exe for the host
>      possibly use ht_exe during the build to generate some [source] file
>      t1t_exe for the [cross]target1
>      t2t_exe for the [cross]target2
>
>  c)  you seldomly compile the same source for the host AND the target, but it may happen.
In case, you are doing unit tests, it’s normal to have the same code running in a test on the host platform and in the final binary on the target.

I think, having more than 1 target platform becomes more and more normal as it becomes more usual to have multiple microcontrollers in a project.

Previously, I have encoded this in the build type. So instead of just having Debug and Release, I had HOST_Debug, HOST_Release NRF51_Debug, NRF51_Release, STM8_Debug, STM8_Release and so on. It doesn’t annoy me very much, that I have to run CMake 3 times to get all the binaries for a release build. The problem that I have, are dependencies between this builds. If I write a tool that (for example) generates source files for one of the target platforms, the build for the host platform must run before the build for that target platform. And when I make changes to that tool, I want the build to regenerate the generated source files.

Keeping track of this dependencies to solve this kind of ordering issues and to allow minimum rebuilds, is the main purpose of any build system. To solve this with CMake, I think we need a way to define the dependencies between build types (in the example above, from the generator from the host build to the generated source file in one of the target builds) and CMake needs to know the build directory for all build types (not only the current).

> The wish-season is coming up, so that's sort of what I would like to
> have. Now it's your turn. No bikeshedding please, only deliveries ;)

How about ``add_dependencies()`` allowing me to define dependencies between different build types? :-)

best regards,
Torsten

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Craig Scott-3


On Tue, Dec 4, 2018 at 6:56 PM Torsten Robitzki <[hidden email]> wrote:
> Am 27.11.2018 um 19:55 schrieb Eric Noulard <[hidden email]>:
>
> However from my point of view and my cross-compiling experience when you cross-compile you have:
>
> 1) the host compiler which is used to compile "host tools"
> 2) the target compiler (may be several of them) to "cross-compile"
>
> My assumption are:
>  a) when you cross-compile your build is a "whole" and you shouldn't have to setup some superbuild
>    structure for building host tools ht_exe and another for target1 tool t1t_exe and another one for target2 tool t2t_exe.
>
>  b) what you want is to build:
>      ht_exe for the host
>      possibly use ht_exe during the build to generate some [source] file
>      t1t_exe for the [cross]target1
>      t2t_exe for the [cross]target2
>
>  c)  you seldomly compile the same source for the host AND the target, but it may happen.

In case, you are doing unit tests, it’s normal to have the same code running in a test on the host platform and in the final binary on the target.

I think, having more than 1 target platform becomes more and more normal as it becomes more usual to have multiple microcontrollers in a project.

Previously, I have encoded this in the build type. So instead of just having Debug and Release, I had HOST_Debug, HOST_Release NRF51_Debug, NRF51_Release, STM8_Debug, STM8_Release and so on. It doesn’t annoy me very much, that I have to run CMake 3 times to get all the binaries for a release build. The problem that I have, are dependencies between this builds. If I write a tool that (for example) generates source files for one of the target platforms, the build for the host platform must run before the build for that target platform. And when I make changes to that tool, I want the build to regenerate the generated source files.

Keeping track of this dependencies to solve this kind of ordering issues and to allow minimum rebuilds, is the main purpose of any build system. To solve this with CMake, I think we need a way to define the dependencies between build types (in the example above, from the generator from the host build to the generated source file in one of the target builds) and CMake needs to know the build directory for all build types (not only the current).

Perhaps a superbuild would be the cleanest approach here? The host tools would be one subproject and the cross-compile builds would depend on the host tools' build. You could then choose to build everything via the top level superbuild or just work on one of the subprojects if that's all you needed once the initial tools build had been done. You could even set up as many different sub-projects for the different architectures as needed. Packaging would require a little more work, but it shouldn't be prohibitively so.

Another alternative is the approach described in this stackoverflow article which performs the host tools build off to the side in a secondary build during configure. This works well when the host tools don't change much (we use it extensively at work with very large, complex hierarchical projects). It wouldn't help though if you need to build more than one cross-compiled architecture.

 
> The wish-season is coming up, so that's sort of what I would like to
> have. Now it's your turn. No bikeshedding please, only deliveries ;)

How about ``add_dependencies()`` allowing me to define dependencies between different build types? :-)

A superbuild would already give you the equivalent capability.


--
Craig Scott
Melbourne, Australia


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Eric Noulard

Le dim. 9 déc. 2018 à 12:24, Craig Scott <[hidden email]> a écrit :
On Tue, Dec 4, 2018 at 6:56 PM Torsten Robitzki <[hidden email]> wrote:
> Am 27.11.2018 um 19:55 schrieb Eric Noulard <[hidden email]>:
>
> My assumption are:
>  a) when you cross-compile your build is a "whole" and you shouldn't have to setup some superbuild
>    structure for building host tools ht_exe and another for target1 tool t1t_exe and another one for target2 tool t2t_exe.
>
>  b) what you want is to build:
>      ht_exe for the host
>      possibly use ht_exe during the build to generate some [source] file
>      t1t_exe for the [cross]target1
>      t2t_exe for the [cross]target2
>
>  c)  you seldomly compile the same source for the host AND the target, but it may happen.

In case, you are doing unit tests, it’s normal to have the same code running in a test on the host platform and in the final binary on the target.

I think, having more than 1 target platform becomes more and more normal as it becomes more usual to have multiple microcontrollers in a project.

Yes that's why I thought it was worth going further than host + target, but host + tgt1 + tg2 + ....
 

Previously, I have encoded this in the build type. So instead of just having Debug and Release, I had HOST_Debug, HOST_Release NRF51_Debug, NRF51_Release, STM8_Debug, STM8_Release and so on. It doesn’t annoy me very much, that I have to run CMake 3 times to get all the binaries for a release build. The problem that I have, are dependencies between this builds. If I write a tool that (for example) generates source files for one of the target platforms, the build for the host platform must run before the build for that target platform. And when I make changes to that tool, I want the build to regenerate the generated source files.

Keeping track of this dependencies to solve this kind of ordering issues and to allow minimum rebuilds, is the main purpose of any build system. To solve this with CMake, I think we need a way to define the dependencies between build types (in the example above, from the generator from the host build to the generated source file in one of the target builds) and CMake needs to know the build directory for all build types (not only the current).

Perhaps a superbuild would be the cleanest approach here? The host tools would be one subproject and the cross-compile builds would depend on the host tools' build. You could then choose to build everything via the top level superbuild or just work on one of the subprojects if that's all you needed once the initial tools build had been done. You could even set up as many different sub-projects for the different architectures as needed. Packaging would require a little more work, but it shouldn't be prohibitively so.

I guess the tough part is to find a [light] way to specify dependencies between host target build and the various target builds.
  
Another alternative is the approach described in this stackoverflow article which performs the host tools build off to the side in a secondary build during configure. This works well when the host tools don't change much (we use it extensively at work with very large, complex hierarchical projects). It wouldn't help though if you need to build more than one cross-compiled architecture.

> The wish-season is coming up, so that's sort of what I would like to
> have. Now it's your turn. No bikeshedding please, only deliveries ;)

How about ``add_dependencies()`` allowing me to define dependencies between different build types? :-)

A superbuild would already give you the equivalent capability.

Not as easy as it seems right? 
I bet you know it well as you listed the dependencies shortcoming of adding dependencies for External_ProjectAdd in your book (§27.1.4).

--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: dependencies of cross compiliations

Craig Scott-3


On Mon, Dec 10, 2018 at 7:57 PM Eric Noulard <[hidden email]> wrote:

Le dim. 9 déc. 2018 à 12:24, Craig Scott <[hidden email]> a écrit :
On Tue, Dec 4, 2018 at 6:56 PM Torsten Robitzki <[hidden email]> wrote:
> Am 27.11.2018 um 19:55 schrieb Eric Noulard <[hidden email]>:
>
> My assumption are:
>  a) when you cross-compile your build is a "whole" and you shouldn't have to setup some superbuild
>    structure for building host tools ht_exe and another for target1 tool t1t_exe and another one for target2 tool t2t_exe.
>
>  b) what you want is to build:
>      ht_exe for the host
>      possibly use ht_exe during the build to generate some [source] file
>      t1t_exe for the [cross]target1
>      t2t_exe for the [cross]target2
>
>  c)  you seldomly compile the same source for the host AND the target, but it may happen.

In case, you are doing unit tests, it’s normal to have the same code running in a test on the host platform and in the final binary on the target.

I think, having more than 1 target platform becomes more and more normal as it becomes more usual to have multiple microcontrollers in a project.

Yes that's why I thought it was worth going further than host + target, but host + tgt1 + tg2 + ....
 

Previously, I have encoded this in the build type. So instead of just having Debug and Release, I had HOST_Debug, HOST_Release NRF51_Debug, NRF51_Release, STM8_Debug, STM8_Release and so on. It doesn’t annoy me very much, that I have to run CMake 3 times to get all the binaries for a release build. The problem that I have, are dependencies between this builds. If I write a tool that (for example) generates source files for one of the target platforms, the build for the host platform must run before the build for that target platform. And when I make changes to that tool, I want the build to regenerate the generated source files.

Keeping track of this dependencies to solve this kind of ordering issues and to allow minimum rebuilds, is the main purpose of any build system. To solve this with CMake, I think we need a way to define the dependencies between build types (in the example above, from the generator from the host build to the generated source file in one of the target builds) and CMake needs to know the build directory for all build types (not only the current).

Perhaps a superbuild would be the cleanest approach here? The host tools would be one subproject and the cross-compile builds would depend on the host tools' build. You could then choose to build everything via the top level superbuild or just work on one of the subprojects if that's all you needed once the initial tools build had been done. You could even set up as many different sub-projects for the different architectures as needed. Packaging would require a little more work, but it shouldn't be prohibitively so.

I guess the tough part is to find a [light] way to specify dependencies between host target build and the various target builds.
  
Another alternative is the approach described in this stackoverflow article which performs the host tools build off to the side in a secondary build during configure. This works well when the host tools don't change much (we use it extensively at work with very large, complex hierarchical projects). It wouldn't help though if you need to build more than one cross-compiled architecture.

> The wish-season is coming up, so that's sort of what I would like to
> have. Now it's your turn. No bikeshedding please, only deliveries ;)

How about ``add_dependencies()`` allowing me to define dependencies between different build types? :-)

A superbuild would already give you the equivalent capability.

Not as easy as it seems right? 
I bet you know it well as you listed the dependencies shortcoming of adding dependencies for External_ProjectAdd in your book (§27.1.4).

For a strict superbuild arrangement, handling straight dependencies to get build order correct is not too bad. This is what I was thinking of with my earlier comment that super builds essentially give you the equivalent capability as just using add_dependencies(). You can also usually define CMAKE_PREFIX_PATH to pass sub-project install locations between the sub-projects and they each find what they need from the others without much further help. The top level superbuild is then really just specifying which sub-projects depend on which other ones. That does require a little bit of boilerplate, agreed, but not crazy amounts.

Where it starts getting out of hand is when you want to have a main build that directly wants/needs to refer to targets from sub-projects brought in by ExternalProject. Then you have to go working out where everything is without much help from CMake and that can be a soul-destroying exercise for complex multi-platform projects. Packaging suffers from this if done from the top superbuild project, but the difficulty can sometimes be reduced by doing the packaging itself as another subproject and importing what it needs from the others (you can then use the various imported targets, etc. to locate things), but it's still not particularly clean.

But yes, I'm agreeing with you, it isn't all that simple once you start considering the full software delivery process (i.e. going beyond more than just getting things to build).

--
Craig Scott
Melbourne, Australia


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake