find_file - strange behavior when using Android toolchain

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

find_file - strange behavior when using Android toolchain

Stephan Menzel
Hello all,

I'm trying to adapt my CMake based toolchain to Android and I'm noticing very strange behavior that I'd like to ask about.
My toolchain is C++ based with some dependencies such as Boost, OpenSSL or protobuf. So far it works on a variety of platforms such as Windows (MSVC), several Linuxes, including Raspbian on ARM. So I considered myself well prepared for Android, thinking it can't be that different. How wrong I was... I'm cross compiling from a Linux system, which is able to build the source in question just fine with clang and gcc. Most recent Android SDK and bundled NDK.

As soon as I started building and finding my 3rd party dependencies I noticed that many commands in my own scripts or others that try to find a file (like a header) or a lib fail despite the file being there.
Like here in this case for example (randomly taken from the AWS SDK installed CMake finders):

find_file(AWSSDK_CORE_HEADER_FILE Aws.h
            "${AWSSDK_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            "${AWSSDK_DEFAULT_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            )

I have double checked many times that the file clearly is there in that exact position correct casing and the process has full read permissions. I can open it for reading and I can't see anything strange about it. Just some header.  But AWSSDK_CORE_HEADER_FILE is NOTFOUND. The finders for Boost and OpenSSL have the same problem whenever they use find_file or similar and so have my own.

So far I have worked around the issue by hard setting the variables in question before the script runs. In this case I would look at my directories, set AWSSDK_CORE_HEADER_FILE to the file path and run it again. Then the test will pass but that can't be it. Also, a huge library like the AWS SDK has so many such commands and variables that I cannot possibly override all of them.

Why does that command fail at all? I looked at the docs for find_file but I cannot see any mention of cases where the file is not found because of some toolchain or cross compile settings

I have tried with the CMake that comes bundled in the NDK as well, same results. And again, it works fine on ARM in general when the ndk toolchain is not involved. I can only assume something in "ndk-bundle/build/cmake/android.toolchain.cmake" sets some global CMake setting that disables find_file. But why would they do this? And why would such a breaking setting even exist? None of that makes sense to me. 

Anybody have some input please?

Cheers,
Stephan
       


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: find_file - strange behavior when using Android toolchain

Eric Dönges
Am 31.07.19 um 08:20 schrieb Stephan Menzel:
Hello all,

I'm trying to adapt my CMake based toolchain to Android and I'm noticing very strange behavior that I'd like to ask about.
My toolchain is C++ based with some dependencies such as Boost, OpenSSL or protobuf. So far it works on a variety of platforms such as Windows (MSVC), several Linuxes, including Raspbian on ARM. So I considered myself well prepared for Android, thinking it can't be that different. How wrong I was... I'm cross compiling from a Linux system, which is able to build the source in question just fine with clang and gcc. Most recent Android SDK and bundled NDK.

As soon as I started building and finding my 3rd party dependencies I noticed that many commands in my own scripts or others that try to find a file (like a header) or a lib fail despite the file being there.
Like here in this case for example (randomly taken from the AWS SDK installed CMake finders):

My experience has been that find_file and friends behave in unexpected ways when you set CMAKE_SYSROOT. In our toolchain files, we set

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

as suggested in the cmake-toolchains(7) documentation. We then add the path to the top-level directory where we keep all our thirdparty dependencies to CMAKE_FIND_ROOT_PATH, as otherwise find_package and find_path cannot find anything even when we give them the absolute path where to search (the find_file documentation suggests that the elements of the CMAKE_FIND_ROOT_PATH are prepended to the paths specified in the find_file arguments, but that does not seem to happen if the search paths are subpaths of the CMAKE_FIND_ROOT_PATH). My guess would be that the NDK does something similar, meaning you would need to add the path where to look for AWS to CMAKE_FIND_ROOT_PATH.

CMake feature request: it would be real nice to have a verbose mode for the find_XXX functions that tell you exactly where CMake is looking for stuff and why to help debug problems like this.

With kind regards,
Eric

--

Dr. Eric Dönges
Senior Software Engineer

MVTec Software GmbH | Arnulfstr. 205 | 80634 Munich | Germany
[hidden email] | Tel: +49 89 457 695-0 | www.mvtec.com

 Sign up for our MVTec Newsletter!

Geschäftsführer: Dr. Wolfgang Eckstein, Dr. Olaf Munkelt
Amtsgericht München HRB 114695

 

MVTec Software GmbH Logo

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: find_file - strange behavior when using Android toolchain

ugesh reddy
Hi,

I also have the same issue with find_file/find_path/find_package when I use the Android NDK and Eric's explanation make sense.

One way to solve this

find_file(AWSSDK_CORE_HEADER_FILE Aws.h
            "${AWSSDK_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            "${AWSSDK_DEFAULT_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            NO_CMAKE_FIND_ROOT_PATH
            NO_CMAKE_SYSROOT
            )

Regards,
Ugesh Gurram.

On Wed, Jul 31, 2019 at 9:16 AM Eric Doenges <[hidden email]> wrote:
Am 31.07.19 um 08:20 schrieb Stephan Menzel:
Hello all,

I'm trying to adapt my CMake based toolchain to Android and I'm noticing very strange behavior that I'd like to ask about.
My toolchain is C++ based with some dependencies such as Boost, OpenSSL or protobuf. So far it works on a variety of platforms such as Windows (MSVC), several Linuxes, including Raspbian on ARM. So I considered myself well prepared for Android, thinking it can't be that different. How wrong I was... I'm cross compiling from a Linux system, which is able to build the source in question just fine with clang and gcc. Most recent Android SDK and bundled NDK.

As soon as I started building and finding my 3rd party dependencies I noticed that many commands in my own scripts or others that try to find a file (like a header) or a lib fail despite the file being there.
Like here in this case for example (randomly taken from the AWS SDK installed CMake finders):

My experience has been that find_file and friends behave in unexpected ways when you set CMAKE_SYSROOT. In our toolchain files, we set

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

as suggested in the cmake-toolchains(7) documentation. We then add the path to the top-level directory where we keep all our thirdparty dependencies to CMAKE_FIND_ROOT_PATH, as otherwise find_package and find_path cannot find anything even when we give them the absolute path where to search (the find_file documentation suggests that the elements of the CMAKE_FIND_ROOT_PATH are prepended to the paths specified in the find_file arguments, but that does not seem to happen if the search paths are subpaths of the CMAKE_FIND_ROOT_PATH). My guess would be that the NDK does something similar, meaning you would need to add the path where to look for AWS to CMAKE_FIND_ROOT_PATH.

CMake feature request: it would be real nice to have a verbose mode for the find_XXX functions that tell you exactly where CMake is looking for stuff and why to help debug problems like this.

With kind regards,
Eric

--

Dr. Eric Dönges
Senior Software Engineer

MVTec Software GmbH | Arnulfstr. 205 | 80634 Munich | Germany
[hidden email] | Tel: +49 89 457 695-0 | www.mvtec.com

 Sign up for our MVTec Newsletter!

Geschäftsführer: Dr. Wolfgang Eckstein, Dr. Olaf Munkelt
Amtsgericht München HRB 114695

 

MVTec Software GmbH Logo
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--
 Ugesh

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: find_file - strange behavior when using Android toolchain

ugesh reddy
Hi,

Sorry a small correction.

There is no such thing as "NO_CMAKE_SYSROOT".

find_file(AWSSDK_CORE_HEADER_FILE Aws.h
            "${AWSSDK_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            "${AWSSDK_DEFAULT_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            NO_CMAKE_FIND_ROOT_PATH
            NO_CMAKE_SYSTEM_PATH
            )

Regards,
Ugesh

On Wed, Jul 31, 2019 at 9:49 AM ugesh reddy <[hidden email]> wrote:
Hi,

I also have the same issue with find_file/find_path/find_package when I use the Android NDK and Eric's explanation make sense.

One way to solve this

find_file(AWSSDK_CORE_HEADER_FILE Aws.h
            "${AWSSDK_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            "${AWSSDK_DEFAULT_ROOT_DIR}/${AWSSDK_INSTALL_INCLUDEDIR}/aws/core"
            NO_CMAKE_FIND_ROOT_PATH
            NO_CMAKE_SYSROOT
            )

Regards,
Ugesh Gurram.

On Wed, Jul 31, 2019 at 9:16 AM Eric Doenges <[hidden email]> wrote:
Am 31.07.19 um 08:20 schrieb Stephan Menzel:
Hello all,

I'm trying to adapt my CMake based toolchain to Android and I'm noticing very strange behavior that I'd like to ask about.
My toolchain is C++ based with some dependencies such as Boost, OpenSSL or protobuf. So far it works on a variety of platforms such as Windows (MSVC), several Linuxes, including Raspbian on ARM. So I considered myself well prepared for Android, thinking it can't be that different. How wrong I was... I'm cross compiling from a Linux system, which is able to build the source in question just fine with clang and gcc. Most recent Android SDK and bundled NDK.

As soon as I started building and finding my 3rd party dependencies I noticed that many commands in my own scripts or others that try to find a file (like a header) or a lib fail despite the file being there.
Like here in this case for example (randomly taken from the AWS SDK installed CMake finders):

My experience has been that find_file and friends behave in unexpected ways when you set CMAKE_SYSROOT. In our toolchain files, we set

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

as suggested in the cmake-toolchains(7) documentation. We then add the path to the top-level directory where we keep all our thirdparty dependencies to CMAKE_FIND_ROOT_PATH, as otherwise find_package and find_path cannot find anything even when we give them the absolute path where to search (the find_file documentation suggests that the elements of the CMAKE_FIND_ROOT_PATH are prepended to the paths specified in the find_file arguments, but that does not seem to happen if the search paths are subpaths of the CMAKE_FIND_ROOT_PATH). My guess would be that the NDK does something similar, meaning you would need to add the path where to look for AWS to CMAKE_FIND_ROOT_PATH.

CMake feature request: it would be real nice to have a verbose mode for the find_XXX functions that tell you exactly where CMake is looking for stuff and why to help debug problems like this.

With kind regards,
Eric

--

Dr. Eric Dönges
Senior Software Engineer

MVTec Software GmbH | Arnulfstr. 205 | 80634 Munich | Germany
[hidden email] | Tel: +49 89 457 695-0 | www.mvtec.com

 Sign up for our MVTec Newsletter!

Geschäftsführer: Dr. Wolfgang Eckstein, Dr. Olaf Munkelt
Amtsgericht München HRB 114695

 

MVTec Software GmbH Logo
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--
 Ugesh


--
 Ugesh

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: find_file - strange behavior when using Android toolchain

Stephan Menzel
In reply to this post by Eric Dönges
Thank you, Eric and Ugesh,

Am Mi., 31. Juli 2019 um 09:16 Uhr schrieb Eric Doenges <[hidden email]>:

My experience has been that find_file and friends behave in unexpected ways when you set CMAKE_SYSROOT. In our toolchain files, we set

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

as suggested in the cmake-toolchains(7) documentation. We then add the path to the top-level directory where we keep all our thirdparty dependencies to CMAKE_FIND_ROOT_PATH, as otherwise find_package and find_path cannot find anything even when we give them the absolute path where to search (the find_file documentation suggests that the elements of the CMAKE_FIND_ROOT_PATH are prepended to the paths specified in the find_file arguments, but that does not seem to happen if the search paths are subpaths of the CMAKE_FIND_ROOT_PATH). My guess would be that the NDK does something similar, meaning you would need to add the path where to look for AWS to CMAKE_FIND_ROOT_PATH.


I have just looked that up and yes, it would explain what I'm seeing. CMAKE_SYSROOT is being set by the toolchain file and it would appear as this would make all search hints and paths implicitly relative.

I'm now trying to restructure my 3rd party management for this target to enable me to install my dependencies into the subdirs set as sysroot, hoping that they will then be found.

Thanks for the suggestion, that looks promising. I will get back with more info on if and how that turned out.

CMake feature request: it would be real nice to have a verbose mode for the find_XXX functions that tell you exactly where CMake is looking for stuff and why to help debug problems like this.

Seconded.
I always insert debug message() statements all over the place and then often forget to clean them up. Also, this way I can only verify the search paths are correct but not why find_file fails.

Cheers,
Stephan

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: find_file - strange behavior when using Android toolchain

Eric Noulard


Le mer. 31 juil. 2019 à 11:19, Stephan Menzel <[hidden email]> a écrit :
Thank you, Eric and Ugesh,

Am Mi., 31. Juli 2019 um 09:16 Uhr schrieb Eric Doenges <[hidden email]>:

My experience has been that find_file and friends behave in unexpected ways when you set CMAKE_SYSROOT. In our toolchain files, we set

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

as suggested in the cmake-toolchains(7) documentation. We then add the path to the top-level directory where we keep all our thirdparty dependencies to CMAKE_FIND_ROOT_PATH, as otherwise find_package and find_path cannot find anything even when we give them the absolute path where to search (the find_file documentation suggests that the elements of the CMAKE_FIND_ROOT_PATH are prepended to the paths specified in the find_file arguments, but that does not seem to happen if the search paths are subpaths of the CMAKE_FIND_ROOT_PATH). My guess would be that the NDK does something similar, meaning you would need to add the path where to look for AWS to CMAKE_FIND_ROOT_PATH.


I have just looked that up and yes, it would explain what I'm seeing. CMAKE_SYSROOT is being set by the toolchain file and it would appear as this would make all search hints and paths implicitly relative.

I'm now trying to restructure my 3rd party management for this target to enable me to install my dependencies into the subdirs set as sysroot, hoping that they will then be found.

Thanks for the suggestion, that looks promising. I will get back with more info on if and how that turned out.

CMake feature request: it would be real nice to have a verbose mode for the find_XXX functions that tell you exactly where CMake is looking for stuff and why to help debug problems like this.

Seconded.

+1
 
I always insert debug message() statements all over the place and then often forget to clean them up. Also, this way I can only verify the search paths are correct but not why find_file fails.

so that you my leave your message(DEBUG ...) message while not being displayed at default loglevel.

You may use the --loglevel command line option to get them. The default level being displayed is STATUS.


--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake