LD_LIBRARY_PATH not used by the linker

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

LD_LIBRARY_PATH not used by the linker

Ruben Di Battista
Hello, 
I'm running cmake on a system where I have module-loaded software such that `LD_LIBRARY_PATH` gets populated by the module (actually I'm using Lmod) function.

In my project, I have a dependency on VTK, and VTK could be compiled with Python wrapping. 

This is what I have in my main CMakeFiles.txt

project(HG)
set(HG_VERSION MAJOR 0)
set(HG_VERSION MINOR 1)

# We use VTK extensively
find_package(VTK REQUIRED)
include(${VTK_USE_FILE})


add_subdirectory(src)

# :: Tests :: 
set(BUILD_TESTS FALSE CACHE BOOL "Set this to TRUE if you want to build and \
    run the tests ")

if(BUILD_TESTS)
    # We use catch2 for tests
    find_package(Catch2 QUIET
        HINTS
        "${CMAKE_BINARY_DIR}/third_party"
        )

    if(NOT Catch2_FOUND)
        message(WARNING "Catch2 was not found in the system. \
        It will be downloaded from its git repository and installed")

        # Bootstrap Catch2
        include(DownloadCatch2)

        # Try again
        find_package(Catch2 REQUIRED)
    endif(NOT Catch2_FOUND)

    enable_testing()
    add_subdirectory(tests)
    
endif(BUILD_TESTS)

In the `tests` directory I have:

set(TEST_TARGET "test_${PROJECT_NAME}")

# find_package(PythonInterp REQUIRED)
# find_package(PythonLibs REQUIRED)
# message(FATAL_ERROR ${PYTHON_LIBRARIES})

# Generate test executable
add_executable(${TEST_TARGET}
    test_hg.cxx
    test_polydata_wrapper.cxx)
target_link_libraries(${TEST_TARGET} ${PROJECT_NAME} Catch2::Catch2 ${VTK_LIBRARIES})

Now, when I compile the project, when linking the tests target:
cmake -DBUILD_TESTS:BOOL=ON ..

[100%] Linking CXX executable test_HG
PATH_TO_VTK_LIB_DIR/vtkWrappingPython27Core-7.1.so.1: error: undefined reference to '_PyUnicodeUCS2_AsDefaultEncodedString'
PATH_TO_VTK_LIB_DIR/vtkWrappingPython27Core-7.1.so.1: error: undefined reference to 'PyUnicodeUCS2_AsUTF8String'
PATH_TO_VTK_LIB_DIR/vtkWrappingPython27Core-7.1.so.1: error: undefined reference to 'PyUnicodeUCS2_DecodeUTF8'
PATH_TO_VTK_LIB_DIR/vtkWrappingPython27Core-7.1.so.1: error: undefined reference to 'PyUnicodeUCS2_GetSize'
collect2: error: ld returned 1 exit status
make[2]: *** [tests/test_HG] Error 1
make[1]: *** [tests/CMakeFiles/test_HG.dir/all] Error 2
make: *** [all] Error 2

That is because the linker is linking against the system Python library instead of the one in the LD_LIBRARY_PATH.

Checking the command in VERBOSE=1 mode, I get that the c++ command is linking against python using `-lpython2.7` instead of the fullpath to the library.

To confirm that, explicitly adding to LDFLAGS the path to the python library:
LDFLAGS='-L/path/to/the/dir/of/libpython2.7.so' cmake -DBUILD_TESTS:BOOL=ON' allows the project to compile correctly.

So now, why the linker does not look into LD_LIBRARY_PATH? Is that normal? And if it is, how should I be supposed to make the linker look into LD_LIBRARY_PATH?

Thanks in advance, 







--
          _  
-.     .´  |∞∞∞∞
  ',  ;    |∞∞∞∞∞∞
    ˜˜     |∞∞∞∞∞∞∞∞∞ RdB 
    ,.,    |∞∞∞∞∞∞
  .'   '.  |∞∞∞∞
-'       `'


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: LD_LIBRARY_PATH not used by the linker

Roger Leigh
On 02/08/18 20:03, Ruben Di Battista wrote:

> I'm running cmake on a system where I have module-loaded software such
> that `LD_LIBRARY_PATH` gets populated by the module (actually I'm using
> Lmod) function.

LD_LIBRARY_PATH is used by the *runtime* linker, ld.so.  It is not used
when linking at *build* time, with ld.


Regards,
Roger
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: LD_LIBRARY_PATH not used by the linker

Ruben Di Battista
Hi Roger, 
so I explicitly passing the LDFLAGS is the way to go?

On Thu, Aug 2, 2018 at 9:41 PM Roger Leigh <[hidden email]> wrote:
On 02/08/18 20:03, Ruben Di Battista wrote:

> I'm running cmake on a system where I have module-loaded software such
> that `LD_LIBRARY_PATH` gets populated by the module (actually I'm using
> Lmod) function.

LD_LIBRARY_PATH is used by the *runtime* linker, ld.so.  It is not used
when linking at *build* time, with ld.


Regards,
Roger
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--
          _  
-.     .´  |∞∞∞∞
  ',  ;    |∞∞∞∞∞∞
    ˜˜     |∞∞∞∞∞∞∞∞∞ RdB 
    ,.,    |∞∞∞∞∞∞
  .'   '.  |∞∞∞∞
-'       `'


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: LD_LIBRARY_PATH not used by the linker

Roger Leigh
It's the quick and easy way to do it.

However, I'd suggest looking at getting CMake to use the proper path
without it.  Can you use any module-specific variables like
PYTHON_LIBRARY to override the search.  Or even just add to
CMAKE_PREFIX_PATH.  (Does VTK support imported targets?)

Regards,
Roger

On 02/08/18 20:43, Ruben Di Battista wrote:

> Hi Roger,
> so I explicitly passing the LDFLAGS is the way to go?
>
> On Thu, Aug 2, 2018 at 9:41 PM Roger Leigh <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 02/08/18 20:03, Ruben Di Battista wrote:
>
>      > I'm running cmake on a system where I have module-loaded software
>     such
>      > that `LD_LIBRARY_PATH` gets populated by the module (actually I'm
>     using
>      > Lmod) function.
>
>     LD_LIBRARY_PATH is used by the *runtime* linker, ld.so.  It is not used
>     when linking at *build* time, with ld.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: LD_LIBRARY_PATH not used by the linker

Ruben Di Battista
Thanks for your answer, 

The thing is that, since the Python dependency comes from VTK, I checked in the VTK install directory all the CMake configuration files (In particulare the UseVTK.cmake) file to see if they keep the Python path somewhere. 

And it happens that yes, they do. 
If I run: `grep -e 'python2.7' -r <PATH_TO_VTK/lib/cmake/vtk-7.1>`, I actually get that in the file `VTKTargets.cmake`and in the `Modules/vtkPython.cmake` they actually keep the path of the python used to compile VTK.

For example in VTKTargets.cmake, you find:

set_target_properties(vtkWrappingPythonCore PROPERTIES
  INTERFACE_LINK_LIBRARIES "vtkCommonCore;/home2/softmassot/eb/Compiler/GCCcore/7.2.0/Python/2.7.14-bare/lib/libpython2.7.so;vtksys"
)

So when I link my test target to ${VTK_LIBRARIES}, shouldn't it also be able to retrieve the right python library from the target properties?


On Thu, Aug 2, 2018 at 9:56 PM Roger Leigh <[hidden email]> wrote:
It's the quick and easy way to do it.

However, I'd suggest looking at getting CMake to use the proper path
without it.  Can you use any module-specific variables like
PYTHON_LIBRARY to override the search.  Or even just add to
CMAKE_PREFIX_PATH.  (Does VTK support imported targets?)

Regards,
Roger

On 02/08/18 20:43, Ruben Di Battista wrote:
> Hi Roger,
> so I explicitly passing the LDFLAGS is the way to go?
>
> On Thu, Aug 2, 2018 at 9:41 PM Roger Leigh <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 02/08/18 20:03, Ruben Di Battista wrote:
>
>      > I'm running cmake on a system where I have module-loaded software
>     such
>      > that `LD_LIBRARY_PATH` gets populated by the module (actually I'm
>     using
>      > Lmod) function.
>
>     LD_LIBRARY_PATH is used by the *runtime* linker, ld.so.  It is not used
>     when linking at *build* time, with ld.
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake


--
          _  
-.     .´  |∞∞∞∞
  ',  ;    |∞∞∞∞∞∞
    ˜˜     |∞∞∞∞∞∞∞∞∞ RdB 
    ,.,    |∞∞∞∞∞∞
  .'   '.  |∞∞∞∞
-'       `'


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: LD_LIBRARY_PATH not used by the linker

Robert Maynard
When you dumb out the contents of VTK_LIBRARIES do you see
';/home2/softmassot/eb/Compiler/GCCcore/7.2.0/Python/2.7.14-bare/lib/libpython2.7.so'.

In general CMake will only stop using the absolute path of a library
when the library resides in an implicit directory provided by the
compiler. On Thu, Aug 2, 2018 at 4:13 PM Ruben Di Battista
<[hidden email]> wrote:

>
> Thanks for your answer,
>
> The thing is that, since the Python dependency comes from VTK, I checked in the VTK install directory all the CMake configuration files (In particulare the UseVTK.cmake) file to see if they keep the Python path somewhere.
>
> And it happens that yes, they do.
> If I run: `grep -e 'python2.7' -r <PATH_TO_VTK/lib/cmake/vtk-7.1>`, I actually get that in the file `VTKTargets.cmake`and in the `Modules/vtkPython.cmake` they actually keep the path of the python used to compile VTK.
>
> For example in VTKTargets.cmake, you find:
>
> set_target_properties(vtkWrappingPythonCore PROPERTIES
>   INTERFACE_LINK_LIBRARIES "vtkCommonCore;/home2/softmassot/eb/Compiler/GCCcore/7.2.0/Python/2.7.14-bare/lib/libpython2.7.so;vtksys"
> )
>
> So when I link my test target to ${VTK_LIBRARIES}, shouldn't it also be able to retrieve the right python library from the target properties?
>
>
> On Thu, Aug 2, 2018 at 9:56 PM Roger Leigh <[hidden email]> wrote:
>>
>> It's the quick and easy way to do it.
>>
>> However, I'd suggest looking at getting CMake to use the proper path
>> without it.  Can you use any module-specific variables like
>> PYTHON_LIBRARY to override the search.  Or even just add to
>> CMAKE_PREFIX_PATH.  (Does VTK support imported targets?)
>>
>> Regards,
>> Roger
>>
>> On 02/08/18 20:43, Ruben Di Battista wrote:
>> > Hi Roger,
>> > so I explicitly passing the LDFLAGS is the way to go?
>> >
>> > On Thu, Aug 2, 2018 at 9:41 PM Roger Leigh <[hidden email]
>> > <mailto:[hidden email]>> wrote:
>> >
>> >     On 02/08/18 20:03, Ruben Di Battista wrote:
>> >
>> >      > I'm running cmake on a system where I have module-loaded software
>> >     such
>> >      > that `LD_LIBRARY_PATH` gets populated by the module (actually I'm
>> >     using
>> >      > Lmod) function.
>> >
>> >     LD_LIBRARY_PATH is used by the *runtime* linker, ld.so.  It is not used
>> >     when linking at *build* time, with ld.
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> https://cmake.org/mailman/listinfo/cmake
>
>
>
> --
>           _
> -.     .´  |∞∞∞∞
>   ',  ;    |∞∞∞∞∞∞
>     ˜˜     |∞∞∞∞∞∞∞∞∞ RdB
>     ,.,    |∞∞∞∞∞∞
>   .'   '.  |∞∞∞∞
> -'       `'
> https://rdb.is
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake