Issue
I am attempting to cross compile a project for my raspberry pi (32 bit armv8) on my ubuntu machine. I set up a toolchain using crosstool-NG and compiling works, but linking fails. My CMake toolchain file is as follows:
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
set(CMAKE_SYSROOT /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/sysroot)
SET(CMAKE_C_COMPILER /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/bin/armv8-rpi3-linux-gnueabihf-gcc)
SET(CMAKE_CXX_COMPILER /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/bin/armv8-rpi3-linux-gnueabihf-g++)
SET(CMAKE_FIND_ROOT_PATH /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf)
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
# for libraries and headers in the target directories
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
I have a few libraries that are not being found by the linker. The problem is with my sysroot. I copied over /usr and /lib from the pi to my sysroot folder that was created by crosstool-NG. The libraries that are being linked reside in sysroot/usr/lib/arm-linux-gnueabihf.
However, the linker is not checking that directory, it checks sysroot/usr/lib and a few others, but for some reason it does not check the arm-linux-gnueabihf directory.
On the pi, I can run ld -lasound --verbose and get the following output:
attempt to open //usr/local/lib/arm-linux-gnueabihf/libasound.so failed
attempt to open //usr/local/lib/arm-linux-gnueabihf/libasound.a failed
attempt to open //lib/arm-linux-gnueabihf/libasound.so failed
attempt to open //lib/arm-linux-gnueabihf/libasound.a failed
attempt to open //usr/lib/arm-linux-gnueabihf/libasound.so succeeded
so its finding it correctly on the pi.
When I do the same using the ld generated by crosstool-NG:
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/home/stone/x-tools/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/lib/libasound.so failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/home/stone/x-tools/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/lib/libasound.a failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/usr/local/lib/libasound.so failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/usr/local/lib/libasound.a failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/lib/libasound.so failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/lib/libasound.a failed
attempt to open /home/stone/development/raspberry_pi/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/bin/../sysroot/usr/lib/libasound.so failed
It fails to find it because it doesnt look in the arm-linux-gnueabihf directory. I copied the library .so file from sysroot/usr/lib/arm-linux-gnueabihf to sysroot/usr/lib and it compiles and links successfully, but I would like to make it so that I do not have to do that. How can I make the linker check that arm-linux-gnueabihf directory?
EDIT: I also copied /etc/ld.so.conf and /etc/ld.so.conf.d from the pi into my sysroot, but it does not seem to affect the linker.
EDIT: After further research it looks like this might be due to something with gcc-multiarch. I am not sure what that is but hopefully I can figure this out
EDIT: I verified that indeed the linker is not searching the arm-linux-gnueabihf path:
./armv8-rpi3-linux-gnueabihf-ld --verbose | grep -i "search"
outputs:
SEARCH_DIR("=/home/stone/x-tools/armv8-rpi3-linux-gnueabihf/armv8-rpi3-linux-gnueabihf/lib"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib");
Solution
Alright I got it all figured out. The repo where I learned the tricks to getting this to work can be found here. I am grateful for this repo, without it I would not have gotten this to work.
The issue lies with the linker. The linker that comes with plain old binutils that gets pulled in by crosstool-ng does not search the arm-linux-gnueabihf subdirectories. These directories exist because of Debian multiarch, and in order to get a linker that looks in such a directory, binutils needs to be patched. The instructions that follow assume you have already generated a ct-ng config based on the rpi3 sample.
On the raspberry pi, I installed the binutils-source package:
$ sudo apt install binutils-source
This will give access to a patch file located in /usr/src/binutils/patches.
On the host machine, you need to add a patches directory in the directory where you are building your toolchain (where the ct-ng config file is). It must have a specific structure that reflects the structure of packages in the crosstool-ng repo (ie. inside patches/ you must have a directory with the package name, inside that you must have the version number, and inside that is where you place the patch.):
$ cd your_toolchain_directory
$ mkdir -p patches/binutils/2.31.1/
Now we can copy the patch file from the pi to the host. The specific patch we need is 129_multiarch_libpath.patch:
$ cd patches/binutils/2.31.1/
$ scp pi@raspberrypi:/usr/src/binutils/129_multiarch_libpath.patch
Now that we have the patch, we need to update the config file to tell ct-ng to include local patches as well as enable the multiarch flag for gcc. You can do this using ct-ng menuconfig so change back to the directory with the config file and run:
$ ct-ng menuconfig
In the paths and misc options you need to change the patches origin to bundled then local, and then add ${CT_TOP_DIR}/patches as the local patch directory. This should produce the following lines under the extracting section of the .config file:
CT_PATCH_BUNDLED_LOCAL=y
CT_PATCH_ORDER="bundled,local"
CT_PATCH_USE_LOCAL=y
CT_LOCAL_PATCH_DIR="${CT_TOP_DIR}/patches"
Next you need to add the --enable-multiarch flag in the gcc options. Again using menuconfig, go to the C compiler settings. Add --enable-multiarch to the gcc extra config setting. The config file should have the following in the gcc settings:
CT_CC_GCC_EXTRA_CONFIG_ARRAY="--enable-multiarch"
Save the config. The last thing we need to do is export a variable.
$ export DEB_TARGET_MULTIARCH=arm-linux-gnueabihf
Now we can build the toolchain:
$ ct-ng build
Once its done, you can change into the bin directory of the generated toolchain and run:
$ ./ld --verbose | grep -i "search"
and you should see that the arm-linux-gnueabihf directory is now in the search paths. What a process.
Answered By - Stone Preston
