Issue
I've finally cross-compiled and linked a binary on an Ubuntu Xenial x86_64 host to run on the Raspberry Pi 4's armhf.
My toolchain's from href="https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads" rel="nofollow noreferrer">ARM and placed in $TOOLCHAIN.
My sysroot is a loop-mounted Raspberry OS image placed in $RASPBIAN_ROOT.
This is a sample compilation line:
$TOOLCHAIN/gcc-arm-8.3-2019.03/bin/arm-linux-gnueabihf-g++ -std=c++11 --sysroot=$RASPBIAN_ROOT \
-D_ARM_ -DOTHER_DEFINES \
-v -w -fexceptions -fpermissive -pipe \
-mcpu=cortex-a72 -mfpu=neon-vfpv4 -mfloat-abi=hard \
-Wabi-tag -D_GLIBCXX_USE_CXX11_ABI=0 \
-fno-use-cxa-atexit \
-g \
-I . -I .. -I extlib1/include -I extlib2/include -I $RASPBIAN_ROOT/usr/include/libxml2 \
-I $TOOLCHAIN/gcc-arm-8.3-2019.03/arm-none-linux-gnueabihf/libc/usr/include \
-I $RASPBIAN_ROOT/usr/include/arm-linux-gnueabihf \
-I $RASPBIAN_ROOT/usr/include \
-c File.cpp -o obj/linux/armhf/debug/File.o
There are also some .c files. And this is the linking line:
$TOOLCHAIN/gcc-arm-8.3-2019.03/bin/arm-linux-gnueabihf-ld.gold \
-L ../localdependency \
-L $RASPBIAN_ROOT/opt/vc/lib \
-L $RASPBIAN_ROOT/usr/lib/arm-linux-gnueabihf \
-L $RASPBIAN_ROOT/lib \
-L$RASPBIAN_ROOT/usr/lib \
-L $RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8 -o bin/linux/armhf/debug/executable obj/linux/armhf/debug/File.o ... \
$RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8/crtbegin.o \
$RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8/crtend.o \
$RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8/libgcc_eh.a \
--verbose --sysroot=$RASPBIAN_ROOT \
-lbcm_host -lvcos -lvchiq_arm -lcurl -lxml2 \
-lpthread -lz -lm -ldl -lstdc++ -lc -lgcc -lgcc_s
...which seems to produce an executable. (Both lines edited for readability.)
However, when running on the pi, i get a segfault:
$ ./executable param1 param2
Segmentation fault
If i try to debug, it's not even reaching main():
$ gdb executable
GNU gdb (Raspbian 8.2.1-2) 8.2.1
...
(gdb) b main
Breakpoint 1 at 0x2c7c74: file main.cpp, line 7029.
(gdb) r param1 param2
Starting program: /home/user/executable param1 param2
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Program received signal SIGSEGV, Segmentation fault.
0xb6f97420 in __libc_enable_asynccancel () at ../nptl/cancellation.c:33
33 ../nptl/cancellation.c: No such file or directory.
(gdb) bt
#0 0xb6f97420 in __libc_enable_asynccancel () at ../nptl/cancellation.c:33
#1 0xb6f77968 in __GI___libc_write (nbytes=407, buf=0xb6fdc838 <banner>, fd=1) at ../sysdeps/unix/sysv/linux/write.c:26
#2 __GI___libc_write (fd=fd@entry=1, buf=0xb6fdc838 <banner>, nbytes=nbytes@entry=407) at ../sysdeps/unix/sysv/linux/write.c:24
#3 0xb6ec98d8 in __libc_print_version () at version.c:71
#4 __libc_main () at version.c:71
#5 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
While i see the "warning: Unable to find dynamic linker breakpoint function." (and am trying to figure out where in the packages does ld.so come from) i'm more inclined to believe my toolchain/sysroot/libs is not properly configured.
A simple helloWorld cross-compiles and executes fine:
#include <iostream>
int main(int argc, char* argv[]) {
int i = 1;
std::cout << "hello world " << i << std::endl;
return 0;
}
$ ./rpihello
hello world 1
$ ldd rpihello
linux-vdso.so.1 (0xbef33000)
/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0xb6f90000)
libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0xb6e30000)
libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb6dae000)
libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb6d81000)
libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6c33000)
/lib/ld-linux-armhf.so.3 (0xb6fa5000)
Edit 1
Since nptl/cancellation.c seems to come from libc, i double-checked i had all the necessary packages, and i believe i do (otherwise i'd be having widespread problems with the Pi):
pi$ cat /etc/ld.so.conf.d/\*
/opt/vc/lib
# Multiarch support
/usr/local/lib/arm-linux-gnueabihf
/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/libfakeroot
# libc default configuration
/usr/local/lib
pi$ apt list --installed | grep ^libc
libc-ares2/now 1.14.0-1 armhf [installed,upgradable to: 1.14.0-1+deb10u1]
libc-bin/stable,now 2.28-10+rpi1 armhf [installed]
libc-dev-bin/stable,now 2.28-10+rpi1 armhf [installed,automatic]
libc-l10n/stable,now 2.28-10+rpi1 all [installed,automatic]
libc6-dbg/stable,now 2.28-10+rpi1 armhf [installed]
libc6-dev/stable,now 2.28-10+rpi1 armhf [installed]
libc6-pic/stable,now 2.28-10+rpi1 armhf [installed]
libc6/stable,now 2.28-10+rpi1 armhf [installed]
libcc1-0/stable,now 8.3.0-6+rpi1 armhf [installed,automatic]
pi$ apt list --installed | grep ^libgcc
libgcc-6-dev/stable,now 6.5.0-1+rpi1+b1 armhf [installed,automatic]
libgcc-8-dev/stable,now 8.3.0-6+rpi1 armhf [installed,automatic]
libgcc1-dbg/stable,now 1:8.3.0-6+rpi1 armhf [installed,automatic]
libgcc1/stable,now 1:8.3.0-6+rpi1 armhf [installed]
So i still think i borked the cross-compilation/linking somehow.
Edit 2
I've managed to compile my codebase on the raspberry pi and the produced binary exhibits the same results as the cross-compiled one. So either i've misscompiled or i have both my sysroot and the pi's system a bit borked.
Edit 3
rpi $ readelf -dW myexecutable
Dynamic section at offset 0x664e58 contains 48 entries:
Tag Type Name/Value
0x00000003 (PLTGOT) 0x6716d4
0x00000002 (PLTRELSZ) 5856 (bytes)
0x00000017 (JMPREL) 0x2d940
0x00000014 (PLTREL) REL
0x00000011 (REL) 0x2d2e0
0x00000012 (RELSZ) 1632 (bytes)
0x00000013 (RELENT) 8 (bytes)
0x00000015 (DEBUG) 0x0
0x00000006 (SYMTAB) 0x8148
0x0000000b (SYMENT) 16 (bytes)
0x00000005 (STRTAB) 0xfb48
0x0000000a (STRSZ) 94185 (bytes)
0x6ffffef5 (GNU_HASH) 0x26b34
0x00000004 (HASH) 0x2930c
0x00000001 (NEEDED) Shared library: [libbcm_host.so]
0x00000001 (NEEDED) Shared library: [libvcos.so]
0x00000001 (NEEDED) Shared library: [libvchiq_arm.so]
0x00000001 (NEEDED) Shared library: [libjson.so]
0x00000001 (NEEDED) Shared library: [libmylocallib.so]
...
0x00000001 (NEEDED) Shared library: [libpthread.so.0]
0x00000001 (NEEDED) Shared library: [libz.so.1]
0x00000001 (NEEDED) Shared library: [libm.so.6]
0x00000001 (NEEDED) Shared library: [libdl.so.2]
0x00000001 (NEEDED) Shared library: [libstdc++.so.6]
0x00000001 (NEEDED) Shared library: [libc.so.6]
0x00000019 (INIT_ARRAY) 0x66dd60
0x0000001b (INIT_ARRAYSZ) 244 (bytes)
0x0000001a (FINI_ARRAY) 0x66de54
0x0000001c (FINI_ARRAYSZ) 4 (bytes)
0x6ffffff0 (VERSYM) 0x2c1b0
0x6ffffffe (VERNEED) 0x2d0f0
0x6fffffff (VERNEEDNUM) 9
0x00000000 (NULL) 0x0
I found this interesting:
rpi $ file myexecutable
myexecutable: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/libc.so.1, with debug_info, not stripped
rpi $ file `which ls`
/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=67a394390830ea3ab4e83b5811c66fea9784ee69, stripped
Shouldn't my executable be using ld as the interpreter?
Edit 4
I added[2] --dynamic-linker=/lib/ld-linux-armhf.so.3 to my linker line, but now i get no stack:
Program received signal SIGSEGV, Segmentation fault.
0x00000000 in ?? ()
(gdb) bt
#0 0x00000000 in ?? ()
(gdb)
Running strace executable the last lines are these:
rt_sigprocmask(SIG_BLOCK, NULL, ~[ILL TRAP BUS FPE KILL SEGV STOP RTMIN RT_1], 8) = 0
rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xb5e55120}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
clock_gettime(CLOCK_MONOTONIC_COARSE, {tv_sec=6336, tv_nsec=768744979}) = 0
gettimeofday({tv_sec=1631533208, tv_usec=497816}, NULL) = 0
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 3
read(3, "\314\4bJn\36\16\240f\203\216I\257\306\251r\202\v\232\263\210\374\222\276TN\33,[\3A\233", 32) = 32
close(3) = 0
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 3
read(3, "\301\231\203~G6Ey_\262Y\241\34\216F\2448w\2365Z\213\376\223\260\322zwo\334+e", 32) = 32
close(3) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV +++
Segmentation fault
/dev/urandom does exist though.
Edit 5
$ readelf -h executable
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: ARM
Version: 0x1
Entry point address: 0x0
Start of program headers: 52 (bytes into file)
Start of section headers: 40514564 (bytes into file)
Flags: 0x5000400, Version5 EABI, hard-float ABI
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 8
Size of section headers: 40 (bytes)
Number of section headers: 37
Section header string table index: 36
I don't think entry point address should be 0x0.
Edit 6
I've compiled the code on a pi3 and on an arm-qemu chroot - same result.
Solution
The solution I found was to cross-compile with the toolchain from raspberrypi.org and set --sysroot= to a Raspbian image. Their releases more or less follow Debian's and always support all existing Raspberry Pies.
Both CXX and LD point to their g++, meaning it's GCC who determines linking flags. If necessary I can use -Wl,..
Other than -Wabi-tag -D_GLIBCXX_USE_CXX11_ABI=0, CFLAGS is pretty standard.
Raspbian changed their download URL format between Buster 10.3 (2020-02-13) and 10.4 (2020-05-28) and started providing full and lite versions. I used the latter.
Since these images typically have almost no free space, I first extend the image file, partition (p2) and underlying filesystem, so I can install the -dev packages that satisfy the dependencies of my project by chrooting into the sysroot via quemu-arm.
Answered By - vesperto Answer Checked By - Cary Denson (WPSolving Admin)
