Issue
Some background: I'm writing a bare-metal C++ app/OS for the Raspberry Pi 4B (in 64-bit mode, so booting kernel8.elf off of an SD card) and I've been running into strange crashes/hangs (where logging to the screenbuffer just stops with no explanation) while doing pretty normal C++ tasks such as:
- Constructing an object in main() (though making the same object a global seemed to work)
- Compiling at -O0, while -O2 and -O3 tend to work
- Calling member functions
My first instinct was to look for UB in my program, and I'm pretty certain that isn't the problem. The ctor in question just calls some inline ASM to read the PI's model number and set the MMIO base address accordingly.
I'm unable to post my sourcecode publicly, but I think (and hope) that this is an issue with how I'm compiling/linking my executable.
I'd prefer to compile/link against the C++20 stdlib, since I want to use std::arrays and optionals throughout this project.
Here's the makefile I'm using
ARCH=cortex-a72
TRIPLE=aarch64-none-elf
XDIR:=/PATH/TO/GCC/$(TRIPLE)
GCC_VERSION=11.2.1
XBINDIR:=$(XDIR)/bin
XLIBDIR1:=$(XDIR)/lib
XLIBDIR2:=$(XDIR)/lib/gcc/$(TRIPLE)/$(GCC_VERSION)
XLIBDIR3:=$(XDIR)/$(TRIPLE)/lib
AR:=$(XBINDIR)/$(TRIPLE)-ar
ASM:=$(XBINDIR)/$(TRIPLE)-gcc
CC:=$(XBINDIR)/$(TRIPLE)-gcc
CXX:=$(XBINDIR)/$(TRIPLE)-g++
LD:=$(XBINDIR)/$(TRIPLE)-ld
OBJCOPY:=$(XBINDIR)/$(TRIPLE)-objcopy
RANLIB:=$(XBINDIR)/$(TRIPLE)-ranlib
SIZE:=$(XBINDIR)/$(TRIPLE)-size
STRIP:=$(XBINDIR)/$(TRIPLE)-strip
# COMPILE OPTIONS
WARNINGS=-Wall -Wextra -Wpedantic
OPTS=-O3
CFLAGS:=-g $(OPTS) -pipe -flto=auto -static-pie -fsigned-char $(WARNINGS) -mcpu=$(ARCH)\
-static -ffreestanding -nostartfiles
CXXFLAGS:=$(CFLAGS) -std=c++20 -fno-exceptions -fno-unwind-tables -fno-rtti
LDFLAGS:= -Wl,-nmagic\
-Wl,-Tlinker.ld\
-L.\
-L$(XLIBDIR1)\
-L$(XLIBDIR2)\
-L$(XLIBDIR3)\
-lc\
-lgcc\
-lstdc++\
-Wl,-gc-sections
RM=rm -f
# Source files and include dirs
SOURCES := $(wildcard src/*.cc) $(wildcard src/*.c) $(wildcard src/*.S) $(wildcard test/*cc)
# Create .o and .d files for every .cc and .S (hand-written assembly) file
OBJECTS := $(patsubst %.c, %.o, $(patsubst %.S, %.o, $(patsubst %.cc,%.o,$(SOURCES))))
DEPENDS := $(patsubst %.c, %.d, $(patsubst %.S, %.d, $(patsubst %.cc,%.d,$(SOURCES))))
INC=-Iinclude
# .PHONY means these rules get executed even if
# files of those names exist.
.PHONY: all clean
# The first rule is the default, ie. "make",
# "make all" and "make kernel8.elf" mean the same
all: kernel8.elf
clean:
$(RM) $(OBJECTS) $(DEPENDS) kernel8.elf kernel8.img
# Linking the executable from the object files
kernel8.elf kernel8.img &: $(OBJECTS) linker.ld
$(CXX) $(INC) $(CXXFLAGS) $(filter-out %.ld, $^) -o $@ $(LDFLAGS)
$(OBJCOPY) $@ -O binary kernel8.img
-include $(DEPENDS)
QEMUCMD=qemu-system-aarch64 -M raspi3b\
-kernel kernel8.img\
-display none\
-serial null\
-serial stdio\
-semihosting\
-d unimp
debug: CFLAGS += -DDEBUG -Og
debug: kernel8.img
$(QEMUCMD) -s -S
run: kernel8.img
$(QEMUCMD)
%.o: %.cc Makefile
$(CXX) $(INC) $(CXXFLAGS) -MMD -MP -c $< -o $@ $(LDFLAGS)
%.o: %.c Makefile
$(CXX) $(INC) $(CFLAGS) -MMD -MP -c $< -o $@ $(LDFLAGS)
Here's linker.ld:
ENTRY(_start)
SECTIONS {
. = 0x80000;
.text : { KEEP(*(.text.boot)) *(.text .text.* .gnu.linkonce.t*) }
.rodata : { *(.rodata .rodata.* .gnu.linkonce.r*) }
PROVIDE(_data = .);
.data : { *(.data .data.* .gnu.linkonce.d*) }
.bss (NOLOAD) : {
. = ALIGN(16);
__bss_start = .;
__bss_start__ = __bss_start;
*(.bss .bss.*)
*(COMMON)
__bss_end = .;
}
__bss_size = SIZEOF(.bss);
. = ALIGN(4096);
.init_array : {
__init_array_start = .;
KEEP (*(.init_array*))
__init_array_end = .;
}
__end = .;
/* needed for certain newlib routines that (potentially) call _sbrk */
end = __bss_end;
__end__ = end;
__dso_handle = 0;
}
And finally, here's the .S file that I define _start in:
//https://www.rpi4os.com/part1-bootstrapping/
.section ".text.boot" // Make sure the linker puts this at the start of the kernel image
.global _start // Execution starts here
.global exit
_start:
// Check processor ID is zero (executing on main core), else hang
mrs x1, mpidr_el1
and x1, x1, #3
// We're not on the main core, so hang in an infinite loop
cbnz x1, exit
// We're on the main core!
// initialize SP
ldr x0, =_start
mov sp, x0
// init global objects and BSS before handing control over to C++
bl init
// Jump to our main() routine in C++ (make sure it doesn't return)
bl main
// if main returns, just spin
exit: b exit
init
just calls std::fill to zero .bss, then iterates through the ctors in init_array.
For my compiler, I'm using the arm GNU toolchain on an x86-64 linux host, which I downloaded from here.
I'm hoping there's something that's just obviously wrong with my setup, since I've tried all the usual debugging steps from C++-land, to no avail. The exact same code works when called inline, but hangs/crashes when executed by a method. Thanks!
Solution
I figured out the cause of the crashes I was seeing. I was calling memset before having set up the MMU, and newlib's optimized memset was generating SIMD stores (referring to q0) which were causing an alignment fault and a subsequent exception. I ended up rolling my own memset and using that instead.
Answered By - Zen Answer Checked By - Robin (WPSolving Admin)