Building embedded ARM systems with Crosstool-NG

This tutorial deals with creating a minimal embedded ARM system from scratch. The approach taken here requires no additional hardware beyond the development host and the source code of the required softwares. QEMU is used to run and debug the ARM system.

This tutorial details the following topics:

• Building a toolchain for ARM using Crosstool-NG.
• Building QEMU with ARM system emulation support.
• Building an ARM Linux kernel.
• Building and extending a tiny filesystem from scratch, based on BusyBox.
• Debugging ARM applications inside QEMU.

I intentionally kept the default settings when applicable for simplicity and clarity reasons. Many things can be tweaked; you are encouraged to try alternate ways to become familiar with embedded systems.

Prerequisites

When writing this tutorial, I used the following environment:

• Ubuntu 11.10 (Oneiric Ocelot) 32-bit as a host system
• Crosstool-NG v1.13.2
• Linux kernel v3.2
• BusyBox v1.19.3
• QEMU v0.14.1
• GNU Hello v2.7

Created a working folder:

$ mkdir -p ~/dev-embedded/ctool-ng

Building Crosstool-NG

Crosstool-NG allows building cross-compiling toolchains for multiple target architectures. This versatile tool provides a menuconfig configuration interface, which makes it quite easy to use. Crosstool-NG comes with a set of ready-made configuration files for various typical setups and supports glibc, uclibc and eglibc.

Install needed packages:

$ sudo apt-get install build-essential libncurses5-dev
$ sudo apt-get install automake libtool bison flex texinfo
$ sudo apt-get install gawk curl cvs subversion gcj-jdk
$ sudo apt-get install libexpat1-dev python-dev

Download, configure and install Crosstool-NG locally using —local option (Note that is possible to install it globally):

$ cd ~/dev-embedded/ctool-ng
$ wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.13.2.tar.bz2
$ tar xjf crosstool-ng-1.13.2.tar.bz2
$ cd crosstool-ng-1.13.2
$ ./configure --local
$ make
$ make install

Crosstool-NG comes with a set of ready-made configuration files for various typical setups called samples. They can be listed by running:

$ ./ct-ng list-samples

Use the ARM EABI cross compiler for use on linux. Configure and produce this toolchain by issuing:

$ ./ct-ng arm-unknown-linux-gnueabi

Refine your configuration by running the menuconfig interface:

$ ./ct-ng menuconfig

At this point it is up to you to select what options you want. You can explore the available options by traveling through the menus and looking at the help for some of the options.

I recommend the following minimal change:

In Path and misc options, set Number of parallel jobs to 2 times the number of CPU cores in your workstation. This causes a faster building of your toolchain.

Disable Fortran and Java programming languages, if you are not going to use them:
In C compiler options, uncheck Fortran and Java.

Build Crosstool-NG:

$ ./ct-ng build

The operation Retrieving needed toolchain components’ tarballs starts. The ./build.log file is useful to track issues. If you get a problem related to fetching a tarball then try downloading the same version from another location or mirror into ./.build/tarballs folder and rerun the ./ct-ng build command.

Compilation takes a long time, be patient.

For the subsequent steps in this tutorial, you need to add path of executables you just installed to your PATH environment variable by issuing the following command:

$ export PATH=$PATH:$HOME/x-tools/arm-unknown-linux-gnueabi/bin

If you want to make it session wide then add the previous command to ~/.bashrc or ~/.pam_environment.

Now, you can verify that your ARM toolchain works by typing:

$ arm-unknown-linux-gnueabi-gcc -v

Building an ARM Linux kernel

This ARM kernel, built with a predefined configuration for the versatile platform board, will be used in the QEMU virtual machine.

Download and configure the kernel:

$ cd ~/dev-embedded/ctool-ng
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.tar.bz2
$ tar xjf linux-3.2.tar.bz2
$ cd linux-3.2
$ make ARCH=arm versatile_defconfig
$ make ARCH=arm menuconfig

The last command brings up the kernel configuration menu, go to the Kernel Features section, check Use the ARM EABI to compile the kernel then exit (Make sure to save your changes).

Compile the kernel:

$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- all

Once it’s done building, the compressed kernel image is located at ./arch/arm/boot/zImage.

Building QEMU

QEMU is a high performance full system simulator supporting both emulation and virtualization. QEMU supports emulating several CPU architectures that are different from the host running the simulation. QEMU is used as the core device model by both Xen and KVM.

Install needed packages:

$ sudo apt-get install zlib1g-dev libsdl-dev

Download, configure and build QEMU from source:

$ cd ~/dev-embedded/ctool-ng
$ wget http://download.savannah.gnu.org/releases/qemu/qemu-0.14.1.tar.gz
$ tar xzf qemu-0.14.1.tar.gz
$ cd qemu-0.14.1
$ ./configure --enable-sdl --disable-kvm --enable-debug --target-list=arm-softmmu
$ make

The executable located at ./qemu-0.14.1/arm-softmmu/qemu-system-arm is the full system emulator for the ARM architecture.

Building GNU Hello

The very simple GNU Hello program, a minimal example of a GNU package that prints Hello, world‘ when you run it, will be debugged in the QEMU virtual machine.

Download, configure and build GNU Hello:

$ cd ~/dev-embedded/ctool-ng
$ wget http://ftp.gnu.org/gnu/hello/hello-2.7.tar.gz
$ tar xzf hello-2.7.tar.gz
$ cd hello-2.7
$ CC=arm-unknown-linux-gnueabi-gcc ./configure --host=arm-unknown-linux-gnueabi
$ make

The ARM-executable is located at ./hello-2.7/src/hello.

Building BusyBox

BusyBox combines tiny versions of many common UNIX utilities into a single small executable, including a shell and an init process. BusyBox is used extensively to build embedded systems of many types and is customizable so that only the needed utilities are built.

Download, configure and build BusyBox from source:

$ cd ~/dev-embedded/ctool-ng
$ wget http://busybox.net/downloads/busybox-1.19.3.tar.bz2
$ tar xjf busybox-1.19.3.tar.bz2
$ cd busybox-1.19.3
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- defconfig
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- install

A folder, containing a minimal root filesystem, called _install is created.

Customizing the filesystem

In order to debug the GNU Hello program, a customization of our minimal filesystem created in the previous section is required. First, copy the following:

• gdbserver built with arm-unknown-linux-gnueabi.
• GNU hello executable compiled previously.

Copy required executables:

$ cd ~/dev-embedded/ctool-ng
$ cd busybox-1.19.3/_install
$ CROSSTOOLNG_DIR=$HOME/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/
$ cp $CROSSTOOLNG_DIR/debug-root/usr/bin/gdbserver usr/bin/
$ cp ../../hello-2.7/src/hello usr/bin/

The minimal filesystem still lacks some shared libraries to run busybox, gdbserver and hello programs. These missing libraries can be enumerated using arm-unknown-linux-gnueabi-readelf (created when building the ARM toolchain) as following:

$ arm-unknown-linux-gnueabi-readelf ./usr/bin/hello -a |grep lib
$ arm-unknown-linux-gnueabi-readelf ./usr/bin/gdbserver -a |grep lib
$ arm-unknown-linux-gnueabi-readelf ../busybox -a |grep lib

After analyzing these commands output, you likely just needed to add the following libraries:

$ mkdir -p lib
$ cp $CROSSTOOLNG_DIR/sysroot/lib/ld-linux.so.3 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libgcc_s.so.1 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libm.so.6 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libc.so.6 lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/libdl.so.2 lib/

To add networking capabilities to the guest ARM system side and optionally use the same keyboard layout as your host machine, you need to create additional folders in the filesystem:

$ mkdir -p proc sys dev etc etc/init.d

Optionally, create a BusyBox-compatible keymap, in the filesystem, from the currently loaded keymap of your host:

$ sudo busybox dumpkmap > etc/host.kmap

The default initialization script that BusyBox searches for is /etc/init.d/rcS. Instead of using inittab, this is the preferred method to initialize an embedded system based on BusyBox. Create the rcS script as follows:

$ cat > ./etc/init.d/rcS << EOF
#!/bin/sh

# Mount the /proc and /sys filesystems
mount -t proc none /proc
mount -t sysfs none /sys

# Populate /dev
/sbin/mdev -s

# Enable the localhost interface
ifconfig lo up

# Manually configure the eth0 interface. Note that QEMU has a built-in 
# DHCP server that assigns addresses to the hosts starting from 10.0.2.15.
ifconfig eth0 10.0.2.15 netmask 255.255.255.0
route add default gw 10.0.2.1

# Comment this line if changing your keyboard layout is not needed
loadkmap < /etc/host.kmap
EOF

Add executable permission to the rcS script:

$ chmod +x etc/init.d/rcS

Create a compressed filesystem image:

$ find . | cpio -o --format=newc | gzip > ../../rootfs.img.gz

Running and debugging ARM applications

QEMU has all required components prepared (a compressed kernel image and a customized compressed filesystem image) and is ready to run the embedded ARM system.

Run the platform with the following command:

$ cd ~/dev-embedded/ctool-ng
$ ./qemu-0.14.1/arm-softmmu/qemu-system-arm -M versatilepb -m 128M -kernel ./linux-3.2/arch/arm/boot/zImage -initrd ./rootfs.img.gz -append "root=/dev/ram rdinit=/sbin/init" -redir tcp:65000::65000

The redir option will redirect any TCP communication from port 65000 of host system to port 65000 of the guest ARM system. This arbitrarily unused port number is chosen for testing purposes.

The ARM system will boot in a QEMU window then displays the message Please press Enter to activate this console. Press enter and you should be presented with an interactive shell. Start the debugging server as follows:

$ gdbserver --multi localhost:65000

Open a new terminal window, and install the ddd debugger:

$ sudo apt-get install ddd

Edit a configuration file, for example exec-commands.dbg, using the following commands:

$ cd ~/dev-embedded/ctool-ng
$ cat > ./exec-commands.dbg << EOF 
set solib-absolute-prefix nonexistantpath
set solib-search-path $HOME/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/
file ./hello-2.7/src/hello
target extended-remote localhost:65000
set remote exec-file /usr/bin/hello
break main
run
EOF

Run the ddd debugger as follows:

$ export PATH=$PATH:$HOME/x-tools/arm-unknown-linux-gnueabi/bin
$ ddd --debugger arm-unknown-linux-gnueabi-gdb -x exec-commands.dbg

You should see a breakpoint in the main function in hello.c file.