Building embedded ARM systems with Crosstool-NG

This tutorial deals with creating a minimal embedded ARM system from scratch. The approach taken here requires no additional hardware beyond the development host and the source code of the required softwares. QEMU is used to run and debug the ARM system.

This tutorial details the following topics:

    • Building a toolchain for ARM using Crosstool-NG.
    • Building QEMU with ARM system emulation support.
    • Building an ARM Linux kernel.
    • Building and extending a tiny filesystem from scratch, based on BusyBox.
    • Debugging ARM applications inside QEMU.

I intentionally kept the default settings when applicable for simplicity and clarity reasons. Many things can be tweaked; you are encouraged to try alternate ways to become familiar with embedded systems.


When writing this tutorial, I used the following environment:

Created a working folder:

$ mkdir -p ~/dev-embedded/ctool-ng

Building Crosstool-NG

Crosstool-NG allows building cross-compiling toolchains for multiple target architectures. This versatile tool provides a menuconfig configuration interface, which makes it quite easy to use. Crosstool-NG comes with a set of ready-made configuration files for various typical setups and supports glibc, uclibc and eglibc.

Install needed packages:

$ sudo apt-get install build-essential libncurses5-dev
$ sudo apt-get install automake libtool bison flex texinfo
$ sudo apt-get install gawk curl cvs subversion gcj-jdk
$ sudo apt-get install libexpat1-dev python-dev

Download, configure and install Crosstool-NG locally using —local option (Note that is possible to install it globally):

$ cd ~/dev-embedded/ctool-ng
$ wget
$ tar xjf crosstool-ng-1.13.2.tar.bz2
$ cd crosstool-ng-1.13.2
$ ./configure --local
$ make
$ make install

Crosstool-NG comes with a set of ready-made configuration files for various typical setups called samples. They can be listed by running:

$ ./ct-ng list-samples

Use the ARM EABI cross compiler for use on linux. Configure and produce this toolchain by issuing:

$ ./ct-ng arm-unknown-linux-gnueabi

Refine your configuration by running the menuconfig interface:

$ ./ct-ng menuconfig

At this point it is up to you to select what options you want. You can explore the available options by traveling through the menus and looking at the help for some of the options.

I recommend the following minimal change:

In Path and misc options, set Number of parallel jobs to 2 times the number of CPU cores in your workstation. This causes a faster building of your toolchain.

Disable Fortran and Java programming languages, if you are not going to use them:
In C compiler options, uncheck Fortran and Java.

Build Crosstool-NG:

$ ./ct-ng build

The operation Retrieving needed toolchain components’ tarballs starts. The ./build.log file is useful to track issues. If you get a problem related to fetching a tarball then try downloading the same version from another location or mirror into ./.build/tarballs folder and rerun the ./ct-ng build command.

Compilation takes a long time, be patient.

For the subsequent steps in this tutorial, you need to add path of executables you just installed to your PATH environment variable by issuing the following command:

$ export PATH=$PATH:$HOME/x-tools/arm-unknown-linux-gnueabi/bin

If you want to make it session wide then add the previous command to ~/.bashrc or ~/.pam_environment.

Now, you can verify that your ARM toolchain works by typing:

$ arm-unknown-linux-gnueabi-gcc -v

Building an ARM Linux kernel

This ARM kernel, built with a predefined configuration for the versatile platform board, will be used in the QEMU virtual machine.

Download and configure the kernel:

$ cd ~/dev-embedded/ctool-ng
$ wget
$ tar xjf linux-3.2.tar.bz2
$ cd linux-3.2
$ make ARCH=arm versatile_defconfig
$ make ARCH=arm menuconfig

The last command brings up the kernel configuration menu, go to the Kernel Features section, check Use the ARM EABI to compile the kernel then exit (Make sure to save your changes).

Compile the kernel:

$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- all

Once it’s done building, the compressed kernel image is located at ./arch/arm/boot/zImage.

Building QEMU

QEMU is a high performance full system simulator supporting both emulation and virtualization. QEMU supports emulating several CPU architectures that are different from the host running the simulation. QEMU is used as the core device model by both Xen and KVM.

Install needed packages:

$ sudo apt-get install zlib1g-dev libsdl-dev

Download, configure and build QEMU from source:

$ cd ~/dev-embedded/ctool-ng
$ wget
$ tar xzf qemu-0.14.1.tar.gz
$ cd qemu-0.14.1
$ ./configure --enable-sdl --disable-kvm --enable-debug --target-list=arm-softmmu
$ make

The executable located at ./qemu-0.14.1/arm-softmmu/qemu-system-arm is the full system emulator for the ARM architecture.

Building GNU Hello

The very simple GNU Hello program, a minimal example of a GNU package that prints Hello, world‘ when you run it, will be debugged in the QEMU virtual machine.

Download, configure and build GNU Hello:

$ cd ~/dev-embedded/ctool-ng
$ wget
$ tar xzf hello-2.7.tar.gz
$ cd hello-2.7
$ CC=arm-unknown-linux-gnueabi-gcc ./configure --host=arm-unknown-linux-gnueabi
$ make

The ARM-executable is located at ./hello-2.7/src/hello.

Building BusyBox

BusyBox combines tiny versions of many common UNIX utilities into a single small executable, including a shell and an init process. BusyBox is used extensively to build embedded systems of many types and is customizable so that only the needed utilities are built.

Download, configure and build BusyBox from source:

$ cd ~/dev-embedded/ctool-ng
$ wget
$ tar xjf busybox-1.19.3.tar.bz2
$ cd busybox-1.19.3
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- defconfig
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- install

A folder, containing a minimal root filesystem, called _install is created.

Customizing the filesystem

In order to debug the GNU Hello program, a customization of our minimal filesystem created in the previous section is required. First, copy the following:

  • gdbserver built with arm-unknown-linux-gnueabi.
  • GNU hello executable compiled previously.

Copy required executables:

$ cd ~/dev-embedded/ctool-ng
$ cd busybox-1.19.3/_install
$ CROSSTOOLNG_DIR=$HOME/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/
$ cp $CROSSTOOLNG_DIR/debug-root/usr/bin/gdbserver usr/bin/
$ cp ../../hello-2.7/src/hello usr/bin/

The minimal filesystem still lacks some shared libraries to run busybox, gdbserver and hello programs. These missing libraries can be enumerated using arm-unknown-linux-gnueabi-readelf (created when building the ARM toolchain) as following:

$ arm-unknown-linux-gnueabi-readelf ./usr/bin/hello -a |grep lib
$ arm-unknown-linux-gnueabi-readelf ./usr/bin/gdbserver -a |grep lib
$ arm-unknown-linux-gnueabi-readelf ../busybox -a |grep lib

After analyzing these commands output, you likely just needed to add the following libraries:

$ mkdir -p lib
$ cp $CROSSTOOLNG_DIR/sysroot/lib/ lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/ lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/ lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/ lib/
$ cp $CROSSTOOLNG_DIR/sysroot/lib/ lib/

To add networking capabilities to the guest ARM system side and optionally use the same keyboard layout as your host machine, you need to create additional folders in the filesystem:

$ mkdir -p proc sys dev etc etc/init.d

Optionally, create a BusyBox-compatible keymap, in the filesystem, from the currently loaded keymap of your host:

$ sudo busybox dumpkmap > etc/host.kmap

The default initialization script that BusyBox searches for is /etc/init.d/rcS. Instead of using inittab, this is the preferred method to initialize an embedded system based on BusyBox. Create the rcS script as follows:

$ cat > ./etc/init.d/rcS << EOF

# Mount the /proc and /sys filesystems
mount -t proc none /proc
mount -t sysfs none /sys

# Populate /dev
/sbin/mdev -s

# Enable the localhost interface
ifconfig lo up

# Manually configure the eth0 interface. Note that QEMU has a built-in 
# DHCP server that assigns addresses to the hosts starting from
ifconfig eth0 netmask
route add default gw

# Comment this line if changing your keyboard layout is not needed
loadkmap < /etc/host.kmap

Add executable permission to the rcS script:

$ chmod +x etc/init.d/rcS

Create a compressed filesystem image:

$ find . | cpio -o --format=newc | gzip > ../../rootfs.img.gz

Running and debugging ARM applications

QEMU has all required components prepared (a compressed kernel image and a customized compressed filesystem image) and is ready to run the embedded ARM system.

Run the platform with the following command:

$ cd ~/dev-embedded/ctool-ng
$ ./qemu-0.14.1/arm-softmmu/qemu-system-arm -M versatilepb -m 128M -kernel ./linux-3.2/arch/arm/boot/zImage -initrd ./rootfs.img.gz -append "root=/dev/ram rdinit=/sbin/init" -redir tcp:65000::65000

The redir option will redirect any TCP communication from port 65000 of host system to port 65000 of the guest ARM system. This arbitrarily unused port number is chosen for testing purposes.

The ARM system will boot in a QEMU window then displays the message Please press Enter to activate this console. Press enter and you should be presented with an interactive shell. Start the debugging server as follows:

$ gdbserver --multi localhost:65000

Open a new terminal window, and install the ddd debugger:

$ sudo apt-get install ddd

Edit a configuration file, for example exec-commands.dbg, using the following commands:

$ cd ~/dev-embedded/ctool-ng
$ cat > ./exec-commands.dbg << EOF 
set solib-absolute-prefix nonexistantpath
set solib-search-path $HOME/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/
file ./hello-2.7/src/hello
target extended-remote localhost:65000
set remote exec-file /usr/bin/hello
break main

Run the ddd debugger as follows:

$ export PATH=$PATH:$HOME/x-tools/arm-unknown-linux-gnueabi/bin
$ ddd --debugger arm-unknown-linux-gnueabi-gdb -x exec-commands.dbg

You should see a breakpoint in the main function in hello.c file.

10 thoughts on “Building embedded ARM systems with Crosstool-NG

  1. navarromoral

    A few years ago, this whole process took hours if not some days. Thank you very much for summarizing it in the very essential steps. It saved me hours!

  2. Pingback: How to I get crosstool-ng C++ compiler working

  3. Jagan

    I tried the same steps above…but came up with below issue…

    VFS: Mounted root (ext2 filesystem) on device 1:0.
    Freeing init memory: 108K
    Kernel panic – not syncing: Attempted to kill init! exitcode=0x0000000b

    [] (unwind_backtrace+0x0/0xf0) from [] (panic+0x74/0x1b0)
    [] (panic+0x74/0x1b0) from [] (do_exit+0x640/0x6f0)
    [] (do_exit+0x640/0x6f0) from [] (do_group_exit+0x3c/0xac)
    [] (do_group_exit+0x3c/0xac) from [] (get_signal_to_deliver+0x134/0x4b8)
    [] (get_signal_to_deliver+0x134/0x4b8) from [] (do_signal+0x30/0x5b0)
    [] (do_signal+0x30/0x5b0) from [] (do_notify_resume+0x60/0x6c)
    [] (do_notify_resume+0x60/0x6c) from [] (work_pending+0x28/0x2c)

    Request for help.


    1. briolidz Post author


      One possible error is that your filesystem is missing some libraries to
      run your embedded binaries. See “arm-unknown-linux-gnueabi-readelf” usage
      in this post to fix it.


  4. Ismail Zemni

    Thank you Kamel for this jobs. I have search a long time for a tutorial that summarize all these steps on Internet an no result. thak you to save my time.

    but I have also a question:
    when running arm-unknown-linux-gnueabi-readelf tool to determine all missing libraries, where you find exactly the name of the missed library? (only I have seen library !)

    1. briolidz Post author

      As described in this guide, one method to find out shared library dependencies of a executable is by running arm-unknown-linux-gnueabi-readelf as follows (pay attention to NEEDED and “Shared library” keywords):
      arm-unknown-linux-gnueabi-readelf your_executable -a |grep lib

      Depending on your development environment you may end up copying a different set of shared libraries to your target. I also encourage you to read readelf documentation.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s