Some time ago, ARM Holdings presented the new virtualization extensions for its processor architecture, which are now present on some models of the Cortex family, like the Cortex-A7 and Cortex-A15. Though it’s a quite recent technology, both KVM and Xen hypervisors already support such extensions, allowing to run virtualized Guests in the same way you can already do on x86.
It’s true that current SoCs (System-on-Chip) and development boards doesn’t provide a number of cores and RAM memory that invite to run a significant number of Guests on them, but these are the first steps towards stabilization of ARM virtualization, paving the way for the future server-oriented ARM processors. On the other hand, this is also an interesting option for running alternative operating systems (like the *BSD family) on ARM hardware, without dealing with the extremely heterogeneous nature of it.
In this guide, where going to see how you can enable KVM virtualization on the Olinuxino-A20-MICRO development board.
About the Olinuxino-A20-MICRO board
This is one of the latest ARM-based boards produced by Olimex, an European company specialized in electronics and devoted to Open Hardware. This board is powered by an Allwinner A20 SoC with a dual-core Cortex-A7 processor and 1 GB of RAM memory.
This is pretty similar to the popular Cubieboard2, and most of the steps described here also can be applied to this one with minor variations.
Requirements
- An OlinuXino-A20-MICRO board (it should work with Olinuxino-A20-LIME too, but I haven’t tested it).
- A 4GB (or bigger) SD card.
- (OPTIONAL but encouraged) An USB to UART adapter. The one sold by Olimex (USB-Serial-Cable-F) works nicely.
- A PC with a recent GNU/Linux distribution.
Step 1: Installing an ARM cross-compiler
To be able to compile u-boot and Linux, we need a cross-compiler for ARM with EABI and Hard Float options. Most recent distributions provide this king of cross-compiler and all its dependencies on their repositories. In our case, using Ubuntu 14.04, the main package is called gcc-arm-linux-gnueabihf
:
slp@ubuntu1404:~$ sudo apt-get install gcc-arm-linux-gnueabihf
Step 2: Compiling the boot loader (u-boot)
When we power our board, the initial boot loader, residing in SoC’s internal memory, searches for a secondary boot loader on the SD card.
The official Debian images provided by Olimex already provide this boot loader, but we need to replace it for a version which detects when the CPU is on HYP mode, and avoids dropping to SVC32 (a requirement for KVM initialization).
We create a directory to store all the software we’re going to build, and then we get the u-boot code from the appropriate git repository:
slp@ubuntu1404:~$ mkdir kvm-arm slp@ubuntu1404:~$ cd kvm-arm/ slp@ubuntu1404:~/kvm-arm$ git clone -b sunxi-next --depth=1 git://github.com/jwrdegoede/u-boot-sunxi.git Clonar en «u-boot-sunxi»... remote: Counting objects: 9155, done. remote: Compressing objects: 100% (7854/7854), done. remote: Total 9155 (delta 1708), reused 3512 (delta 1113) Receiving objects: 100% (9155/9155), 13.73 MiB | 1.58 MiB/s, done. Resolving deltas: 100% (1708/1708), done. Checking connectivity... hecho.
Then we build u-boot, specifying the configuration for our board, and the cross-compiler prefix:
slp@ubuntu1404:~/kvm-arm/u-boot-sunxi$ make CROSS_COMPILE=arm-linux-gnueabihf- A20-OLinuXino_MICRO_config Configuring for A20-OLinuXino_MICRO - Board: sun7i, Options: A20_OLINUXINO_M,CONS_INDEX=1,STATUSLED=226,SPL,SUNXI_EMAC slp@ubuntu1404:~/kvm-arm/u-boot-sunxi$ make CROSS_COMPILE=arm-linux-gnueabihf-
If we’re building on a multicore machine, we should consider adding the option -j4
for compiling in parallel.
If the build ends successfully, we’ll have a file named u-boot-sunxi-with-spl.bin
, ready to be written to the SD card.
Step 3: Preparing an startup script
To avoid the need of manually entering all the commands on each boot, u-boot searches for a file named boot.scr
on various block devices, and loads it if found. This file contains a list of commands to be run one by one, in binary format. We’re going to create our own startup script with the appropriate commands.
On the u-boot-sunxi
, open your favorite text editor and create the file boot.cmd
, with the following contents:
setenv kernel_addr_r 0x46000000 # 8M setenv fdt_addr 0x49000000 # 2M setenv fdt_high 0xffffffff # Load fdt in place instead of relocating fatload mmc 0 ${kernel_addr_r} /uImage setenv bootargs "console=ttyS0,115200 ro root=/dev/mmcblk0p2 rootwait" fatload mmc 0 ${fdt_addr} /sun7i-a20-olinuxino-micro.dtb bootm ${kernel_addr_r} - ${fdt_addr}
Save the file, and create its binary counterpart with mkimage
:
slp@ubuntu1404:~/kvm-arm/u-boot-sunxi$ ./tools/mkimage -C none -A arm -T script -d boot.cmd boot.scr Image Name: Created: Wed Jul 23 13:48:35 2014 Image Type: ARM Linux Script (uncompressed) Data Size: 365 Bytes = 0.36 kB = 0.00 MB Load Address: 00000000 Entry Point: 00000000 Contents: Image 0: 357 Bytes = 0.35 kB = 0.00 MB
Now we have our startup script on the file boot.scr
.
Paso 4: Building the kernel
As its sadly common in the ARM world, the SoC maker (Allwinner, in this case), releases an initial version of Linux with support for a certain model, updates it for a few months, and then stops maintaining it. It doesn’t invest any resources into trying to get the support pushed to mainline Linux neither. The result is that kernel support for such SoC gets “stuck” on a certain Linux version, lacking support for newer features, like KVM.
The goods news is that some hackers at the linux-sunxi community have been working on getting support for Allwinner SoCs into mainline. At the moment of writing this, Linux 3.16 already supports a big part of the hardware, and also features KVM support, among other cool stuff.
So let’s get the latest kernel (3.16-rc6 in our case) and uncompress it on the previously created sources directory:
slp@ubuntu1404:~/kvm-arm$ wget https://www.kernel.org/pub/linux/kernel/v3.x/testing/linux-3.16-rc6.tar.xz slp@ubuntu1404:~/kvm-arm$ tar xf linux-3.16-rc6.tar.xz
Next, we create a default configuration file, and launch the kernel configuration menu (if this last step fails, you’ll probably need to install the libncurses5-dev package, or the equivalent on your distributions):
slp@ubuntu1404:~/kvm-arm$ cd linux-3.16-rc6/ slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make CROSS_COMPILE=arm-linux-gnueabihf- ARCH=arm sunxi_defconfig # # configuration written to .config # slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make CROSS_COMPILE=arm-linux-gnueabihf- ARCH=arm menuconfig
On the configuration menu, enable to following options:
General setup -> Control Group support
System Type -> Support for Large Physical Address Extension
Device Drivers -> Block devices -> Loopback device support
Virtualization
Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
Exit from menuconfig, saving the new configuration.
Finally, launch a build for kernel and DTBs (Flattened Device Tree):
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ export PATH=$PATH:/home/slp/kvm-arm/u-boot-sunxi/tools slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- LOADADDR=0x40008000 uImage dtbs
This will take a few minutes (less that the usual x86 kernel build, due to the smaller number of options built in). When finished, you’ll have the kernel (arch/arm/boot/uImage
) and the DTB (arch/arm/boot/dts/sun7i-a20-olinuxino-micro.dtb
).
Step 5: Preparing the SD card
Now that we have all the pieces ready, we need to prepare an SD card for booting our new kernel. If you already have an SD with the official Debian image from Olimex, you can reuse it. If this is not your case, download it from the Olinuxino-A20-MICRO page and dump it to the SD card.
Once you have an SD card with such image, write the new u-boot to it (we assume the device block for the SD card is mmcblk0
, you should check which one is in your case):
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ cd ~/kvm-arm slp@ubuntu1404:~/kvm-arm$ sudo dd if=u-boot-sunxi/u-boot-sunxi-with-spl.bin of=/dev/mmcblk0 bs=1k seek=8
Next, mount the first partition and copy the startup script, the kernel and the DTB:
slp@ubuntu1404:~/kvm-arm$ sudo mount /dev/mmcblk0p1 /mnt slp@ubuntu1404:~/kvm-arm$ sudo cp u-boot-sunxi/boot.scr /mnt slp@ubuntu1404:~/kvm-arm$ sudo cp linux-3.16-rc6/arch/arm/boot/uImage /mnt slp@ubuntu1404:~/kvm-arm$ sudo cp linux-3.16-rc6/arch/arm/boot/dts/sun7i-a20-olinuxino-micro.dtb /mnt slp@ubuntu1404:~/kvm-arm$ sudo umount /mnt
Now, the SD card is ready for the first boot.
Step 6: First boot and QEMU installation
If you have the USB-to-UART cable, when booting you’ll be able to see the messages coming from u-boot and Linux. If you don’t have one, wait a few minutes and try connecting with SSH (we assume you’ve already configured your network before).
To be able to launch our first Guest, we need a recent QEMU version. The one that comes with Debian “wheezy” is quite old, so you have to build a newer one:
Download the build dependencies:
slp@a20:~$ sudo apt-get build-dep qemu slp@a20:~$ sudo apt-get install libpixman-1-dev
Then, create a directory for the sources, download QEMU 2.0 and uncompress it:
slp@a20:~$ mkdir srcs slp@a20:~$ cd srcs slp@a20:~/srcs$ wget http://wiki.qemu-project.org/download/qemu-2.0.0.tar.bz2 --2014-07-24 06:28:00-- http://wiki.qemu-project.org/download/qemu-2.0.0.tar.bz2 Resolving wiki.qemu-project.org (wiki.qemu-project.org)... 140.211.15.109 Connecting to wiki.qemu-project.org (wiki.qemu-project.org)|140.211.15.109|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 12839647 (12M) [application/x-bzip2] Saving to: `qemu-2.0.0.tar.bz2' 100%[======================================>] 12,839,647 2.08M/s in 8.2s 2014-07-24 06:28:09 (1.49 MB/s) - `qemu-2.0.0.tar.bz2' saved [12839647/12839647] slp@a20:~/srcs$ tar xf qemu-2.0.0.tar.bz2
Now run the configure
script with options for enabling KVM (it is enabled automatically if supported, but this way we ensure we get warned if it’s not going to be build with it) and for building the ARM emulation target only:
slp@a20:~/srcs$ cd qemu-2.0.0/ slp@a20:~/srcs/qemu-2.0.0$ ./configure --enable-kvm --target-list=arm-softmmu
Finally, build and install it (by default, the new binaries will reside in /usr/local/bin
):
slp@a20:~/srcs/qemu-2.0.0$ make slp@a20:~/srcs/qemu-2.0.0$ sudo make install
Step 7: Building a kernel for the Guest
When running QEMU with KVM, the hardware emulated is a Versatile Express A15, one the reference platforms provided by ARM Holdings. So, for our Guest we need a kernel which supports this board.
Return to the Linux directory we used on Step 3, load the default configuration for Versatile Express A15, and launch menuconfig:
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- vexpress_defconfig slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig
Enable these options:
General setup -> Configure standard kernel features (expert users)
General setup -> open by fhandle syscalls
Enable the block layer -> Support for large (2TB+) block devices and files
And generate a kernel in zImage format:
slp@ubuntu1404:~/kvm-arm/linux-3.16-rc6$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage
Step 8: Running our first Guest
In addition to the kernel, we also need a root filesystem with the distribution we want to use for our userland. As we’re going to virtualize an ARMv7 CPU, the best option is using a earmv7hf distribution. In this guide, we’re going to use a minimal OpenSuSE image (JeOS), but you can choose the one of your preference.
Create a directory with the files needed by the Guest:
slp@a20:~/srcs/qemu-2.0.0$ mkdir -p ~/kvm-arm/opensuse slp@a20:~/srcs/qemu-2.0.0$ cd ~/kvm-arm/opensuse slp@a20:~/kvm-arm/opensuse$
Copy the kernel built on Step 7, and the DTB for the Versatile Express A15:
slp@a20:~/kvm-arm/opensuse$ scp slp@10.x.x.x:~/kvm-arm/linux-3.16-rc6/arch/arm/boot/zImage . slp@10.x.x.x's password: zImage 100% 2558KB 2.5MB/s 00:00 slp@a20:~/kvm-arm/opensuse$ scp slp@10.x.x.x:~/kvm-arm/linux-3.16-rc6/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb . slp@10.x.x.x's password: vexpress-v2p-ca15-tc1.dtb 100% 12KB 12.0KB/s 00:00
Now we create a raw image with OpenSuSE’s userland:
slp@a20:~/kvm-arm/opensuse$ wget http://download.opensuse.org/ports/armv7hl/factory/images/openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-1.12.1-Build175.2.tbz slp@a20:~/kvm-arm/opensuse$ qemu-img create -f raw opensuse-factory.img 1G Formatting 'opensuse-factory.img', fmt=raw size=1073741824 slp@a20:~/kvm-arm/opensuse$ sudo losetup /dev/loop0 opensuse-factory.img [sudo] password for slp: slp@a20:~/kvm-arm/opensuse$ sudo mkfs.ext4 /dev/loop0 mke2fs 1.42.5 (29-Jul-2012) Discarding device blocks: done Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done slp@a20:~/kvm-arm/opensuse$ sudo mount /dev/loop0 /mnt slp@a20:~/kvm-arm/opensuse$ sudo tar xf openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-1.12.1-Build175.2.tbz -C /mnt slp@a20:~/kvm-arm/opensuse$ sudo umount /mnt slp@a20:~/kvm-arm/opensuse$ sudo losetup -d /dev/loop0
And finally, launch your first Guest:
slp@a20:~/kvm-arm/opensuse$ sudo qemu-system-arm -enable-kvm -m 512m -M vexpress-a15 -cpu host -kernel zImage -dtb vexpress-v2p-ca15-tc1.dtb -append "root=/dev/vda console=ttyAMA0 rootwait" -drive if=none,file=opensuse-factory.img,id=factory -device virtio-blk-device,drive=factory -net nic -net user -monitor null -serial stdio -nographic Booting Linux on physical CPU 0x0 Initializing cgroup subsys cpuset Linux version 3.16.0-rc6 (slp@ubuntu1404) (gcc version 4.8.2 (Ubuntu/Linaro 4.8.2-16ubuntu4) ) #12 SMP Thu Jul 24 11:48:10 CEST 2014 CPU: ARMv7 Processor [410fc074] revision 4 (ARMv7), cr=10c53c7d CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache Machine model: V2P-CA15 Memory policy: Data cache writealloc PERCPU: Embedded 7 pages/cpu @9fbdf000 s7552 r8192 d12928 u32768 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048 Kernel command line: root=/dev/vda console=ttyAMA0 rootwait PID hash table entries: 2048 (order: 1, 8192 bytes) Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) Memory: 513156K/524288K available (4604K kernel code, 188K rwdata, 1284K rodata, 235K init, 150K bss, 11132K reserved) ... [ OK ] Started OpenSSH Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Welcome to openSUSE Factory "Bottle" - Kernel 3.16.0-rc6 (ttyAMA0). linux login:
Now we can start playing with this Guest. The default credentials are root/linux
:
linux login: root Password: linux Last login: Thu Jul 24 11:57:53 on ttyAMA0 Have a lot of fun... linux:~ # uname -a Linux linux 3.16.0-rc6 #12 SMP Thu Jul 24 11:48:10 CEST 2014 armv7l armv7l armv7l GNU/Linux linux:~ # cat /proc/cpuinfo processor : 0 model name : ARMv7 Processor rev 4 (v7l) Features : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xc07 CPU revision : 4 Hardware : ARM-Versatile Express Revision : 0000 Serial : 0000000000000000
That’s all, now you’re ready for ARM virtualization! 😉
Recent Comments
peter
28 July, 2014 at 5:05 pmVery detailed tutorial.
Popolon
8 August, 2014 at 12:07 amOn the dd line:
sudo dd if=u-boot-sunxi/u-boot-sunxi-with-spl.bin of=/dev/mmcblk0 bs=1k count=8
Isn’t need be replaced by?
sudo dd if=u-boot-sunxi/u-boot-sunxi-with-spl.bin of=/dev/mmcblk0 bs=1k skip=8
After this :
http://linux-sunxi.org/Bootable_SD_card#SD_Card_Layout
The First 8k is for partition table (that you didn’t defined/speek about). SPL and uboot start at 8K.
slp
8 August, 2014 at 10:04 amYou’re right. Instead of “count” it should be “seek” (“skip” would omit the blocks from the input).
I’ve just fixed it on the article. Thanks!
Comments are closed.