Enabling KVM virtualization for Raspberry Pi 2

As I wrote on my previous post, Enabling HYP mode on the Raspberry Pi 2, the newest machine from the Raspberry Pi Foundation features a Cortex-A7 with Virtualization Extensions, but it isn’t possible to make use of such feature out of the box.

In that article I showed that it was possible to start the kernel in HYP mode. Now, I’ll cover the rest of steps needed for enabling KVM virtualization and running your first guest OS.


Isolating a core

With KVM on ARM, interrupts are dealt in a very particular way. If a hardware interrupt is received when a core in running in guest mode, the execution is interrupted, a full world switch takes place returning to the kernel in host mode, then interrupts are reenabled and the core traps again, this time following the usual route.

Apparently, the BCM2836 (the core of the RPi2) doesn’t like the behavior, and if you try to run a guest on a core that can receive physical interrupts, you’ll find your RPi2 hangs within a few seconds. Without a JTAG of a full development board, debugging this problem is very, very hard.

Luckily, there’s a pretty simple workaround. Using the kernel option isolcpus when can isolate a core, so Linux doesn’t assign any tasks (including IRQ lines) to it.

In our case, we’re going to isolate the core number 3 (starting from zero). This is pretty straightforward, just edit /boot/cmdline.txt and add isolcpus=3 at the of the line.
 

A kernel with VGIC emulation for the RPi2

On my previous article about RPi2 emulation, I wrote about the BCM2836 and its lack of a GIC, which makes emulation a bit harder. To work around this issue, I’ve implemented VGIC emulation inside the Linux kernel.

To be able to use both the customized bootloader and this modified version of the kernel, we need to generate a bundle with both components, and the dtb concatenated at end.

 
The easy way. Using a prebuilt image

I’ve uploaded a prebuilt image here (md5sum: 356788d260e1a93376c5b8acbb63da13), which contains the bootloader, some padding (to put the zImage at 0x8000), the kernel and the dtb. Simply replace your kernel with it (save a copy first!):

mv /boot/kernel7.img /boot/kernel7.img.bak
cp kernel7.img /boot

Then you’ll need to add the kernel_old=1 option to your config.txt:

echo "kernel_old=1" >> /boot/config.txt

That’s it! On the next boot, Linux should say something like this:

[    0.154131] CPU: All CPU(s) started in HYP mode.
[    0.154158] CPU: Virtualization extensions available.

 
Building it yourself (Part I: the bootloader)

If you want to build the image by yourself, you’ll need to grab an ARM cross toolchain. I’m using the one from the GCC ARM Embedded project, which works just fine. Then add it to your $PATH, grab the code for my repo, and build it:

export PATH=$PATH:/home/slp/sources/gcc-arm-none-eabi-4_9-2014q4/bin
git clone https://github.com/slp/rpi2-hyp-boot.git
cd rpi2-hyp-boot
make

That should produce a bootblk.bin, with contains the boot code and 32k of padding.

 
Building it yourself (Part II: kernel and dtb)

You can find a complete guide for building the kernel on a variety of host systems here Raspberry Pi Kernel Compilation. Read that and the grab the sources from my repo and make sure the following options are enabled:

  • Patch physical to virtual translations at runtime
  • General setup -> Control Group support
  • System Type -> Support for Large Physical Address Extension
  • Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)
  • Boot options -> Supplement the appended DTB with traditional ATAG information
  • Device Drivers -> Block devices -> Loopback device support
  • Virtualization
  • Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
  • DISABLE Virtualization -> KVM support for Virtual GIC
  • ENABLE Virtualization -> KVM support for Emulated GIC

Now build both kernel and dtb:

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- zImage dtbs

 
Building it yourself (Part III: bundling all pieces)

This is the easy part, just concatenate them:

cat rpi2-hyp-boot/bootblk.bin linux/arch/arm/boot/zImage linux/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > kernel7.img

Now you can use it in the same way as the prebuilt image provided above.

 
Patched QEMU installation

To be able to launch our first Guest, we need a recent and patched (to ensure the guest is running in the isolated core) QEMU. The one that comes with Raspbian is quite old, so you have to build a newer one:

Download the build dependencies:

pi@raspberrypi:~$ sudo apt-get build-dep qemu
pi@raspberrypi:~$ sudo apt-get install libpixman-1-dev

Then, create a directory for the sources, download QEMU 2.2 and uncompress it:

pi@raspberrypi:~$ mkdir srcs
pi@raspberrypi:~$ cd srcs
pi@raspberrypi:~/srcs$ wget http://wiki.qemu-project.org/download/qemu-2.2.0.tar.bz2
pi@raspberrypi:~/srcs$ tar xf qemu-2.2.0.tar.bz2

We need to apply this patch to make sure the QEMU runs our guest in the core we isolated with option isolcpus:

pi@raspberrypi:~/srcs$ cd qemu-2.2.0/
pi@raspberrypi:~/srcs/qemu-2.2.0$ patch -p1 < ~/qemu-cpu-affinity.patch

Now run the configure script with options for enabling KVM (it is enabled automatically if supported, but this way we ensure we get warned if it's not going to be build with it) and for building the ARM emulation target only:

pi@raspberrypi:~/srcs/qemu-2.2.0$ ./configure --enable-kvm --target-list=arm-softmmu

Finally, build and install it (by default, the new binaries will reside in /usr/local/bin):

pi@raspberrypi:~/srcs/qemu-2.2.0$ make
pi@raspberrypi:~/srcs/qemu-2.2.0$ sudo make install

 
A kernel for the Guest

When running QEMU with KVM, the hardware emulated is a Versatile Express A15, one the reference platforms provided by ARM Holdings. So, for our Guest we need a kernel which supports this board, and the corresponding dtb.

I've uploaded prebuilt binaries for both kernel (md5sum: 7c4831e852d6dda2145dd04fe3c2b464) and dtb (md5sum: 249885543f0fcca2ce7a011ef5157e7d).

If you want to build the kernel yourself, you'll need a vanilla kernel (the one from RPi2's repo wouldn't build). In case of doubt, just grab the lastest stable from kernel.org. Use the default configuration for Vestatile Express A15 (make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- vexpress_defconfig) and make sure you enable these options:

  • General setup -> Configure standard kernel features (expert users)
  • General setup -> open by fhandle syscalls
  • Enable the block layer -> Support for large (2TB+) block devices and files

Then build both kernel and dtb:

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- zImage dtbs

This will generate linux/arch/arm/boot/zImage and linux/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb. Copy both files to your Raspberry.

 
Running our first Guest

In addition to the kernel, we also need a root filesystem with the distribution we want to use for our userland. As we’re going to virtualize an ARMv7 CPU, the best option is using a earmv7hf distribution. In this guide, we’re going to use a minimal OpenSuSE image (JeOS), but you can choose the one of your preference.

Create a directory with the files needed by the Guest:

pi@raspberrypi:~$ mkdir -p ~/kvm-arm/opensuse
pi@raspberrypi:~$ cd ~/kvm-arm/opensuse
pi@raspberrypi:~/kvm-arm/opensuse$

Now we create a raw image with OpenSuSE’s userland:

pi@raspberrypi:~/kvm-arm/opensuse$ wget http://download.opensuse.org/ports/armv7hl/factory/images/openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-Current.tbz
pi@raspberrypi:~/kvm-arm/opensuse$ qemu-img create -f raw opensuse-factory.img 1G
Formatting 'opensuse-factory.img', fmt=raw size=1073741824
pi@raspberrypi:~/kvm-arm/opensuse$ sudo losetup /dev/loop0 opensuse-factory.img
pi@raspberrypi:~/kvm-arm/opensuse$ sudo mkfs.ext4 /dev/loop0
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

pi@raspberrypi:~/kvm-arm/opensuse$ sudo mount /dev/loop0 /mnt
pi@raspberrypi:~/kvm-arm/opensuse$ sudo tar xf openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-Current.tbz -C /mnt
pi@raspberrypi:~/kvm-arm/opensuse$ sudo umount /mnt
pi@raspberrypi:~/kvm-arm/opensuse$ sudo losetup -d /dev/loop0

And finally, launch your first Guest:

pi@raspberrypi:~/kvm-arm/opensuse$ sudo qemu-system-arm -enable-kvm -smp 1 -m 256 -M vexpress-a15 -cpu host -kernel /home/pi/vexpress-zImage -dtb /home/pi/vexpress-v2p-ca15-tc1.dtb -append "root=/dev/vda console=ttyAMA0 rootwait" -drive if=none,file=/home/pi/opensuse-factory.img,id=factory -device virtio-blk-device,drive=factory -net nic,macaddr=02:fd:01:de:ad:34 -net tap -monitor null -serial stdio -nographic
audio: Could not init `oss' audio driver
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpuset
Linux version 3.19.1 (slp@linux-ni2o) (gcc version 4.5.3 (GCC) ) #1 SMP Wed Mar 18 10:52:22 CET 2015
CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: V2P-CA15
Memory policy: Data cache writealloc
PERCPU: Embedded 9 pages/cpu @8fddd000 s7232 r8192 d21440 u36864
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
Kernel command line: root=/dev/vda console=ttyAMA0 rootwait
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 252936K/262144K available (4839K kernel code, 184K rwdata, 1324K rodata, 256K init, 147K bss, 9208K reserved, 0K cma-reserved)
...
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

Welcome to openSUSE Factory "Tumbleweed" - Kernel 3.19.1 (ttyAMA0).


linux login:

Now we can start playing with this Guest. The default credentials are root/linux:

linux login: root
Password: 
Last login: Tue Mar 17 18:01:41 on ttyAMA0
Have a lot of fun...
linux:~ # cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 5 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xc07
CPU revision	: 5

Hardware	: ARM-Versatile Express
Revision	: 0000
Serial		: 0000000000000000

 
The end
And that’s it, now you can extract the full potential of your Raspberry Pi 2!

Speaking for myself, I’m going to work on getting MMIO support on NetBSD (without this, the performance is really awful), and then I’ll be able to build an hybrid Linux/NetBSD SD card image, which was my initial motivation for all this work 😉

Recent Comments

  • mkoryak

    18 March, 2015 at 2:02 pm

    Last line of the article is “And that’s it, now you can extract the full potential of your Raspberry Pi 2!”
    Can someone explain this to me? I know almost nothing about virtualization, but isn’t it be definition slower than straight up OS install?

    • Wonseok

      9 August, 2015 at 3:14 pm

      I think that “full potential” means he enable hardware-assited virtualization feature(ARM VE) and porting KVM/ARM on raspberry Pi2. You know, when you got any ARM boards, vendor always hide the hardware-assited virtualization feature because in normal development processes(i.e., linux, RTOS and android), you don’t need to use the feature. Therefore, they always provide BSP(board support package) for normal development. If you want to use hardware-assisted virtualization feature for ARM Board, you must modify firmware or bootloader to enable HYP mode. Do you want to know how the board boot with HYP mode, you need to study about ARM boot procedure.

      Anyway, as you know, virtualization give us some benefits e.g. supporting multiple os onto single platform but give us some drawbacks e.g. performacen degradation. This is the fact and it make us to use virtualizaion uncomfortably way in terms of performace. Sometimes, however, system have to work long time without downtime, for example, factory automation. Those kind of system, they don’t consider performance of system, they only consider how long can system work or reducing downtime without human intervention(It’s unpredictable behavior). In this case, virtualization offers many benefits.

  • Nico Maas

    18 March, 2015 at 2:30 pm

    I have not tested it myself, but this is some remarkable job! Thanks for your awesome work :)!

    • Sergio L. Pascual

      18 March, 2015 at 5:06 pm

      Thanks!

      • Wiki

        8 November, 2016 at 11:55 pm

        Hi Sergio
        Please can you help me to enable networking in qemu guest machine, i shall be really greatfull, as i have tried many tutorials but it is not working.
        when i execute command with out network parameters its work but when i use network parameter i get error like unable to open /dev/net/tun.
        i also followed following and many other tutorials
        https://en.wikibooks.org/wiki/QEMU/Networking

        Thank you

  • Matthias Cramer

    18 March, 2015 at 2:42 pm

    Thats a great article, thanks. Would it be possible to do the oposite in cpu isolation that you did and dedicate 1 core to the native kernel and interropts and have 3 cores for virtualisation? This way it would be feasable to have more that one VM running…

    • Sergio L. Pascual

      18 March, 2015 at 5:09 pm

      Sure, you can use something like isolcpus=1,2,3 on your /boot/cmdline.txt. You’ll need to tweak QEMU a little too, to be able to specify which core you want to use from the command line (in my patch, the core number is hardcoded).

      • Dada

        8 August, 2015 at 2:10 pm

        Hi,
        Ok to use multi core for several VM instance. But how to allow a single VM to use several cores. Even if I specify several cores with the smp option when qemu is launching, only one core is used by the VM !
        Thanks

  • Alex Ellis

    18 March, 2015 at 5:22 pm

    Ninja mastery at play. Great article and good examples. What next?

  • MY123

    19 March, 2015 at 6:51 pm

    The RPi2 has ARM JTAG on the GPIOs.

  • Mihai

    1 June, 2015 at 3:21 pm

    Awesome tutorial. I failed initially to configure networking, but with some qemu net nic,tap hacks it finally worked, following all your steps.

    • Garrett

      20 November, 2015 at 6:51 pm

      Would you have the details on how you got the networking working on the QEMU image?

    • Wiki

      6 November, 2016 at 11:37 pm

      Hi Mihai,
      Please can you help me to enable networking in qemu guest machine, i shall be really greatfull, as i have tried many tutorials but it is not working.

      Thank you

  • nik

    2 July, 2015 at 4:49 am

    I got opensuse to boot but without the -net tap option. When i use the -net tap option, I get the following error: qemu-system-arm: -net tap: Device ‘tap’ could not be initialized.
    Could you suggest how you fixed this error and enabled networking?

  • Icey590

    15 July, 2015 at 1:22 pm

    For your premade kernel, could you upload the modules?

    • Wonseok

      9 August, 2015 at 3:20 pm

      In many normal cases, you could not insert the modules into prebuilt kernel.
      However, If prebuilt kernel has configuration for your module, you can insert your module.

      Answer is that depends on your prebuilt kernel.

  • Fab

    16 July, 2015 at 5:25 pm

    Great tutorial. I have managed to run the guest (Linaro) on top of the Raspbian by following your tutorial. Now I am trying to install wiringPi library on the guest and trying to control the GPIO of the raspberryPi. The library is successfully installed but when I use the command “gpio readall” I get the message”I see Hardware: ARM-Versatile Express – expecting BCM2708 or BCM2709″. I have checked the list of available machines for the qemu-system-arm (qemu-system-arm -M help) but BCM2708 or BCM2709 are not listed. How can I access the GPIO of the raspberryPi from the guest (Linaro) ? Any suggestion or help will be highly appreciated. Thanks in advance.

  • Kinnefix

    21 July, 2015 at 3:11 pm

    Hi, is it possible to compile the kernel from your repo for rpi2 with kernel version 4.0?

    • Sergio L. Pascual

      27 July, 2015 at 9:00 am

      That would require some effort to port the code to 4.x, but shouldn’t be hard.

  • Stefan

    5 August, 2015 at 9:13 pm

    Hello,

    I followed the steps and everything was compiled sucessfully on RPi2. During boot I get:

    [ 0.000000] Kernel command line: … isolcpus=3

    [ 0.154274] CPU: All CPU(s) started in HYP mode.
    [ 0.154301] CPU: Virtualization extensions available.

    [ 0.972826] kvm [1]: timer IRQ99
    [ 0.976111] kvm [1]: Hyp mode initialized successfully

    But when trying to start Qemu (Patched 2.2.1) I get the following:
    MiniPi ~ # qemu-system-i386 -enable-kvm -cpu n270 -drive file=/root/HS-debian.qcow,if=ide,index=0,media=disk -net nic -net tap,ifname=tap0,script=no,downscript=no -m 512M -usb -vnc :1 -daemonize

    “kvm” accelerator not found.
    No accelerator found!

    Is KVM not supported for i386 on RPi2?

    No way to accellerate? Read some threads emulating WinXP-Machines that where referring this post.

    Thanks and best regards,
    Stefan

    • Sergio L. Pascual

      17 August, 2015 at 9:43 am

      Nope, by its own nature, virtualization only works when both Guest and Host use the same architecture. Otherwise, it would be emulation.

  • chisight

    16 September, 2015 at 10:15 pm

    Might taskset work rather than patching qemu?

    for using core 2 and 3 config.txt would have:
    isolcpus=2,3
    the command line would become:
    taskset 0x0000000C qemu-system-arm -enable-kvm -smp 2 -m 256 -M vexpress-a15 -cpu host -kernel /home/pi/vexpress-zImage -dtb /home/pi/vexpress-v2p-ca15-tc1.dtb -append “root=/dev/vda console=ttyAMA0 rootwait” -drive if=none,file=/home/pi/opensuse-factory.img,id=factory -device virtio-blk-device,drive=factory -net nic,macaddr=02:fd:01:de:ad:34 -net tap -monitor null -serial stdio -nographic

  • Sebastjan

    18 October, 2015 at 1:44 am

    Is it possible for i386 emu instead of arm with your guide ? Well done sir!

  • Mallapuram Phanirajkiran

    26 October, 2015 at 7:46 pm

    Nice Article. YUP. Done it!!

  • Vincent

    30 October, 2015 at 4:00 pm

    Hi, I’ve tried your guide and it works very well. I was wondering, have you tried with newer version of the Kernel something like 4.0+?

  • Quora

    17 November, 2015 at 9:38 pm

    How can I build Cloud Computing Infrastructure on top of my Raspberry Pi?

    Let me explain the key elements for setting up your own Cloud Computing Infrastructure. 1. Hardware that supports virtualization. So that you can boot up instances and allocate them to your clients. 2. A pool of public IP’s that will be allocated to y…

  • Garrett

    18 November, 2015 at 12:25 pm

    Great blog by the way! Just wondering if anybody managed to get the networking enabled on their OpenSUSE VM? I have the VM up and running but I’m getting a few errors on start-up, most of which are in relation to networking. Any suggestions welcome!

    • Christian

      29 February, 2016 at 11:31 pm

      I used
      -netdev user,id=net0 -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55
      for network.

      With tap network a bridge has to be set up an configured beforhand.
      -netdev tap,id=net0,script=/etc/qemu-ifup -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55

      /etc/network/interfaces:
      auto lo br0
      [other stuff]
      # Bridge setup
      iface br0 inet dhcp
      bridge_ports eth0

      /etc/qemu-ifup:
      #!/bin/sh
      set -x

      switch=br0

      if [ -n “$1” ];then
      tunctl -u `whoami` -t $1
      ip link set $1 up
      sleep 0.5s
      brctl addif $switch $1
      exit 0
      else
      echo “Error: no interface specified”
      exit 1
      fi

      • Wiki

        9 November, 2016 at 12:24 am

        hi
        can you help me to enable networking in qemu guest machine, i shall be really greatfull, as i have tried many tutorial but it is not working.

        Thank you

  • Travis M

    5 December, 2015 at 6:32 am

    Excellent tutorial! I am a Sr Engineer with Virtuozzo and we have been making Container / VM Virtualization for 15yrs but this makes me wonder how small I could make a cluster. What I would love to know is if you could load our bare metal Virtuozzo load, Linux based, on 5x Pi’s and then cluster the SD’s together from the 5x using our software defined storage option built into Virtuozzo giving you fail over between the 5 Pi’s… I have done it with 5x Lenovo Notebooks but it would be way cool done on some Raspberry’s. Any thoughts on this or any desire in playing with some copies of our solution to see if it would work? Just an awesome walk through either way. Thanks for sharing and email me directly if you want to work on this project with me…

  • su xing yu

    17 February, 2016 at 3:48 am

    i am trying to run car diagnosis software( vw vcds and toyota MiniVCI), which only support windows, on my raspberry pi2 . this is very helpful

  • Virtualization Blog |

    4 March, 2016 at 12:17 am

    […] Enabling KVM virtualization for … – I think that “full potential” means he enable hardware-assited virtualization feature(ARM VE) and porting KVM/ARM on raspberry Pi2. You know, when you … […]

  • Ian

    7 May, 2016 at 9:44 pm

    Found your article and I am interested to know if you have done this with the Raspberry Pi 3? Are there any differences?

      • Pedrinho

        10 November, 2016 at 1:11 pm

        I’m curious about it too – in other words, what differences there are between the method used here and the method to be used on a Raspberry Pi 3… Do Pi2 and Pi3 use the very same kernell? And what about the updates that were released after this tutorial? Has something changed in a such a way that forces us to change this method? (Sorry for my horrible English, hope you can understand me, I don’t use translator bots like Google)

  • Wiki

    20 July, 2016 at 4:13 pm

    Hi IAN
    Did youget anything to do same thing on Pi 3?
    Am also looking for some stuff for Pi3. If you get something then please also share it with me.

    Thanks

  • Wiki

    6 November, 2016 at 11:36 pm

    Hi All,
    If someone can help me to enable networking in qemu guest machine, i shall be really greatfull, as i ave tried many tutorial but it is not working.

    Thank you

  • Eric

    18 February, 2017 at 6:29 pm

    Hello there, thanks for the post, I have tried to follow your step however I got a stuck right after the first step.
    After I tried to use your prebuild kernel image and reboot my raspberry pi it will just stuck on a color screen.
    I’ve reinstalled my jessie system a few time got the same result, any suggestion will be appreciated.

    thank you.

  • caprice-j

    21 February, 2017 at 5:08 am

    FYI: I bought RP3 last week, and before I use the pre-built kernel mentioned in this article, “CPU: All CPU(s) started in HYP mode.” appears in /var/log/messages after booting. I guess some changes are done in kernel code (I’m using shipped Raspbian).

  • Muhammad Faisal

    28 February, 2017 at 11:58 pm

    Hi I am new to virtualization…So kindly forgive my ignorance.

    One thing I could not understand is the Use of QEMU here..We have ARM cortex A7 which has hardware support for virtualization and then KVM which is ported here then what is the job of QEMU?

    with regards,
    Muhammad Faisal

Comments are closed.