As I wrote on my previous post, Enabling HYP mode on the Raspberry Pi 2, the newest machine from the Raspberry Pi Foundation features a Cortex-A7 with Virtualization Extensions, but it isn’t possible to make use of such feature out of the box.
In that article I showed that it was possible to start the kernel in HYP mode. Now, I’ll cover the rest of steps needed for enabling KVM virtualization and running your first guest OS.
Isolating a core
With KVM on ARM, interrupts are dealt in a very particular way. If a hardware interrupt is received when a core in running in guest mode, the execution is interrupted, a full world switch takes place returning to the kernel in host mode, then interrupts are reenabled and the core traps again, this time following the usual route.
Apparently, the BCM2836 (the core of the RPi2) doesn’t like the behavior, and if you try to run a guest on a core that can receive physical interrupts, you’ll find your RPi2 hangs within a few seconds. Without a JTAG of a full development board, debugging this problem is very, very hard.
Luckily, there’s a pretty simple workaround. Using the kernel option isolcpus when can isolate a core, so Linux doesn’t assign any tasks (including IRQ lines) to it.
In our case, we’re going to isolate the core number 3 (starting from zero). This is pretty straightforward, just edit /boot/cmdline.txt and add isolcpus=3 at the of the line.
A kernel with VGIC emulation for the RPi2
On my previous article about RPi2 emulation, I wrote about the BCM2836 and its lack of a GIC, which makes emulation a bit harder. To work around this issue, I’ve implemented VGIC emulation inside the Linux kernel.
To be able to use both the customized bootloader and this modified version of the kernel, we need to generate a bundle with both components, and the dtb concatenated at end.
The easy way. Using a prebuilt image
I’ve uploaded a prebuilt image here (md5sum: 356788d260e1a93376c5b8acbb63da13), which contains the bootloader, some padding (to put the zImage at 0x8000), the kernel and the dtb. Simply replace your kernel with it (save a copy first!):
mv /boot/kernel7.img /boot/kernel7.img.bak cp kernel7.img /boot
Then you’ll need to add the kernel_old=1 option to your config.txt:
echo "kernel_old=1" >> /boot/config.txt
That’s it! On the next boot, Linux should say something like this:
[ 0.154131] CPU: All CPU(s) started in HYP mode. [ 0.154158] CPU: Virtualization extensions available.
Building it yourself (Part I: the bootloader)
If you want to build the image by yourself, you’ll need to grab an ARM cross toolchain. I’m using the one from the GCC ARM Embedded project, which works just fine. Then add it to your $PATH, grab the code for my repo, and build it:
export PATH=$PATH:/home/slp/sources/gcc-arm-none-eabi-4_9-2014q4/bin git clone https://github.com/slp/rpi2-hyp-boot.git cd rpi2-hyp-boot make
That should produce a bootblk.bin, with contains the boot code and 32k of padding.
Building it yourself (Part II: kernel and dtb)
You can find a complete guide for building the kernel on a variety of host systems here Raspberry Pi Kernel Compilation. Read that and the grab the sources from my repo and make sure the following options are enabled:
Patch physical to virtual translations at runtimeGeneral setup -> Control Group supportSystem Type -> Support for Large Physical Address ExtensionBoot options -> Use appended device tree blob to zImage (EXPERIMENTAL)Boot options -> Supplement the appended DTB with traditional ATAG informationDevice Drivers -> Block devices -> Loopback device supportVirtualizationVirtualization -> Kernel-based Virtual Machine (KVM) support (NEW)- DISABLE
Virtualization -> KVM support for Virtual GIC - ENABLE
Virtualization -> KVM support for Emulated GIC
Now build both kernel and dtb:
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- zImage dtbs
Building it yourself (Part III: bundling all pieces)
This is the easy part, just concatenate them:
cat rpi2-hyp-boot/bootblk.bin linux/arch/arm/boot/zImage linux/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > kernel7.img
Now you can use it in the same way as the prebuilt image provided above.
Patched QEMU installation
To be able to launch our first Guest, we need a recent and patched (to ensure the guest is running in the isolated core) QEMU. The one that comes with Raspbian is quite old, so you have to build a newer one:
Download the build dependencies:
pi@raspberrypi:~$ sudo apt-get build-dep qemu pi@raspberrypi:~$ sudo apt-get install libpixman-1-dev
Then, create a directory for the sources, download QEMU 2.2 and uncompress it:
pi@raspberrypi:~$ mkdir srcs pi@raspberrypi:~$ cd srcs pi@raspberrypi:~/srcs$ wget http://wiki.qemu-project.org/download/qemu-2.2.0.tar.bz2 pi@raspberrypi:~/srcs$ tar xf qemu-2.2.0.tar.bz2
We need to apply this patch to make sure the QEMU runs our guest in the core we isolated with option isolcpus:
pi@raspberrypi:~/srcs$ cd qemu-2.2.0/ pi@raspberrypi:~/srcs/qemu-2.2.0$ patch -p1 < ~/qemu-cpu-affinity.patch
Now run the configure script with options for enabling KVM (it is enabled automatically if supported, but this way we ensure we get warned if it's not going to be build with it) and for building the ARM emulation target only:
pi@raspberrypi:~/srcs/qemu-2.2.0$ ./configure --enable-kvm --target-list=arm-softmmu
Finally, build and install it (by default, the new binaries will reside in /usr/local/bin):
pi@raspberrypi:~/srcs/qemu-2.2.0$ make pi@raspberrypi:~/srcs/qemu-2.2.0$ sudo make install
A kernel for the Guest
When running QEMU with KVM, the hardware emulated is a Versatile Express A15, one the reference platforms provided by ARM Holdings. So, for our Guest we need a kernel which supports this board, and the corresponding dtb.
I've uploaded prebuilt binaries for both kernel (md5sum: 7c4831e852d6dda2145dd04fe3c2b464) and dtb (md5sum: 249885543f0fcca2ce7a011ef5157e7d).
If you want to build the kernel yourself, you'll need a vanilla kernel (the one from RPi2's repo wouldn't build). In case of doubt, just grab the lastest stable from kernel.org. Use the default configuration for Vestatile Express A15 (make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- vexpress_defconfig) and make sure you enable these options:
General setup -> Configure standard kernel features (expert users)General setup -> open by fhandle syscallsEnable the block layer -> Support for large (2TB+) block devices and files
Then build both kernel and dtb:
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- zImage dtbs
This will generate linux/arch/arm/boot/zImage and linux/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb. Copy both files to your Raspberry.
Running our first Guest
In addition to the kernel, we also need a root filesystem with the distribution we want to use for our userland. As we’re going to virtualize an ARMv7 CPU, the best option is using a earmv7hf distribution. In this guide, we’re going to use a minimal OpenSuSE image (JeOS), but you can choose the one of your preference.
Create a directory with the files needed by the Guest:
pi@raspberrypi:~$ mkdir -p ~/kvm-arm/opensuse pi@raspberrypi:~$ cd ~/kvm-arm/opensuse pi@raspberrypi:~/kvm-arm/opensuse$
Now we create a raw image with OpenSuSE’s userland:
pi@raspberrypi:~/kvm-arm/opensuse$ wget http://download.opensuse.org/ports/armv7hl/factory/images/openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-Current.tbz pi@raspberrypi:~/kvm-arm/opensuse$ qemu-img create -f raw opensuse-factory.img 1G Formatting 'opensuse-factory.img', fmt=raw size=1073741824 pi@raspberrypi:~/kvm-arm/opensuse$ sudo losetup /dev/loop0 opensuse-factory.img pi@raspberrypi:~/kvm-arm/opensuse$ sudo mkfs.ext4 /dev/loop0 mke2fs 1.42.5 (29-Jul-2012) Discarding device blocks: done Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done pi@raspberrypi:~/kvm-arm/opensuse$ sudo mount /dev/loop0 /mnt pi@raspberrypi:~/kvm-arm/opensuse$ sudo tar xf openSUSE-Factory-ARM-JeOS.armv7-rootfs.armv7l-Current.tbz -C /mnt pi@raspberrypi:~/kvm-arm/opensuse$ sudo umount /mnt pi@raspberrypi:~/kvm-arm/opensuse$ sudo losetup -d /dev/loop0
And finally, launch your first Guest:
pi@raspberrypi:~/kvm-arm/opensuse$ sudo qemu-system-arm -enable-kvm -smp 1 -m 256 -M vexpress-a15 -cpu host -kernel /home/pi/vexpress-zImage -dtb /home/pi/vexpress-v2p-ca15-tc1.dtb -append "root=/dev/vda console=ttyAMA0 rootwait" -drive if=none,file=/home/pi/opensuse-factory.img,id=factory -device virtio-blk-device,drive=factory -net nic,macaddr=02:fd:01:de:ad:34 -net tap -monitor null -serial stdio -nographic
audio: Could not init `oss' audio driver
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpuset
Linux version 3.19.1 (slp@linux-ni2o) (gcc version 4.5.3 (GCC) ) #1 SMP Wed Mar 18 10:52:22 CET 2015
CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: V2P-CA15
Memory policy: Data cache writealloc
PERCPU: Embedded 9 pages/cpu @8fddd000 s7232 r8192 d21440 u36864
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 65024
Kernel command line: root=/dev/vda console=ttyAMA0 rootwait
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 252936K/262144K available (4839K kernel code, 184K rwdata, 1324K rodata, 256K init, 147K bss, 9208K reserved, 0K cma-reserved)
...
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
Starting Update UTMP about System Runlevel Changes...
[ OK ] Started Update UTMP about System Runlevel Changes.
Welcome to openSUSE Factory "Tumbleweed" - Kernel 3.19.1 (ttyAMA0).
linux login:
Now we can start playing with this Guest. The default credentials are root/linux:
linux login: root Password: Last login: Tue Mar 17 18:01:41 on ttyAMA0 Have a lot of fun... linux:~ # cat /proc/cpuinfo processor : 0 model name : ARMv7 Processor rev 5 (v7l) BogoMIPS : 38.40 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xc07 CPU revision : 5 Hardware : ARM-Versatile Express Revision : 0000 Serial : 0000000000000000
The end
And that’s it, now you can extract the full potential of your Raspberry Pi 2!
Speaking for myself, I’m going to work on getting MMIO support on NetBSD (without this, the performance is really awful), and then I’ll be able to build an hybrid Linux/NetBSD SD card image, which was my initial motivation for all this work 😉
Recent Comments
mkoryak
18 March, 2015 at 2:02 pmLast line of the article is “And that’s it, now you can extract the full potential of your Raspberry Pi 2!”
Can someone explain this to me? I know almost nothing about virtualization, but isn’t it be definition slower than straight up OS install?
Wonseok
9 August, 2015 at 3:14 pmI think that “full potential” means he enable hardware-assited virtualization feature(ARM VE) and porting KVM/ARM on raspberry Pi2. You know, when you got any ARM boards, vendor always hide the hardware-assited virtualization feature because in normal development processes(i.e., linux, RTOS and android), you don’t need to use the feature. Therefore, they always provide BSP(board support package) for normal development. If you want to use hardware-assisted virtualization feature for ARM Board, you must modify firmware or bootloader to enable HYP mode. Do you want to know how the board boot with HYP mode, you need to study about ARM boot procedure.
Anyway, as you know, virtualization give us some benefits e.g. supporting multiple os onto single platform but give us some drawbacks e.g. performacen degradation. This is the fact and it make us to use virtualizaion uncomfortably way in terms of performace. Sometimes, however, system have to work long time without downtime, for example, factory automation. Those kind of system, they don’t consider performance of system, they only consider how long can system work or reducing downtime without human intervention(It’s unpredictable behavior). In this case, virtualization offers many benefits.
Nico Maas
18 March, 2015 at 2:30 pmI have not tested it myself, but this is some remarkable job! Thanks for your awesome work :)!
Sergio L. Pascual
18 March, 2015 at 5:06 pmThanks!
Wiki
8 November, 2016 at 11:55 pmHi Sergio
Please can you help me to enable networking in qemu guest machine, i shall be really greatfull, as i have tried many tutorials but it is not working.
when i execute command with out network parameters its work but when i use network parameter i get error like unable to open /dev/net/tun.
i also followed following and many other tutorials
https://en.wikibooks.org/wiki/QEMU/Networking
Thank you
Matthias Cramer
18 March, 2015 at 2:42 pmThats a great article, thanks. Would it be possible to do the oposite in cpu isolation that you did and dedicate 1 core to the native kernel and interropts and have 3 cores for virtualisation? This way it would be feasable to have more that one VM running…
Sergio L. Pascual
18 March, 2015 at 5:09 pmSure, you can use something like
isolcpus=1,2,3on your/boot/cmdline.txt. You’ll need to tweak QEMU a little too, to be able to specify which core you want to use from the command line (in my patch, the core number is hardcoded).Dada
8 August, 2015 at 2:10 pmHi,
Ok to use multi core for several VM instance. But how to allow a single VM to use several cores. Even if I specify several cores with the smp option when qemu is launching, only one core is used by the VM !
Thanks
Alex Ellis
18 March, 2015 at 5:22 pmNinja mastery at play. Great article and good examples. What next?
MY123
19 March, 2015 at 6:51 pmThe RPi2 has ARM JTAG on the GPIOs.
Mihai
1 June, 2015 at 3:21 pmAwesome tutorial. I failed initially to configure networking, but with some qemu net nic,tap hacks it finally worked, following all your steps.
Garrett
20 November, 2015 at 6:51 pmWould you have the details on how you got the networking working on the QEMU image?
Wiki
6 November, 2016 at 11:37 pmHi Mihai,
Please can you help me to enable networking in qemu guest machine, i shall be really greatfull, as i have tried many tutorials but it is not working.
Thank you
nik
2 July, 2015 at 4:49 amI got opensuse to boot but without the -net tap option. When i use the -net tap option, I get the following error: qemu-system-arm: -net tap: Device ‘tap’ could not be initialized.
Could you suggest how you fixed this error and enabled networking?
Icey590
15 July, 2015 at 1:22 pmFor your premade kernel, could you upload the modules?
Wonseok
9 August, 2015 at 3:20 pmIn many normal cases, you could not insert the modules into prebuilt kernel.
However, If prebuilt kernel has configuration for your module, you can insert your module.
Answer is that depends on your prebuilt kernel.
Fab
16 July, 2015 at 5:25 pmGreat tutorial. I have managed to run the guest (Linaro) on top of the Raspbian by following your tutorial. Now I am trying to install wiringPi library on the guest and trying to control the GPIO of the raspberryPi. The library is successfully installed but when I use the command “gpio readall” I get the message”I see Hardware: ARM-Versatile Express – expecting BCM2708 or BCM2709″. I have checked the list of available machines for the qemu-system-arm (qemu-system-arm -M help) but BCM2708 or BCM2709 are not listed. How can I access the GPIO of the raspberryPi from the guest (Linaro) ? Any suggestion or help will be highly appreciated. Thanks in advance.
Kinnefix
21 July, 2015 at 3:11 pmHi, is it possible to compile the kernel from your repo for rpi2 with kernel version 4.0?
Sergio L. Pascual
27 July, 2015 at 9:00 amThat would require some effort to port the code to 4.x, but shouldn’t be hard.
Stefan
5 August, 2015 at 9:13 pmHello,
I followed the steps and everything was compiled sucessfully on RPi2. During boot I get:
…
[ 0.000000] Kernel command line: … isolcpus=3
…
[ 0.154274] CPU: All CPU(s) started in HYP mode.
[ 0.154301] CPU: Virtualization extensions available.
…
[ 0.972826] kvm [1]: timer IRQ99
[ 0.976111] kvm [1]: Hyp mode initialized successfully
…
But when trying to start Qemu (Patched 2.2.1) I get the following:
MiniPi ~ # qemu-system-i386 -enable-kvm -cpu n270 -drive file=/root/HS-debian.qcow,if=ide,index=0,media=disk -net nic -net tap,ifname=tap0,script=no,downscript=no -m 512M -usb -vnc :1 -daemonize
“kvm” accelerator not found.
No accelerator found!
Is KVM not supported for i386 on RPi2?
No way to accellerate? Read some threads emulating WinXP-Machines that where referring this post.
Thanks and best regards,
Stefan
Sergio L. Pascual
17 August, 2015 at 9:43 amNope, by its own nature, virtualization only works when both Guest and Host use the same architecture. Otherwise, it would be emulation.
chisight
16 September, 2015 at 10:15 pmMight taskset work rather than patching qemu?
for using core 2 and 3 config.txt would have:
isolcpus=2,3
the command line would become:
taskset 0x0000000C qemu-system-arm -enable-kvm -smp 2 -m 256 -M vexpress-a15 -cpu host -kernel /home/pi/vexpress-zImage -dtb /home/pi/vexpress-v2p-ca15-tc1.dtb -append “root=/dev/vda console=ttyAMA0 rootwait” -drive if=none,file=/home/pi/opensuse-factory.img,id=factory -device virtio-blk-device,drive=factory -net nic,macaddr=02:fd:01:de:ad:34 -net tap -monitor null -serial stdio -nographic
Sebastjan
18 October, 2015 at 1:44 amIs it possible for i386 emu instead of arm with your guide ? Well done sir!
Mallapuram Phanirajkiran
26 October, 2015 at 7:46 pmNice Article. YUP. Done it!!
Vincent
30 October, 2015 at 4:00 pmHi, I’ve tried your guide and it works very well. I was wondering, have you tried with newer version of the Kernel something like 4.0+?
Quora
17 November, 2015 at 9:38 pmHow can I build Cloud Computing Infrastructure on top of my Raspberry Pi?
Let me explain the key elements for setting up your own Cloud Computing Infrastructure. 1. Hardware that supports virtualization. So that you can boot up instances and allocate them to your clients. 2. A pool of public IP’s that will be allocated to y…
Garrett
18 November, 2015 at 12:25 pmGreat blog by the way! Just wondering if anybody managed to get the networking enabled on their OpenSUSE VM? I have the VM up and running but I’m getting a few errors on start-up, most of which are in relation to networking. Any suggestions welcome!
Christian
29 February, 2016 at 11:31 pmI used
-netdev user,id=net0 -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55
for network.
With tap network a bridge has to be set up an configured beforhand.
-netdev tap,id=net0,script=/etc/qemu-ifup -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55
/etc/network/interfaces:
auto lo br0
[other stuff]
# Bridge setup
iface br0 inet dhcp
bridge_ports eth0
/etc/qemu-ifup:
#!/bin/sh
set -x
switch=br0
if [ -n “$1” ];then
tunctl -u `whoami` -t $1
ip link set $1 up
sleep 0.5s
brctl addif $switch $1
exit 0
else
echo “Error: no interface specified”
exit 1
fi
Wiki
9 November, 2016 at 12:24 amhi
can you help me to enable networking in qemu guest machine, i shall be really greatfull, as i have tried many tutorial but it is not working.
Thank you
Travis M
5 December, 2015 at 6:32 amExcellent tutorial! I am a Sr Engineer with Virtuozzo and we have been making Container / VM Virtualization for 15yrs but this makes me wonder how small I could make a cluster. What I would love to know is if you could load our bare metal Virtuozzo load, Linux based, on 5x Pi’s and then cluster the SD’s together from the 5x using our software defined storage option built into Virtuozzo giving you fail over between the 5 Pi’s… I have done it with 5x Lenovo Notebooks but it would be way cool done on some Raspberry’s. Any thoughts on this or any desire in playing with some copies of our solution to see if it would work? Just an awesome walk through either way. Thanks for sharing and email me directly if you want to work on this project with me…
su xing yu
17 February, 2016 at 3:48 ami am trying to run car diagnosis software( vw vcds and toyota MiniVCI), which only support windows, on my raspberry pi2 . this is very helpful
Virtualization Blog |
4 March, 2016 at 12:17 am[…] Enabling KVM virtualization for … – I think that “full potential” means he enable hardware-assited virtualization feature(ARM VE) and porting KVM/ARM on raspberry Pi2. You know, when you … […]
Ian
7 May, 2016 at 9:44 pmFound your article and I am interested to know if you have done this with the Raspberry Pi 3? Are there any differences?
Elias
23 August, 2016 at 7:32 am+1
Pedrinho
10 November, 2016 at 1:11 pmI’m curious about it too – in other words, what differences there are between the method used here and the method to be used on a Raspberry Pi 3… Do Pi2 and Pi3 use the very same kernell? And what about the updates that were released after this tutorial? Has something changed in a such a way that forces us to change this method? (Sorry for my horrible English, hope you can understand me, I don’t use translator bots like Google)
Wiki
20 July, 2016 at 4:13 pmHi IAN
Did youget anything to do same thing on Pi 3?
Am also looking for some stuff for Pi3. If you get something then please also share it with me.
Thanks
Wiki
6 November, 2016 at 11:36 pmHi All,
If someone can help me to enable networking in qemu guest machine, i shall be really greatfull, as i ave tried many tutorial but it is not working.
Thank you
Eric
18 February, 2017 at 6:29 pmHello there, thanks for the post, I have tried to follow your step however I got a stuck right after the first step.
After I tried to use your prebuild kernel image and reboot my raspberry pi it will just stuck on a color screen.
I’ve reinstalled my jessie system a few time got the same result, any suggestion will be appreciated.
thank you.
caprice-j
21 February, 2017 at 5:08 amFYI: I bought RP3 last week, and before I use the pre-built kernel mentioned in this article, “CPU: All CPU(s) started in HYP mode.” appears in /var/log/messages after booting. I guess some changes are done in kernel code (I’m using shipped Raspbian).
Muhammad Faisal
28 February, 2017 at 11:58 pmHi I am new to virtualization…So kindly forgive my ignorance.
One thing I could not understand is the Use of QEMU here..We have ARM cortex A7 which has hardware support for virtualization and then KVM which is ported here then what is the job of QEMU?
with regards,
Muhammad Faisal
Comments are closed.