Run a kvm-support vm on Raspberry pi 2

Raspberry Pi 2 uses bcm2709 which is an ARMV7 processor.  

cat /proc/cpuinfo

01 02 03 04 05 06 07 08 09 10 11

Hardware    : BCM2709 Revision    : a01041   model name  : ARMv7 Processor rev 5 (v7l) BogoMIPS    : 64.00 Features    : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part    : 0xc07 CPU revision    : 5

 

The processor is CORTEX-A7 with virtualization extension. However, this feature is not supported in current raspbian system.

This article is about how to implement virtulaization for Raspberry Pi 2.

We use uboot+zImag+DTB to boot machine here.

 

First, enable HYP mode.

[    0.154021] CPU: All CPU(s) started in SVC mode

Rsapberry Pi 2 jumps to SVC mode instead of HYP mode.

Solution:

Build a bootloader which properly sets HYP mode before jumping to 0x8000.

So, build a new u-boot, and connect to zImage.

git clone https://github.com/slp/rpi2-hyp-boot.git
cd rpi2-hyp-boot
make

this will get a bootblk.bin file which contains boot code and 32K padding, then  

mv /boot/kernel7.img /boot/kernel7.img.bak
cat bootblk.bin /boot/kernel7.img.bak > /boot/kernel7.img
echo "kernel_old=1" >> /boot/config.txt

This will make raspberry pi get into HYP mode.

[    0.154131] CPU: All CPU(s) started in HYP mode.
[    0.154158] CPU: Virtualization extensions available.

However, it is not over yet, we need run kvm-support vm finally.

Second, modification for the host kernel.

make sure the following options are enabled:

Patch physical to virtual translations at runtime

General setup -> Control Group support

System Type -> Support for Large Physical Address Extension

Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)

Boot options -> Supplement the appended DTB with traditional ATAG information

Device Drivers -> Block devices -> Loopback device support

Virtualization

Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)

DISABLE Virtualization -> KVM support for Virtual GIC

ENABLE Virtualization -> KVM support for Emulated GIC

Then cross-compiler this kernel.

wget http://releases.linaro.org/14.04/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux.tar.bz2

tar -xvf  gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux.tar.bz2

Add the path of gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux/bin to $PATH.

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- zImage dtbs

Last, wrap all things to a new kernel, kernel7.img by follewing step:

cat rpi2-hyp-boot/bootblk.bin linux/arch/arm/boot/zImage linux/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > kernel7.img

Edit /boot/cmdline.txt and add isolcpus=3 at the of the line

 

Third, modifcation on QEMU.

Rember to let QEMU support CPU affitiy.

wget http://wiki.qemu-project.org/download/qemu-2.2.0.tar.bz2
patch -p1 < ~/qemu-cpu-affinity.patch

After all setup, we need to build a guest vm kernel and dtb, build a rootfs. use qemu to boot it.

path-to-qemu/arm-softmmu/qemu-system-arm \

-enable-kvm -smp 1 -m 256 -M vexpress-a15 -cpu host \
-kernel vexpress-zImage \
-dtb vexpress-v2p-ca15-tc1.dtb \
-append "root=/dev/vda console=ttyAMA0 rootwait" \
-drive if=none,file=opensuse-factory.img,id=factory \
-device virtio-blk-device,drive=factory \
-monitor null -serial stdio \
-nographic

The only tricky part is to build rootfs img file.

with all three steps above we could run a kvm-support machine succesfully.

 

Reference :

1. https://www.raspberrypi.org/documentation/linux/kernel/building.md

2. http://blog.flexvdi.com/2015/03/17/enabling-kvm-virtualization-on-the-raspberry-pi-2/

3. https://lwn.net/Articles/557132/