I got curious about following up these two threads making Cog a unikernel... Cog in the cloud http://forum.world.st/Cog-in-the-cloud-td4796514.html Hosting the Pharo VM on MirageOS http://forum.world.st/Hosting-the-Pharo-VM-on-MirageOS-td4799424.html#none One path forward is to use rumpkernel, which "provides a means to run existing POSIX applications as unikernels on Xen" https://blog.xenproject.org/2015/08/06/on-rump-kernels-and-the-rumprun-unike... Deploying real-world software today as unikernels on Xen with Rumprun http://events.linuxfoundation.org/sites/events/files/slides/xdps15-talk-fina... So I thought I would try the "Building Rumprun Unikernels" tutorial here... https://github.com/rumpkernel/wiki/wiki/Tutorial:-Building-Rumprun-Unikernel... which I report here in case it was interest. The tutorial is based on QEMU (and I think I saw a comment somewhere that Cog doesn't run so well on QEMU and needs Bochs), so after that I adapted it from QEMU to Xen. 64-BIT DEBIAN 8 JESSIE & XEN So first I installed Debian 8 Jessie 64-bit on my new laptop $ uname -a
...x86_64
Then installed Xen and after a reboot could choose it from the boot menu. $ apt-get update $ apt get install xen-system-amd64 libxen-dev REBOOTED Tested manual selection from menu to boot into Xen dom0. Checked Xen was operating with... $ xl info 32-BIT DEBIAN 8 JESSIE & XEN However later I hit some complications determining whether the rump unikernel was being compiled 32-bit or 64-bit in order to mix in with 32-bit Pharo, so to simplify things I did a complete reinstall of my system as 32-bit. $ uname -a
...686-pae
Then installed Xen again Xen only has a 64-bit hypervisor, but this was not immediately available under 32-bit OS. I can't exactly remember but I think I just needed to add multiarch before it would find the Xen package. $ apt-get update $ apt-get install multiarch $ apt get install xen-system-amd64 libxen-dev Otherwise it may have needed something like "dpkg --add-architecture ..." REBOOTED First test, manually selected to boot into Xen dom0, after which as root the following checked Xen was operating... $ xl info Then set it to automatically boot into Xen... $ dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen $ update-grub REBOOTED ==================================== DID THE QUICK START TUTORIAL USING GENERIC HW ON QEMU 0. Installed support tools and libs $ apt-get install build-essential git-core cmake 1. Built the rumprun platform $ mkdir -p ~/Repos/rumprun-hw $ cd ~/Repos/rumprun-hw $ git clone http://repo.rumpkernel.org/rumprun $ cd rumprun $ git submodule update --init $ CC=cc ./build-rr.sh hw Took note of this output needed later...
toolchain tuple: i486-rumprun-netbsdelf cc wrapper: i486-rumprun-netbsdelf-gcc
$ echo $(pwd) /Users/ben/Repos/rumpkernel-hw/rumprun $ export PATH="${PATH}:$(pwd)/rumprun/bin" 2. Built a toy application $ mkdir -p test $ cd test $ cat > helloer.c << EOF .... $ cc -o helloer helloer.c $ i486-rumprun-netbsdelf-gcc -o helloer-rumprun helloer.c $ rumprun-bake hw_generic helloer-rumprun.bin helloer-rumprun 3. Ran the toy application $ su $ rumprun qemu -i helloer-rumprun.bin In another terminal, check its running $ xl list
Name ID Mem ...etc Domain-0 0 7583 rumprun-hellor-rumprun.bin 1 64
The app takes over the terminal its run from, so from another terminal... $ xl destroy rumprun-hellor-rumprun.bin ==================================== ADAPTED THE QUICK START TUTORIAL FOR XEN PARAVIRTUALIZATION 0. Started a new terminal to get an environment with a clean PATH 1. Built the rumprun platform $ mkdir -p ~/Repos/rumprun-xen $ cd ~/Repos/rumprun-xen $ git clone http://repo.rumpkernel.org/rumprun $ cd rumprun $ git submodule update --init $ CC=cc ./build-rr.sh xen #<<<< Took note of this output needed later...
toolchain tuple: i486-rumprun-netbsdelf cc wrapper: i486-rumprun-netbsdelf-gcc
$ echo $(pwd) /Users/ben/Repos/rumpkernel-xen/rumprun $ export PATH="${PATH}:$(pwd)/rumprun/bin" 2. Built a toy application $ mkdir -p test $ cd test $ cat > helloer.c << EOF .... $ cc -o helloer helloer.c $ i486-rumprun-netbsdelf-gcc -o helloer-rumprun helloer.c $ rumprun-bake xen_pv helloer-rumprun.bin helloer-rumprun 3. Ran toy application $ cd .. $ su $ export PATH="${PATH}:`pwd`/rumprun/bin $ cd test $ rumprun xen -i helloer-rumprun.bin $ xl list
Name ID Mem ...etc Domain-0 0 7583 rumprun-hellor-rumprun.bin 1 64
This took over that terminal, so from another terminal... $ xl destroy rumprun-hellor-rumprun.bin ==================================== TRIED IT USING CMAKE $ cat > CMakeLists.txt << EOF see this link for text... https://github.com/rumpkernel/wiki/wiki/Tutorial:-Building-Rumprun-Unikernel... $ mkdir host_bld $ cd host_bld $ cmake -G "Unix Makefiles" .. $ make $ ./helloer $ cd .. $ mkdir rumprun_bld $ cd rumprun_bld Now with the given cmake line I got an error message: Could not find toolchain file: ../../rumprun/rumprun/share/x86_64-rumprun-netbsd-toolchain.cmake $ find ../.. -name x86_64-rumprun-netbsd-toolchain.cmake found it, so that the following worked... $ cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=../../rumprun/rumprun-x86_64/share/x86_64-rumprun-netbsd-toolchain.cmake .. $ make $ rumprun-bake xen_pv helloer.bin helloer $ rumprun-bake hw_generic helloer.bin helloer $ rumprun xen -i helloer.bin Next I'll report on initial steps to get Cog working on Xen as a Rump unikernel... cheers -ben
Ben, great to hear you're working on this! I'm watching with bated breath! BTW, you might find that Docker is easier and more general; I believe there's no Xen for ARM for example, and ARM is increasingly interesting with up coming extremely cheap 64-bit versions of Pi and PINE. Good luck!! On Fri, Dec 11, 2015 at 10:27 AM, Ben Coman <btc@openinworld.com> wrote:
I got curious about following up these two threads making Cog a unikernel...
Cog in the cloud http://forum.world.st/Cog-in-the-cloud-td4796514.html
Hosting the Pharo VM on MirageOS http://forum.world.st/Hosting-the-Pharo-VM-on-MirageOS-td4799424.html#none
One path forward is to use rumpkernel, which "provides a means to run existing POSIX applications as unikernels on Xen"
https://blog.xenproject.org/2015/08/06/on-rump-kernels-and-the-rumprun-unike...
Deploying real-world software today as unikernels on Xen with Rumprun
http://events.linuxfoundation.org/sites/events/files/slides/xdps15-talk-fina...
So I thought I would try the "Building Rumprun Unikernels" tutorial here...
https://github.com/rumpkernel/wiki/wiki/Tutorial:-Building-Rumprun-Unikernel... which I report here in case it was interest. The tutorial is based on QEMU (and I think I saw a comment somewhere that Cog doesn't run so well on QEMU and needs Bochs), so after that I adapted it from QEMU to Xen.
64-BIT DEBIAN 8 JESSIE & XEN
So first I installed Debian 8 Jessie 64-bit on my new laptop $ uname -a
...x86_64
Then installed Xen and after a reboot could choose it from the boot menu. $ apt-get update $ apt get install xen-system-amd64 libxen-dev
REBOOTED
Tested manual selection from menu to boot into Xen dom0. Checked Xen was operating with... $ xl info
32-BIT DEBIAN 8 JESSIE & XEN
However later I hit some complications determining whether the rump unikernel was being compiled 32-bit or 64-bit in order to mix in with 32-bit Pharo, so to simplify things I did a complete reinstall of my system as 32-bit. $ uname -a
...686-pae
Then installed Xen again Xen only has a 64-bit hypervisor, but this was not immediately available under 32-bit OS. I can't exactly remember but I think I just needed to add multiarch before it would find the Xen package. $ apt-get update $ apt-get install multiarch $ apt get install xen-system-amd64 libxen-dev Otherwise it may have needed something like "dpkg --add-architecture ..."
REBOOTED
First test, manually selected to boot into Xen dom0, after which as root the following checked Xen was operating... $ xl info
Then set it to automatically boot into Xen... $ dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen $ update-grub REBOOTED
==================================== DID THE QUICK START TUTORIAL USING GENERIC HW ON QEMU
0. Installed support tools and libs
$ apt-get install build-essential git-core cmake
1. Built the rumprun platform
$ mkdir -p ~/Repos/rumprun-hw $ cd ~/Repos/rumprun-hw $ git clone http://repo.rumpkernel.org/rumprun $ cd rumprun $ git submodule update --init $ CC=cc ./build-rr.sh hw Took note of this output needed later...
toolchain tuple: i486-rumprun-netbsdelf cc wrapper: i486-rumprun-netbsdelf-gcc
$ echo $(pwd) /Users/ben/Repos/rumpkernel-hw/rumprun $ export PATH="${PATH}:$(pwd)/rumprun/bin"
2. Built a toy application
$ mkdir -p test $ cd test $ cat > helloer.c << EOF .... $ cc -o helloer helloer.c $ i486-rumprun-netbsdelf-gcc -o helloer-rumprun helloer.c $ rumprun-bake hw_generic helloer-rumprun.bin helloer-rumprun
3. Ran the toy application
$ su
$ rumprun qemu -i helloer-rumprun.bin
In another terminal, check its running $ xl list
Name ID Mem ...etc Domain-0 0 7583 rumprun-hellor-rumprun.bin 1 64
The app takes over the terminal its run from, so from another terminal... $ xl destroy rumprun-hellor-rumprun.bin
==================================== ADAPTED THE QUICK START TUTORIAL FOR XEN PARAVIRTUALIZATION
0. Started a new terminal to get an environment with a clean PATH
1. Built the rumprun platform
$ mkdir -p ~/Repos/rumprun-xen $ cd ~/Repos/rumprun-xen $ git clone http://repo.rumpkernel.org/rumprun $ cd rumprun $ git submodule update --init $ CC=cc ./build-rr.sh xen #<<<< Took note of this output needed later...
toolchain tuple: i486-rumprun-netbsdelf cc wrapper: i486-rumprun-netbsdelf-gcc
$ echo $(pwd) /Users/ben/Repos/rumpkernel-xen/rumprun $ export PATH="${PATH}:$(pwd)/rumprun/bin"
2. Built a toy application
$ mkdir -p test $ cd test $ cat > helloer.c << EOF .... $ cc -o helloer helloer.c $ i486-rumprun-netbsdelf-gcc -o helloer-rumprun helloer.c $ rumprun-bake xen_pv helloer-rumprun.bin helloer-rumprun
3. Ran toy application
$ cd .. $ su $ export PATH="${PATH}:`pwd`/rumprun/bin $ cd test $ rumprun xen -i helloer-rumprun.bin $ xl list
Name ID Mem ...etc Domain-0 0 7583 rumprun-hellor-rumprun.bin 1 64
This took over that terminal, so from another terminal... $ xl destroy rumprun-hellor-rumprun.bin
==================================== TRIED IT USING CMAKE
$ cat > CMakeLists.txt << EOF see this link for text...
https://github.com/rumpkernel/wiki/wiki/Tutorial:-Building-Rumprun-Unikernel...
$ mkdir host_bld $ cd host_bld $ cmake -G "Unix Makefiles" .. $ make $ ./helloer
$ cd .. $ mkdir rumprun_bld $ cd rumprun_bld
Now with the given cmake line I got an error message: Could not find toolchain file: ../../rumprun/rumprun/share/x86_64-rumprun-netbsd-toolchain.cmake
$ find ../.. -name x86_64-rumprun-netbsd-toolchain.cmake found it, so that the following worked...
$ cmake -G "Unix Makefiles"
-DCMAKE_TOOLCHAIN_FILE=../../rumprun/rumprun-x86_64/share/x86_64-rumprun-netbsd-toolchain.cmake .. $ make $ rumprun-bake xen_pv helloer.bin helloer $ rumprun-bake hw_generic helloer.bin helloer $ rumprun xen -i helloer.bin
Next I'll report on initial steps to get Cog working on Xen as a Rump unikernel...
cheers -ben
-- _,,,^..^,,,_ best, Eliot
On 11 Dec 2015, at 20:09, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Ben,
great to hear you're working on this! I'm watching with bated breath! BTW, you might find that Docker is easier and more general; I believe there's no Xen for ARM for example, and ARM is increasingly interesting with up coming extremely cheap 64-bit versions of Pi and PINE. Good luck!!
@Eliot: There is Xen for ARMv8 (because ARM is interesting for multi-core servers) @Ben: Awesome. Will be interesting how small one can get it (with still functioning TCP/IP)
On Sat, Dec 12, 2015 at 4:38 AM, Holger Freyther <holger@freyther.de> wrote:
On 11 Dec 2015, at 20:09, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Ben,
great to hear you're working on this! I'm watching with bated breath! BTW, you might find that Docker is easier and more general; I believe there's no Xen for ARM for example, and ARM is increasingly interesting with up coming extremely cheap 64-bit versions of Pi and PINE. Good luck!!
@Eliot: There is Xen for ARMv8 (because ARM is interesting for multi-core servers)
Indeed, ARM seems a very good fit for the Xen hypervisor. http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions_white... Some technical info on ARM virtualization [1] 8:50 technical info starts (note, sound broken first 0:30) 11:55 nice overview of architecture virtualization support * RasPi 2 with Cortex-A7 would thus seem support Xen virtualization (but other revision not) * BeagleBoneBlack with Cortex-A8 would seem not to :( but maybe the next upgrade, the future comes around fast sometimes... * PINE with Cortex-A53 would support Xen virtualization 14:00 shows privilege model where perhaps something like the car drive-by-wire system runs in the "Secure world", and entertainment apps (e.g. Pharo/Squeak) run under the hypervisor. I'm not sure Docker provides provides this separation. 24:05 not that I understand a "2MB section" but it seems useful our VM fits within that 26:00 Handles 256 running VMs per CPU. 31:00 Timers [1] https://www.youtube.com/watch?v=uvh6Mu3j1oY
@Ben: Awesome. Will be interesting how small one can get it (with still functioning TCP/IP)
StackVM = 1.4MB StackVM.unikernel = 15MB Pharo.image += 30MB so without trying gives a system ~45MB + application code Alternatively pharo-minimal.image = 6.2MB giving a system ~21MB + application code. For comparison [2] discusses typically having 1GB docker images, but they can reasonably trim it to 85MB - but how much functionality do they lose (e.g. we can retain out GUI debugger). So even though raw Go apps get down to 3.5MB, I think the rump kernel puts us in the ball park. [2] https://labs.ctl.io/optimizing-docker-images/?hvid=1OW0br cheers -ben
Whoops I notice an erroneous line in that recipe... On Sat, Dec 12, 2015 at 2:27 AM, Ben Coman <btc@openinworld.com> wrote:
I got curious about following up these two threads making Cog a unikernel... So I thought I would try the "Building Rumprun Unikernels" tutorial here... https://github.com/rumpkernel/wiki/wiki/Tutorial:-Building-Rumprun-Unikernel... which I report here in case it was interest. The tutorial is based on QEMU (and I think I saw a comment somewhere that Cog doesn't run so well on QEMU and needs Bochs), so after that I adapted it from QEMU to Xen. ... $ cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=../../rumprun/rumprun-x86_64/share/x86_64-rumprun-netbsd-toolchain.cmake .. $ make $ rumprun-bake xen_pv helloer.bin helloer ERRONEOUS> $ rumprun-bake hw_generic helloer.bin helloer $ rumprun xen -i helloer.bin
cheers -ben
Here is the recipe for my first successful build and run through to an error... StackVM.unikernel: could not find any display driver This is done under 32-bit Debian 8 Jessie. Attached is StackRumpXenUnikernelConfig.st # Built the Rumprun Platform $ mkdir -p cogunikernel $ git clone http://repo.rumpkernel.org/rumprun $ cd rumprun $ git submodule update --init $ CC=cc ./build-rr.sh xen
toolchain tuple: i486-rumprun-netbsdelf cc wrapper: i486-rumprun-netbsdelf-gcc installed to "/home/ben/Repos/cogunikernel/rumprun/./rumprun"
# Generated Cog VM sources $ cd ../../cogunikernel $ git clone https://github.com/pharo-project/pharo-vm.git $ cd pharo-vm/image $ ./newImage.sh $ ./pharo-ui generator.image Tools > File Browser Filed in StackRumpXenUnikernelConfig.st From class comment, evaluated... StackRumpXenUnikernelConfig new generateSources; generate. # Built the StackVM and StackVM.unikernel $ bash build.sh # Tested them... $ cd ../results $ ./StackVM
bash: ./StackVM: cannot execute binary file: Exec format error Good. That was the expected result.
$ rumprun xen -i StackVM.unikernel
StackVM.unikernel: could not find any display driver
# Stop unikernel machine running, from another terminal... $ xl list $ xl destroy rumprun-StackVM.unikernel ================ Now I basically just poked, cut and slashed the config until it could compile and boot up the unikernel, without regard for what that may break. Making the VM *actually* work is bigger step that for now is beyond me, but I'll have a bite at it. Really my motivation in doing this after I stumbled on the rumprun tutorial, was I wanted to crack the seal on it to interest others capable :). I'd be interested if the above recipe is successful for others. Todo: * incorporate rumprun as an auto-built third party library, rather than outside, perhaps using git submodule like the rump kernel already does (if these can be nested) * determine how to deal with different rumprun toolchains (depending on which platform the target) * move the baking step from the build script to the generated CMake configuration * determine how to deal with lack of display * determine how an image file might be passed to the VM - maybe baking the VM and image into a ram disk (?) cheers -ben
There were a couple of minor log errors below, so I attach a script that does it all, tested on 32-bit Debian Jessie. cheers -ben On Sat, Dec 12, 2015 at 2:57 PM, Ben Coman <btc@openinworld.com> wrote:
Here is the recipe for my first successful build and run through to an error... StackVM.unikernel: could not find any display driver
This is done under 32-bit Debian 8 Jessie. Attached is StackRumpXenUnikernelConfig.st
# Built the Rumprun Platform
$ mkdir -p cogunikernel $ git clone http://repo.rumpkernel.org/rumprun $ cd rumprun $ git submodule update --init $ CC=cc ./build-rr.sh xen
toolchain tuple: i486-rumprun-netbsdelf cc wrapper: i486-rumprun-netbsdelf-gcc installed to "/home/ben/Repos/cogunikernel/rumprun/./rumprun"
# Generated Cog VM sources
$ cd ../../cogunikernel $ git clone https://github.com/pharo-project/pharo-vm.git $ cd pharo-vm/image $ ./newImage.sh $ ./pharo-ui generator.image Tools > File Browser Filed in StackRumpXenUnikernelConfig.st From class comment, evaluated... StackRumpXenUnikernelConfig new generateSources; generate.
# Built the StackVM and StackVM.unikernel
$ bash build.sh
# Tested them...
$ cd ../results $ ./StackVM
bash: ./StackVM: cannot execute binary file: Exec format error Good. That was the expected result.
$ rumprun xen -i StackVM.unikernel
StackVM.unikernel: could not find any display driver
# Stop unikernel machine running, from another terminal... $ xl list $ xl destroy rumprun-StackVM.unikernel
================
Now I basically just poked, cut and slashed the config until it could compile and boot up the unikernel, without regard for what that may break. Making the VM *actually* work is bigger step that for now is beyond me, but I'll have a bite at it. Really my motivation in doing this after I stumbled on the rumprun tutorial, was I wanted to crack the seal on it to interest others capable :). I'd be interested if the above recipe is successful for others.
Todo: * incorporate rumprun as an auto-built third party library, rather than outside, perhaps using git submodule like the rump kernel already does (if these can be nested) * determine how to deal with different rumprun toolchains (depending on which platform the target) * move the baking step from the build script to the generated CMake configuration * determine how to deal with lack of display * determine how an image file might be passed to the VM - maybe baking the VM and image into a ram disk (?)
cheers -ben
participants (3)
-
Ben Coman -
Eliot Miranda -
Holger Freyther