A Popper Template for Linux Kernel Research (part 1)

TLDR: We describe how to follow the Popper convention to manage and automate experiments that modify the Linux Kernel. This first part deals with building the kernel and provisioning a VM. In part 2, we cover the experiment execution and validation of results using this template.

A typical exploration involving Linux consists of code that implements one or more new features in the kernel, and then run experiments in order to test a hypothesis. In this case, Popper is followed to manage and track the changes done to the experiment, analysis and validation of results.


In this post we will show how to hack the kernel in a DevOps oriented way by following Popper, with the goal of making it easier for others to reproduce results.

The above diagram depicts the series of steps that we will go over in some level of detail:

  1. Create a repository for your experiment (0).
  2. Fork the kernel repo and clone it into your machine (1, 2).
  3. Build new kernel packages (3, 4, 5).
  4. Provision a VM with the custom kernel (6, 7).

For 3 and 4, we will create scripts and, for illustration purposes, we will write these scripts from scratch. These scripts are part of a Popper template, so one can avoid writing them from scratch by quickly importing it to an existing Git repository (see last last section).

Creating a Repo for the Experiment

We want to create a repository that will hold the scripts, tests, analysis, results and possibly a manuscript describing all. In order to do that, we first initialize a git repository:

$ mkdir mypaper
$ cd mypaper
$ git init
Initialized empty Git repository in mypaper/.git/

$ echo "# repository on " > README.md
$ git commit -m "first commit"

For more on how to Learn Enough Git to be Dangerous, see here, here, or here.

We then create a repository at GitHub and link it to the local repository that was just created. To do this, you need an account at GitHub (see a guide here). Once this is done, the repository is linked by doing the following:

$ git remote add origin $your_repo_url

where $your_repo_url is the URL to the repository at GitHub.

Add the kernel source repo

We now will add a kernel source repository as a “sub-repository” to the mypaper repo that we just created above. To do so, we first fork the kernel source repository at https://github.com/torvalds/linux. To fork the repository, you need an account at GitHub. For more info on how to fork a repo, take a look at this guide.

Once you have a repo (we’ll assume this repo resides at https://github.com/$youruser/linux), then you need to add it to the mypaper repository as a submodule. To do so:

$ cd mypaper
$ mkdir -p experiments/kernel-experiment
$ git submodule add --depth 1 https://github.com/$youruser/linux experiments/kernel-experiment/linux
$ git commit -m "adding my kernel repository as a submodule"

The --depth 1 flag passed to the submodule add subcommand instructs git to make the submodule shallow so that the whole repo isn’t pulled in. To learn more about submodules, see here, here or here.

Build Kernel Packages

Now, we want to build the kernel in a reproducible manner. One way to do this is to use Docker. We create a experiments/kernel-experiment/docker folder and create an image with all the dependencies of our build environment. Instead of pasting the contents of this folder, you can take a look at them here. To learn more about Docker visit this page.

This container automates all the steps involved in building a kernel. To build, we do the following:

set -e -x

# build the container
docker build -t kernel-ci docker/

# build the kernel by passing the source folder
docker run --rm -ti \
  -v `pwd`/linux:/linux \

mv linux/*deb vagrant/debs/

The script above is placed in a experiments/kernel-experiment/build-kernel.sh script and becomes part of the mypaper repo.

Provision a VM with new Packages

In order to test our kernel, we’ll use a VirtualBox VM. Instead of interacting directly with the VirtualBox commands, we use the awesome Vagrant tool. We first grab a vagrant box that runs Debian Jessie. This is done by defining a Vagrantfile like the following:

Vagrant.configure("2") do |config|
  config.vm.box = "debian/jessie64"

  # Enable provisioning with a shell script.
  config.vm.provision "shell", inline: <<-SHELL
    sudo dpkg -i /vagrant/debs/*.deb

The above script can be placed in a experiments/kernel-experiment/vagrant/ folder. The output of the building process described in the previous section generates packages and puts them in this experiments/kernel-experiment/vagrant/debs folder. In order to provision the VM with these packages, we do:

set -e -x

# bring the VM up
vagrant up

# provision
vagrant reload --provision
vagrant reload

Again, we put the above script in the experiments/kernel-experiment/vagrant/provision.sh script so this becomes part of our mypaper repository. After the above executes, we have a VM running with our customized kernel. To login to the VM, we can do:

$ cd mypaper/experiments/kernel-experiment/vagrant/
$ vagrant ssh

vagrant@jessie:~$ uname -a
Linux jessie 4.9.0-ci #3 SMP Wed Jan 25 01:00:13 UTC 2017 x86_64 

To learn more about Vagrant, look at here.

Importing the Popper Template

The scripts that we created in the previous sections are part of a Popper template, so one can avoid writing them from scratch by quickly importing the template to a repository. To do so, install the Popper CLI and, assuming you have already create the mypaper repo (first subsection above), we do:

$ popper init
$ popper experiment add linux-cgroups my-linux-exp

You can ignore for now why this template is named linux-cgroups. (after reading part 2 this will make more sense). The contents of the experiments/my-linux-exp would look like the following:

$ tree experiments/my-linux-exp/
├── build-kernel.sh
├── docker
│   ├── Dockerfile
│   ├── buildkernel.sh
│   ├── config-4.6.0-0.bpo.1-amd64
│   └── kernel_config.sh
├── run.sh
└── vagrant
    └── Vagrantfile

    2 directories, 7 files

Next, we add the submodule for the Linux repository as we did in above in a linux/ folder. After this, the .build-kernel.sh command will build kernel packages in vagrant/debs. The provisioning of the VM is done in run.sh.

Written on January 23, 2017