How to Build and Compile a Custom Linux Kernel for EC2

I have a long running goal that I'm trying to reach with all these pv-grub for EC2 posts. That goal is to find the smallest/tightest usable node that can be created for EC2. The next step in that path requires a custom Linux kernel. What follows is how to build the latest Linux kernel so that it works on EC2 using pv-grub.

It is important to have a recent kernel since most of what is needed to get a kernel to work on EC2 is now incorporated into the source. These instructions assume the latest kernel is and I've used them with 2.6.35 as well but keep that in mind since the one patch that is required could eventually be merged in. Before getting started it may help to read over how to compile the Linux kernel normally and then my post on running CentOS 5.5 on EC2 using pv-grub.

Before you begin you will need a place to build the kernel. For these instructions I used an EC2 instance to build the kernel but you don't have to. I also installed the kernel on the same EC2 instance when I was done. The AMI I used was Amazon's EBS boot starter ( ami-b232d0db : amazon/getting-started-with-ebs-boot ).

The following steps go over building and installing the kernel in detail:

  1. Download the latest Linux kernel or the version I'm using:
  2. Configure the source to run on EC2:
    make menuconfig

    You will need to make sure the following options are set in the configuration:

    • "Processor type and features" -> "High Memory Support" -> Make sure it is set to 64GB
    • "Processor type and features" -> "PAE (Physical Address Extension) Support" -> enable
    • "Processor type and features" -> "Paravirtualized guest support" -> enable
    • "Processor type and features" -> "Paravirtualized guest support" -> "Xen guest support" -> enable
    • "Device Drivers" -> "Block devices" -> "Xen virtual block device support" -> enable either as a module or built in
    • "Device Drivers" -> "Network device support" -> "Xen network device frontend driver" -> enable either as a module or built in

    If you want you can make the device drivers modules but you have to have them so it is probably best to just compile them into the kernel itself. If you want to compare your config file with the one I used you can download mine here: kernel-

    The starter AMI I used needed ncurses development libraries and gcc installed for menuconfig to work:

    yum install ncurses-devel gcc

    If you want to know a little more about what is being enabled in this step see the "Building with domU support" section of XenParavirtOps.

  3. Apply the following patch to disable XSAVE:
    --- a/arch/x86/xen/enlighten.c	2010-08-05 20:35:13.000000000 -0400
    +++ b/arch/x86/xen/enlighten.c	2010-08-05 20:35:22.000000000 -0400
    @@ -776,6 +776,7 @@
     	cr4 &= ~X86_CR4_PGE;
     	cr4 &= ~X86_CR4_PSE;
    +	cr4 &= ~X86_CR4_OSXSAVE;
    cd /path/to/root/of/kernel/source
    patch -p1 < /tmp/kernel.patch

    Note that the above patch file has tabs in it. Make sure there is a tab before each cr4 line and the native_write_cr4 line. If you want to you can download a copy of the patch with the tabs in it to be sure.

  4. Build the kernel and install it if you are on the same machine you want to install it on. If you need help compiling the kernel refer to the kernel compile howto.

    After this step you have a kernel, modules and initrd that you can use. The remaining steps go over using it.

  5. Configure the /boot/grub/menu.lst file on the target AMI to use the new kernel, the following is an example of the contents of the file:
    default 0
    timeout 1
    title Test
         root (hd0)
         kernel /boot/vmlinuz- root=/dev/xvda1
         initrd /boot/initrd-

    Note that the root device here is /dev/xvda1 instead of /dev/sda1. This is caused by the XSAVE patch.

  6. Verify that your /etc/fstab file is correct. If your previous root device was /dev/sda1 it is going to be /dev/xvda1 now. The contents of the fstab file I used follow:
    /dev/xvda1                              /                       ext3    defaults 1 1
    /dev/mapper/swapVG-swapFS               swap                    swap    defaults 0 0
    /dev/mapper/storageVG-storageFS         /mnt                    ext3    defaults 0 0
    none                                    /dev/pts                devpts  gid=5,mode=620 0 0
    none                                    /dev/shm                tmpfs   defaults 0 0
    none                                    /proc                   proc    defaults 0 0
    none                                    /sys                    sysfs   defaults 0 0
  7. Make a snapshot of the volume and register it as an AMI:
    ec2-create-snapshot -d "Snapshot Description" volume-id
    ec2-register -n "CustomKernel" -d "Custom kernel AMI" --root-device-name /dev/sda1 -b /dev/sda1=snap-id:15:true

    Note that the devices here are /dev/sda1 and not /dev/xvda1. That is a little confusing but the AWS system doesn't see the devices in the same way your AMI will once it is booted.

  8. Start the instance. In my case using the hd0 pv-grub kernel:
    ec2-run-instances -z us-east-1a -g your-group -k your-keypair -n 1 --kernel aki ami-from-step-7

If all goes well you should be able to run dmesg and see a boot message something like the following at the top:

Reserving virtual address space above 0xf5800000
Linux version (root@domU) (gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)) #2 SMP Mon Aug 23 20:00:01 EDT 2010
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 000000006a400000 (usable)
NX (Execute Disable) protection: active

With the ability to create a custom kernel for EC2 the next step is to prune the OS itself down to the bare minimum.

Leave a Reply

Your email address will not be published.