Debian on Amazon EC2: Booting issues

Recently I've been playing around with PyPy and its new support for C extensions. I've been wanting to jump into it more, but outside of my desktop at work, I don't really have a suitable environment for hacking on stuff, especially for non-work related stuff. My linux box at home is a MythTV DVR; probably not the best place to do PyPy "translations". Clearly I need another dev box.

For a while I've been meaning to start playing around with Amazon EC2, so I figured this might be a good opportunity, plus I just found out recently they have a free service tier for the first year. Unfortunately, the default Amazon images are Fedora, and I'm more of a Debian guy, so I wanted to install a Debian machine image. Being the paranoid sort, I don't really want to install somebody else's machine image; plus it's more fun to do it myself!

I set up a normal 64-bit Linux AMI, mounted a second "target" EBS volume, and ran the ec2ubuntu script to build an AMI. (I had to make a couple of changes and run a few things by hand to get the script to work.) It downloads Debootstrap, and installs Debian to a target directory/volume, using chroot as necessary. When I was done setting up the image, I created a snapshot (using the management console), ran ec2-register to create an AMI, and launched it using the management console. The ec2-register command was:

ec2-register -a x86_64 -b /dev/sda1=<snapshot-id>:2:false -n 'Debian-testing' -d 'Debian testing 2011-05-22' --kernel <hd0 pvgrub aki id>

The biggest challenge was to get the AMI to actually boot. Many of the guides out there pre-date Amazon's use of PV-GRUB. Previously, in order to boot an EC2 image, you had to use one of Amazon's pre-made Kernels, and you had to have their kernel modules installed on your image. Now you can have Amazon boot your own custom kernel image using PV-GRUB, and in fact this is how Amazon's pre-made Linux AMIs work. Troubleshooting and debugging the boot process is particularly frustrating; all you get is a dump of the console log (visible in the AWS Management Console). In the end for your image to boot, you need a few things: grub, the correct Amazon kernel ID, and a working /boot/grub/menu.lst.

By default, Debian comes with GRUB 2. I'm not sure if this works with EC2. I installed the grub-legacy package instead, as this matched the version of PV-GRUB I saw on the console. It's possible you don't even need a version of grub installed, as long as you have a working menu.lst.

The Amazon kernel IDs are listed in the Amazon EC2 Developers Guide, but it's pretty misleading, or flat out wrong. You need to use an "hd0" PV-GRUB AKI if the root device contains your /boot/grub/menu.lst directly and and an "hd00" PV-GRUB AKI if the root device is partitioned and your menu.lst is on the first partition. So, even though I'm using an EBS-backed instance, I needed an "hd0" AKI, because my root EBS volume is not partitioned. In us-west-1, for a 64-bit OS I used AKI "aki-9ba0f1de".

Setting up the /boot/grub/menu.lst correctly was the hardest part for me. The biggest issue is that grub (or the OS?) sees the "root device" as /dev/xvda1 instead of /dev/sda1, so the "kernel" line of the menu.lst needs to say /dev/xvda1 instead of /dev/sda1.  (The console output when this was incorrect was very misleading; I was getting errors like "Gave up waiting for root device" and "Alert! /dev/sda1 does not exist.") I'm not sure why it did not occur to me sooner that I could use the Amazon Linux AMI's menu.lst as an example, but this realization proved to be helpful. Anyways, my full menu.lst is:

default 0
timeout 1

title EC2
        root (hd0)
        kernel /boot/vmlinuz-2.6.38-2-amd64 root=/dev/xvda1
        initrd /boot/initrd.img-2.6.38-2-amd64

After correcting my menu.lst, everything booted just fine. I was able to ssh in, as root. The init.d scripts added by the ec2ubuntu script copied over the correct SSH key, so logging in was easy.

I'm now the proud "owner" of a dev machine running Debian "in the cloud". My next step is to configure it, but that will have to wait for later.