API requests are taking longer than usual due to a surge in post volume.  We're working on a longer term solution to add capacity and apologize for the inconvenience.   We'll keep you updated as we go.

It sucks that your users can’t access your site and you can’t sign up new users. But the problem goes beyond that. At a startup, you’re trying to build features as fast as possible. You’re iterating and launching features every single day.

So your site breaking or having other ops related issues is a distraction. It means you stop building product, and probably means you have to go learn about the gory details of MySQL, nginx, or Delayed Job.

A couple years ago when Posterous started growing like a weed, we decided we needed a sys ops person. Our architecture was complex, issues kept coming up, and we needed to plan for scale.

We looked for someone for a while, but it was hard to find a good ops lead (and it’s probably harder now). You are going to trust your baby with this person!

After doing some research, we decided to outsource ops to a 3rd party called Bitpusher. We found this to be way more effective than to add a dedicated ops engineer to our small team.

Some of the advantages of using Bitpusher vs doing Ops in house:

1. Bitpusher costs us less per month than a full time hire. Plus we don’t have to pay benefits.
2. Instead of a single person, we get an ops team. This means we have 24/7 on call support.
3. Bitpusher has a lot of experience running ops. They have seen all kinds of architectures, and have used all sorts of technologies. They not only run our boxes, but gave us guidance in how to architect for scale.
4. We had to move datacenters (from Slicehost to Rackspace) and they were able to take on this entire process. They architected the system, transferred the data over, handled monitoring and backups, and more. This was something we weren’t looking forward to doing ourselves.

If you’re interested in working with Bitpusher, let me know and I’ll gladly put you in touch. Full disclosure: we get a referral fee if we send them customers!

To build ubuntu physical ubuntu servers we use ubuntu preseed.

This works great but if you use a static preseed file you end up building a host that doesn’t have its hostname or static ip address set. This means that you have to manually set it afterward and we decided to automate it.

BTW it took us a while to figure out how to set a static ip in a preseed file. We blogged about it here: network-preseeding-debianubuntu-with-a-static

To do this we wrote a small sinatra app that dynamically generates the preseed file with the hostname and static ip address.

This is done by looking up the mac address of the requested host from the arp table and comparing it to a pipe delimited file that contains the mac address, what the static ip should be and its hostname.

The list is stored in a file named ip2mac.txt and was populated by a script.

The ip2mac.txt file looks like this:

172.28.0.71|a4:ba:db:35:e6:09|chi-devops11a
 172.28.0.72|78:2b:cb:03:c5:44|chi-devops11b

Instead of calling a static preseed file from the pxelinux.cfg/default file we instead make a request to the sinatra app which generates it dynamically. The line in the default file we use looks like this:

append console=tty0 console=ttyS1,115200n8 initrd=ubuntu-10.04-server-amd64-  initrd.gz auto=true priority=critical preseed/url=http://172.27.0.115:4567/lucid-preseed-noraid interface=eth0 netcfg/dhcp_timeout=60 console-setup/ask_detect=false console-setup/layoutcode=us console-keymaps-at/keymap=us locale=en_US --

When the request is made the sinatra app does the following:

*  1. looks up the mac address of the request from the apr table
*  2. compares the mac address to the matching line in ip2mac.txt
*  3. uses the ip and hostname to populate hostname and ip variables in the preseed file
*  4. returns the preseed file to the host making the request

The code:

require 'rubygems' # skip this line in Ruby 1.9
  require 'sinatra'
  require "erb"
  require 'logger'

  def log(message)
    flog = Logger.new('foo.log')
    flog.info(message)
  end

  def lookup_mac(mac)
    rr = Array.new
    hostfile = File.open("ip2mac.txt","r")
    hostfile.readline
    hostfile.each do |line|
      list_ip,list_mac,name = line.split('|')
      if mac.match(list_mac)
    rr.push(list_ip)
      end
    end
  return rr[0]
  end

  def get_mac_address()
    ip =  @env['REMOTE_ADDR']
    cmd = "arp -n " + ip.chomp + " | grep -v Address | awk '{print \$3}'"
    mac  = `#{cmd}`
   return mac
  end

  def rev_lookup(ip)
    cmd = "host " + ip + " | awk '{print \$5}'"
    hostname = `#{cmd}`
fqdn = hostname.chop.chop
return fqdn
  end

  get '/lucid-preseed-noraid' do
    mac = get_mac_address()
    log(mac)
    ips = lookup_mac(mac)
    log(ips)
    fqdns = rev_lookup(ips)
    @ip = ips
    @fqdn = fqdns
    log(fqdns)
    erb :lucid_preseed_noraid
  end

  get '/lucid-preseed-nosrv' do
    mac = get_mac_address()
    log(mac)
    ips = lookup_mac(mac)
    log(ips)
    fqdns = rev_lookup(ips)
    @ip = ips
    @fqdn = fqdns
    log(fqdns)
    erb :lucid_preseed_nosrv
  end

  get '/' do
    "ops11"
  end

To start the sinatra app just run the following:

ruby preseeder.rb

I had a hard time figuring this out and there seems to be lots of conflicting information out there so I thought I'd write down my thoughts about this immediately after I figured it out.

I was trying to get a fully automated network install of Ubuntu 10.04 working with a static ip address that gets set up in the preseed file.  My entire preseed was working but the static ip address was not.  The example preseed has the following entries:

# If you prefer to configure the network manually, uncomment this line and

# the static network configuration below.

#d-i netcfg/disable_dhcp boolean true



# If you want the preconfiguration file to work on systems both with and

# without a dhcp server, uncomment these lines and the static network

# configuration below.

#d-i netcfg/dhcp_failed note

#d-i netcfg/dhcp_options select Configure network manually



# Static network configuration.

#d-i netcfg/get_nameservers string 192.168.1.1

#d-i netcfg/get_ipaddress string 192.168.1.42

#d-i netcfg/get_netmask string 255.255.255.0

#d-i netcfg/get_gateway string 192.168.1.1

#d-i netcfg/confirm_static boolean true

Great...so I should just be able to uncomment the disable_dhcp and the Static network configuration entries and it should work right? WRONG. I went through various permutations and could not get it to work. Finally, I broke down and decided to RTFM and found this:

http://d-i.alioth.debian.org/manual/en.amd64/apbs04.html#preseed-network

It reads:

“Although preseeding the network configuration is normally not possible when using network preseeding (using preseed/url”), you can use the following hack to work around that, for example if you'd like to set a static address for the network interface. The hack is to force the network configuration to run again after the preconfiguration file has been loaded by creating a “preseed/run” script containing the following commands:

killall.sh; netcfg

So....I added the following to my preseed.cfg file

d-i preseed/early_command string /bin/killall.sh; /bin/netcfg

and boom the static network config works. Yay!