API requests are taking longer than usual due to a surge in post volume.  We're working on a longer term solution to add capacity and apologize for the inconvenience.   We'll keep you updated as we go.

Permalink | Leave a comment  »

It sucks that your users can’t access your site and you can’t sign up new users. But the problem goes beyond that. At a startup, you’re trying to build features as fast as possible. You’re iterating and launching features every single day.

So your site breaking or having other ops related issues is a distraction. It means you stop building product, and probably means you have to go learn about the gory details of MySQL, nginx, or Delayed Job.

A couple years ago when Posterous started growing like a weed, we decided we needed a sys ops person. Our architecture was complex, issues kept coming up, and we needed to plan for scale.

We looked for someone for a while, but it was hard to find a good ops lead (and it’s probably harder now). You are going to trust your baby with this person!

After doing some research, we decided to outsource ops to a 3rd party called Bitpusher. We found this to be way more effective than to add a dedicated ops engineer to our small team.

Some of the advantages of using Bitpusher vs doing Ops in house:

1. Bitpusher costs us less per month than a full time hire. Plus we don’t have to pay benefits.
2. Instead of a single person, we get an ops team. This means we have 24/7 on call support.
3. Bitpusher has a lot of experience running ops. They have seen all kinds of architectures, and have used all sorts of technologies. They not only run our boxes, but gave us guidance in how to architect for scale.
4. We had to move datacenters (from Slicehost to Rackspace) and they were able to take on this entire process. They architected the system, transferred the data over, handled monitoring and backups, and more. This was something we weren’t looking forward to doing ourselves.

If you’re interested in working with Bitpusher, let me know and I’ll gladly put you in touch. Full disclosure: we get a referral fee if we send them customers!

Permalink | Leave a comment  »

To build ubuntu physical ubuntu servers we use ubuntu preseed.

This works great but if you use a static preseed file you end up building a host that doesn’t have its hostname or static ip address set. This means that you have to manually set it afterward and we decided to automate it.

BTW it took us a while to figure out how to set a static ip in a preseed file. We blogged about it here: network-preseeding-debianubuntu-with-a-static

To do this we wrote a small sinatra app that dynamically generates the preseed file with the hostname and static ip address.

This is done by looking up the mac address of the requested host from the arp table and comparing it to a pipe delimited file that contains the mac address, what the static ip should be and its hostname.

The list is stored in a file named ip2mac.txt and was populated by a script.

The ip2mac.txt file looks like this:

172.28.0.71|a4:ba:db:35:e6:09|chi-devops11a
 172.28.0.72|78:2b:cb:03:c5:44|chi-devops11b

Instead of calling a static preseed file from the pxelinux.cfg/default file we instead make a request to the sinatra app which generates it dynamically. The line in the default file we use looks like this:

append console=tty0 console=ttyS1,115200n8 initrd=ubuntu-10.04-server-amd64-  initrd.gz auto=true priority=critical preseed/url=http://172.27.0.115:4567/lucid-preseed-noraid interface=eth0 netcfg/dhcp_timeout=60 console-setup/ask_detect=false console-setup/layoutcode=us console-keymaps-at/keymap=us locale=en_US --

When the request is made the sinatra app does the following:

*  1. looks up the mac address of the request from the apr table
*  2. compares the mac address to the matching line in ip2mac.txt
*  3. uses the ip and hostname to populate hostname and ip variables in the preseed file
*  4. returns the preseed file to the host making the request

The code:

require 'rubygems' # skip this line in Ruby 1.9
  require 'sinatra'
  require "erb"
  require 'logger'

  def log(message)
    flog = Logger.new('foo.log')
    flog.info(message)
  end

  def lookup_mac(mac)
    rr = Array.new
    hostfile = File.open("ip2mac.txt","r")
    hostfile.readline
    hostfile.each do |line|
      list_ip,list_mac,name = line.split('|')
      if mac.match(list_mac)
    rr.push(list_ip)
      end
    end
  return rr[0]
  end

  def get_mac_address()
    ip =  @env['REMOTE_ADDR']
    cmd = "arp -n " + ip.chomp + " | grep -v Address | awk '{print \$3}'"
    mac  = `#{cmd}`
   return mac
  end

  def rev_lookup(ip)
    cmd = "host " + ip + " | awk '{print \$5}'"
    hostname = `#{cmd}`
fqdn = hostname.chop.chop
return fqdn
  end

  get '/lucid-preseed-noraid' do
    mac = get_mac_address()
    log(mac)
    ips = lookup_mac(mac)
    log(ips)
    fqdns = rev_lookup(ips)
    @ip = ips
    @fqdn = fqdns
    log(fqdns)
    erb :lucid_preseed_noraid
  end

  get '/lucid-preseed-nosrv' do
    mac = get_mac_address()
    log(mac)
    ips = lookup_mac(mac)
    log(ips)
    fqdns = rev_lookup(ips)
    @ip = ips
    @fqdn = fqdns
    log(fqdns)
    erb :lucid_preseed_nosrv
  end

  get '/' do
    "ops11"
  end

To start the sinatra app just run the following:

ruby preseeder.rb

Permalink | Leave a comment  »

I had a hard time figuring this out and there seems to be lots of conflicting information out there so I thought I'd write down my thoughts about this immediately after I figured it out.

I was trying to get a fully automated network install of Ubuntu 10.04 working with a static ip address that gets set up in the preseed file.  My entire preseed was working but the static ip address was not.  The example preseed has the following entries:

# If you prefer to configure the network manually, uncomment this line and

# the static network configuration below.

#d-i netcfg/disable_dhcp boolean true



# If you want the preconfiguration file to work on systems both with and

# without a dhcp server, uncomment these lines and the static network

# configuration below.

#d-i netcfg/dhcp_failed note

#d-i netcfg/dhcp_options select Configure network manually



# Static network configuration.

#d-i netcfg/get_nameservers string 192.168.1.1

#d-i netcfg/get_ipaddress string 192.168.1.42

#d-i netcfg/get_netmask string 255.255.255.0

#d-i netcfg/get_gateway string 192.168.1.1

#d-i netcfg/confirm_static boolean true

Great...so I should just be able to uncomment the disable_dhcp and the Static network configuration entries and it should work right? WRONG. I went through various permutations and could not get it to work. Finally, I broke down and decided to RTFM and found this:

http://d-i.alioth.debian.org/manual/en.amd64/apbs04.html#preseed-network

It reads:

“Although preseeding the network configuration is normally not possible when using network preseeding (using preseed/url”), you can use the following hack to work around that, for example if you'd like to set a static address for the network interface. The hack is to force the network configuration to run again after the preconfiguration file has been loaded by creating a “preseed/run” script containing the following commands:

killall.sh; netcfg

So....I added the following to my preseed.cfg file

d-i preseed/early_command string /bin/killall.sh; /bin/netcfg

and boom the static network config works. Yay!

Permalink | Leave a comment  »

Man, it's been a while since we've updated this Space. Well, no more! I'm here to tell you a little bit about the technologies behind the Posterous Spaces redesign.

As you may have noticed, Posterous has taken on a new name, and a new look. To go along with those cosmetic changes, we've rewritten our app from the ground up to take advantage of cutting edge technologies. Using these technologies has allowed our team to work at a feverish pace to deliver you a Posterous that is faster, more engaging, and more fun!

At the core of our new stack is our very own Posterous API. This has effectively made us the biggest consumer of our own API. That's pretty nifty, but I'm not going to talk about the API today. 

To interact with the API, we've used some awesome front-end technologies. Among all the awesome stuff we've been able to work with while creating Spaces, the most notable are Backbone, CoffeeScript, Haml.js, Sass, and Compass. Today I will touch on our use of Backbone.

Backbone provides some basic structure to a large JavaScript codebase. This has allowed us to create readable, and most importantly reusable, classes that separate functionality from presentation, which is a constant struggle in front-end programming.

Lest this turn into a primer on MVC 101, I'll just outline how we're using Backbone classes in our application:

Models & Collections: These serve as an interface to our API. For each model we want to interact with in the front-end—for example, a post—we create a subclass of Posterous.Model (a subclass itself of Backbone.Model). If we want to deal with lists of a particular model, as we do with lists of posts, we must also create a subclass of Posterous.Collection (a subclass of Backbone.Collection). With both a Posterous.Model and Posterous.Collection, we now have a link between the front-end and our RESTful API.

Routers: For those familiar with Rails development, a Backbone.Router is very similar to your routes.rb file. For each URL on Posterous Spaces, a router fires and tells our app to render a view (or sometimes two views in the case of multi-column layouts).

Views: The meat of our business logic occurs here. The term "View" is a bit misleading to us; we tend to use Backbone views in a manner similar to UIViewControllers in the iOS world. Views observe events, and fire responses. Views also render templates that we have built in Haml.js. 

A typical page in Posterous Spaces is actually a tree of views and subviews, each observing behavior within its outermost DOM elements. For example: when you click on the "Reader" tab, we are actually instantiating a ReaderListView, which in turn contains many PostListItemView instances. Within the PostListItemView, we instantiate a LikeButtonView, among other things.

I'll leave it at that, for now. I know this is just a birds-eye view of our architecture, so please feel free to ask any questions you may have about our use of Backbone (or other front-end technologies) in the comments.

We're hiring!

If you're interested in using cutting-edge technologies to build user interfaces that delight millions of people, definitely check out our open job listings page!

Permalink | Leave a comment  »

We like t-shirts.  And we love people who build on the Posterous API.  So we've decided to give the first 50 in-person attendees on July 16th the official Posterous Hack Day shirt.

Don't forget to tell us you are coming

Permalink | Leave a comment  »

Fresh off the launch of our new API, we're hosting a Hack Day on Saturday July 16th to provide anyone using the Posterous API with direct access to our development team.

Who: Any developer interested in using the Posterous API to build something cool.  So far, we know of mobile apps and a few web services built on top of Posterous. Whether you need ideas on what to build, are looking to team up with someone else or are already working on an app, the Posterous dev team will be here to help.
 
What:  API overview sessions, office hours for anyone with questions and end of day demos.  Plenty of food, red bull and beer.  After we're done, we'll take everyone out for drinks.   
 
When: Saturday July 16th 10am - 6pm PDT
 
Where: Posterous HQ at 2973 16th Street in San Francisco.   Also available via IRC ( #posterous-dev on freenode).

Why: Learn the best ways to use our technology stack. You'll meet other Rails experts. You could win a prize.

How:  Tell us that you are coming.  Give  us your API feature requests in the comments so we can add them before the 16th.  Then just show up with a laptop, your brain and your appetite.

July 12th update: The first 50 in-person attendees will receive the official Posterous Hack Day shirt.  

Permalink | Leave a comment  »

Today, we're happy to announce a new API that allows third-party developers to access the full Posterous technology stack.

The new API gives developers unprecedented access to methods and actions that were formerly available only to the core Posterous engineering team including the ability to create sites, add users for those sites and assigning  custom themes for each user.   Additionally, we've added API methods for retrieving and manipulating data around sites, users, posts, comments, and a number of other Posterous data types.

Aside from just adding new endpoints, we've also designed the API to be super easy to use. The new API is RESTful and presents a clean and concise set of URLs whose intent is easy to parse and understand. Moreover, we've designed the API site to be a powerful developer tool for working with the new API. Not only does this site document every single available method, it also allows developers to interface directly with the API from their browsers. Using this new tool, developers can experiment by dynamically changing the parameters and inspecting the response.

The use cases for the new API are impressive - whether you are empowering your users to be editors like Pulse did or distributing your content in real-time like Turner Broadcasting did for March Madness, our API can power it.  Another great example is Oxfam, who are using the Posteorus API to to drive awareness and participation in their recently announced campaign to grow a better future.   Anyone visiting the Oxfam grow site will soon be able to sign up and create a blog, hosted on the grow.gd domain (e.g. chris.grow.gd) with a custom theme developed by Obox.  All of the Grow blogs will contain an embedded grow widget to educate consumers on the growing food crisis and solicit their ideas for creating a different future.   And, of course, anyone signing up through Oxfam will have a full Posterous account and access to all our feature.

We can't wait to see what you do with it and welcome your feedback on how we can make it better. Start by checking out our new API site.

Permalink | Leave a comment  »

I am reposting this from my personal blog because the solution I described is in use at Posterous, and I just think it was a really interesting problem to solve.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>example</title>
  <style>
    body { margin: 30px;}
    article { position: relative; }
    .accel { -webkit-perspective: 100; width: 100px; height: 100px; }
    .inner { -webkit-transform-style: preserve-3d; -webkit-transform: rotateY(30deg); width: 100px; height: 100px; background: blue; }
  </style> 
</head>
<body>
  <article>
    <h1>Anti-aliased</h1>
    <div class="accel">
      <div class="inner">
        Hardware Accelerated
      </div>
    </div>
  </article>
  <article>
    <h1>Not anti-aliased</h1>
    <div class="accel">
      <div class="inner">
        Hardware Accelerated
      </div>
    </div>
  </article>
</body>
</html>

To be fair, the bottom heading *was* anti-aliased, but it wasn't sub-pixel anti-aliased, and the anti-aliasing still left the text looking jagged.

The solution(s)

My old foe, position:relative, was the culprit (you sneaky, sneaky property). Due to other elements on the page, the article element had position:relative. Removing this property got rid of the aliasing problem! I'm not sure why this was the case, but it worked for me.

Another solution is to explicitly set -webkit-font-smoothing:antialiased. The downside to this is your anti-aliasing will not match the rest of the page (since the rest of the page is sub-pixel anti-alised). I prefer sub-pixel anti-aliasing, but that's just me.

Anyway, I know this is a pretty obscure problem to have, but I'm sure there's at least one of you out there scratching your head about it.

N.B.: It looks like the latest Webkit nightly fixes this problem, so look forward to better looking text.

We're Hiring

If problems like this interest you, or your share my hatred for anti-aliasing bugs, Posterous is hiring UI Engineers (among other positions).

Permalink | Leave a comment  »

Cache performance is essential to site performance, but most folks don’t understand their cache at a deep enough level to make proper engineering decisions. Tools like SimCache can help by predicting cache performance ahead of potentially costly ops decisions.

Introduction

Modern web applications depend heavily on caching to maintain site performance and reduce loads on their primary databases. However, in many cases, caching strategies are deployed in an ad hoc fashion, without much understanding of how underlying usage patterns affect cache performance. In practice, developers tend to spin up a cache (typically Memcache) and continue adding capacity until site performance is “good enough.” However, without deeper understanding, capacity planning and performance tuning will become harder as traffic grows or usage patterns change.

Posterous was no exception; early in our history, we began to use Memcache heavily to quell the increasing load on our MySQL servers, sizing our Memcache cluster with simple heuristics. However, as we began to use Memcache in different ways and as our traffic grew, the cache began to act erratically, leading to site performance issues, despite rapidly increasing the size of our Memcache cluster. We realized that understanding how our cache performed given our observed usage patterns was essential for appropriately sizing our cache. To do so, we developed SimCache, a tool for predicting cache performance based on observed usage patterns, and used it to plan the second version of our cache, based on Redis.

Background

As a consumer-oriented blogging platform, Posterous is an extremely read-heavy app. Moreover, our usage patterns are extremely “long-tail”; at any given moment, we’ll serve thousands of requests for a heavily-visited site like the Gap’s consumer facing blog but just a few for Mrs. Henry’s sixth grade class blog.

To serve our normal stream of requests, we had been using a fairly large Memcache cluster to store formatted blog posts. However, cache performance began to act erratically, with wild swings in the speed of some requests that we couldn’t really understand. Moreover, the cache was being asked to serve a growing number of requests:

Understanding that this was unacceptable, I began working closely with Chris Burnett, another engineer at Posterous, to assess the degradation in cache performance.

Assessing the Situation

The importance of proper logging and measurement cannot be overstated when assessing the performance of a given caching strategy. Without collecting statistics on your cache performance, you’re essentially blind, with no understanding of how well your cache is working, or how it could be improved.

For a typical key => value cache, collecting statistics is pretty easy to implement. Anytime a key is accessed, simply log whether or not the cache request resulted in a hit or a miss:

Feb 28 01:14:54 hit!  key = posts/1432
Feb 28 01:14:54 hit!  key = posts/2442
Feb 28 01:14:55 miss! key = posts/2970
Feb 28 01:14:55 hit!  key = posts/6917
Feb 28 01:14:57 miss! key = posts/9363
Feb 28 01:14:57 hit!  key = posts/2969

Such simple data can reveal a wealth of insights. Most important is the cache’s miss rate: how frequently do we need to regenerate data? It is the miss rate that ultimately impacts site performance. Using such data, we were shocked to discover that we were caching a lot less than we thought, and that our cache actually behaved quite erratically, with a greater than 2x difference between peak and trough miss rates (1 = baseline):

Using SimCache to Choose a Caching Strategy

Given our initial assessment, it was clear that we would need to increase the size of our cache. But by how much? Could we expect much improvement if we increased the cache size by a third? What about doubling the cache? Would that sufficiently improve site performance? To answer these questions, I wrote a tool called SimCache which would replay our observed cache access patterns against a simulated cache of a given size, measuring how cache size would affect cache miss rates and other important metrics of caching performance. Using SimCache, we tested how cache performance varied if we increased our existing cache size (red) by:

• 33% (green)
• 66% (blue)
• 100% (magenta):

The data indicated that our cache was too small by a factor of almost 2x. Moreover, the undersized cache was resonsible for the wild swings in miss rate. Keys were evicted from the cache far too soon; as the cache size was steadily increased, the variation in miss rate went down dramatically, leading to better consistency in hit rates from our cache.

Using the simulation results, we increased our cache to the appropriate size. Of course, it is important to collect statistics afterwards to verify if the change had its intended effect. In our case, the results were pretty good. At time=0, the newer cache was inserted, resulting in a spike in miss rate. However, as the larger cache began the fill, the measured cache performance (green points) matched the predicted cache performance (blue line) very well:

Conclusion

Using the data from SimCache allowed us to understand why our cache performance was degraded and how to improve it. Moroever, by predicting the required cache size ahead of time, we avoided costly “ops iteration” —– i.e., we did not have to add servers, wait to see if site performance improved, add more cache, rinse and repeat. Instead we were able to size our cache appropriatey from the beginning.

Interested in working on problems like this? We’re hiring.

Thanks to J. Hui, C. Burnett, R. Pearson, D. Meredith, and G. Tan for reading and commenting on different drafts of this post.

Permalink | Leave a comment  »

While building out our API I have at times needed to validate the JSON that we are returning from endpoints or as object representations in the to_json method of our models. Typically I would just parse the returned JSON string and test the resulting value like so.

# contrived rspec but you get the idea
before do
  @post         = Post.new(:title => "Yup, I'm a post")
  @parsed_json  = JSON.parse(@post.to_json)
end

it "should include :title in it's JSON representation" do
  @parsed_json['title'].should == @post.title
end

That works fine and all but once you start validating a model this becomes terribly repetitive and a chore to maintain. After adding a quick patch to Object and a bit of yack shaving, I decided to give it gem cred and publish it.

Enter Pathy

Pathy adds a few convenience methods that make validating the JSON version of an object stupid easy.

Example

require 'pathy'

Object.pathy! # add the methods to all Objects

@json = %[ 
  {
    "string"  : "barr",
    "number"  : 1,
    "array"   : [1,2,3],
    "hash"    : {"one":{"two" : 2}}
  }
]

@obj = JSON.parse(@json)

puts @obj.at_json_path("number")
=> 1
puts @obj.at_json_path("array.1")
=> 2
puts @obj.at_json_path("hash.one.two")
=> 2

Rspec Matcher

it "should work as rspec matcher" do
  @obj.should have_json_path "hash.one"
end

Installation

gem install pathy

Source

https://github.com/twoism/pathy

Permalink | Leave a comment  »

Permalink | Leave a comment  »

With a big thanks to all our beta testers, we are proud to announce Posterous API 2.0.  All the features from the previous version have been upgraded and ported to a RESTful interface.

Getting Started

Our new API requires that you authenticate with HTTP Basic Authentication and provide an API Token with each request. More information on this can be found at http://apidocs.posterous.com.

While most endpoints do require authentication. There are some public endpoints that you can use right away.

Try this:

http://posterous.com/api/2/users/33789/sites/twoism/posts/public?page=1

Want JSONP?

http://posterous.com/api/2/users/33789/sites/twoism/posts/public?callback=someAwesomeMethod

There are plenty more features to discover. So check out the docs at http://apidocs.posterous.com.

Need Examples?

Jarrett Gossett has written a TextMate plugin for creating Markdown posts using the 2.0 API. You can check it out here:

http://jarrettgossett.com/textmate-posterous

Or grab the source on Github:

https://github.com/jarrettgossett/Posterous.tmbundle

Questions? Let us know at help@posterous.com.

Permalink | Leave a comment  »

Posterous serves an enormous amount of data from our servers every day. As our site grows, we have needed to think about new and interesting ways to improve performance. So, we spent the last couple months trying to improve performance any way we could. We stripped out much of the inline Javascript that our theming engine generated, we started using asset bundling and compression, and we audited the site for inefficient database queries. 

Finally, we decided to add full page caching to Posterous blogs. This has resulted in one of our largest performance boosts to date. How we accomplished full page caching on Posterous is the subject of today's post.

Enter Varnish

Varnish, often called an "HTTP Accelerator" or a "Reverse Proxy", is the mechanism we chose to speed up Posterous. Until Varnish, we only relied on fragment caching to save precious CPU cycles. Now that Varnish is in place, we are caching entire pages. This takes a huge amount of load off of our application and database servers, and results in noticeable speed increases for our users.

To be precise: our tests have shown that pages served out of Varnish see a ~67% speed improvement in total page load time. 

Configuration

For those not familiar with Varnish, it is configured using a file known as a "VCL (Varnish Configuration Language)" file. VCL resembles nginx config files, and is divided into subroutines (e.g. vcl_hash, vcl_recv, vcl_pipe). 

You can learn more about VCL here.

We have included the actual VCL file we are using at the bottom of this post.

Dynamic content

While implementing a full page cache, we had quite a few challenges. The first and most important one was how we were going to render dynamic content on pages that are statically cached. For example, in the upper right corner of Posterous sites, there are elements that are unique to each user (like the list of their Posterous sites, their name, etc.). Since we only cache one version of a page, we cannot include this personalized information in the page that gets cached.

We addressed this by gathering information about the static page being served, and made a secondary AJAX call to our servers to account for a user's logged-in state, post view counts, and other dynamic content. Since this AJAX call is significantly less processor-intensive, and it is only made after the page finishes rendering, users see a significant increase in performance.

To make our lives a little easier, we also removed all inline Javascript generated by theme elements. We used HTML5's new data- attributes to add information to HTML elements directly so they can be more efficiently interpreted by Javascript. 

Private posts on blog index pages

Another challenge we faced was allowing site owners and contributors to view private posts within their blog list pages. Since we want to store only one version of a page in the cache, our only alternative was to disable caching if a user is looking at one of his/her own sites. VCL alone didn't have any mechanisms for accomplishing this, so we turned to a really powerful feature of VCL: the ability to embed C code directly in the configuration file.

This snippet did the trick:

C{
  char *host = VRT_GetHdr(sp, HDR_REQ, "\005Host:");
  char *cookie = VRT_GetHdr(sp, HDR_REQ, "\007Cookie:");
  char* result = NULL;
  if (cookie == NULL) {
    cookie = "";
  }
  if (host == NULL) {
    host = "";
  }
  result = strstr(cookie, host);
  if (result != NULL) {
    VRT_SetHdr(sp, HDR_REQ, "\013X-No-Cache:", "YES", vrt_magic_string_end);
  }
}C

if (req.http.X-No-Cache ~ "YES") {
  return(pass);
}

Essentially, we write to a users cookies the list of all sites they own. If the currently-viewed hostname (the current site) matches this cookie, we simply bypass the cache.

We did it this way because even if a malicious user spoofed this cookie, the worst that could happen is that they would see the non-cached version of the page, and since the malicious user isn't actually logged in as the site owner, they won't see any of the private posts.

Lacquer

On the back-end, we are using a gem called Lacquer. The gem was originally developed by Russ Smith (original gem). Lacquer communicates with the Varnish administration port to deal with the purging of stale pages. It also makes it easy to instruct Varnish that an outbound page should (or should not) be cached.

We found there were several limitations and a few bugs with the original gem, so we forked it and added some enhancements, including the ability to purge to multiple Varnish servers in a performant way. You can check out our forked version here.

Wrapping up

At Posterous, we are always looking for awesome ways to improve our service. Varnish is just one of the many enhancements we have added, and we will be writing about other ones very soon.

As always, we love feedback! If you have any suggestions for us, or questions about Varnish, please don't hesitate to ask.

If you found this post interesting, know that Posterous is hiring Infrastructure Engineers, Front-end Engineers, and more.

Reference

Here is the Posterous VCL file, in its full glory:

#-e This is a basic VCL configuration file for varnish.  See the vcl(7)
#man page for details on VCL syntax and semantics.
#
#Default backend definition.  Set this to point to your content
#server.
#

C{
  #include <string.h> 
  #include <stdlib.h> 
  #include <stdio.h>
}C

backend default {
.host = "127.0.0.1";
.port = "8282";
.max_connections = 2000;
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}

sub vcl_hash {
  ### these 2 entries are the default ones used for vcl. Below we add our own.
  set req.hash += req.url;
  set req.hash += req.http.host;

  # This will make sure that the mobile version of our site gets cached under a different hash
  if (req.http.cookie ~ "mobile_view=true") {
    set req.hash += "mobile";
  }
  
  if (req.http.user-agent ~ "(?i)palm|blackberry|nokia|phone|midp|mobi|symbian|chtml|ericsson|minimo|audiovox|motorola|samsung|telit|upg1|windows ce|ucweb|astel|plucker|x320|x240|j2me|sgh|portable|sprint|docomo|kddi|softbank|android|mmp|pdxgw|netfront|xiino|vodafone|portalmmm|sagem|mot-|sie-|ipod|up\\.b|webos|amoi|novarra|cdm|alcatel|pocket|iphone|mobileexplorer|mobile" && !(req.http.user-agent ~ "(?i)ipad") && !(req.http.cookie ~ "full_site=true")) {
    set req.hash += "mobile";
  }
  
  return(hash);
}

#
# Handling of requests that are received from clients.
# First decide whether or not to lookup data in the cache.
#
sub vcl_recv {  
  # Pipe requests that are non-RFC2616 or CONNECT which is weird.
  if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
    return(pipe);
  }

  # Pass requests that are not GET or HEAD
  if (req.request != "GET" && req.request != "HEAD") {
    return(pass);
  }

  # Pass requests for blog pages greater than page 3
  if (req.url ~ "page=([4-9]|[1-9][0-9]+)$") {
    return(pass);
  }

  # Never cache private posts
  if (req.url ~ "\/private\/") {
    return(pass);
  }
  
  # Don't cache the result of a redirect
  if (req.http.Referer ~ "jumpto" || req.http.Origin ~ "poster") {
    return(pass);
  }
  
  # Since we don't site owners and contributors to view the 
  # cached version of their site, we match a special cookie we set
  # with the current host. This assures that site owners see
  # private posts, while other users do not.
  C{
    char *host = VRT_GetHdr(sp, HDR_REQ, "\005Host:");
    char *cookie = VRT_GetHdr(sp, HDR_REQ, "\007Cookie:");
    char* result = NULL;
    if (cookie == NULL) {
      cookie = "";
    }
    if (host == NULL) {
      host = "";
    }
    result = strstr(cookie, host);
    if (result != NULL) {
      VRT_SetHdr(sp, HDR_REQ, "\013X-No-Cache:", "YES", vrt_magic_string_end);
    }
  }C
  
  if (req.http.X-No-Cache ~ "YES") {
    return(pass);
  }

  #
  # Everything below here should be cached
  #

  # Handle compression correctly. Varnish treats headers literally, not
  # semantically. So it is very well possible that there are cache misses
  # because the headers sent by different browsers aren't the same.
  # @see: http://varnish.projects.linpro.no/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.http.Accept-Encoding ~ "gzip") {
      # if the browser supports it, we'll use gzip
      set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate") {
      # next, try deflate if it is supported
      set req.http.Accept-Encoding = "deflate";
    } else {
      # unknown algorithm. Probably junk, remove it
      remove req.http.Accept-Encoding;
    }
  }

  # Clear cookie and authorization headers, set grace time, lookup in the cache
  #unset req.http.Cookie;
  #unset req.http.Authorization;
  set req.grace = 1s;
  return(lookup);
}

#
# Called when entering pipe mode
#  
sub vcl_pipe {
  # If we don't set the Connection: close header, any following
  # requests from the client will also be piped through and
  # left untouched by varnish. We don't want that.
  set req.http.connection = "close";
  return(pipe);
}


#
# Called when the requested object has been retrieved from the
# backend, or the request to the backend has failed
#
sub vcl_fetch {
  # Comments are now fetched via ESI.
  esi;
  
  # Do not cache the object if the backend application does not want us to.
  if (beresp.http.Cache-Control ~ "(no-cache|no-store|private|must-revalidate)") {
    return(pass);
  }

  # Do not cache the object if the status is not in the 200s
  if (beresp.status >= 300) {
    # Remove the Set-Cookie header
    #remove beresp.http.Set-Cookie;
    return(pass);
  }

  #
  # Everything below here should be cached
  #

  # Don't cache the comments ESI
  if (req.url ~ "\/posts\/comments") {
    set beresp.ttl = 0s;
  }

  # Remove the Set-Cookie header
  remove beresp.http.Set-Cookie;

  # Set the grace time
  set beresp.grace = 1s;

  # Static assets aren't served out of Varnish just yet, but when they are, this will
  # make sure the browser caches them for a long time.
  if (req.url ~ "\.(css|js|jpg|jpeg|gif|ico|png)\??\d*$") {
    /* Remove Expires from backend, it's not long enough */
    unset beresp.http.expires;

    /* Set the clients TTL on this object */
    set beresp.http.cache-control = "public, max-age=31536000";

    /* marker for vcl_deliver to reset Age: */
    set beresp.http.magicmarker = "1";
  } else {
    set beresp.http.Cache-Control = "private, max-age=0, must-revalidate";
    set beresp.http.Pragma = "no-cache";
  }

  # return(deliver); the object
  return(deliver);
}

sub vcl_deliver {
  if (resp.http.magicmarker) {
    /* Remove the magic marker */
    unset resp.http.magicmarker;

    /* By definition we have a fresh object */
    set resp.http.age = "0";
  }   

  # Add a header to indicate a cache HIT/MISS
  if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }
}

Permalink | Leave a comment  »

Say you have a large bucket, 10 pounds of sand and 3 pounds of rocks. If you put the sand in first, you won’t have space for the rocks. But if you put the rocks in first, you’ll be able to fit the sand in the gaps between the rocks. That’s similar to how the typical load on a web server behaves for different kinds of requests.

• You’ve got some requests that are fast, sometimes 5ms or less. They’re simple redirects. They’re lightweight. You can plow through them fast. That’s the sand.

• You’ve got some slow requests — those are the rocks. Those hit your databases, or are infrequently used API requests.

If you try to put them together in the wrong order, you’ll get a mess. What should be fast ends up slow. What being slow ends up extra slow.

That’s why you should use haproxy to separate them out. HAProxy is a software load balancer that gives great reporting and fine-grained control over what requests should go to which servers, and how many connections to allow at any given time (minconn / maxconn).

This is an invaluable feather in your cap if you have to run a large production website. Here’s what our haproxy looks like. We have three types of requests: normal Rails requests, getfile requests (the sand — fast S3 redirects) and RSS. Every so often we get inundated by bots who request RSS feeds — and by separating our traffic out, it prevents these bots from overwhelming the rest of the site.

We increase the number of connections (60) that getfile can do to the backend because these requests are absurdly fast and they’re the sand. They should be able to get through at all times. Before we added this separate setting for these requests, they’d get backlogged behind the huge rails requests. It costs nothing for a rails request (200ms) to wait for a 5ms getfile redirect, but the reverse is not true.

Here’s our haproxy config (irrelevant details omitted) that outlines how we put separate haproxy queues to work.

global
   daemon
   pidfile /var/run/haproxy.pid
   user haproxy
   group haproxy

 backend getfile
   server app01  172.32.1.1:80 minconn 60 maxconn 120 check inter 10s rise 1 fall 3 weight 1
   server app02  172.32.1.2:80 minconn 60 maxconn 120 check inter 10s rise 1 fall 3 weight 1
   server app03  172.32.1.3:80 minconn 60 maxconn 120 check inter 10s rise 1 fall 3 weight 1
   server app04  172.32.1.4:80 minconn 60 maxconn 120 check inter 10s rise 1 fall 3 weight 1
   server app05  172.32.1.5:80 minconn 60 maxconn 120 check inter 10s rise 1 fall 3 weight 1
   server app06  172.32.1.6:80 minconn 60 maxconn 120 check inter 10s rise 1 fall 3 weight 1
   server app07  172.32.1.7:80 minconn 60 maxconn 120 check inter 10s rise 1 fall 3 weight 1

 backend rss
   server app08  172.32.1.8:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1

 backend posterous_default
   server app01  172.32.1.1:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1
   server app02  172.32.1.2:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1
   server app03  172.32.1.3:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1
   server app04  172.32.1.4:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1
   server app05  172.32.1.5:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1
   server app06  172.32.1.6:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1
   server app07  172.32.1.7:80 minconn 20 maxconn 40 check inter 10s rise 1 fall 3 weight 1

 frontend posterous 0.0.0.0:8282
   acl getfile_path          path_beg /getfile
   acl rss_path              path_beg /rss

   use_backend getfile if getfile_path
   use_backend rss if rss_path
   default_backend posterous_default

As an added bonus, HAProxy can even be used for rate limiting and fighting denial of service attacks which we’ll be incorporating soon into our configuration. Check back here later to see more about how it works for us in production.

Permalink | Leave a comment  »

Hi everybody! Adam Singer here. Over the coming weeks and months, the rest of the Posterous team and I will be posting to this blog to give you some insight into what powers Posterous. We'll be showing off some cool stuff we've been working on, announce open source projects, and provide some useful tips for those power users out there.

The first installment of the Posterous Technology blog is all about buttons. 

Introduction

Have you ever looked at a button and thought: "This button looks like crap!" This is the kind of thing that keeps me up at night. That's why I made some snazzy buttons that I've dubbed "fancy buttons." Fancy buttons are already in use in our web editor, and we will be adding these buttons throughout the site very soon. Why are these buttons fancy? Well they're fancy for these reasons:

• Made with pure CSS3. Not a single image was used in the making of these buttons.
• Scalable. Just change the font size, and the button scales.
• Colors. Change the background color on the <span> tag and the shine comes with it.
• Compatible. They look usably satisfactory in IE7 and IE8.

The Code

Before we start, here is what the final product looks like (standard, hover, and click states, respectively):

You can see that these buttons have a nice gloss, as well as some drop shadows for a nice 3D effect. The click state also an inset shadow. To achieve these effects, I needed a small truckload of CSS:

a.fancy_button, button.fancy_button { 
  background: #555; 
  background: -webkit-gradient(linear, left top, left bottom, from(rgba(0,0,0,0.4)), to(rgba(77,77,77,0.4)));  
  background: -moz-linear-gradient(top, rgba(0,0,0,0.4), rgba(77,77,77,0.4)); 
  float: left; 
  padding: 4px; 
  text-decoration: none; 
  outline: 0; 
  border-width: 1px; 
  border-style: solid;
  width: auto;
  overflow: visible;
  border-color: transparent transparent #666 transparent; 
  border-color: transparent transparent rgba(202,202,202,0.27) transparent;
  -webkit-border-radius: 12px; -moz-border-radius: 12px; border-radius: 12px;
  -webkit-background-clip: padding-box;
}

a.fancy_button span, button.fancy_button span { 
  display: block; 
  float: left;
  letter-spacing: -1px; 
  border-width: 1px; 
  border-style: solid; 
  border-color: #ccc #444 #111 #444; 
  border-color: rgba(255,255,255,0.7) rgba(0,0,0,0.5) rgba(0,0,0,0.7) rgba(0,0,0,0.5); 
  font: bold 21px/1em Arial;
  color: white; 
  padding: 0.48em 2em; 
  cursor: pointer; 
  text-shadow: rgba(0,0,0,0.45) 0 -1px 0; 
  -webkit-box-shadow: rgba(0,0,0,0.75) 0px 0px 3px; -moz-box-shadow: rgba(0,0,0,0.75) 0px 0px 3px; box-shadow: rgba(0,0,0,0.75) 0px 0px 3px;
  -webkit-border-radius: 7px; -moz-border-radius: 7px; border-radius: 7px;
  background: transparent -webkit-gradient(linear, left top, left bottom, from(rgba(255,255,255,0.6)), color-stop(0.5, rgba(255,255,255,0.15)), color-stop(0.5, rgba(255,255,255,0.01)), to(transparent)); 
  background: transparent -moz-linear-gradient(top, rgba(255,255,255,0.6), rgba(255,255,255,0.15) 50%, rgba(255,255,255,0.01) 50%, transparent);
  filter: progid:DXImageTransform.Microsoft.Gradient(GradientType=0, StartColorstr='#80FFFFFF', EndColorstr='#00FFFFFF'); 
  -webkit-background-clip: padding-box;
}

a.fancy_button:hover span, button.fancy_button:hover span {  
  border-top-color: rgba(255,255,255,0.65); 
  background: -webkit-gradient(linear, left top, left bottom, from(rgba(220,220,220,0.6)), color-stop(0.5, rgba(100,100,100,0.2)), color-stop(0.5, rgba(0,0,0,0.21)), to(rgba(0, 0, 0, 0.20))); 
  background: -moz-linear-gradient(top, rgba(220,220,220,0.6), rgba(100,100,100,0.2) 50%, rgba(0,0,0,0.21) 50%, rgba(0, 0, 0, 0.20));
  filter: progid:DXImageTransform.Microsoft.gradient(GradientType=0, StartColorStr='#99dcdcdc', EndColorStr='#33000000'); 
  -webkit-background-clip: padding-box;
}

a.fancy_button:active span, button.fancy_button:active span {  
  border-top-color: rgba(255,255,255,0.2); 
  border-left-color: rgba(0,0,0,0.4); 
  background: -webkit-gradient(linear, left top, left bottom, from(rgba(150,150,150,0.6)), color-stop(0.5, rgba(60,60,60,0.6)), color-stop(0.5, rgba(40,40,40,0.6)), to(rgba(20, 20, 20, 0.5))); 
  background: -moz-linear-gradient(top, rgba(150,150,150,0.6), rgba(60,60,60,0.6) 50%, rgba(40,40,40,0.6) 50%, rgba(20, 20, 20, 0.5));
  filter: progid:DXImageTransform.Microsoft.gradient(GradientType=0, StartColorStr='#80969696', EndColorStr='#59000000'); 
  -webkit-box-shadow: inset 0 0 18px rgba(0,0,0,0.75), rgba(0,0,0,0.75) 0px 0px 3px; -moz-box-shadow: inset 0 0 18px rgba(0,0,0,0.75), rgba(0,0,0,0.75) 0px 0px 3px; box-shadow: inset 0 0 18px rgba(0,0,0,0.75), rgba(0,0,0,0.75) 0px 0px 3px; 
  -webkit-background-clip: padding-box;
}

a.fancy_button span:active, button.fancy_button span:active { 
  border-top-color: rgba(255,255,255,0.2); 
  border-left-color: rgba(0,0,0,0.4); 
  filter: progid:DXImageTransform.Microsoft.gradient(GradientType=0, StartColorStr='#80969696', EndColorStr='#59000000'); 
  -webkit-background-clip: padding-box;
}

That CSS is definitely quite daunting, but the main reason for the size of the CSS is that many browsers implement different features of CSS, and I wanted to account for as many of them as possible. WebKit-based browsers (Safari and Chrome), Firefox, and Internet Explorer all implement different variants of background gradients and drop shadows. In addition, there were some browser quirks that I had to take care of, such as using -webkit-background-clip to clean up Safari's rendering of rounded corners.

After all this CSS is in place, the HTML is fairly straightforward:

<a class="fancy_button" href="#">
  <span style="background-color: #070;">Post</span>
</a>

I've included the background-color as an inline style for illustration purposes, but you should toss that into your external stylesheet along with the rest of the fancy button code.

Demo

Here's a quick demo of the button in action. Since Posterous already has these styles built-in, I can include it right on the page!

IE

Here is how the button looks in IE. 

Discussion

You may have noticed that I needed an <a> and a <span> to achieve this effect. I'm usually a minimalist when it comes to markup, but I needed to add the extra markup for these reasons:

• Some browsers don't support multiple backgrounds yet, and some don't support multiple shadows.
• The fancy buttons are going to be used on Posterous primarily over dark backgrounds. Because of this, I wanted to add a bit of a border around these buttons so that they look "sunk in".

Changing the font size or the background-color on the <span> will result in buttons of varying sizes and colors. Here are some neat red ones:

Bonus for theme developers: You can use these buttons right now! Simply use the HTML markup above, and the buttons should work—though you may have to override some other styles to get them to look perfect.

Conclusion

So that's it! Our first technology demo has come and gone. If you like what you saw here, or if you have any suggestions for us about making this blog better, we always appreciate your comments!

For those Twitter users out there, we will be autoposting everything from this blog to @posteroustech, so give us a follow!

PS, Posterous is HIRING!

Permalink | Leave a comment  »