As we've been hacking away trying to turn Pogoapp into a reality, we've often had difficulty giving the elevator pitch for what this project is all about. To non-technical people it's tough to communicate something beyond "it's a new kind of web hosting", while many of the tech-minded are familiar with the "platform as a service" concept, but not too clear on the how's and why's and what's. So, while we're wrapping up getting this new website online (thanks RefineryCMS!), we thought we'd use our first blog post to try to explain what we're doing here.
For years the gold-standard approach to running websites on an open source platform has been LAMP, with the core component being a webserver running virtual hosts - a configuration that allows incoming requests to be mapped to different local directories containing HTML and other static assets mixed with code in script files written in a lightweight language like PHP or Perl. Generally the webserver runs a given script for each request - /search.php?query=test results in the search.php script being run by mod_php or as standalone CGI process.
A new model has been gaining traction though, based around an idea of fully-fledged web applications. Instead of Apache as the center of attention, dispatching requests to various files, modern web frameworks like Rails, Sinatra, Django, Express, etc are designed to be run as self-contained web application servers. They do their own complex, dynamic internal routing and may need to hold open client connections and stream data back over HTTP or WebSockets. The processes themselves can have significant startup times and consume anywhere from 20MB to half a gigabyte of RAM.
The old LAMP model of hosting is fundamentally broken for these new web application servers. First off, these web apps almost always depend on a long list of dependencies - specific versions of the language interpreter and libraries that the application was built and tested against, which need to be fetched and compiled on the server. In many cases there's now also additional "pre-deployment" steps, e.g. the Rails 3.x asset pipeline compilation. And the app itself often consists of multiple processes - additional web backends for high availability and/or traffic scaling and background workers for resource-intensive/time-consuming tasks.
For developers with some Linux sysadmin know-how, the cloud has been a godsend for getting quick, cheap cookie-cutter VMs booted and able to run apps, but for the majority of professional and amateur developers who want to run their own code or one of the growing number of open source apps, managing a virtual machine is still a hassle, even in the cloud. And as the dependencies, pre-deployment steps and scaling requirements add up, the complexity continues to grow.
All this points towards the need for a new model of application hosting. The new model is simple: apps and processes, not sites and servers. Ezra Zygmuntowicz at EngineYard was one of the first to articulate the idea in this presentation, and Heroku has done an excellent job laying out what they call the 12factor model. Our platform follows these ideas closely - we have a single LXC container image, and once we've built your application into a "slug" we deploy it out in an isolated, ephemeral container with an overlay filesystem. Because each container is transient, we can run large numbers of virtual machines without the normal maintenance headache that entails - underlying OS configuration/bug/security fixes only need to be applied once.
The idea is simple, but the devil's in the details. Building a system that can generically build and deploy any of a multitude of languages and frameworks, and then dynamically booting an entire virtual machine container for each process the app needs and routing incoming requests to these backend processes is tricky to get right. One of the big problems is just a lack of standardization - many app hosts have specific adapter libraries or configuration files and steps to get apps able to run on their platform, each incompatible with the next.
Surveying the different PaaS hosts, Heroku's buildpack system stood out as making the most conceptual sense. A buildpack is quite simply a set of scripts that can introspect and compile an application. Buildpacks are a clever dependency inversion - a solution to the problem of meeting the ever-expanding variety of different web applications and dependencies without bloating out or fragmenting the underlying hosting platform. Buildpacks can be written by end users and can pull in and build any required dependencies for your app, each time the app is deployed. Best of all, there's lots already out there that will work on Pogoapp with zero to minimal modification.
We've also made it our goal to eventually implement (fairly closely) the Heroku REST API along with some additional API functions used by the command line client. We've very slightly forked the Heroku command line client and named it pogo, and plan to keep our client closely tracking the upstream changes.
Continuing the vein of standardization, we've decided to use the open source Ceph/S3 storage for Pogoapp. Ceph is an exciting new distributed storage system with some pretty serious people behind it, and it includes an S3 compatibility layer, so existing apps/tools designed for storing assets in S3 should work pretty much out-of-the-box. Ceph also powers our build system - your app's compiled archive and a cache that's kept for faster re-builds are both stored in Ceph via S3 so we can scale out builds and deployments easily.
Pogoapp is still quite small (52 app containers as of now, and a dozen or so old chef-based containers we're porting over to buildpacks), but we're starting to open the platform up to new trial users, a few at a time - people who would like to kick the tires on a very rough early system and give us some detailed feedback, even though we've still got a small list of critical bugs and features that need to be finished before we expand much further. But we're also trying to get a sense if there's interest out there in using a paid beta app hosting platform.
Pogoapp is definitely not what we'd call production-ready yet, so we're going to continue hammering away at that goal. But we're a completely bootstrapped company, so we ultimately can't scale a beta up to the point where we're racking up thousands in server bills with free trial accounts. We can either stay small-scale as we continue to work out the kinks, or buy some new hardware and open it up to a larger number of beta users if there's interested developers out there who'd be willing pay at least at-cost for the server resources, and would be willing to ride some roadbumps and offer a lot of feedback.