Official Blog

The previous post outlined our architectural approach to a scalable website. This post outlines our approach to building such website. Our solution in a single word: Automation.

Some hundred years ago industrial production of goods such as cars was still a craft-based production. Ford introduced the moving assembly line and heavy automation. By standardizing productivity increased tenfold. Prices for a model T were cut from $760 to $360*.

A little less known is the information that Ford based his standardization on the work of Taylor. Taylor's gospel also known as "Taylorism" or Scientific Management standardizes tools, employs precise task allocation, does time studies (e.g., screw on each bolt in 15.2 seconds), introduces performance pay, and automates by repetition**.

So do we.

Setting up a server manually is time consuming and error prone. To install Debian – our operating system of choice – we make use of Fully Automated Installation – FAI to, well, automate server setup. FAI uses a collection of Perl and shell scripts for the installation process. 

^{Picture by Ford}

On the next level we use Puppet for automating system administration tasks. Instead of manually running through all those application setup tasks Puppet introduces a declarative language. This language provides a framework to express the relationships between servers, the services they run and the objects that compose those services.

At first it takes some time to get familiar with this language and a couple of early Puppet scripts were a bit frustrating. Yet it is worth it: The time it takes to write the puppet module that potentially runs on dozens of machines is actually often less than executing the administration task on a single server. Once you master the initial obstacles you cut huge chunks of time and pain out of the process of administrating a distributed server and services landscape.

The last part is code deployment. There we use Hudson. In a nutshell, Hudson (like a number of other systems) provides an easy-to-use so-called continuous integration system. The system does two jobs: First, it allows software developers to easily integrate changes to the software and second, it provides for an automated and continuous building and testing of software projects. Once done, Hudson / Puppet push the code to our development and live systems without human intervention.

This automated setup of our core infrastructure increases the productivity of our team dramatically. Instead of investing time at unnecessary places – server setup – we can focus on what matters most: An application that wins the minds and hearts of our users.

Next in the series how to build a scalable startup: Virtualization.

_______

*) Interestingly enough not all of the cost advantages were due to introduction of the assembly line and the involved productivity gains. A lot was owed to a focus on reduced material costs (PDF Document). The equivalent today is greatly reduced hardware costs (More on this and virtualization in the next post).  

**) Taylorism would quickly become the standard for businesses worldwide. Fordism and Taylorism would be hotly contested for unfairness towards workers. But that is another story.

As time goes by: Occasionally we dwell into reminiscence of our first web projects some fifteen years ago. In this long bygone era E-Commerce applications sometimes contained SQL statements, which were included in the webpage itself. 

There was no separation between the presentation layer and the logic layer of the application. In case you changed the underlying SQL table with an implied change of the SQL query you had to touch not only the database but also the above mentioned presentation layer. 

Such interlacing of applications was altogether not really a crispy idea – not even in a time when SQL injection was an unknown term.

Today we rely on the opposite: shared nothing architecture.

To take advantage of simple hardware we sliced the local.ch platform into digestible pieces. For example we set up the local.ch frontend over multiple servers (at latest count over 12) in two data centers. Each server is a standard middle of the road Dell pizza box. We actually only installed this type of server. One down: we simply switch. Heavy workload elsewhere: a simple reconfiguration and the same server is working as load balancer.

The system got way more versatile still when we started to use virtualization (more on this in a next blog post). This plus the use of open source components netted us a dramatic result: throwing small bucks at the problem solves load issues – literally. No haggling with some large account proprietary software vendor, no lengthy hardware delivery times that get you nervous.

A second piece was to take advantage of http. Obvious you say, a website gets called over http. Correct, it simply is a lovable protocol and comes with a bunch of mechanisms that let you take advantage of caching mechanisms, compression, encryption, and authorization. It is easy to understand, robust, tried and tested. And it scales.

The third item is representational state transfer, a method to build web services. Example: The frontend server talks xml with the backend servers over http employing a RESTful method. So long as you don’t change the backend API (resource representation) you can do whatever you want in that backend without breaking the system.

It is of great importance to design and implement these services / resources as small and distinct as reasonably possible. An additional plus is achieved when being able to deploy such services anywhere, i.e. on different machines, different data centers, different countries etc. and still communicate efficiently with each other. At local we had an uptime of over 99.993% with this architecture (that is correct, a downtime of less than 30 minutes over an entire year – one force majeure clause or act of God excluded*).

At Nektoon we do the same thing again with a trick: We outsource the entire hardware platform to Amazon or similar cloud computing services. Our goal: A global and quickly scalable and highly performing platform from day one.

Next in the series how to build a scalable startup: Automation.

_______

*) Yes we admit there was an act of God, a suicidal squirrel more precisely. The incident killed the electricity supply in all of Zurich Nord for a number of hours on 24 August 2008. But then again: The outage was such that the national broadcaster, half Zurich, and countless other sites and services went out of business that day.