Today, 20Min Online, (the most read news website in Switzerland) did an interview with Dorian on Memonic, its origin, the current state and our plans, including a little screencast explaining our application (German).
Update (26 January 2010): Primer in 20 Minutes print today - "Zürcher Start-up organisiert Web-Infos" (pdf).
Yesterday (January 23rd) our company just turned 1. Happy birthday Memonic!
For the occasion we went up the Uetliberg for a nice breakfast with partner and kids.
More pictures in our Flickr Group.
“When our first parents were driven out of Paradise, Adam is believed to have re-marked to Eve: ‘My dear, we live in an age of transition’.” (Inge 1929)
We started out a couple of months ago with this series on scalable startups with a reflection on organizations as lump of butter in a sea of milk. Organizations are social arrangements. Hence our next focus was the core constituent of any organization: Its people. A well-crafted strategy is worthless without a committed crew. The focus on people is core to what we do. Nonetheless, an organization is no end in itself. The fuel that drives the engine forward is customer needs. We cherish users like nothing else: They come first.
We argued forcefully for smaller organizations. Instead of large anonymous corporate monsters we made a case for smaller and nimbler units. We also outlined how to maximize involvement and reduce the useless part of management: Let people decide. To put these decisions into action we rely on a project methodology – Scrum – that finely aligns overall goals, immediate tasks, people and resources.
A particular issue we dealt with is how to get work done. Instead of word wilderness and excel wasteland we draw on the advantages of Wikis for documentation and a powerful task tracking tool. Both tool sets are completely web-based and accessible from everywhere.
In terms of actual setup and operation of an online platform we learned a lesson or two from our previous engagement at local.ch. A must is a shared nothing architecture plus automated building of the software application. To ensure consistency and reliability we made testing a daily priority.
The virtualization options available today are essential for a startup. We can cut down very considerably on our asset investment in servers and other bulky infrastructure items.
The outlined setup of a scalable startup is very much the realization of what Peter Drucker sketched out in his essay on The coming of the new organization over twenty years ago:
“The typical large business 20 years hence will have fewer than half the levels of management of its counterpart today, and no more than a third the managers. In its structure, and in its management problems and concerns, it will bear little resemblance to the typical manufacturing company, circa 1950, which our textbooks still consider the norm. Instead it is far more likely to resemble organizations that neither the practicing manager nor the management scholar pays much attention to today: the hospital, the university, the symphony orchestra. For like them, the typical business will be knowledge-based, an organization composed largely of specialists. … For this reason, it will be what I call an information-based organization.”
A little over two decades later, the startling development of the Internet has driven much of the transformation from an industry-based economy to an information-based economy. Multilevel hierarchies have given way to clusters of business units coordinated by market mechanisms rather than by layers of middle management. The large enterprise structures designed for the business environment of the 1950s and 1960s – firms that typically sought economies of scale through central planning and control mechanisms – are quite likely not the most adroit forms at meeting the current competitive environment that demands both efficiency and effectiveness. Companies track opportunities and resources on a global scale. In an attempt to maximize return on assets, firms perform only those functions for which they possess or can develop expert skills. Activities that can be performed quicker, more effectively or at lower cost by others are outsourced. An intricate network of formal and informal relations ties the firm together. The momentum is paced by information technology.
Back in 1978 Hiltz and Turoff summarized this phenomenon as follows:
“We will become the Network Nation, exchanging vast amounts of both information and socio-emotional communications with colleagues, friends, and ‘strangers’ who share similar interests. ... we will become a ‘global village.’ ... An individual will, literally, be able to work, shop, or be educated by or with persons anywhere in the nation or in the world.”
True.
_______
Series Overview
_______
Drucker P.F., “The Coming of the New Organization”, Harvard Business Review, January - February 1988, pp. 45-53.
Hiltz S.R. and Turoff M., The Network Nation: Human Communication via Computer, London: Addison-Wesley, 1978.
Inge W.R., Dean of St.Paul’s London, Assessments and Anticipations, London, 1929.
To strive for perfection but fall two steps short, is all but human. I remember with mixed emotions the German classes back at school. Quite often we had to do essays on this or that topic. Quite often I got good marks for essay’s idea and its composition just to loose a couple of points for (surely involuntary) grammatical inadequacies.
Writing software is pretty similar: The most elegant lines of code will be undone by mundane errors.
Error recognized, error fixed, or so they say. Yet research shows that on average for every six errors fixed, one new error is introduced. Kind of never catch up game. Similar to the anecdote of Achilles, who is not able to catch up with the turtle.
A scalable website is a complex beast. Even in it’s simplest form it is a web of thousands of lines of code, several servers and services.
Humans perform mostly well in conditions of relative uncertainty. Computers not. Whereas we are able to adapt to situations, computers fail abysmally at tasks that are not clearly specified and where there is no error free code to execute. An unrecognized set of characters in a get request and the receiving web service goes havoc. A minor glitch in a sub-routine and the transaction fails.
Thus the importance of testing.
Testing comes in a number of flavors. Every developer agrees that code should be tested. However, testing on an individual code level is not enough. Especially hard it gets if you have to take over legacy code with little test coverage. My colleague Patrice suggests two simple rules:
Here at Nektoon our goal is to cover as much code with tests as possible and sensible. We don't reach 100% coverage but accept one or two percent less. Going all the way to 100% will drive you crazy, as there is a lot of code, which hasn't been written with testability in mind. Testing such code while possible would be a lot more work.
A certain level of error is an acceptable outcome for a system like ours. A simple solution is some redundancy to be built in.
We consider testing integral to our work. Before automatically building the application (also referred to as continuous integration – see previous post on automation) any piece of code must be thoroughly tested.
The equation is simple: What will the additional time investment in testing yield in terms of benefits? The answer: A lot! Simpler code, more reliability, less onerous bug fixes, and less disappointed users because our service was down.
We believe in testing. Over on his personal blog, Patrice commenced a series specifically dedicated to testing. Over the next few weeks he will write about selection and setup of a testing framework, unit and functional testing, Javascript testing, continuous integration, test-driven development, testability and measuring coverage.
Next in the series how to build a scalable startup: Conclusion.
Quite unnoticed by the casual Internet user a silent revolution took place in the server centers around the globe.
I fondly remember that cold Christmas 2000 when our company delta consulting (later namics), a Web consultancy, moved to a new location. We had to move the server room, too. No easy undertaking. We had dedicated servers for a number of clients, our internal development and staging servers, filers, mail servers etc.
But we had a noble goal: finally some space again! Our old and crammed server room couldn’t hold more servers. Our new room seemed large and spacious. We continued to grow and soon we were once again mired in finding additional space.
Recently I returned to visit my former workplace just to find three quarters of the room converted into a meeting room.
How come?
How does Virtualization work? Companies like Parallels, VM Ware, Citrix and others provide software allowing to emulate or “virtualize” the hardware resources of a machine. The ubiquitous x86 type computer infrastructure was designed to run a single operating system and a single application. This leaves most machines vastly underutilized. Virtualization lets you run multiple virtual machines on a single physical machine, sharing the resources of that single computer across multiple environments.
Sure you loose a bit of performance because the virtualization software mimics the actual physical hardware. With ever faster machines and ever more memory this is no roadblock to virtualization. There are however two big gains: Less space and less initial investment.
At Nektoon we have just a single server for the entire development platform. Roughly twenty or so virtual servers are running on top of it. Especially welcome for our startup: Instead of twenty servers we bought one. Albeit a bit more expensive, our initial investment was still a fraction of a normal setup. Nekton.com
Our live infrastructure is hosted on Amazon Web Services. This has one key advantage: We do not need to invest time and money in setting up a live infrastructure but instead pay-as-we-go at Amazon. With the automation in place (see previous post) we have contained the sheer volume of repetitive setup tasks. Plus we let others solve these tasks.
The distributed infrastructure also allows you to get closer to your customers. In case of physical servers we would have chosen close-by server centers for purely practical reasons (Physical reboot of a machine). Yet somebody from the US will have quite longer response times when accessing servers at some place in Switzerland instead of someplace in the US itself. The distributed infrastructure of Amazon provides a neat framework for this case: Distributed storage and processing power across the globe. So we will store the notes of our American customers in America and vice-versa with notes in Europe.
Next to the technical benefits the impact on the business plan is remarkable: We can run a startup on a shoestring for quite some time.
Next in the series how to build a scalable startup: Testing.
The previous post outlined our architectural approach to a scalable website. This post outlines our approach to building such website. Our solution in a single word: Automation.
Some hundred years ago industrial production of goods such as cars was still a craft-based production. Ford introduced the moving assembly line and heavy automation. By standardizing productivity increased tenfold. Prices for a model T were cut from $760 to $360*.
A little less known is the information that Ford based his standardization on the work of Taylor. Taylor's gospel also known as "Taylorism" or Scientific Management standardizes tools, employs precise task allocation, does time studies (e.g., screw on each bolt in 15.2 seconds), introduces performance pay, and automates by repetition**.
So do we.
Setting up a server manually is time consuming and error prone. To install Debian – our operating system of choice – we make use of Fully Automated Installation – FAI to, well, automate server setup. FAI uses a collection of Perl and shell scripts for the installation process.
Picture by Ford
On the next level we use Puppet for automating system administration tasks. Instead of manually running through all those application setup tasks Puppet introduces a declarative language. This language provides a framework to express the relationships between servers, the services they run and the objects that compose those services.
At first it takes some time to get familiar with this language and a couple of early Puppet scripts were a bit frustrating. Yet it is worth it: The time it takes to write the puppet module that potentially runs on dozens of machines is actually often less than executing the administration task on a single server. Once you master the initial obstacles you cut huge chunks of time and pain out of the process of administrating a distributed server and services landscape.
The last part is code deployment. There we use Hudson. In a nutshell, Hudson (like a number of other systems) provides an easy-to-use so-called continuous integration system. The system does two jobs: First, it allows software developers to easily integrate changes to the software and second, it provides for an automated and continuous building and testing of software projects. Once done, Hudson / Puppet push the code to our development and live systems without human intervention.
This automated setup of our core infrastructure increases the productivity of our team dramatically. Instead of investing time at unnecessary places – server setup – we can focus on what matters most: An application that wins the minds and hearts of our users.
Next in the series how to build a scalable startup: Virtualization.
_______
*) Interestingly enough not all of the cost advantages were due to introduction of the assembly line and the involved productivity gains. A lot was owed to a focus on reduced material costs (PDF Document). The equivalent today is greatly reduced hardware costs (More on this and virtualization in the next post).
**) Taylorism would quickly become the standard for businesses worldwide. Fordism and Taylorism would be hotly contested for unfairness towards workers. But that is another story.
As time goes by: Occasionally we dwell into reminiscence of our first web projects some fifteen years ago. In this long bygone era E-Commerce applications sometimes contained SQL statements, which were included in the webpage itself.
There was no separation between the presentation layer and the logic layer of the application. In case you changed the underlying SQL table with an implied change of the SQL query you had to touch not only the database but also the above mentioned presentation layer.
Such interlacing of applications was altogether not really a crispy idea – not even in a time when SQL injection was an unknown term.
Today we rely on the opposite: shared nothing architecture.
|
The main idea is a distributed computing architecture with no application interlacing. Each node (hardware or software component) is self sufficient and redundantly available. There is no single bottleneck that may break the system when failing. On the other side you can scale to near infinity as Google has shown. To make this happen in our case three items stand out: standard hardware, http, and representational state transfer (REST). |
|
To take advantage of simple hardware we sliced the local.ch platform into digestible pieces. For example we set up the local.ch frontend over multiple servers (at latest count over 12) in two data centers. Each server is a standard middle of the road Dell pizza box. We actually only installed this type of server. One down: we simply switch. Heavy workload elsewhere: a simple reconfiguration and the same server is working as load balancer.
The system got way more versatile still when we started to use virtualization (more on this in a next blog post). This plus the use of open source components netted us a dramatic result: throwing small bucks at the problem solves load issues – literally. No haggling with some large account proprietary software vendor, no lengthy hardware delivery times that get you nervous.
A second piece was to take advantage of http. Obvious you say, a website gets called over http. Correct, it simply is a lovable protocol and comes with a bunch of mechanisms that let you take advantage of caching mechanisms, compression, encryption, and authorization. It is easy to understand, robust, tried and tested. And it scales.
The third item is representational state transfer, a method to build web services. Example: The frontend server talks xml with the backend servers over http employing a RESTful method. So long as you don’t change the backend API (resource representation) you can do whatever you want in that backend without breaking the system.
It is of great importance to design and implement these services / resources as small and distinct as reasonably possible. An additional plus is achieved when being able to deploy such services anywhere, i.e. on different machines, different data centers, different countries etc. and still communicate efficiently with each other. At local we had an uptime of over 99.993% with this architecture (that is correct, a downtime of less than 30 minutes over an entire year – one force majeure clause or act of God excluded*).
At Nektoon we do the same thing again with a trick: We outsource the entire hardware platform to Amazon or similar cloud computing services. Our goal: A global and quickly scalable and highly performing platform from day one.
Next in the series how to build a scalable startup: Automation.
_______
*) Yes we admit there was an act of God, a suicidal squirrel more precisely. The incident killed the electricity supply in all of Zurich Nord for a number of hours on 24 August 2008. But then again: The outage was such that the national broadcaster, half Zurich, and countless other sites and services went out of business that day.
In the contemporary world Michael Foucault, a French philosopher, thinks that organizations are places of controlled violence. Another tradition embodied by the American philosopher John Dewey, views organizations as edifying forums.
Whatever we think of organizations, either as incubators of new cultures, or as defined by the cultural milieu of our societies at large, most, including the cited philosophers, think they are artificial and therefore open to to being remade.
One area how we on purpose remade our organizational setup was the development process and issue tracking. Scrum, our preferred project methodology splits our entire project into stories and tasks assigned to two-week sprints.
Years ago we would handle project tasks with huge Excel sheets. Not a very amusing job, especially if you had to track a larger team. The burden was even greater on the team having to track their tasks.
During these celebratory days of 40 years Moon landing, I always wonder how the folks did it at the time without Wikis and without digital task tracking. 40 years later the problem is solved – digital distributed task tracking does the job. At Nektoon we use Jira.
Jira is a sister product of the Confluence, our Wiki software written by Atlassian. We simply got used to their product after our happy contact with their Wiki software. And additional benefit is, that both are tightly integrated.
Every issue, task, bug, improvement is recorded in our Jira. On a two week basis we assign these issues to individual members of the team. Once you start to work on an issue you register your work: We comment on the progress of our work and clock the time spent on each task.
With this we accumulate a body of data that allows us:
Furthermore the system allows:
For us the relevant part is the transparency of our issue tracking. We are able to manage hundreds of issues, assign accountability and ensure that things get done.
Next in the series how to build a scalable startup: Shared nothing architecture.
Most theories of editing are based on the idea of a single author. Stillinger, an American professor of English literature, calls this the Myth of the solitary genius, and points out that most “works can and frequently do have multiple authors, sometimes with divided and even conflicting intentions among them”.
Friedell, an Austrian historian, is more candid and reckons the discussion about plagiarism to be the most unnecessary controversy in history. Nature does not allow dishonest business, he writes, and goes on to unveil that Alexander borrowed from Philip, Augustine borrowed from Paul, Schiller borrowed from Shakespeare, the latter was influenced by Plutarch. Friedell concludes that dark times like the Middle Ages are always the result of insufficient intellectual larceny. He would have loved the Internet. Goethe, a German poet, rightly questions the sense of asking a well-fed man about all the ox, sheep and pigs he ate to give him strength.
Back in 2005 when we started with local.ch, we had to conceive and build the platform within just six months between October 2005 and March 2006. We were a team of nearly 40 people from six different companies. First, we split the platform into different building blocks (see previous post). Each sub-team had an organizational and conceptual challenge to document and a requirement to update and get updated by all the other sub-teams.
Most of you know what now happens next: An avalanche of E-Mail with zillions of attached specification documents, presentations, Excel sheets and more. After just a few weeks, nobody would ever know again which was the latest presentation, which was the agreed specification, which was the most up-to-date business plan Excel, etc. A nightmare!
What matters at launch – and this differs sharply to Roger Federer’s single achievements – is a team effort focused on a well-executed solution. It is not important who wrote what, but what we wrote. A solid specification is a collaborative effort. Thus we were looking for a highly collaborative platform for specification and documentation: A Wiki.
Wikipedia is well known to all. Within no time it surpassed the venerable Encyclopedia Britannica. Collaborative editing proofed a wining concept. A little less known is that the same technique may be applied to company internal collaboration, too. At least back in 2005 it was a novel concept.
A Wiki enables documents to be written collaboratively using a Web browser. Its main advantages are:
Basically it’s massively distributable collaboration: People located at different places can work on the same document.
I don’t want to hide a number of disadvantages I came across spanning four years of excessive Wiki usage:
The adaption curve was not a linear road. For quite some users to work with a Wiki was a laborious task at first. You edit online and all colleagues immediately see what you do was an odd notion. Plus we realized that the MediaWiki [ http://en.wikipedia.org/wiki/MediaWiki ] syntax proved quite difficult for some users: Some clever clogs simply attached word documents to Wiki pages.
Over Christmas 2005 Toni and myself evaluated a number of alternatives. We wanted a Wiki with a rich text editor for people to avoid the Wiki markup. We choose Confluence from at the time little known Australian startup called Atlassian. A choice we stuck to ever since. As a matter of fact, a couple of friends from liip, namics, Zeix, Blogwerk, PubliGroupe et al. came to learn from us and subsequently introduced and enhanced Wikis in their companies (in case you want a live demo contact any of us).
No surprise then that pretty much the first thing we set up when starting to think about our new company back in 2007 was a Wiki. We could asynchronously work over weekends and late at night and yet still see at the very same time what the others were doing.
Today we collaboratively work in our Wiki on specifications, technical documentation, business issues, project management, bookkeeping, people affairs and we open up our Wiki for our cooperation partners. What counts is not the single genius but the joint effort.
Goethe would have approved.
Next in the series how to build a scalable startup: Process & control.
_______
Stillinger J., “Multiple Authorship and the Myth of Solitary Genius”, Oxford: Oxford University Press, 1991.
Friedell E., "Die Kulturgeschichte der Neuzeit", originally published in 3 volumes, Vienna, 1927-1931, München: Beck, 1996.
Project fiascos usually have managerial consequences, and it was only a matter of time before the ripples from the great computer revolution produced a metamorphosis in the way large and complex IT projects are managed.
Luckily for us, this happened well before we were born.
That was back in a time when computers were the size of a truck. Beasts such as Univac and IBM 700s or on the other side a URAL-2 computer from the Soviet Union.
The invention of computers was closely followed by its Siamese twin: IT project failures. The history of information science is littered with such examples.
Back in the fifties and sixties line managers would perform the work required by his line organization and then throw the “ball” over the fence hoping that someone from IT development would catch it. Once thrown over the fence the line manager would wash his hands off any responsibility. In case something went wrong because the ball was no longer in his yard. The IT people claimed that the requirements were subpar and threw the ball back. However, the real problem with this approach was that the actual customer had no single contact for their questions.
This debate generated some notable books. One I like particularly is Fred Brooks’ “The Mythical Man-Month: Essays on Software Engineering”. It is based on his experiences at managing the IBM OS/360 project.
Back in the nineties a couple of people started to look over the fence to Japanese product engineering and software development techniques. One such technique is called Sashimi. Out of this technique grew Scrum.
The basic issue with the traditional waterfall models of project management is their focus on quality and low cost. The scrum inventors realized that software projects need in addition speed and flexibility. And they observed that there are people actually doing the project work and there are others with a more casual interest in the project’s outcome:
A pig and a chicken are walking down a road. The chicken looks at the pig and says, "Hey, why don't we open a restaurant?" The pig looks back at the chicken and says, "Good idea, what do you want to call it?" The chicken thinks about it and says, "Why don't we call it 'Ham and Eggs'?" "I don't think so," says the pig, "I'd be committed, but you'd only be involved." (Quelle)
The “pigs” are committed to building the software, while everyone else – “chickens” – is just interested but sort of indifferent to the outcome – they never committed themselves fully. This distinction plus the particular agile setup of the methodology are the distinctive marks of Scrum. Today a growing body of evidence suggests that agile methods greatly improve software quality and project success rates.
At local.ch we were using agile methods, building the platform in release cycles of about 4 weeks. Over time some teams started to use Scrum. At Nektoon we set out to apply Scrum fully right from the start.
How do we do it?
First we fixed late last year the strategic goals for 2009. These goals have been broken down into a roadmap for the year. Example: We fixed last year the goal of a beta release late spring. Today we’re one two-week sprint off this goal.
The next level is our product backlog. Here we write down the big tasks. We refer to them as stories. For each story we award a point on a scale from 1 to 5 as an indication of its complexity and difficulty.
Every other Monday morning we sit together and sort through this product backlog. First we prioritize and then in a second step we pick the stories for our next two-week sprint. In turn each story will be split into tasks. For each task we allocate an estimate on the number of hours required for completion. Then we go and work.
Important: We decided to do this not just for engineering stories but for business stories, too. Example: For marketing purposes we currently are producing a brief video clip. Thus we created a story “Video Clip”. This story and associated tasks are part of the current sprint.
Then there is the issue of controlling: Every morning we sit together and quickly go through three questions: What did I do yesterday, what will I do today, is there anything blocking me? To update everybody we create a status update in our Wiki (More on that in the next blog post).
At the end of each sprint we go through each story, demo the developed features or present the business related stories. This is important to get a feedback on overall progress.
You might say for a startup with just five people on its payroll, this is a bit too much. We think it isn’t. The pace and rhythm we set now for our company will be the pace and rhythm of our company in the future. We try to achieve a couple of simple things with this approach:
Next in the series how to build a scalable startup: Abolish MS Office.
| More memonic photos |
