memonicOfficial Blog

In one of your last posts we talked about the development approach of “Mobile First”, which claims that instead of developing a desktop version of a web app first developers should start with the mobile application instead.

Today, we’ll have a closer look on “Responsive Web Design”, which aims to deliver one single application that adapts its output depending on the device it’s being displayed. In other words: The underlying application stays the same, but what you’ll see, when you use it, depends on the device you’re using the application on.

Why use Responsive Web Design?
Admittedly, you’ll have to do a little more to the application to make it fit any device. However, these additional efforts will pay off since you don’t need to develop completely new applications for each platform and device. Eventually, you’ll end up with one single application – and changes in logic or features won’t need you to rework different clients for different devices – it’s a snap to ensure consistency throughout.
However, consistency in functionality does not mean consistency in how things are being delivered. Depending on the device, certain features might need more space or attention in general. With Responsive Web Design, an individual emphasis on each element of the layout can be achieved, while the functionality does not change across different devices. Content basically stays the same – just how it’s being displayed is depending on the device.

How does it work?
The ideology behind Responsive Web Designs are so called “Fluid Layouts”, which means that you don’t even have to care about the exact screen sizes of all layouts. The layout itself will adapt according to certain criterias. For those of you who are knowledgeable about HTML and CSS will know the new CSS3 media queries. With those, you can query the browser for things like screen resolution, screen size, and other things. That way, it’s pretty easy to detect if the user is sitting in front of a desktop PC or is using a mobile phone. Or a tablet. And deliver a design that displays content in an appropriate way for the respective device.

How’s that going to look like at Squirro?
As we adressed in the “Mobile First” post, we’ve got a number of design drafts for a first mobile version. Building upon that one, we then extended to the desktop interface. Which – as Responsive Web Design claims – offers the same functionalities but tied to a different set of input and output devices: high-resultion screens as well as keyboard and mouse. At Memonic, while the clients shared a big portion of the same code basis with the web platform, they were completely own applications. Even the desktop and the mobile versions differed in many ways. So did the features across all clients. Now with Responsive Web Design, backend developers are able to work on features and functions while the frontend designers do their best to assemble them in ways it makes sense for each device.

For those of you fluent in German, a few sneak-peaks of the upcoming desktop application have been revealed in the video interview posted yesterday. For all the others, here are some tiny little extractions of what we’ll come up with in the next weeks and months:

At Memonic we rely on MySQL for most of our data storage. In any relational database system the correct creation of indices is important, otherwise queries will be inefficient and slow. The problem with that is, that the indices often are forgotten, especially when updating an existing query. As a tool to detect queries without proper indices MySQL offers the slow query log. All queries that take more than a certain time are logged there.

We host our platform in Amazon’s cloud. For database we rely on their on their Relational Database Service (RDS) service. As we don’t have root access to those machines we can’t just tail the slow log to see what’s up. Instead RDS optionally writes the slow log into a special system table. From there a query can be used to retrieve the data. See the Amazon RDS FAQ about how to configure the slow log on RDS.

For automated analysis of the slow logs we like to use mk-query-digest. This excellent utility groups all logged queries together by their general type and thus allows a big-picture overview. As an example take these three queries that may have been logged:

SELECT * FROM events WHERE user_id = 'foo';
SELECT * FROM data WHERE created_at >= '2011-12-13 10:12:00';
SELECT * FROM events WHERE user_id = 'bar';

These will be grouped together by mk-query-digest as just two queries:

SELECT * FROM events WHERE user_id = '?';
SELECT * FROM data WHERE created_at >= '?';

This is accompanied with how often each query type was executed, how long it took in total, etc. This is a great way to focus any optimization effort first on the queries that are actually used a lot.

Unfortunately mk-query-digest only works with the normal MySQL slow query log format and can’t access the proprietary table that Amazon RDS keeps. To work around this, we wrote the db2log.py script which we hereby release into the public domain.

It simply dumps the current contents of the RDS slow_log table in the same format that MySQL usually uses for their slow log file. This output can then be piped into mk-query-digest to generate a report.

We ended up doing just that in a daily cron job which sends a mail to our developers.

There is one line which needs further explanation: the filter. We filter out any slow log events that were triggered by either the bi or the memonic users. The former is used for asynchronous generation of some statistics and performance isn’t required for that. The latter we use for ad-hoc queries which we don’t need to optimize.

So there you have it: an automated mechanism to analyze slow MySQL queries. From time to time when deploying a new release a new slow query may pop up. But the next day we are informed about it and can fix the issue.

As I documented before we develop with Python. Python is an interpreted programming language that doesn’t need compilation. And yet, the Jenkins build server plays a crucial role in our development process with a full 95 builds configured at the moment. How come?

In our case the build server’s main job is testing. For each component we have an extensive suite of automated tests that are run after every commit. If all tests are successful, there are additional tasks that get executed, namely:

• Create code coverage reports.
• Create Dojo bundles for JavaScript and CSS files. I documented that extensively in the previous blog post Why we love Dojo at Memonic.
• Minify the JavaScript and CSS files.
• Upload all static files (JavaScript, CSS, images) to Amazon S3. In the live system we serve those images using Amazon CloudFront.
• Create a Python egg (Python’s equivalent to Java’s jar files or Ruby’s gem files).
• Deploy that egg to our development system.

In the next few sections I’ll explain three of those areas in a bit more detail.

Automated Testing

Most of our frontend and backend services have extensive tests. Those are very important for us to ensure a quality level without having to invest a lot of resources into manual testing.

The motivator to maintain those tests is a simple mechanism: automatic deployment is done only after all tests have passed. That means that for a developer it’s actually the easier route to fix the tests than to manually package and deploy the project.

This simple mechanism ensures that all our test suites are always green (or blue if we hadn’t installed the essential greenballs plugin).

The individual test runs are very quick because we split up our whole platform into many small components. There are 95 different projects at the moment.

One test that we added just a short time ago is for translations. In our translation system there are place holders. For example we’d write “Good morning {first_name}” in the source code. The German translator will then translate that to “Guten Morgen {first_name}”. But sometimes it happens that the translator also translates the variable name so the result is “Guten Morgen {vorname}”. That of course wreaks havoc in production and has happened several time. So eventually we ended up checking for those to make sure that all placeholders are copied untouched into the target string.

Continuous Deployment

As mentioned above we do continuous deployment to the development system. On every commit the build for the corresponding project is triggered. The deployment then consists of a few simple steps.

First we build an egg. Eggs are the Python package format. Then those packages are pushed to a web server. From that web server these packages can then be installed using easy_install or pip. And that’s the final step: a script logs into the development server, updates the package and restarts the corresponding process.

This is a process that would take a bit of time if done manually. Thanks to the automation we make sure that it happens every time somebody commits a change.

In fact this has been so successful at Memonic that my claim is this: if you currently suffer from too many red builds in your project then integrate deployment into your build system. And of course only trigger the deployment process after successful builds!

Static files on Amazon CloudFront

An important build step is uploading all static resources to Amazon S3. We then reference all of those files using a CloudFront endpoint.

The host name we use for that is s.memonic.ch. We chose a different top-level domain on purpose to avoid any pollution with cookies which can especially affect caching by proxy servers.

Because we don’t serve the JavaScript files from the same domain as the HTML files, the initial setup was a bit complicated. In the case of Dojo we had to use their cross-domain builds. But after the initial setup cost we never had
any issues with that.

Conclusion

The build process we have in place is relatively complex. Of course those steps outlined above evolved over several years and didn’t all fall into place overnight.

We have achieved the main purpose which is to maximize developer performance. Every time a developer doesn’t have to worry about how to package a service or upload files to S3 is a win.

A few days ago, Chris wrote about our motivation behind the decision for the Titanium Mobile framework for our mobile application development. I’ld like to follow up on this post and extend it with some thoughts on our desktop applications. Even though the average time any device is connected to the internet increases every year and mobile access is on its way to be the prime manner of accessing data, we still want to provide our users with a full offline client for popular desktop devices. The goal was similar as for the mobile app development:

• Runs on the main operating systems (Windwos, Mac and on Linux as a bonus).
• Reuses as much as possible from the look and feel and ui patterns of the web application. Users should feel at home immediately.
• Works offline and integrates into the operating system.
• Is operating system specific where the user would expect it.

Native Apps, Adobe air and similar

Yes, this would have been an option. But hey, we are a team of passionate web developers. We love our Javascript and HTML. And the ones among us with a past in the thick client development environments were not too keen to dive back into this. Not to mention the effort to build it for and the difficulties to get (and keep) these running on all major operating systems and versions. So we were looking for a an HTML5/Javascript type of application that we could pack into an installable desktop application. And lucky enough, Appcelerator offers exactly this.

Appcelerator Titanium Desktop

Titanium Desktop is the lesser known brother of Titanium Mobile which gets all the attention at the moment. The idea of Titanium Desktop however is similar: build a desktop application based on HTML and Javascript. Use Titaniums API to reach out to OS specific functionality like filesystem access. And with this single code base Appcelerator lets you build, package and run your application on any Windows, Mac or Linux system. Great, exactly what we needed. But it gets even better: With the help of a javascript templating engine – embedded javascript was our choice – we were able to not only share the same look and feel very easily but even reuse the same code of our web implementation for the desktop application. With very little overhead we could develop new features for the web version and almost immediately apply it to the desktop application, and vice-versa, by only re-routing the calls to the data layer (done with ajax in the web case) to the local database. And there comes the second big bonus: the same way we were able to share the frontend part with our web implementation, we were able to share the backend part with the mobile devices. As we based our mobile applications on Titanium Mobile and the API differences between Mobile and Desktop are oversee-able for a large part, we kept the database layout and access objects the same for all apps. Also the client side indexing and search or the synchronization of all data is a shared component that we only developed once for all devices. Slightly oversimplified, it could be summarizes as following: next to our web version of Memonic we developd one mobile-frontend and one device-backend and created an application that runs on at least five fundamentally different device types. This all sounds great, doesn’t it? Well, there are a few drawbacks too.

Is Titanium Desktop end of life?

While the mobile product of appcelerator really takes off, the desktop product came to a little stand still. Just compare the two little commit-graphs from github.

Replacing Titanium Developer – the environment to build, test, package and deploy any Titanium Applications – with Titanium Studio after the acquisition of Aptana, did not help either. As Titanium Developer was built with Titanium Desktop and it almost seemed to be the only reason to keep Titanium Desktop going. In late spring this year, we were full of hope to get the next version of Titanium Desktop (v 1.2.) by the end of June. But now, four months later, we are still stuck with v1.1. and only a fourth release candidate of the new version.

Really tight integration into the operating system is difficult.

Deeper integration into the desktop, like easy indexing of the applications data into the OS-own search system or integrations into other applications is difficult. Titanium does not offer an out of the box solution there. You have however the possibility to package and run python, php or ruby with your application. We only use this potential currently for improved detecting of proxy settings and offering the possibility to take screenshots as part of a note.

Documentation is not up to date and support can take its time

Sometimes you will find that the documentation is not exactly accurate. But often an inspection of the corresponding javascript objects usually solves these issues quickly. If not, there is a support team that is ready to answer your questions. But don’t expect wonders, getting answers can take its time and advanced questions sometimes were no answered satisfyingly.

Building a common user experience

Earlier this year we changed our user interface drastically and created a whole new user experience – simpler, more beautiful and easier to use.
In the same go as changing the web platform, our mobile and desktop clients needed to be changed too.

Memonic allows you to have all your knowledge with you, wherever you are and what kind of device you’re using. Save your notes in the office, read them in the train and organize your inbox on your home PC.
For all of our users who are using Memonic on several devices, a key factor is to feel at home on all of them. If you just installed the app on your iPhone, you don’t want to spend time learning how to create notes. Or if you just bought an Android phone there is nothing worse than learn that the Android app is not able to assign tags as your iFriends are able to do.

Sharing the same user experience on all devices was a major plan for our new mobile and desktop apps. Still, we were aware of the fact that each platform has certain interaction patterns that all apps should implement. While iOS users are used to navigate between screens by using the toolbar on top of the screen, Android users are used to the back and menu buttons on their phones. And since the Android OS allows sharing data with other apps on the device, we definitively wanted to add support for this feature too.

Making all of that happen on iPhone, iPad, Android, Windows and Mac, built by 4 people within 2 months? A good choice of technology was needed indeed!

Deciding for a framework

Today’s smartphones are based on different technologies: Objective C is used on iOS, Google’s Java variant on Android, Java ME on Blackberry, …

If you are planning to support multiple mobile platforms you have the choice between using the platform’s core technologies or deciding for a technology that’s available on all devices: HTML and JavaScript.

Native Apps

The traditional approach for targeting platforms is writing native apps using the primary tools and languages and there are good reasons for doing that. You’re always able to use the most recent APIs, you have access to all of the phone’s hardware capabilities like orientation sensor, database, geolocation or hardware acceleration. It’s easy to follow the UI guidelines and create apps that look and behave like all the others on that platform. On the downside you need to learn different languages and maintain multiple code. Often this results in apps created by different people in maybe even different teams with different speed and roadmaps. This makes it difficult to offer the same features and behavior on all targeted platforms.

HTML5

The most common denominator on all modern devices is JavaScript. Since all smartphones are equipped with a web browser, you can create an appealing mobile website which can be accessed whenever the user is online.

HTML5 on mobile and desktop devices is undoubtedly a future safe way that will gain more importance over the next few years. However, besides the fact that implementing offline access might be difficult, your mobile website won’t be available in the iOS AppStore, the Android Market or other app stores. While some people dislike these app markets for their revenue sharing models and sometimes rigid and slow review processes, their marketing aid must not be underestimated. An app store also means an advertising platform where your app is showcased, discussed and found by your future users.

PhoneGap

Frameworks like PhoneGap are wrapping your mobile website into an application which can be distributed on all kind of app stores and downloaded from there like all the other apps. Once installed, the HTML and JavaScript code will be executed in the web browsing engine of your device, without network latency and data transmission costs. And in addition you get a uniform API for accessing the phone hardware.

Sounds great – so what are the drawbacks? Be aware that your app is still a mobile website and your users will notice a different user experience compared to native apps. Frameworks like jQTouch and Sencha Touch are doing quite a good job at creating good looking user interfaces but as of 2011 the difference is still noticeable. You will have to decide yourselves whether this matters or not. We made serious prototyping and testing and decided that we did not want to compromise on the user experience.

Appcelerator Titanium Mobile

The Titanium Mobile framework runs a hybrid approach and claims to be a “native cross platform development framework”.

Titanium Mobile is cross platform: the apps are written in JavaScript, which is interpreted on the mobile device. But is it native? Well – sort of. The whole user interface is composed by native widgets and components, which can be accessed from JavaScript over a bridge to Objective C (iOS) or Java (Android). As you are using this layer for creating the UI and reacting to user input, there is a certain performance penalty compared to really native apps but still, the Titanium Mobile based apps feel more real than their HTML5 counterparts.

A further advantage of Titanium Mobile is that the framework allows you to implement your custom native modules in Objective C or Java and using them from JavaScript. This allows you to extend the API and make use of all features of the underlying platform.

Of course, Titanium Mobile has drawbacks as well. There is a certain performance penalty, which might be less or more visible based on the complexity of your app. And then, the Appcelerator crew needs to keep their API in sync with the supported platforms – if you see the rapid evolution of iOS and Android, that’s not an easy task. Appcelerator is doing a pretty good job but so far they have not been able to release stable versions for Blackberry (currently in Beta) or Windows Phone 7. Thankfully, the module concept allows you to add missing parts or implement performance critical tasks yourself.

Comparing all the different approaches, the Titanium Mobile framework seemed to be the best mix between sharing code across platforms and creating a good user experience.

Our top 5 tips for Titanium Mobile development

1. MVC is your friend

We have created a model-view-controller framework for Titanium Mobile, which allowed us to share 100% of the model and business logic with our desktop apps. The views are easily exchangeable for certain platforms, especially tablets.

2. Separate your styles from the code

A no-brainer for web development but applies for Titanium Mobile development too. Keep all layout information far away from code and be able to adapt the layout without breaking the rest.

3. Use phone specific UI only where it makes sense – but use it

A consistent layout on all clients is important to make your users feel at home. But some things simply need to be done in a platform specific way. Example: the best way to make Android users angry is not correctly supporting the back button…

4. Optimize by creating native modules

The goal of cross platform development frameworks is that you as a coder don’t have to write native code for each platform. However, the great thing about Titanium Mobile is that this door is still open.
Implement your app in 100% JavaScript first and then optimize where needed.
We’re using native code where performance matters. This includes tasks like image scaling or document indexing.

5. Titanium Mobile is open source

Use the source code of Titanium Mobile for understanding how the framework works, even if you don’t plan to extend it yourself. You will learn how to improve your code.
We have our own fork at github where we integrate our own extensions and patches.