memonicOfficial Blog

Short summary: the Memonic web clipper may be broken. If that’s the case re-install the bookmarklet.

Longer version:

For our web site we use several domain names like memonic.com or memonic.ch. Those need to be renewed yearly so we can continue to use them. Unfortunately we messed up with the renewal of one of the domain names that we use for hosting images and other resources.

How does this affect you?

• The Memonic web clipper may be broken. If that’s the case re-install the bookmarklet.
• Images on this blog are currently broken.

We are working as fast as we can on fixing this situation.

I personally apologize for this mistake.

Update Friday September 23: All things back to normal. Again apologies for the inconvenience.

Yesterday evening August 7th, Memonic was down from approximately 19h30 to 01h00 Monday morning (All times Central European Summer Time).

What happened?

A lightning strike caused a power failure in Amazon’s Dublin based server center. Amazon said that lightning struck a transformer near its data center, causing an explosion and fire that knocked out utility service and left it unable to start its generators, resulting in a total power outage.

Right now most services are back, operating normally or near normally. In the case of Memonic you should be able to use all our services normally.

Important for you: No data was lost.

Why are we hosting with Amazon?

We are hosting Memonic on Amazon Web Services’ range of cloud computing services, as do many other startups, too. Why? Amazon provides the full range of hosting services required to run a service such as Memonic: processing power, storage, content delivery and that at reasonable rates. An additional advantage for us: We can stand on the shoulder of a giant. Amazon will always have more dedicated resources (brains and chips) available then we ever can muster. And that pays: No outages in the last 12 months, except yesterday evening.

Our sincere apologies for the downtime.

Memonic is hosted on Amazon’s Web Services. Since today, Sunday August 7th, as of around 7.30pm (CET) Amazon AWS is facing unexplained network issues.

More information on our Memonic Ops Twitter channel and here on this blog.

Update 11.21pm (CET): The AWS RDS database service is still down. Next update no later than tomorrow morning 8am (CET).

Update 1.11am (CET): The site is now up again. Some performance issues remain as not all our servers have come up again.

This post has been updated on August 11, 2011, see below.
—-

On this blog we talk about the tools and services we use and love. Yet sometimes a service fails you. Also our service sometimes has a hiccup. We try to correct the mistake asap and communicate with our users – We love to do customer support – and they seem to love it. This post is about how not do customer support.

If you buy a subscription to our service – thanks if you do – you will be asked to provide your payment details. Credit cards of course but also country specific payment methods such as bank collection in Germany (Bankeinzug). It’s one of those things that you don’t want to do yourself for every country. So we had to choose a payment provider last year. We chose Adyen. Big mistake.

The issue I write here about is not the first issue we had with them but the straw that breaks the camel’s back.

Quite over a month ago I noticed that there were suddenly deposit corrections popping up in the settlement statements – Adyen calls the twice weekly reports “Settlement Batch Reports”. Nothing big I thought, so I looked at the batch overview page, the batch detail page and the detailed settlement report, found other inconsistencies and could simply not align the numbers.

For clarity I keep it to the report #113, the same actually holds true for other reports.

As you see in the overview the payout reads CHF 45.99.

Once I click on the batch number and get the detail page it says CHF 48.09.

Okay that’s two bucks to my detriment. Strange… but wait it gets sillier. Once you download the detailed report you are totally lost. It is for somebody outside Adyen completely intransparent how on earth they tabulate these aggregate numbers given what is displayed here.

No problem, I thought, drop support an email and they gladly explain to you how these numbers align. That was in June. The first mail I wrote on June 20th simply asking that someone explains the differences. To this day I have not received an answer.

Well I received lots of email responses by Kevin Lobbezzo and Gavin Stok from their support department explaining to me that I basically need to look elsewhere. As an example an email from Gavin from June 30th

> On 30 Jun 2011, at 3:49 PM, Adyen Support Department wrote:
>> Hi Dorian.
>> As per Kevin’s email this ticket is now very long and we ask that you log one ticket per issue or set of questions. This avoids the chance of confusion and frustration, and allows us to search for tickets in the future easier if required.
>>
>> If I understand correctly, you would like us to explain the correlation between a settlement report and the settlement batch overview tab. If this is true, please provide a batch number and we’ll reply in a new ticket with the explanation.
>>
>> If there are any other questions you’d like answered please let us know in a new ticket, and please feel free to use our knowledgebase at https://support.adyen.com/ as a starting point.
>>
>>
>> Kind Regards,
>> Gavin Stok
>> (Support Manager)

Sure, at this time I already had provided them the batch reports multiple times. And honestly folks, fix your support tool mess (By now the mail they keep sending me back and forth is 2.1MB big… ) Use a tool like Zendesk. That works nicely even with long threaded issues.

A few more meaningless Adyen support emails later I still do not have an answer. Probably everything is correct, let’s suppose this, but if the other party can’t explain its reports the confidence starts to erode.

By now – a full month later – I thought it’s time to escalate the issue.

At the core it’s a simple question: Would you trust your bank if it sends you a monthly statement that it can’t explain properly? I won’t. Exactly this question I asked Pieter van der Does, the CEO of Adyen, a week ago. No prizes for guessing that I haven’t received any response as of yet.

To be fair, payments ain’t easy. Credit card companies, acquirers, issuers, payment providers, and more are required for something seemingly simply as an online payment. And of course each intermediary wants to earn his living (on your costs, as effectively the consumer is paying for all this with a higher retail price).

Yet this is no excuse to provide the worst ever customer support.

—-

Update August 10, 2011

Four days after this post Christine Nitsch, Senior Account Manager, from Adyen contact me. We could resolve some of the issues. The difference to which I point in the post was due to exchange rate adjustments, that have not been properly reflected in the reporting frontend. Adyen has corrected this since. She also told me that Adyen works on a number of other improvements in the reporting and reassured me that the actual transaction accounting is 100% correct.

June first is, well the first day in the month: The current synchronization license key – we use an external solution built by Synthesis – expired tonight at midnight with the switch of the month.

Truly embarrassing. Our sincere apologies for failing you.

It doesn’t help you to say that we couldn’t detect this, as it is an external component. It’s only relevant that we fix it quickly.

So what next:

• We submitted a fix to Apple on the fast track and hope it gets approved quickly (we keep you posted)
• For the moment please use our mobile site at m.memonic.com.

Keep coming back regularly to this post for updates and/or check for more immediate updates our Memonic Operations account on Twitter.