Keep Calm And Carry On

It went like this:

Andrew: This service adds a moustache to any face! It’s awesome!

Me: I’m busy working.

A: What if we create an extension that would add a moustache to every face! It’d be awesome!

Me: I’m busy working.

A: It’d take only a few minutes anyway.

Me: Ok, whatever…

In the next 7 minutes or so I created a Chrome extension that puts a moustache on every face on every page. It took us another half an hour to test it, fix an occasional bug, draw a few icons, write a description for the Chrome gallery and pay $5 developer fee to Google since I never used my own account for extensions yet.

We tweeted about it and it went viral, to our surprise. We found out that the code doesn’t perform well on the Cedar stack of Heroku, so we had to redeploy to the old Bamboo – all while the usage levels were growing exponentially. Almost immediately it became obvious that our Face.com API key wouldn’t be enough to process all the requests and we asked to raise the limit (fortunately, they raised our limit promptly – thanks). In a matter of minutes we received an email from the American Mustache Institute, and in a matter of hours people began blogging about it (and not only in English). Our servers are still unable to process all the requests; we have to drop some of them and the tweets keep coming.

It was a good fun for a Friday afternoon pairing with the awesome @teabass!

Google Chrome, unlike Firefox, doesn’t allow to detect when the user uninstalls the extension, which is quite useful to understand user behaviour. There is a feature request on crbug.com with a discussion of this feature but it hasn’t been implemented yet.

At InvisibleHand we are really trying to understand user behaviour to improve our extension. In particular, we run A/B tests of our extension to track whether new features make the users to uninstall InvisibleHand less or more. To track the uninstallations in a fairly hacky way, we inject this piece of code as a content script into every single page:

var iFrameId = 'invisiblehand-uninstallation-notifier';
var silenceCount = 0;
var maxSilence = 3;

var timer = setInterval(function() {
  chrome.extension.sendRequest({topic: "heartbeat"}, function(reply) {
    if (reply && reply.alive)
      silenceCount = 0; // we want 3 consecutive silence events just to be sure
    else
      silenceCount++;
    if (silenceCount < maxSilence) return;
    clearInterval(timer);
    if (document.getElementById(iFrameId)) return;
    var iframe = document.createElement("iframe");
    iframe.id = iFrameId;
    iframe.src = "http://productsiframe.invisiblehand.co.uk/uninstall";
    iframe.style.display = "none";
    document.body.insertBefore(iframe, document.body.firstChild);
  })
}, 1000);

This code tries to connect to the background page every second and if it fails (when it fails, the callback is still called but the reply is undefined), we assume the extension was disabled or uninstalled, so we inject an invisible iframe into the current page to notify our server about the uninstall. It’s the server’s responsibility to count only one uninstall per user since there is a high probability that this code will be executed in every open tab. However, it’s not of utmost important to us since we’re tracking not the absolute number of uninstalls but a relative change.

However, there is a big problem with this piece of code apart from it being very hacky. Several users reported that it leaks memory and the tabs with this code running that are left open in background can leak hundreds of MBs of memory. We found it hard to reproduce it but it looks like the issue is only present on the version 11 and not present on version 12 of Google Chrome. Furthermore, some users have reported InvisibleHand crashing a lot when this code was deployed. Again, we were unable to reproduce it reliably but I tend to believe this setInterval() has something to do with memory leaks and crashes.

We usually deploy this code only for short periods of time (hours) to run a specific A/B test and roll it back (release a new version without this content script) as soon as possible. I’m really looking to the day when it’ll be possible to listen for the uninstallation event.

If you know why this code may be causing memory leaks and/or crashes, please let me know in the comments.

As it turns out, you’ll get

Net::SMTPFatalError: 555 5.5.2 Syntax error.

if your from address looks like ‘Evgeny Shadchnev <email@gmail.com>”. Setting it to just ‘email@gmail.com’ helps.

Textmate is great but Find in Project is slow as molasses, so everybody’s using Ack in Project bundle. The problem is it doesn’t work with Ruby 1.9 (fails with to_plist ArgumentError) and the problem is Textmate, which provides plist.c that is incompatible with Ruby 1.9. The solution:

# Updating osx-plist for Ruby 1.9 compatibility
$> git clone git://github.com/kballard/osx-plist.git
$> cd osx-plist/ext/plist
$> ruby extconf.rb && make
$> cp plist.bundle /Applications/TextMate.app/Contents/SharedSupport/Support/lib/osx/

The credit for the finding the solution goes to James.