Tumblr Engineering

Tumblr SDK updates

bryan — Thu, 25 Apr 2013 10:57:39 -0400

Alongside yesterday’s Tumblr for iOS 3.3.1 release, I’m happy to also announce a couple of enhancements to the Tumblr iOS SDK (version 1.0.2 is now available in the CocoaPods repository).

Tumblr for iOS now exposes URL schemes that can be invoked in order to create text, quote, link, and chat posts. Additionally, developers can specify callback URLs that will send the user back to their apps once the Tumblr post is either created or cancelled. These endpoints are now wrapped and exposed by the TMTumblrAppClient class, or the URLs can be accessed directly if you’d prefer (they’re documented here).

Lastly, Tumblr for iOS has been able to create photo and video posts using media shared from other apps for some time now, via Apple’s standard UIDocumentInteractionController. As of 3.3.1, third-party applications can now also specify captions and tags to accompany the photos and videos being shared. The annotations used to do so are documented here.

If you’re using the Tumblr iOS SDK in your app or plan to integrate going forward, please get in touch and let me know if there’s anything I can do to make this even easier.

My Philosophy On Alerting

robewaschuk — Thu, 25 Apr 2013 09:52:10 -0400

robewaschuk:

I wrote some stuff while I was at Google about writing clean alerts and keeping an oncall rotation sane; after some cleanup they’ve allowed me to make it public. Of course, this represents my opinions and not Google’s. They do reflect what we think are best practices at Tumblr, though. We’re hiring Site Reliability Engineers.

Check out My Philosophy On Alerting.

Summary

When you are auditing or writing alerting rules, consider these things to keep your oncall rotation happier:

Pages should be urgent, important, actionable, and real.

They should represent either ongoing or imminent problems with your service.

Err on the side of removing noisy alerts.

You should almost always be able to classify the problem into one of
availability & basic functionality

latency

correctness (completeness, freshness and durability of data)

and feature-specific problems.

Symptoms are a better way to capture more problems more comprehensively and robustly with less effort.

The further up your serving stack you go, the more problems you catch in a single rule. Balance this with being able to distinguish what’s going on.

TMCache: fast object caching for iOS & OS X

jstn — Wed, 24 Apr 2013 13:06:10 -0400

A simple write-through cache can be a powerful optimization for any system needing fast, frequent access to the same resource (i.e., almost everything ever). It’s especially true for a limited memory environment like iOS, which is also partially bound by processor and bandwidth constraints. Something as common as downloading and displaying an image touches all these pain points, and that makes it a great place to explore the potential of caching, especially if it’s something your app does as often as the Tumblr mobile apps.

Cocoa itself does a fair bit of caching on its own without intervention, but it has many limitations. In the case of downloading an image using NSURLConnection, the entire server response is stored in a shared NSURLCache, but the inner details are infamously murky and entirely dependent on HTTP headers from the server. After you’ve loaded the image into a UIImageView it’s one memory warning away from being lost, or even sooner if it gets slated for recycling from a table cell scrolling off screen.

Today we’re proud to announce the open sourcing of TMCache, which is designed for this and any other situation where there’s a need to persist “expensive” objects and access them rapidly. TMCache is an object cache for iOS and OS X, suitable for any object conforming to the NSCoding protocol (including the basic Foundation data types, collections, and many UIKit objects like UIImage). It consists of two parallel caches, one in memory and one on disk, both coordinated locklessly by GCD. Like NSCache, TMCache will automatically remove objects from memory when the app receives a warning or goes into the background. Unlike NSCache, TMCache can do things like transparently restore objects from disk, access objects asynchronously using blocks, and automatically limit the size of the cache by age or size.

TMCache is available under the Apache 2.0 license on GitHub or as a CocoaPod, including full documentation in HTML and docset format. Let us know if you find it useful!

Open Source - Memcache Top

mobocracy — Tue, 23 Apr 2013 12:53:07 -0400

mobocracy:

We rely on memcache pretty heavily at Tumblr, with over 10TB of cache memory available across the stack. One of the things we’ve historically had a challenging time with at Tumblr is finding hot keys. A hot key is a memcache key getting dramatically more activity than other keys. This can have a significant performance impact on your cache backend.

We spent the past few days working on a C++ implementation of mctop*, which we’re happy to release today as memkeys. We do some pretty interesting stuff in memkeys to keep from dropping packets, some of which is documented in the wiki. I’m particularly proud of the striped lock-free queue implementation. In some basic benchmarks I found that memkeys dropped less than 2% of packets when seeing 1Gb/s of traffic. Additionally, the latency between a packet being picked up, parsed, processed, and reported on averages less than 1ms. Here is a screenshot of memkeys in action.

Interested in stuff like this? We’re hiring.

Footnote: Etsy created the excellent mctop tool which aims to be like unix top for memcache, showing you which keys are getting the most activity. Unfortunately (as noted in the known issues), mctop drops packets. It drops a lot of packets. This can be really problematic because depending on the packets being dropped, you’re getting a really incomplete view of your cache story.

Tumblr Engineering @ Percona Live MySQL Conference

evan — Thu, 18 Apr 2013 17:39:00 -0400

We’re pleased to announce that Tumblr’s Database Engineering team will be attending the Percona Live MySQL Conference next week in Santa Clara, CA!

We’ll be giving a talk on our open source automation software, Jetpants, which has helped us scale to over 175 billion distinct rows of relational data to date. We’re also looking forward to attending a number of amazing sessions from our friends at Percona, Facebook, Oracle, Palomino, Etsy, and more.

If you haven’t registered yet, use code SpeakMySQL to save 15%. Hope to see you there!

Last week, two of our engineers, Wolf and JC, got to stay up all...

codingjester — Thu, 11 Apr 2013 13:46:00 -0400

Last week, two of our engineers, Wolf and JC, got to stay up all night eating pizza with some really talented young hackers at two different hackathons — 2013 hackNY Student Hackathon at Columbia University and Photo Hack Day at Facebook HQ.

After a long day of hacking, pizza, sleep deprivation, more pizza, and some DIY tacos, participants demoed their hacks, including a few for the Tumblr API. We were absolutely blown away by the creativity and quality of the projects and look forward to seeing what you’ll build at the next hackathon!

The spring 2013 hackNY Student Hackathon is this weekend at...

codingjester — Wed, 03 Apr 2013 15:56:50 -0400

The spring 2013 hackNY Student Hackathon is this weekend at Columbia University! For 24 hours, students will collaborate on creative coding challenges, and then present their hacks to a panel of judges. Prizes and bragging rights will be awarded to the winners.

One of our engineers, The Wolf, will be demoing the Tumblr API, helping students build awesome stuff, and eating leftover pizza.

Details

When: Saturday, 4/6 at 2pm – Sunday, 4/7 at 2pm

Where: Fu Foundation School for Engineering and Applied Science, Columbia University

How: Sign up here

If all-night hacking isn’t your thing, be sure to check out all the student hacks at DemoFest (tickets are free!) this Sunday from 12-2 pm.

See you soon!

Core Data as a cache

bryan — Mon, 25 Mar 2013 17:16:00 -0400

Core Data is great but automatic migrations can be tricky. Migrations can take a long time, which could result in your app being terminated if it is happening on the main thread during application launch. Performing migrations on a background thread is also a bad idea, meaning your application really needs to be able to fully launch without a Core Data stack whatsoever in order to safely migrate. This can be a huge change to make to an existing app.

If you’re really only using Core Data as a cache, you don’t actually need to perform a migration. Simply check if the existing store is compatible with your managed object model and if not, delete and recreate it. Here’s my approach for doing so.

Many thanks to Marcus Zarra for being extremely helpful both over Twitter and on Stack Overflow. You should totally buy his Core Data book.

Over the past few weeks we’ve open-sourced quite a few official...

bryan — Fri, 22 Mar 2013 16:08:34 -0400

Over the past few weeks we’ve open-sourced quite a few official client libraries for the Tumblr API. Today I’m proud to announce the Tumblr iOS SDK, an Objective-C library for easily integrating Tumblr data into your iOS (or OS X) applications, however you see it.

The Tumblr SDK for iOS contains a few different components to start:

Authentication (OAuth and xAuth implementations)
A full wrapper around all of our API endpoints
Inter-app communication (this is pretty limited at the moment but we plan to expand it quite a bit going forward)
A UIActivity stub for easy inclusion of a Tumblr button in a standard Apple UIActivityViewController

We’ve been using this SDK in production for quite some time now and are thrilled to finally be able to share it with you.

If you’re interested in integrating with Tumblr on iOS or OS X in a way that the SDK doesn’t currently facilitate, please get in touch. I’m very interested in hearing any and all feedback on how we can make this as easy as possible.

seejohnrun: tumblr.php I’m here to announce another new...

codingjester — Fri, 15 Mar 2013 13:52:34 -0400

seejohnrun:

tumblr.php

I’m here to announce another new addition to our list of official API clients. This week we have tumblr.php - it’s available on GitHub and composer via packagist. It is tested, is PSR-2 compatible, and is well documented.

Like the other clients we’ve been announcing (most recently JS and Java) it has full support for all of the Tumblr v2 API endpoints.

Time to make something cool!

Using entropy to route web traffic

codingjester — Fri, 01 Mar 2013 11:40:05 -0500

adamlaiacano:

Earlier this week, Blake asked me for some help with a problem he’s working on. He has a couple of hash functions that are being used to route web traffic to a number of different servers. A hash function takes an input, such as a blog’s url, and outputs a number between 0 and 2^32. Say we have 1000 servers, that means that each one will handle about 430 million points in the hash-space.

The data looked something like this (with fake blog names, of course):
##       blog.name        H1        H2        H3        H4
## 1 23.tumblr.com 3.137e+09 1.866e+09 6.972e+08 5.792e+08
## 2 19.tumblr.com 1.875e+09 2.545e+08 2.606e+09 1.312e+09
## 3 34.tumblr.com 1.366e+09 2.236e+09 1.106e+09 3.640e+09
## 4 43.tumblr.com 2.639e+09 1.098e+09 8.755e+08 1.507e+09
## 5 90.tumblr.com 6.564e+08 5.397e+07 3.084e+09 2.961e+09
## 6 29.tumblr.com 2.476e+09 4.532e+08 2.787e+08 4.894e+08
One important thing to point out before we get started is that this has to be a representative sample of the request data. Despite the wild popularity of my personal blog, it doesn’t get a sliver of the traffic that Beyonce gets. That fact needs to be represented in the sample data, meaning that her blog should appear in more rows of the sample data than mine.

Plot the data

The first thing I ever do is plot data to get a sense of what I’m working with and what I’m trying to accomplish. The density plots below show the distribution of values in the hash space for each algorithm. If you’re not familiar with kernel density plots, you can imagine this to be a smoothed (and prettier) version of a histogram. For the electrical engineers out there, it’s the sum of the convolution of a kernel function (usually a gaussian), with an impulse function at each of the points on the x-axis (represented here by dots).

By comparison, here are the density plots of a “near-ideal” example (1000 pulls from a uniform distribution) and a bad example (all assigned to the value 2e+09). The worst case example here shows the shape of the kernel function.

Calculate the entropy

In information theory, entropy is the minimum number of bits required (on average) to identify an encoded symbol (stay with me here…). If we’re trying to transmit a bunch of text digitally, we would encode the alphabet where each “symbol” is a letter that will be represented in 1’s and 0’s. In order to transmit our message quickly, we want to use as few bits as possible. Since the letter “e” appears more frequently than the letter “q”, we want the symbol for “e” to have fewer bits than “q”. Make sense?

Huffman Coding is one encoding algorithm. There’s an example implementation here, which assigns the code 100 to “e” and 1111110010 to “q”. The more “uneven” your symbol distribution is, the fewer bits it will take, on average, to transmit your message (meaning you’ll have a lower entropy). The entropy value is a lower bound for the actual weighted average of the symbol lengths. There are special cases where some encoding algorithms get closer to the entropy value than others, but none will ever surpass it.

The actual entropy formula is:

\[ H(x)=-\sum_{i=0}^{N-1} p_i log_2(p_i) \]

Where $ H(x) $ is the entropy, and $ p_i $ is the probability of that symbol $ i $ will appear. In the example I linked to above, $ p_e = 0.124 $ and $ p_q=0.0009 $, so it makes sense that e’s symbol is so much shorter. In the example, the average number of bits per symbol is $ \sum S_i p_i = 4.173 \frac{bits}{symbol} $, where $ S_i $ is the number of bits in the symbol. The entropy, from the above equation, is $ 4.142 \frac{bits}{symbol} $.

The example problem of web traffic distribution is a little different. We’re not actually encoding anything, but rather trying to make the theoretical lower bound for average number of bits/signal as high as possible.

We can consider each server to be a symbol, and the amount of traffic that it recieves is decided by the hash function that we’re trying to choose. If one of our servers is the equivalent of the letter “e”, it’s going to be totally overloaded while the “q” isn’t going to be handling much traffic at all. We want each symbol (server) to appear (receive traffic) equally often.

So to calculate the entropy, we’ll take a histogram of the hash values with 20 buckets (representing the 20 servers). That will give us the number of requests that go to each server. Dividng that by the total number of requests gives us each server’s probability of handing the next incoming request. These are the $ p_i $ values that we need in order to calculate the entropy. In code, it looks like this:
calc.entropy <- function(hash.value) {
    h = hist(hash.value, plot = FALSE, breaks = seq(0, 2^32, length.out = 21))
    probs = h$counts/sum(h$counts)
    print(probs)
    entropy = -sum(probs * log2(probs))
}
The entropy values for our four hash functions are:
##   hash.function entropy
## 1            H1   4.203
## 2            H2   4.226
## 3            H3   4.254
## 4            H4   4.180
And while we’re at it, here’s the entropy of our best/worst case example that we plotted earlier.
##    hash.function entropy
## 1     near.ideal   4.309
## 2 worst.possible   0.000
Why is the worst case value 0? Because if all traffic is going to one server, we wouldn’t need any bits at all to tell us which server the request is going to. The theoretical limit for a histogram with 20 buckets is: $ -20\frac{1}{20}log_2{\frac{1}{20}} = 4.32 $, which we’re close to but can never exceed.

All of our hash functions appear to be working pretty well, especially for such a small sample size that I’m using for this blog post. It looks like our winner is H3.

To summarize here’s what we did to find the optimal hashing function:

Get a representative sample of your web traffic

Run each request through the hashing function

Take a histogram of the resulting values with N bins, where N is the number of servers you have available

Divide the bin counts by the total number of requests in your sample to get the probability of handing a request for each server

Calculate the entropy, $ H(x)=-\sum_{i=0}^{N-1} p_i log_2(p_i) $, for each hash function

There are more considerations to take, like setting an upper bound on the value of $ p_i $ to ensure that no single server ever gets so busy that it can’t handle its load.

If you want to read up more on information theory, Elements of Information Theory by Thomas Cover and Joy Thomas is an excellent book that is reasonably priced (used) on Amazon.

There is also, of course, Claude Shannon’s landmark paper from 1948 “A Mathematical Theory of Communication”“, in which he essentially defines the entire field.

seejohnrun: tumblr Java client: jumblr Last week we announced...

codingjester — Mon, 25 Feb 2013 11:10:00 -0500

seejohnrun:

tumblr Java client: jumblr

Last week we announced our first official API client, for JavaScript (read: the announcement). This week we’re back to announce the release of an official Java client, Jumblr:
Blog blog = client.blogInfo("seejohnrun.tumblr.com");
for (Post post : blog.posts()) {
    post.like(); // you're too kind..
}
Like its JavaScript counterpart, Jumblr comes with full support for all of the API V2 endpoints. Check out more detail on the github page!

If you’re interested in following along with any of our open-source work, check out our GitHub page.

Community is extremely important us here at Tumblr. That goes...

codingjester — Fri, 22 Feb 2013 13:54:09 -0500

Community is extremely important us here at Tumblr. That goes double for engineers and developer evangelists. After spending all day giving talks at the API Strategy & Practice Conference, many of the speakers who were in town came over to TumblrHQ for the monthly Dev Evangelist Meetup here in NYC. It was a great time had by all and we were even able to get a group shot of everyone!

If you’re looking to be a part of a team that encourages community, We’re hiring.

seejohnrun: tumblr.js JavaScript client Today I’m excited to...

codingjester — Tue, 19 Feb 2013 17:43:33 -0500

seejohnrun:

tumblr.js JavaScript client

Today I’m excited to announce the release of tumblr.js, the first of several official API clients we’ll be rolling out over the next few months.

You can install it now with npm, and start making something awesome:
var tumblr = require('tumblr.js');
var client = tumblr.createClient({
  consumer_key: 'consumer_key',
  consumer_secret: 'consumer_secret',
  token: 'oauth_token',
  token_secret: 'oauth_token_secret'
});

// Name all of the authenticating user's blogs
client.userInfo(function (err, data) {
  data.user.blogs.forEach(function (blog) {
    console.log(blog.name);
  });
});
It comes with full support for all of the API V2 endpoints including tag search, following, liking, and post creation. For more detail, see the GitHub page.

More to come soon!

0xa: We have Engineering Summer Internships @ Tumblr! We’re...

0xa — Tue, 29 Jan 2013 18:43:25 -0500

0xa:

We have Engineering Summer Internships @ Tumblr! We’re looking for aspiring software engineers with a passion for open source software to join us for a summer of programming and fun. You will be integrated into a small engineering team working on a real-world project as part of

Product Engineering: Using PHP and JavaScript, create new and improve site features that keep our growing millions of users doing amazing things with their tumblelogs.

Search Engineering: Research, expand and refine Tumblr’s search infrastructure and search features.

Platform Engineering: Write highly optimized distributed services that manage data and requests in real time, helping our site scale to billions of posts.

Infrastructure Engineering: Work hands on with the Network team configuring, deploying and maintaining JunOS network devices.

Mobile Engineering: Come work on the application that’s putting Tumblr in millions of users’ pockets. A passion for IOS and Android is needed.

Polish your code samples, then send us your resume via our Engineering Summer Internship job page.

(We’re also hiring full time Engineers at every level of our technical stack. Learn more on Tumblr’s Jobs page).

Tumblr for iPad's custom UIPopoverController

bryan — Thu, 20 Dec 2012 16:35:28 -0500

When Peter was designing Tumblr for iPad, he came up with a great custom popover design:

Implementing his design was a challenge that I wanted to share.

iOS’s UIKit framework includes UIPopoverBackgroundView, an abstract class which can be subclassed to provide a custom background for a popover. Sounds perfect, right? UIPopoverBackgroundView let me use custom assets provided by Peter and Ben for the background and arrow, but the design also included a custom inset stroke/shadow. Fortunately, UIPopoverBackgroundView makes it easy to hide the default inset:

+(BOOL)wantsDefaultContentAppearance {
    return NO;
}

Unfortunately, adding its replacement wasn’t so easy.

First I subclassed UIPopoverController and added the custom inset to the view hierarchy when the popover is presented (_insetShadow is a resizable UIImageView):

UIView *superview = self.contentViewController.view.superview;
[superview addSubview:_insetStroke];

_insetStroke.frame = superview.bounds;

Simple enough, unless the popover’s content view controller is a UINavigationController.

In that case, the view hierarchy looks different and we have to resort to something like:

UIViewController *controller = self.contentViewController;

BOOL isNavigationController = [controller isKindOfClass:[UINavigationController class]];

UIView *superview = isNavigationController 
        ? controller.view 
        : controller.view.superview;

[superview addSubview:_insetStroke];

CGRect frame = controller.view.superview.bounds;

if (isNavigationController && 
        !((UINavigationController *)controller).isNavigationBarHidden) {
    float navBarOffset = 36;

    frame.origin.y += navBarOffset;
    frame.size.height -= navBarOffset;   
}

_insetStroke.frame = frame;

Ugly, but it works.

Except it doesn’t work on iOS 5 because wantsDefaultContentAppearance wasn’t added until iOS 6. On iOS 5 we need to manually traverse the view hierarchy to find the default stroke and remove it. But keep in mind that the view hierarchy looks different depending on whether or not the content view controller is a navigation controller. We end up with something like this:

if (!IS_IOS_6) {
    for (UIView *subview in [superview subviews]) {
        if (isNavigationController) {
            for (UIView *subsubview in [subview subviews])
                if ([subsubview class] == [UIImageView class])
                    subsubview.hidden = YES;
        } else {
            if ([subview class] == [UIImageView class] && subview != _insetStroke)
                subview.hidden = YES;
        }
    }
}

Orr Sella: Introducing tumblr4s: A Scala Library For The Tumblr API

mobocracy — Mon, 10 Dec 2012 18:14:55 -0500

Orr Sella: Introducing tumblr4s: A Scala Library For The Tumblr API:

orrsella:

Back in February of this year I stumbled upon an amazing article on HighScalability.com about Tumblr’s infrastructure. It’s a great read for many reasons (go read it). One of the greatest take-aways I had from this article, was this “new” programming language called Scala. I had never heard of…

Awesome stuff. Thanks Orr!

adamlaiacano: John Myles White was kind enough to come by...

radioon — Thu, 06 Dec 2012 10:02:00 -0500

adamlaiacano:

John Myles White was kind enough to come by Tumblr HQ this week and give a talk about the advantages that MAB (Multi-Armed Bandit) testing provide over traditional A/B testing.

Most of the content is drawn from his ebook “Bandit Algorithms for Website Optimization”.

adamlaiacano: The Bad Data Handbook is finally out! It’s the...

codingjester — Tue, 13 Nov 2012 13:23:27 -0500

adamlaiacano:

The Bad Data Handbook is finally out! It’s the first time I’ve contributed to a publication and I’m incredibly excited about it.

The top required skills of a data scientist are generally considered to be: mathematical know-how, programming capabilities, and some sort of domain knowledge enabling them to ask (and then answer) relevant questions.

This book is about all of the other garbage that you have to put up with along the way. Each chapter is written by someone who has spent more time than they probably would have liked dealing with a specific issue, and they provide some tips and pitfalls. I haven’t read the other chapters yet, but some of the included topics are:

Test drive your data to see if it’s ready for analysis

Work spreadsheet data into a usable form

Handle encoding problems that lurk in text data

Develop a successful web-scraping effort (that’s me)

Use NLP tools to reveal the real sentiment of online reviews

Address cloud computing issues that can impact your analysis effort

Avoid policies that create data analysis roadblocks

Take a systematic approach to data quality analysis

I hope at least a few of you buy it and enjoy it.

Tumblr for iPhone is now 100% native

bryan — Thu, 08 Nov 2012 09:57:00 -0500

My appreciation for those who build web browsers has grown dramatically over the past few months.

With today’s 3.2 release I’m happy to announce that Tumblr for iPhone is now 100% native. We love web technologies at Tumblr but believe this change provides a much better experience for this particular product.

HTML is the first-class citizen at Tumblr. All posts, whether authored in plain text, Markdown, or using our WYSIWIG editor, are stored as HTML. This is what we use to build your dashboard both on the website and in the mobile apps, via our API.

The decision to use a web view to render lists of posts in the 3.0 rewrite (June 2012) had nothing to do with cross-platform compatibility or ease of development and everything to do with needing to render (somewhat) arbitrary HTML, provided by our users. This makes building a Tumblr iOS app of the highest caliber an interesting challenge.

As noted by Zach Williams, we used GRMustache to render the post lists and Zepto.js was used to implement the web view’s behavior, with some slight modications (the longTap event didn’t work exactly the way we wanted, tap wouldn’t play HTML5 audio, and we needed to prevent touches while scrolling). CSS classes were used instead of :active pseudo-classes so they could be removed programmatically, as scrolling began. Our JavaScript-native “bridge” was basically this example.

Suitable scrolling performance was difficult to achieve in the web view as lists of Tumblr posts are usually extremely media-heavy. Some of the measures we took included using images in place of border-radius and box-shadow CSS and scaling/compressing photos on our servers, to the exact size needed on the phone.

We considered writing a JavaScript dequeuing mechanism but didn’t end up doing so. Since our 3.0 release, Airbnb open-sourced their “UITableView for the web” in ∞.js and LinkedIn published a detailed overview of how they approached a similar problem in their iPad application. Anyone attempting to build a web/native hybrid application like we did should take a look at both of these resources first.

I’m excited about the speed and stability that I believe going fully native has brought to our app, but please let me know how we can make the Tumblr iOS experience even better.