• Key-Value-Observing is a divisive API, to say the least. Despite it’s (well-documented) flaws, I personally tend to favor it when wanting to know if a property’s value changes, but most developers I talk to (including both of my fellow iOS developers here at Tumblr) tend to prefer overriding setters. Here’s a case where I think KVO works slightly better than overwriting setters does.

    Say you have a custom view controller subclass with a property, and changes to that property’s value will result in some modifications being made to the view controller’s view or subviews. An example from the Tumblr codebase does exactly this:

    - (void)setContainerScrollable:(BOOL)containerScrollable {
        if (_containerScrollable != containerScrollable) {
            _containerScrollable = containerScrollable;
    
            self.container.scrollEnabled = containerScrollable;
            self.tableView.scrollEnabled = !containerScrollable;
        }
    }
    

    Looks simple enough, right? Now you can simply do the following:

    TMContainerViewController *controller = [[TMContainerViewController alloc] init];
    controller.containerScrollable = YES;
    

    Of course, there’s a problem with this. Since we’re calling the custom setter before the controller’s view has necessarily loaded, it won’t have the desired effect. At best, the subviews we operate on inside the overridden setter will be nil and our custom behavior won’t be applied. At worst, we’ll refer to self.view in our implementation, the view will be loaded prematurely, and something unexpected could occur.

    So how can we fix this? One way is to make sure our setter is called again after the view is loaded, and prevent against the custom logic being executed beforehand:

    - (void)setContainerScrollable:(BOOL)containerScrollable {
        if (_containerScrollable != containerScrollable) {
            _containerScrollable = containerScrollable;
    
            if ([self isViewLoaded]) {
                self.container.scrollEnabled = containerScrollable;
                self.tableView.scrollEnabled = !containerScrollable;
            }
        }
    }
    
    - (void)viewDidLoad {
        // View set-up
    
        self.containerScrollable = self.isContainerScrollable;
    }
    

    This should work, but calling a getter and passing it’s return value to the same property’s setter doesn’t strike me as being particularly elegant. What if we factor out our custom logic into a separate private instance method?

    - (void)updateViewsForContainerScrollability {
        self.container.scrollEnabled = self.isContainerScrollable;
        self.tableView.scrollEnabled = !self.isContainerScrollable ;
    }
    
    - (void)setContainerScrollable:(BOOL)containerScrollable {
        if (_containerScrollable != containerScrollable) {
            _containerScrollable = containerScrollable;
    
            if ([self isViewLoaded]) {
                [self updateViewsForContainerScrollability];
            }
        }
    }
    
    - (void)viewDidLoad {
        // View set-up
    
        [self updateViewsForContainerScrollability];
    }
    

    This is a fine solution, and will work as expected. That being said, let’s look at another approach to the same problem using KVO.

    Here’s what our observation code looks like:

    - (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change 
                           context:(void *)context {
        if (context == TMContainerViewControllerKVOContext) {
             if (object == self && [keyPath isEqualToString:@"containerScrollable"]) {
                self.container.scrollEnabled = self.isContainerScrollable;
                self.tableView.scrollEnabled = !self.isContainerScrollable;
            }
        }
        else {
            [super observeValueForKeyPath:keyPath ofObject:object change:change context:context];
        }
    }
    

    Now, we’re still faced with the same issue of needing to ensure that this code runs both A) as soon as the view is loaded and B) any time the property’s value is changed going forward. Thankfully, NSKeyValueObservingOptionInitial provides this exact behavior.

    - (void)viewDidLoad {
        // View set-up
    
      [self addObserver:self forKeyPath:@"containerScrollable"
                options:NSKeyValueObservingOptionInitial
                context:TMContainerViewControllerKVOContext];
    }
    

    Since we’re no longer overriding the setter, the property’s value can be changed completely independently of the view being initialized. When our view is set-up, we add an observer that is called immediately with the initial property value, and called again whenever the property is changed in the future.

    KVO code can be messy, can result in problems if used incorrectly, and certainly isn’t the best tool to use in all situations. But if you ask me, this is a pretty good example of when it does come in handy.

  • Efficient distribution of cacheable http requests at Tumblr

    Tumblr uses Haproxy for load balancing http requests and other types of tcp traffic to pools of application servers in order to evenly distribute our workloads and ensure that we’re using our machines efficiently. Haproxy supports consistent hashing of requests to specific servers which is a requirement for efficiently caching responses. Consistent hashing ensures that only a few mappings (of requests to servers) are redistributed when a server is added or removed from the pool which results in a higher maintained hit ratio when dealing with caching systems than a traditional modulo distribution method.

    An important aspect of the hashing algorithm is to ensure that load is distributed evenly across the pool. Haproxy uses the SDBM hashing function by default, but about a year ago we began doing testing to determine the efficiency of alternative algorithms using the same hashing criteria as we see in production. We found that the hashing algorithm most suited for the input given to our Haproxy instances at Tumblr was DJB2.

    The improvement we see when using DJB2 is best illustrated via a graph of connections from Tumblr’s varnish pools to the application nodes. The Varnish pools are consistently hashed backends to Haproxy. The effect of switching the algorithm is an improvement in the load distribution on the varnish pool which is reflected in the convergence of connections to application nodes ultimately leading to better cache usage.

    image

    Please continue reading for more on the details of hashing as implemented in Haproxy and the patch we have pushed upstream which lets you try out these options.

    Hash functions strive to have little correlation between input and output. The heart of a hash function is its mixing step. The behavior of the mixing step largely determines the degree in which a hash function is collision resistant. Hash functions that are collision resistant are more likely to provide an even distribution of load.

    The purpose of the mixing function is to spread the effect of each message bit throughout all the bits of the internal state. Ideally every bit in the hash state is affected by every bit in the message and perform that operation as quickly as possible for the sake of program performance. A function is said to satisfy the strict avalanche criterion if, whenever a single input bit is complemented (toggled between 0 and 1), each of the output bits should change with a probability of one half for an arbitrary selection of the remaining input bits.

    To guard against a combination of hash function and input that results in high rate of collisions, Haproxy implements an avalanche algorithm on the result of the hashing function. The avalanche is always applied when using the consistent hashing directive. It is intended to provide a good distribution for little input variations. The result is quite suited to fit over a 32-bit space with enough variations so that a randomly picked number falls equally before any server position, which is ideal for consistently hashed backends, a common use case for caches.

    In additional tests involving alternative algorithms for hash input and an option to trigger avalanche, we found different algorithms perform better on different criteria. DJB2 performs well when hashing ascii text which makes it a good choice for hashing http host headers. Other alternatives perform better on numbers and are a good choice when using source ip. The results also vary by use of the avalanche flag.

    What are the effects of the DJB2 and avalanche algorithms on your production deployments? Would it not be great to have an option that lets you play with the hash function and determine via configuration if avalanche is beneficial to you?

    Now you can. Tumblr’s patches for enabling the hashing alternative and an option to trigger avalanche are now available in Haproxy 1.5. Let us know the results of your testing.

  • keithmcknight:

really happened the other night. true story.

    keithmcknight:

    really happened the other night. true story.

  • Last month, Corinne and I attended the Grace Hopper Celebration of Women in Computing. As product engineers, we were delighted to meet other women in our field and celebrate their technical expertise.

To increase diversity in the tech space, Tumblr is a proud supporter of Hacker School and their initiatives. This fall, we’re sponsoring Daphne, a coder in the current batch, and we can’t wait to see what amazing things she’ll create next.

    Last month, Corinne and I attended the Grace Hopper Celebration of Women in Computing. As product engineers, we were delighted to meet other women in our field and celebrate their technical expertise.

    To increase diversity in the tech space, Tumblr is a proud supporter of Hacker School and their initiatives. This fall, we’re sponsoring Daphne, a coder in the current batch, and we can’t wait to see what amazing things she’ll create next.

  • mt:

codingjester talking about tumblr APIs at yahoodevelopers Hack USA!

    mt:

    codingjester talking about tumblr APIs at yahoodevelopers Hack USA!

  • My Summer at Tumblr: Luke Cycon

My summer as an Engineering Intern at Tumblr? You might expect me to say I found myself in a chaotic new world. Truth is, it was pretty tame. Now don’t mistake tame for uninteresting or easy. Of those, I can assure you, my summer was neither.

True to my interests, I worked on Tumblr’s back-end systems. In terms of size, the code-bases I worked on were larger than what I was used to, but not by a tremendous amount. What made for an interesting summer of development was the volume of data with which my code was expected to handle. I primarily worked on our Firehose. One could imagine that amount of data that flows through there. I spent my summer ironing our issues, refactoring “messy” parts of the code-base, and implementing runtime performance boosts.

I am quite proud of the features I contributed to the code-bases I worked with here. Small things, such as seeing a performance graph change shape, or the resident memory of a service drop because of a feature you added is powerful. Every improvement I made, I did so knowing that it would positively affect the experiences of the millions of creators that use Tumblr.

Working mainly in Scala, a language I consider to be my second favorite, I felt “comfortable” editing the code from day one. As the weeks passed, I began to notice my understanding of both my code-bases and the libraries we worked with growing quite intimate. I no longer required the documentation nearby to use a library, I no longer needed to look up the behavior of “that one constructor” for a class in the core library. From there, I saw the true power of the abstractions in which we deal as engineers. To say I grew as developer would be an understatement.

One thing I learned over the summer? I couldn’t tell you if I wanted. What this summer has meant to me is more than a collection of tips and tricks that are nice to know, it’s been a chance to grow as a software engineer. Working in such a violently creative environment with a group of such incredible and talented people has been an amazing experience. I am excited to head back to California, but it will be hard to leave this place and these people behind. I suppose I won’t have to, not entirely at least. Moving forward, I will be working part-time with Tumblr from school.

Tumblr is something different in a powerful way, and it has been nothing short of exhilarating to be a part of it.

    My Summer at Tumblr: Luke Cycon

    My summer as an Engineering Intern at Tumblr? You might expect me to say I found myself in a chaotic new world. Truth is, it was pretty tame. Now don’t mistake tame for uninteresting or easy. Of those, I can assure you, my summer was neither.

    True to my interests, I worked on Tumblr’s back-end systems. In terms of size, the code-bases I worked on were larger than what I was used to, but not by a tremendous amount. What made for an interesting summer of development was the volume of data with which my code was expected to handle. I primarily worked on our Firehose. One could imagine that amount of data that flows through there. I spent my summer ironing our issues, refactoring “messy” parts of the code-base, and implementing runtime performance boosts.

    I am quite proud of the features I contributed to the code-bases I worked with here. Small things, such as seeing a performance graph change shape, or the resident memory of a service drop because of a feature you added is powerful. Every improvement I made, I did so knowing that it would positively affect the experiences of the millions of creators that use Tumblr.

    Working mainly in Scala, a language I consider to be my second favorite, I felt “comfortable” editing the code from day one. As the weeks passed, I began to notice my understanding of both my code-bases and the libraries we worked with growing quite intimate. I no longer required the documentation nearby to use a library, I no longer needed to look up the behavior of “that one constructor” for a class in the core library. From there, I saw the true power of the abstractions in which we deal as engineers. To say I grew as developer would be an understatement.

    One thing I learned over the summer? I couldn’t tell you if I wanted. What this summer has meant to me is more than a collection of tips and tricks that are nice to know, it’s been a chance to grow as a software engineer. Working in such a violently creative environment with a group of such incredible and talented people has been an amazing experience. I am excited to head back to California, but it will be hard to leave this place and these people behind. I suppose I won’t have to, not entirely at least. Moving forward, I will be working part-time with Tumblr from school.

    Tumblr is something different in a powerful way, and it has been nothing short of exhilarating to be a part of it.

  • Tumblr Summer Intern: Walter Menendez

    This summer I got to join Tumblr as a search team intern. This was a dream come true: I spend a lot of my day on Tumblr sharing content with others and generally indulging in the specific Tumblr lingo and lifestyle, so to do it all day was just too unreal. (Fun fact: I got hired through Tumblr, thanks to a post on this very same blog!) I didn’t expect to like New York as much as I did and I’m so heartbroken that I have to leave Tumblr and go back to school.

    The search team is a really cool team to work on because we do a lot of really critical, quantitative thinking about the highly subjective, qualitative content on Tumblr. My projects definitely required a thorough understanding of Tumblr’s users and their idiosyncrasies in order for our data to make any sense. It was cool to just go into our databases, scrape some data and do all kinds of things with it. It definitely opened up my view of Tumblr as a community and, well, I also found a ton of stuff to follow/reblog while doing so. 

    I go to MIT, where a lot of my research background is in data visualization and large data sets, so one of my first projects was to work on a data visualization for our trending post stream. The search team had been working on trending content for quite some time now, focusing on the three core forms of content on the site: blogs, tags, and posts. Blogs and tags had a decent amount of front end work but trending posts not so much. We basically had a list of post URLs that were deemed trending based on our metrics, but we had no idea what they looked like, nor did we really have any intuition on to what extent they were trending. To fix that, I built a D3.js based visualization, while grabbing posts from our PHP framework. There was a fair amount of tech involved! From processing JSONs to animating images, the visualization took a fair bit of work. At the moment, it’s more of an internal thing but hopefully, users all over Tumblr will be able to see it as well.

    My second project was much more data intensive and focused on search traffic analysis. Tumblr had rolled out trending tags to mobile during my time here, which was a great way to discover more content on Tumblr. However, those metrics and algorithms were only based on post creation. Since more people consume content than they do create it, there was a lot of interesting data lying in tags and their search counts. I started scraping data from our page logs and effectively did a very high level count over hundreds of thousands of tags. Afterwards, I would save the data into Redis and would then compute some statistics and ultimately rank the tags based on how trending they were.

    This analysis was a really cool project because every morning as I ran my scripts to collect and process the tag data, I got to see the progression of current events all of a sudden going from single digit counts to thousands of hits. It was also a great way to stay up to date with news as I would often Google a tag that I had no idea what it was referring to. Another cool part was comparing my ranking to our current rankings in production, and seeing just how aligned and different we were.

    My time at Tumblr definitely couldn’t have been possible had it not been for the staff here. They’re all so knowledgable and approachable and hilarious, so while my roommates would groan about going to “work”, I would skip on my merry way to a blogger’s haven. Plus, we have dogs. Who wouldn’t rush to the office to see those?

    Forever reblog!

    - Walter

  • Tumblr Summer Intern: Eric Leong

    I spent this summer as an intern on Tumblr’s Android team. I joined as the team had just transitioned to Android Studio, so we were all learning how to work with the new IDE together while familiarizing myself with the code. On the first day I even made a pull request to ignore a folder created when installing Android Studio. My tasks throughout the summer were to implement features the team hadn’t gotten around to yet, allowing me to brush the dust off my Java and touch the various parts of the codebase at my own pace.

    I mainly focused on user-facing feature development, which involved grappling with the intricacies of Android animations and the Tumblr app. My first task was to develop the notifications widget and revamp the existing post creation widgets, which turned out to be a great way to play with the Tumblr codebase and the newest features of the Android framework. I had full responsibility for the features I developed, with the design and QA team assigning bugs to me, which was both exciting and humbling. I had the most fun working on features I knew no one else had implemented, like “swipe back”, and realizing that I was pushing the boundaries of the Android platform.

    It was an incredible experience to watch my code end up in the hands of our Android users and watch my friends play with the features I built. I’m also proud that I had a chance to fix issues and add functionality that I wanted to see in the app. Even when I did not directly work on a feature, my feedback helped shape the design and implementation.

    I had an amazing summer working with the Tumblr team, who provided guidance amidst a flurry of questions and showed me how design and code can come together to produce a beautiful app. Me and my fellow interns were brought closer together by our love of photography and the Tumblr community.

    It was an unforgettable summer at Tumblr where I played my small role in bringing creators the audience they deserve.

    - Eric

  • Tumblr Summer Intern:  Jared Stern
Working at Tumblr as a summer engineering intern, I was flung headfirst into a mysterious world replete with terms I did not understand, people I did not know, and a codebase that defied all attempts at understanding. I asked many questions, and slowly realized that with a little thinking I could figure some things out myself. Slowly, I got to know the tools and languages (now I know some PHP!) and gained a bit of confidence in dealing with our code.Much of my work dealt with a couple of Tumblr’s back-end services, which handle a lot of the heavy lifting for the web application—mostly moving data around so the app can quickly access everything it needs. I was given quite a bit of responsibility, which was exciting and instructive and frightening, particularly when I broke an important thing on my third Tuesday. Luckily, my other work was a bit less eventful. 
I eventually got more comfortable (a bit more comfortable, anyway!) with deploying changes to our code. In addition to services, I worked on a few small projects closer to the site’s front end. I fixed a bug in the bookmarklet that caused posts to be created incorrectly. I fixed an issue in our internal administration site so the site would be more responsive at peak times. On one occasion, I worked with a member of our support team to fix a single tumblelog that curiously would not load past its thirteenth page. It was a pleasure to be able to fix an actual user’s actual problem.Through all of this I had the honor of working with the fabulous Tumblr team, people who were friendly and helpful and knowledgeable, and who went out of their way to help me along. All told, it was a marvelously challenging, interesting, and educational summer.
- Jared

    Tumblr Summer Intern:  Jared Stern

    Working at Tumblr as a summer engineering intern, I was flung headfirst into a mysterious world replete with terms I did not understand, people I did not know, and a codebase that defied all attempts at understanding. I asked many questions, and slowly realized that with a little thinking I could figure some things out myself. Slowly, I got to know the tools and languages (now I know some PHP!) and gained a bit of confidence in dealing with our code.

    Much of my work dealt with a couple of Tumblr’s back-end services, which handle a lot of the heavy lifting for the web application—mostly moving data around so the app can quickly access everything it needs. I was given quite a bit of responsibility, which was exciting and instructive and frightening, particularly when I broke an important thing on my third Tuesday. Luckily, my other work was a bit less eventful.

    I eventually got more comfortable (a bit more comfortable, anyway!) with deploying changes to our code. In addition to services, I worked on a few small projects closer to the site’s front end. I fixed a bug in the bookmarklet that caused posts to be created incorrectly. I fixed an issue in our internal administration site so the site would be more responsive at peak times. On one occasion, I worked with a member of our support team to fix a single tumblelog that curiously would not load past its thirteenth page. It was a pleasure to be able to fix an actual user’s actual problem.

    Through all of this I had the honor of working with the fabulous Tumblr team, people who were friendly and helpful and knowledgeable, and who went out of their way to help me along. All told, it was a marvelously challenging, interesting, and educational summer.

    - Jared

  • Tumblr Summer Intern: Jacob Haip
This summer I interned as a product engineer in the Creativity team under Pau Santesmasses.  I was tasked with removing handlebars.js and prototype.js, two javascript libraries we were using for HTML tempting and simplifying javascript code.  Nothing was wrong with these libraries but we were also including underscore.js and jQuery, libraries with similar features, so it was a bit of a waste.  To help free Tumblr of these libraries, over the course of the summer I rewrote ~15,000 lines of code to port prototype.js to jQuery and handlebars.js to underscore.js.
The transition was a challenge not because the libraries were that different but because of the how scattered the old code was throughout the website.  I’m happy to end the summer having removed handlebars.js and all but a couple isolated pieces of prototype.js.  It was also cool to have touched so many pieces of Tumblr in the process.
Users should enjoy faster load times throughout the website.  Here in the office this is about a 1/3 of a second faster load time and this is even more noticeable on slower internet connections.  It’s also nice to know that the rest of the people in the engineering team will have two less libraries they have to worry about.
It was a pleasure to be a part of the Tumblr team during such an exciting time: the Yahoo! acquisition and Tumblr gaining more of the attention it deserves as the place for brands.  The other interns were amazing and I loved how I got to know all of them and not just the engineering interns.  I look forward to joining my friends once again at MIT but I will miss all the people I have met here.  MIT maybe be good at science but it sure isn’t good at art and I will miss the creativity of all the people in the office and being a part of the truly creative community on Tumblr.
- Jacob

    Tumblr Summer Intern: Jacob Haip

    This summer I interned as a product engineer in the Creativity team under Pau Santesmasses.  I was tasked with removing handlebars.js and prototype.js, two javascript libraries we were using for HTML tempting and simplifying javascript code.  Nothing was wrong with these libraries but we were also including underscore.js and jQuery, libraries with similar features, so it was a bit of a waste.  To help free Tumblr of these libraries, over the course of the summer I rewrote ~15,000 lines of code to port prototype.js to jQuery and handlebars.js to underscore.js.

    The transition was a challenge not because the libraries were that different but because of the how scattered the old code was throughout the website.  I’m happy to end the summer having removed handlebars.js and all but a couple isolated pieces of prototype.js.  It was also cool to have touched so many pieces of Tumblr in the process.

    Users should enjoy faster load times throughout the website.  Here in the office this is about a 1/3 of a second faster load time and this is even more noticeable on slower internet connections.  It’s also nice to know that the rest of the people in the engineering team will have two less libraries they have to worry about.

    It was a pleasure to be a part of the Tumblr team during such an exciting time: the Yahoo! acquisition and Tumblr gaining more of the attention it deserves as the place for brands.  The other interns were amazing and I loved how I got to know all of them and not just the engineering interns.  I look forward to joining my friends once again at MIT but I will miss all the people I have met here.  MIT maybe be good at science but it sure isn’t good at art and I will miss the creativity of all the people in the office and being a part of the truly creative community on Tumblr.

    - Jacob