• Moving things out of critical rendering path

    javascript:

    Tumblr’s web pages are heavy.

    Naturally, the heaviest strain on the network comes from the content - filled with heavy gifs and other content. However, we also load embarrassingly large amounts of JavaScript. There is one particularly heavy JavaScript file that contains our vendor libraries (e.g., jQuery, Backbone, etc), which we call the vendor bundle.

    It was loaded right on the top of the page, significantly adding to the critical rendering path. But we don’t need it there, being one of the first things user loads. Moving the vendor bundle down the page might result in performance gains and a more responsive page load time - all are great goals that Tumblr’s Core Web team is set on accomplishing.

    After a few months of patching ancient, legacy inline scripts, and running performance tests, the script is finally moved to the bottom of the page! Even more, with this effort we decided to really dig into performance - we conducted A/B testing of that move (establishing performance split testing framework along the way) to see what exactly what effect the move would have on our site’s performance.

    Results

    Starry eyed and excited to see improvements across the board (that file exists on nearly every page of tumblr, after all) we jumped head on into the data to find ourselves amused and slightly disappointed.

    Turns out, the performance gains that we expected to see from moving a heavy file out of a critical rendering path were not there. The 75th percentile of page load times across the board remained the same. We suspected that analyzing on a more granular would reveal performance gains on certain pages or even certain browsers - but results were pretty uniform. See for yourself - below are boxplots for performance on the dashboard:

    image

    Was our approach to taking measurements incorrect? We revisited numbers again and again. We looked into finer detail at various pages. We excluded IE9 after hypothesizing that it might skewing our results. We measured across several metrics that we had at our disposal (we sample actual navigation timings from actual users). Results remained the same.

    Outcome

    As much as we were disappointed, we were also glad that we ended up knowing precisely what effect we had on the performance. Oftentimes we take blind faith that best practices will lead to best results, and fail to realize that myriads of factors weigh in on actual outcomes.

    If anything - we learned a lot from this experience. We gained better insight into our codebase and established an approach to measuring performance in future experiments!

  • Data Lasso

    javascript:

    There I was. Stranded. Alone with my hive query export of many thousands of records, earned by a tireless series of painfully refined select statements, needing to identify what the outliers were in this madness of data.

    “I have data. Now what…Crap.” I mumbled to myself, realizing that I am limited to a few unpromising options.

    Flexing my brain muscle, I tried to recollect bits and pieces I knew on mighty R. “What was the name of the library? ggdraw? ggchart? Dammit it’s ggplot.” Prospect of trying to remember cryptic R work was dooming, weighing heavily on a tired engineer who had enough of suffering for the day.

    Then a shameful thought passed through my mind: “Just open it in MS Excel. No one has to know.” Countless minutes passed as I was still looking at a beach ball of death, spinning as the naive program tried to open my data set and obviously failing.

    Desperation fell on a lonely engineer. There’s got to be a better way. A way to easily visualize the data set as a whole and not needing to write code for it. Just plug in the data set and specify what to show from it. A scalable solution. Firmly deciding that no one should have to be stranded in that situation, a determination came to write a tool that will solve that gap. So was born the Data Lasso.

    Data Lasso

    Data Lasso is a visualization tool that allows exploration of arbitrary set of data in 3D. It is built to be agnostic to the structure and formatting of data.

    There is no setup. You don’t need to prepare your data, .csv, .tsv or .json will do. It can easily visualize half a million entries. All of that in a three dimensional space, so you can have complete freedom of looking at your data.

    image

    Data Lasso can help answer such fundamental questions as:

    • What are the outliers in this multi-dimensional data set?
    • How one dimension of data correlates to another one? Another two?
    • How do you find signal in what is, otherwise, simply noise?

    Under the hood

    Future

    WebGL. The future is upon us, and 3D in a browser is a reality. Using three.js to help wrangle WebGL, Data Lasso benefits from that extra dimension a lot. Rotating, moving and zooming in 3D space gives additional freedom to look at your data closely.

    Data Lasso can visualize around half a million entries - all thanks to shaders that allow to take rendering off the CPU and pass it on to the GPU. Shaders alone might have been the single most important breakthrough for Data Lasso, enabling those smooth 60fps even with large data sets.

    Goes well with your stack

    At it’s core, It is built to be extensible by means of modules, that can hook right into Data Lasso event bus. That allows you to set up data flow out of Data Lasso that is customized for your needs, or add a UI on top of the Data Lasso to be specific to your data.

    Data Lasso can be used standalone, but it was not made into an npm module for no reason - add it to your stack and serve it up from your systems.

    Data Lasso was used inside Tumblr for several months, and shown itself to be an extremely useful visualization tool, filling a big gap in a workflow of working with data.

    Now it’s open sourced. Go check it out.

  • Resizing Gifs On-Demand

    Why dynamically resize images in the first place?

    Before the dawn of on-demand resizing at Tumblr, every posted image was resized into seven or eight different sizes, and each was saved into our backing media store (a massive S3 bucket). This made serving our images very fast—just grab the size you want right from the bucket! While this was great, it also meant that any changes to our image processing would not affect any images we had already saved (billions of images). If we were to upgrade image quality, add a new size crop, or change how we handle taking down media, the effect would only be marginal…what a bummer! The cost of storing all the resizes as separate files (petabytes of data!), along with a lack of agility moving forward, motivated us to adopt a dynamic resizing and serving strategy.

    We began with resizing jpg and png images on-demand instead of persisting each different resize crop in our S3 bucket. This has been a great success; our “Dynamic Image Resizer” churns through over 6,000 images a second, at a roundtrip request latency of only 250ms per image. Not having to store the resizes saves us tens of thousands of dollars a month! So, the natural question was, can we also do this for gifs and make a “Dynamic Gif Resizer?”

    The problem with resizing gifs on-demand

    Gifs, as a medium, are a wonderful thing. They capture a special or hilarious moment and repeat it back to you, forever. However, the actual Graphics Interchange Format leaves much to be desired. Last touched in 1989, the format is woefully outdated, and this begets massive, low quality animated images. When compared to video format counterparts (H.264 and the like), the gif file size can be tens of times larger at similar visual quality. Many companies have punted on the gif file format entirely; imgur released their gifv format, which wraps an mp4 video. Instagram will loop your video clips, but will flatten gifs to a still image. However, as the true “home of the gif,” Tumblr isn’t ever giving up on your gif files!

    Resize it faster

    A while ago, one of my colleagues @dngrm posted about updates we made in our gif resizing technology. Essentially, we switched our gif resizing library from ImageMagick to gifsicle with great success—we got lower latency and higher-quality results. In order to resize a gif in a realistic timeframe for on-demand resizing and serving, we proposed some changes to gifsicle that parallelizes the resizing step. Since a gif is just a stack of image frames, we figured that resizing them using a thread pool could lead to a performance improvement. Luckily for us (and the world!), gifsicle author Eddie Kohler accepted and merged our changes into gifsicle. With this new threaded resize option in gifsicle, we gained about a 1.5-2x speed-up in resizing an average gif against using the vanilla gifsicle. This brought down the average wall time of a gif resize to about 100ms. The entire gif resize request (downloading the upstream image, resizing the gif, and serving the response) is now only 400ms on average.

    Cache is king

    To make all this possible, tumblr heavily relies on CDNs to cache massive amounts of static content and avoid repeated work. Thanks to this, the Dynamic Gif Resizer only gets a little over 1,000 resize requests per second, thanks to an incredibly-high cache hit ratio on our CDNs.

    On top of that, we rely on the usage of conditional GET requests and 304 Not Modified responses to cut down on the amount of real work we must do in the resizing level. The number of 304s we serve fluctuates between 30-50% for all gif responses, which saves us a tremendous amount of compute time!

    Putting it all together

    The resizer itself is an nginx server with a custom module that does the upstreaming and resizing, and is written in C. The jpg/png resizer utilizes OpenCV for image manipulation, while the gif resizer uses the aforementioned gifsicle library.
    Our fleet of resizers and their surrounding architecture are housed in AWS. The main motivation for this was colocation to our image store (S3) and the ability to automatically scale our instance count up and down, depending on time of day (our traffic pattern is heavily cyclic over a 24h window). The rest of tumblr’s architecture is housed in our own DC. Below is a minimalistic diagram of our resizer setup.

    image

    Thanks

    Both the Image Resizer and Gif Resizer were massive undertakings, and a lot of people deserve credit for fantastic work:
    Massive thanks to co-developer @naklin, and to @neerajrajgure who helped with improvements.
    To @dngrm, @michaelbenedict, @heinstrom, @yl3w, and @jeffreyweston for architecture help and sage advice.
    To our AWS enterprise support team Frank Cincotta, Shaun Qualheim, Darrell DeCosta, and Dheeraj Achra.
    And to Eddie Kohler who helped clean up my ugly gifsicle changes and let them be a part of his library.

    Questions? Comments?

    Talk to me on tumblr using our new messaging system! My tumblr is @hashtag-content

    image

    Originally posted by gameraboy

  • awnerd:

    Culture Matters: Casper Suite for People Who Fear Going Corporate

    A few weeks ago, I presented at JNUC 2015 about an emerging approach in IT – a people-centric approach. IT teams have a significant impact on the culture of the organizations we support. Embracing this influence is challenging. But doing so can be transformative.

    We covered a lot of ground:

    • The urgent business case for nurturing individual expression
    • Ways to build trust between IT and other departments
    • How choosing the right words can change how people think (and can be key when needing buy-in from managers and employees)

    Each year, JAMF produces videos of every session from the 3-day JAMF Nation User Conference, and makes them freely available. I was thrilled to be asked to present, and I’m honored to have been chosen as a highlight of this year’s conference.

    Take a look and let me know what you think.

  • awnerd:



  Culture Matters: Casper Suite for People Who Fear Going Corporate
  
  This session will be a journey through Tumblr’s experience of rolling out the Casper Suite at an Internet startup that deeply embraces creativity and individual expression. Andrew will focus on the cultural and technical implications of introducing the Casper Suite to a group of people that have every right to question “corporate” answers to challenging problems. Learn from some of Tumblr’s brightest, and take a peak at what drives their creativity.
  
  Presenter: Andrew Montgomery has been practicing Macintosh system and server administration for SMBs in New York City for over 15 years. He has spent much of that time advocating tirelessly for an approach to IT that grows increasingly aware of the profound impact we have on the culture of the organizations we support —and the exciting opportunities this presents. He has been with Tumblr for half of its existence, growing their fleet of Macs from 80 to more than 400. He supports their corporate IT systems at their headquarters in New York City, and lives in Harlem with his wife, Erin.


The JAMF Nation User Conference is the largest gathering of Apple system administrators in the world.

I’ll be speaking about the incredible power of allowing cultural considerations to drive IT decisions.

    awnerd:

    Culture Matters: Casper Suite for People Who Fear Going Corporate

    This session will be a journey through Tumblr’s experience of rolling out the Casper Suite at an Internet startup that deeply embraces creativity and individual expression. Andrew will focus on the cultural and technical implications of introducing the Casper Suite to a group of people that have every right to question “corporate” answers to challenging problems. Learn from some of Tumblr’s brightest, and take a peak at what drives their creativity.

    Presenter: Andrew Montgomery has been practicing Macintosh system and server administration for SMBs in New York City for over 15 years. He has spent much of that time advocating tirelessly for an approach to IT that grows increasingly aware of the profound impact we have on the culture of the organizations we support —and the exciting opportunities this presents. He has been with Tumblr for half of its existence, growing their fleet of Macs from 80 to more than 400. He supports their corporate IT systems at their headquarters in New York City, and lives in Harlem with his wife, Erin.

    The JAMF Nation User Conference is the largest gathering of Apple system administrators in the world.

    I’ll be speaking about the incredible power of allowing cultural considerations to drive IT decisions.

  • Hi, I’m Michelle and I was a product engineering intern at Tumblr this summer. I worked with the ever so awesome JazzFalcons, whose focus is on growth and retention of the user base. For the first few weeks of the internship, I spent my time familiarizing myself with the code base by doing lots of bug fixes. Soon after that I worked on the tumblr logo animation that was for pride weekend. That was really cool because it was the first time the letters were animated along with the dot. My next project was working on the new user sign-up process. Tag Bingo is what the new users go through to help them pick blogs to follow and populate their dashboard. I helped implement the blog card carousel display Working on this project gave me the chance to really focus on how to make the on boarding process for a new user as seamless and intuitive as possible. And for my final project I worked with the Scala team and their intern Brendan to build a web interface for their service deploy tool. Originally they were using the command line to deploy their services to different machines, but they wanted a UI where they can both create and deploy these services.As a long time user, this was my dream job. I’m I was treated like an actual member of the team and the projects that I worked on went live. I’m forever honored and thankful that I got to work with the really smart and amazing people at Tumblr. This summer was everything I could ever ask for and more.This post fell through the cracks at the end of last summer somehow and didn’t make it to the site. As sad as that is, there is a happy end to this story: Michelle recently joined Tumblr as a full-time engineer!

    Hi, I’m Michelle and I was a product engineering intern at Tumblr this summer. I worked with the ever so awesome JazzFalcons, whose focus is on growth and retention of the user base. For the first few weeks of the internship, I spent my time familiarizing myself with the code base by doing lots of bug fixes. Soon after that I worked on the tumblr logo animation that was for pride weekend. That was really cool because it was the first time the letters were animated along with the dot.

    My next project was working on the new user sign-up process. Tag Bingo is what the new users go through to help them pick blogs to follow and populate their dashboard. I helped implement the blog card carousel display Working on this project gave me the chance to really focus on how to make the on boarding process for a new user as seamless and intuitive as possible.

    And for my final project I worked with the Scala team and their intern Brendan to build a web interface for their service deploy tool. Originally they were using the command line to deploy their services to different machines, but they wanted a UI where they can both create and deploy these services.

    As a long time user, this was my dream job. I’m I was treated like an actual member of the team and the projects that I worked on went live. I’m forever honored and thankful that I got to work with the really smart and amazing people at Tumblr. This summer was everything I could ever ask for and more.


    This post fell through the cracks at the end of last summer somehow and didn’t make it to the site. As sad as that is, there is a happy end to this story: Michelle recently joined Tumblr as a full-time engineer!

  • Backboard

    We’ve open sourced Backboard, a motion-driven animation framework for Android. It’s built on top of Rebound, and makes it easier to animate views based on user interaction. We use Backboard in the Tumblr Android app to make our app more responsive and fun.

  • Adventures in making a CocoaPods Plugin

    brianmichel:

    So, there I was, trying to comprehend why I couldn’t grab some dependencies that had been resolved by including a specific pod only to realize that it was a problem with not being able to clone in a specific way. In my case it was a specification that had a source URL over SSH as opposed to HTTP. An hour or two later I figured out a workaround by replacing the specific source URL with an HTTPS URL instead of using the specified git URL. Meaning I would transform something like this:

    git@cool-git-server.com:Organization/repo.git

    into something like this:

    https://access-token-here@cool-git-server.com/Organization/repo.git

    Let’s talk about this transformed structure for a second first. It’s almost a standard https URL that you’ve seen dozens of time on GitHub and other places. But what’s this access-token-here bit? It’s a GitHub personal access token!1

    We now know the structure of the transform we need to create, all we need is a way for us to tell CocoaPods to run code that will perform the transform. We’re in luck, since CocoaPods 0.28, there has been a plugins interface! Check out their blog announcement here.

    After reading the blog post, I was sure this was the right way forward, and there were even linked examples, and an informal interface, which was all good stuff. The following made me very happy to see they had thought about this.

    Plugin support allows one to tweak the internals of CocoaPods and to insert additional commands.

    Here’s where things get a bit murky. The listed example only covers 1/3rd of the extensibility options offered by this plugin interface. If you need to make a plugin that patches internals (dangerous), or uses a pre or post-install hook (a sanctioned CocoaPods interface) you’re on your own. Thankfully, due to the informal interface conventions, every other plugin must begin with cocoapods- so let’s take a look at RubyGems.org and find some other plugins.

    I ended up checking out the cocoapods-src plugin which will checkout all of the sources for a pod after you’ve finished running pod install. I chose this because it utilized a post-install hook as it’s trigger, and figured this would closely mirror what a pre-install hook plugin would do. Here’s the bit of code in the gem that helped me get going in the right direction:

    Pod::HooksManager.register(:post_install) do |installer_context|
      Pod::Src::Downloader.new(installer_context).download
    end
    

    Great, I’m off and running by changing that to this:

    Pod::HooksManager.register(:pre_install) do |installer_context|
        configuration = SourceURLRewriter::Configuration.new(installer_context)
        SourceURLRewriter::Rewriter.new(installer_context, configuration).rewrite!
    end
    

    I installed my gem locally, ran pod install, and…nothing happened.

    At this point the effort of this project hockeysticked. Without much documentation to go off of, I setup my machine as if I was going to contribute to the CocoaPods project (instructions here). My plan was to search through the code for some symbols I’d been using such as :pre_install and work my way through what was going on.

    I found that, in Pod::HooksManager::Hook, the use of hooks without specifying a plugin name has been deprecated. Okay, so now I just need to specify a name for this plugin, no problem. In the same file there is the signature for the register function. I can use that to figure out how to call this function with a name:

    Pod::HooksManager.register('url_rewriter_plugin', :pre_install) do |installer_context|
        configuration = SourceURLRewriter::Configuration.new(installer_context)
        SourceURLRewriter::Rewriter.new(installer_context, configuration).rewrite!
    end
    

    I rebuilt my gem, installed it locally, ran pod install, and…nothing happened, again.

    At this point, I’m losing the desire to even want to try and build a plugin, and start entertaining the idea in my mind that I can figure out a different way to solve this problem.

    After reading more of the source, and reaching out for help from the masterful @segiddins, he explained that the name of plugin you register must be exactly the same as the gem you publish as your plugin. In retrospect this totally makes sense, and I don’t know of another way this could work.

    So I renamed the pre-installation hook that I’m registering, built my gem, installed it locally, and ran pod install, and…stuff happened!

    However, at this point I realized that my problem was a bit deeper than simply grabbing URLs of the dependencies specified in my Podfile and changing them. Since after dependency resolution there are likely many other dependencies that you need to build, but are not directly specified, I would need to find a way to rewrite these URLs at the time of downloading vs. just changing them in memory on the Podfile object.

    WARNING: This gem was designed to be a stop gap solution and monkey patching is dangerous, YMMV!

    I worked with Sam a bit more, and we decided that a solution, albeit a dangerous one, would be to patch the Git downloader object in CocoaPods. Since this was designed to be a temporary solution, I felt comfortable doing this and then working other channels to resolve the actual network issue. Ultimately, I ended up with a CocoaPods plugin that leveraged something like the following code in my cocoapods_plugin.rb file to deliver a solution:

    module Pod
      module Downloader
        # Reopening the Git downloader class
        class Git
          alias_method :git_url_rewriter_url, :url
          def url
            source_url_rewriter.url_for(git_url_rewriter_url)
          end
    
          def source_url_rewriter
            @source_url_rewriter ||= SourceURLRewriter::Rewriter.new
          end
        end
      end
    end
    

    I can’t stress how dangerous this is, and ultimately not guaranteed to last forever. I am willingly taking advantage of the nature of Ruby to add behavior into the CocoaPods library that suits my needs in the short term. The decision to patch any library in this fashion should not be taken lightly with the expectation that this will break in the future.

    Now that I had this working, I wanted to be able to provide these rewrite options within my plugin declaration so that it was highly visible in the Podfile what was going to happen. For this I looked at the cocoapods-keys plugin to see how they configured their plugin. Our goal is to be able to provide syntax like this:

    plugin 'cocoapods-git_url_rewriter', {
        'git@cool-git-server.com:' => 'https://access-token-here@cool-git-server.com/'
    }
    

    Which leads us towards an internal method that looks like this:

    def user_options
      @options ||= podfile.plugins['cocoapods-git_url_rewriter']
      Pod::UI.notice('No options have been specified for rewriting') unless @options
      @options
    end
    

    On the Podfile object, there exists a hash of plugin which can be referenced by the name of your plugin. The value returned from this is the hash you passed in when you declared the plugin usage in your Podfile. This allows us to keep sensitive info out of the plugin, while being very descriptive of what will happen to our dependencies.

    Once this had been done, I could test my gem and see that the resources I previously was unable to reach are now being resolved, and downloaded correctly. Additionally, I was able to provide a simple solution that was transparent to my fellow developers and extensible for future use (but hopefully not needed). I ran into some problems that I feel like others developing plugins might hit which is why I felt the need to document them here.

    Please feel free to reach out if you have questions I’m @brianmichel and I couldn’t have done this without help from the awesome @segiddins, seriously, he’s great.


    1. Personal access tokens are a solution provided by GitHub and GitHub Enterprise that allow you to specify them instead of OAuth tokens or the basic username:password combination which will grant the caller access to a specific resource. These tokens are the kind of thing that you create once, get one time to look at it, and then it just becomes an opaque resource on your account. Additionally, they can be configured with different limitations (check out the options in the screenshot).

  • Functions As Factories

    cocoa:

    Factories are a fairly well understood design pattern in software development.  The benefits of using factories include:

    1. Abstracting constructors away from clients.
    2. Encapsulating data that clients do not need to know about.
    3. Allowing for more testable code by enforcing the idea of passing objects into initializers instead of referencing singletons directly.

    This post will show the power of a few Swift features as well as of first­class functions.

    Consider a setup like this in a piece of software that needs to send network requests:

    struct APIURL {
        let APICreds: APICredentials
        let baseURL: String
        let route: String
        let queryParams: [String: String]
        
        func URLString() -> String {
            func finalQueryParams() -> [String : String] {
               return .....
            }
            return encodedURLWithQueryParams(baseURL + route, params: finalQueryParams())
        }
        
        private func encodedURLWithQueryParams(URL: String, params: [String: String]) -> String{
            return ....
        }
    }
    struct APINetworkRequest {
        let userCredentials: UserCredentials
        let URL: APIURL
        
        private func sign() {
            // Sign the request using the user’s credentials
        }
    }

    Many parts of your application will need to create requests for different routes, and as such will need to create new APINetworkRequest instances (and therefore, new APIURLs). In order to do this, these parts of your application will need to reference the API credentials, the application API credentials, the base URL, and query parameters. We can pass these items from wherever we first create them (maybe in application(application:, didFinishLaunchingWithOptions:) or after a login screen) to whichever objects need them, or we can create globally accessible singletons that other parts of our codebase can access. 

    Unfortunately, these solutions are both sub par; passing this many values throughout our codebase via initializers is cumbersome, and spreading knowledge of singletons throughout the application is to tightly couple, making testing very difficult.

    A solution to this could be a factory that can construct an APINetworkRequest for each route, which might look something like this:

    struct APINetworkRequestFactory {
        let userCredentials: UserCredentials
        let APICreds: APICredentials
        let baseURL: String
        
        func secretAdminInfo(adminId: String) -> APINetworkRequest {
            return APINetworkRequest(userCredentials: userCredentials,
                URL: APIURL(APICreds: APICreds, baseURL: baseURL, route: "sekret/info/admin", queryParams: ["id" : adminId]))
        }
        
        func search(query: String) -> APINetworkRequest {
            return APINetworkRequest(userCredentials: userCredentials,
                URL: APIURL(APICreds: APICreds, baseURL: baseURL, route: "search", queryParams: ["query" : query]))
        }
        
        func translate(term: String, toLanguage: String) -> APIURL {
            return APIURL(APICreds: APICreds, baseURL: baseURL, route: "translate", queryParams: ["term" : term, "to_language" : toLanguage])
        }
        
        func allData() -> APIURL {
            return APIURL(APICreds: APICreds, baseURL: baseURL, route: "alldata", queryParams: [String: String]())
        }
    }

    Instead of passing four or more values through each initializer, we can just pass around this factory and use it to conveniently construct APINetworkRequests for any route.

    Could we make this even more painless, however? Can we achieve the same benefit by using a function as our “factory”?

    We can!

    Let’s see what a function that can create all of these routes might look like. First, let’s create an enum to denote which route we want to construct a request for:

    enum Route {
        case SecretAdminInfo
        case Search
        case Translate
        case AllData
    }

    Now, we can create a function that returns an APINetworkRequest, with a signature that looks something like this:

    func APINetworkFactoryFunction(APICreds: APICredentials, userCredentials: UserCredentials, baseURL: String, route: Route) -> APINetworkRequest

    Since Swift has first class functions, this function can be passed as a method argument in the same way that we would’ve otherwise passed an instance of our custom factory class.

    Inside this APINetworkFactoryFunction function we can simply use a switch statement to switch on the route argument to construct an APINetworkRequest for each API route just like the APINetworkRequestFactory did.

    There is one problem, however, as you can tell by looking at different method signatures on APINetworkRequestFactory: different routes take different arguments. This makes sense since they are used to perform very different tasks and, as such, require different inputs. With this enum in its current state, we cannot create the variety of fully constructed network requests that our application needs.

    One solution to this is to use a feature of Swift Enumerations called Associated Values. We can re-declare our Route enum to look like this:

    enum Route {
        case SecretAdminInfo(adminId: String)
        case Search(query: String)
        case Translate(term: String, toLanguage: String)
        case AllData()
    }

    Now, the switch statement inside of our APINetworkFactoryFunction can use these values in order to create our APINetworkRequests. The APINetworkFactoryFunction implementation could look something like this:

    func APINetworkFactoryFunction(APICreds: APICredentials, userCredentials: UserCredentials, baseURL: String, route: Route) -> APINetworkRequest {
        switch route {
        case let .SecretAdminInfo(adminId):
            // Create and return an APINetworkRequest using adminId
        case let .Search(query):
            // Create and return an APINetworkRequest using query
        case let .Translate(term, toLanguage):
            // Create and return an APINetworkRequest using term and toLanguage
        case let .AllData():
            // Create and return an APINetworkRequest
        }
    }

    So now we can create the proper APINetworkRequests just like our original APINetworkRequestFactory did. However, invoking this function requires having all of it’s arguments at the call site. Our goal with introducing the factory in the first place was to avoid passing these arguments throughout the codebase. We really want to pass a function around that takes only a Route and returns a APINetworkRequest that already knows around all the other arguments. This is a great example of where function currying can be used. Swift provides special syntax for creating curried functions, which we will use.

    So we can turn the function signature we had before into this:

    func APINeworkFactoryFunction(APICreds: APICredentials, userCredentials: UserCredentials, baseURL: String)(route: Route) -> APINetworkRequest

    Notice how the last argument is in its own set of parenthesis. APINetworkFactoryFunction’s type is now (APICredentials, UserCredentials, String) -> (route: Route) -> APINetworkRequest – a function that returns a function that returns an APINetworkRequest.

    For example, if we wanted only a function that takes a Route (a function of type (route: Route) -> APINetworkRequest) we can create one by currying our original function:

    let functionToPassToClients = APINeworkFactoryFunction(APICredentials(key: "key", secret: "sekret"), UserCredentials(token: "token", tokenSecret: "tokenSekret"), "https://tumblr.com")

    This is very powerful, as the function now provides exactly the same functionality as the APINetworkRequestFactory. But can this be even more flexible?

    Let’s say you don’t yet have all of the inputs needed to create a new Route instance; you can instead create just a function that takes a Route in parts. For example you might have the application credentials at launch but you don’t have the user credentials until after login. We can solve this by further currying our function so that you pass arguments in one at a time, like so:

    func APINeworkFactoryFunction(APICreds: APICredentials)(userCredentials: UserCredentials)(baseURL: String)(route: Route) -> APINetworkRequest

    You can set the API credentials first:

    let factoryFunction = APINeworkFactoryFunction(APICredentials(key: "key", secret: "sekret"))

    and then when you get the user’s credentials you can ‘add’ them in:

    let factoryFunctionIncludingUserCredentials = factoryFunction(userCredentials: UserCredentials(token: "token", tokenSecret: "tokenSekret"))

    Finally, when you’re ready to to pass a route-creating function to clients, you can pass the function returned by this call:

    let functionToPassToClients = factoryFunctionIncludingUserCredentials(baseURL: "https://tumblr.com")

    Why might having your factory be a curried function be more desirable than having a struct with functions for each route?

    1. Function currying makes it harder/impossible for developers in the future to make private fields public or mutable, since the inputs are frozen within the function and can not be changed or exposed.
    2. Function currying allows for partial creation of the factory without having to have an intermediary, mutable object. 

    This is not to claim that this is the best ​way to use factories in Swift; this is just an example of how factories, first­class functions, function currying, and Swift enumerations with associated values can be used to solve real world problems.

  • Bookends and Remember

    We’ve open-sourced a couple of Android utilities that we use in the Tumblr app for Android. Check it out:

    Bookends

    A UI widget that allows for headers and footers on lists backed by RecyclerView.

    As we were upgrading our app to migrate from ListView to RecyclerView, we found it kind of silly that RecyclerView doesn’t support headers by default. So we built a little wrapper that’ll do this for you.

    Remember

    An in-memory data store backed by shared preferences.

    SharedPreferences are useful but since they’re backed by disk, they can have unpredictable performance characteristics – you’re not guaranteed to always be in memory, and in the case of write operations, you have to hit disk (possibly asynchronously) and remember what you wrote.

    Remember takes care of that by putting a write-through cache in front of SharedPreferences. It also gives you a bunch of desirable consistency and concurrency characteristics – access can happen from any number of threads concurrently, and doing a write followed by a read will always return the value you just put. (Even if the value hasn’t been written to disk yet).


    Both of these projects are open-sourced under the Apache license, and are available at our Github page. Let us know what you think!