Alex Kahn Blog

Gotham Ruby Conference 2010: A Recap

2010-05-26T00:00:00-04:00

Over the weekend I attended the fourth Gotham Ruby Conference (GoRuCo). The conference was well-organized, the content was top-notch and there were tons of great people there.

Photo by sd

The day opened with a welcome from a Pace University’s Jonathan Hill. Jonathan, who is assistant dean of Pace’s Computer Science program, described the longstanding relationship between Pace and the New York Ruby community, pointing out that GoRuCo has been held at Pace since its first year. Pace uses Ruby and Rails extensively in its Computer Science program and, in turn, its graduates often find employment through this community. Jonathan expressed real appreciation for the health and energy of the Ruby community in New York. It was neat to feel this connection and know that Pace was more than just a rented space for the conference.

Nick Gauthier presented the first talk, Grease Your Suite, on making slow test suites run faster. This talk demonstrated how he improved a test suite in a Rails project from 13 minutes down to under 20 seconds. To do this, Nick used lifo’s fast_context, his own Hydra for distributing tests across CPU cores or machines, and Ruby Enterprise Edition for its ability to tune Ruby’s garbage collection. He also gained speed by mocking PaperClip’s image manipulation, using fixtures to replace truly-resized images. Nick got further performance by tweaking a journaling setting for the ext4 filesystem (Nick is a staunch Linux-on-the-desktop user), as well as by disabling atime. Nick noted that the improvements from these tweaks gain as much in speed as running tests on a solid state drive. With such a fast test suite, the Continuous Integration server can return to its true calling—ensuring a green build in a production-like environment. While I don’t have any projects that are burdened by tests that take too long, I nonetheless left this talk with strategies to grease my suite. Get all the details in Nick’s slides.

The next talk, by Aman Gupta, was on a new Ruby profiling tool, Memprof. Aman provided a tour of the tools he’d worked with for profiling Ruby’s memory use: ObjectSpace.each_object, gdb.rb, bleak_house, and patching Ruby in order for it to provide heap dumps. These tools are all helpful, he said, but they require writing patching or running special versions of Ruby, or don’t provide complete enough information on the contents of the objects in memory or their context in code. Memprof, created by Joe Damato with contributions from Aman, solves the above problems: Installing it is simple (gem install memprof), use is simple (Memprof.dump_all, for example), it outputs to JSON, and provides all the necessary information and context about objects in memory, including references between objects. Memprof’s JSON output makes it easy to dump its output into a database like MongoDB, which memprof.com does. Aman’s talk struck at just the right level: deep enough to keep us interested, but not so deep as to lose the audience. And if you’ve ever checked out Aman and Joe’s blog you’ll know how deep this this rabbit hole goes. My thanks to Aman for that.

Managing Rubyists, a talk by Luke Melia, was directed at Ruby developers who manage a team. This was the only non-technical talk of the day and it was one of the highlights for me. In the talk, Luke laid out 15 principles or techniques for managers to follow for their team and its members to achieve happiness, motivation, productivity and growth. I was blown away by the care Luke took with this subject and the regard he has for the developers he works with. Luke takes his craft as a manager as seriously as Rubyists take their code. Check out Luke’s slides and notes for the details. If you’re a manager, be sure to check out Luke’s recommended reading.

After lunch, Chris Williams (of JSConf fame) presented Rails Caching à la JavaScript. In his talk, Chris made the case a straightforward approach to working around performance issues with Rails: using page caching to avoid the Rails stack altogether for most requests, and making JSONP AJAX requests to retrieve the personalized aspects of a page. Even though this AJAX request is handled by Rails, its rendering overhead is offloaded to the client (Chris recommends mustache.js for client-side rendering). Despite Chris’ preface that the technique described is not ideal for most Rails applications, the talk drew skepticism and ire from the audience. Nonetheless, the talk was enjoyable, humorous, and made a strong argument for a straightforward caching strategy.

Next, Alex MacCaw showcased Bowline (pronounced “bowlin’” like the knot), a new framework for creating cross platform desktop apps using Ruby, JavaScript and WebKit. Alex gave a tour of developing an app with Bowline, showing off its in-memory model layer, SuperModel, its bindings between Ruby and JavaScript and neat features such as seamless background updates for installed applications and synchronizing data across remote client apps. In all, Bowline looks to be a very ambitious and promising project.

Paul Dix, author of the forthcoming Service Oriented Architecture with Ruby on Rails, presented on how to create web service clients using ActiveModel to scale an app. This is not scaling an app for traffic, but how to scaling as complexity and team size grow. Paul demonstrated how to leverage ActiveModel to implement client models with validations, dirty tracking, callbacks and even persistence. If you are struggling to deal with complexity in a large application and to abstract components into separate services, Paul offers an excellent approach. Definitely look out for his book this August.

The last talk of the day, Scaling to Hundreds of Millions of Requests, was presented by James Golick. In this talk, James outlined how he improved performance and scaled a very active community site. James shared his experience moving from Amazon EC2 to physical hardware and the immense performance increase this brought. James also went over his trials with MongoDB and using MySQL “non-relationally” with Friendly before finally settling on Cassandra. As James argued in a recent blog post Cassandra is one of the NoSQL databases that truly scales horizontally. Look for James speaking further on this topic at Railsconf.

The final hour of the conference comprised of “Rejectconf,” a round of lightning talks and a chance to speak for people whose talks weren’t accepted for the conference program. Some highlights were Pat Nakajima presenting his Bookmarklet Factory (“how to downvote 100 Gizmodo comments”); Edward Ocampo-Gooding presenting on open data in Ottawa; Aaron Quint presenting on Jim, a JavaScript manager, bundler and compressor; and Michael Bernstein presenting on Ohm, the Redis object mapper with an extensive contrib collection. Then it was beer time.

In all it was a great conference. Thanks to the conference organizers and hello to all the folks I met. And hello to all the folks I didn’t get a chance to meet but wish I had. Keep your eyes peeled on the conference site; videos will be posted in the next two weeks.

Up and Running With LaTeX

2010-05-13T00:00:00-04:00

Recently I’ve had the pleasure of getting acquainted with LaTeX. For those that don’t know, LaTeX is a typesetting system that turns marked-up text files into precise, readable, print-ready documents. LaTeX is specialized for creating technical documents containing math and equations, but it’s also well-suited for any structured document. LaTeX is not just the command-line tool for procesing .tex files, though—it’s an entire ecosystem consisting of libraries, packages and tools for myriad purposes. The size of this ecosystem is staggering. At times, it’s frustrating. But I’ve grown to appreciate it the more I’ve learned.

What’s so great about LaTeX? I love that it demands explicitness. It doesn’t try to guess what you mean. Want a hyphen? Use -. Want a slightly longer dash, suitable for numerical ranges? That’s --. Using a dash for punctuation? --- inserts an em dash. The same explicitness applies to quotes. How does LaTeX know what is an opening quote and what is a closing quote? You tell it. Use two backticks (``) for an opening quote and two apostrophes for a closing quote (''). These are two examples of the attention to detail that LaTeX promotes.

Some things are more automatic. Hyphenation, for example, is handled by LaTeX. And it’s smart.¹ The system sometimes misses possible hyphenation points, but it’s very good at avoiding incorrect hyphenation, such as “new-spapers.” It also allows manual control of hyphenation. For example, put \- in the middle of a word to identify a possible hyphenation point. I also found some of these line breaking commands helpful.

LaTeX excels at working with tables, references, bibliographies, footnotes, indexes (and math, of course)—features whose use ranges from tedious to hellish in word processing software. Perhaps its greatest strength is that unlike word processing software, LaTeX forces authors to focus on the content and structure of a document, rather than its presentation. LaTeX takes care of what it looks like. It’s the anti-WYSIWYG.

Installing LaTeX

The easiest way to get started with LaTeX on Mac OS X is to install MacTeX. Once you have MacTeX installed, take a look in /usr/local/texlive (MacTeX is a specialized version of the TeX Live distribution) to see what was installed. You’ll want to add /usr/local/texlive/2009/bin/universal-darwin to your $PATH. Also make note of /usr/local/texlive/2009/texmf-local/tex/latex—this is where you’ll install additional LaTeX packages.

The Memoir Class

You can get far with LaTeX’s built in document classes: book, report, article, etc. But for my document, I needed a bit more flexibility. I turned to the Memoir document class. The goal of this class is to combine LaTeX’s built-in book class with some of the more common packages that help customize the design of a document. The result is a large class, but one that offers convenience, since it reduces the need for installing additional packages.

It’s difficult to summarize what Memoir offers because it does so much. Instead, I’ll just share my own experience: Memoir made laying out my document a breeze. I continually got the impression that the creators of Memoir had thought of everything. Refer to Memoir’s exhaustive, 500-page manual (PDF) to learn about what it offers. It’s epic.

To install Memoir (or any document class or package), download it from CTAN, compile it using the instructions in the README, and then copy the entire memoir directory to the installation directory mentioned above. Then, after running texhash, you should be able to compile documents using the Memoir class.

Printing 2-Up

The document I was preparing was a long packet of reading. I wanted to print this document with two pages per sheet, as a photocopied packet from a book would be layed out. Although there are ways to bend LaTeX’s strong desire to print one logical page per physical sheet of paper (among these are booklet and twoup), a more prudent alternative is pdfpages. pdfpages is a package for embedding—you guessed it—PDF pages into LaTeX documents. Conveniently, it has the ability to embed multiple pages of a PDF document into a single page of a LaTeX document. The end result: multiple logical pages per physical sheet. After producing a PDF of your original document, create another LaTeX document which embeds the compiled PDF, manipulating it to render two PDF pages on one LaTeX page. Some code might make this clearer:

\documentclass[landscape]{article}
\usepackage{pdfpages}

\begin{document}
\includepdf[nup=2x1,pages=-]{my-document.pdf}
\end{document}

The document class used here is irrelevant, but it’s important to set landscape orientation. The interesting line here is the \includepdf directive. The nup=2x1 tells pdfpages to place two pages from the PDF file side-by-side in our document. pages=- tells pdfpages to include all pages of the PDF file. When you compile this document, the output contains two logical pages laid out on a landscape-oriented physical sheet.

For optimal 2-up printing, you’ll need to adjust your original document’s margins. LaTeX assumes that you are printing a book, and thus need a large inner margin (where the binding would be) and a smaller outer margin. Memoir makes it easy to override these defaults:

% Set inner and outer margins to .5 and 1 inch, respectively
\setlrmarginsandblock{.5in}{1in}{*}
% Set top and bottom margins to .8 inches
\setulmarginsandblock{.8in}{.8in}{*}
% The {*} parameter tells LaTeX to use the measurements
% provided rather than attempt to calculate a ratio

pdfpages has numerous options for arranging booklet-style documents, which you can read about in its manual (PDF). Get the package here.

Better Typography

One point of frustration I had with LaTeX was its font handling. The routine for preparing a typeface to be used with LaTeX was so elaborate that it scared me away. But then I came across XeLaTeX, a project that brings modern, convenient typography to LaTeX.

Using XeLaTeX solved two problems for me. One was using Unicode characters in my documents. For some reason, LaTeX doesn’t support this by default, requiring external packages for this purpose. Using XeLaTeX, it was as simple as using Unicode characters in an HTML document. XeLaTeX also makes it dropdead simple to use any OpenType or TrueType typeface installed on your system. Replacing LaTeX’s handsome but staid Computer Modern with the dignified, refined Baskerville was as simple as \setmainfont[Mapping=tex-text]{Baskerville}.

Editing LaTeX in Vim

The Latex-Suite plugin for Vim is a lifesaver. It provides syntax highlighting, folding (handy for hiding sections or chapters you’re not currently working on) and numerous mappings that I haven’t taken the time to learn. One exception is the mapping for compiling. To compile your document to PDF, use ,ll. The compilation log is displayed in a preview window, and any compilation errors pop up in a QuickFix window. To set Latex-Suite to use xelatex to compile your document, add let g:Tex_CompileRule_pdf = 'xelatex -interaction=nonstopmode $*' to your .vimrc.

Notes on Redis: Data Modeling, Hashes, and Namespaces

2010-03-30T00:00:00-04:00

What is Redis?

Redis is a simple database. It stores its data in memory and saves a snapshot to disk in the background periodically. It can be used as a simple key-value store like Memcached, but it also supports more complex data types such as lists, sets, sorted sets and hashes. These will come in handy. Redis is versatile and is used for a plethora of tasks: as a job queue, as a cache store, for tracking downloads of Ruby Gems, for storing A/B testing results, and even as the backend for a chat server.

Redis has been receiving a lot of praise lately and is the subject of lot of excitement online. This praise is deserved – it’s an excellent piece of software. I’ve had the pleasure of using Redis recently and here are some of my findings.

Our Use Case

On mediaFAIL, we wanted to track how many mentions a fail receives on Twitter. We wanted to show a count of a fail’s Twitter mentions and roll that number into a fail’s votes – this feature lies somewhere between act.ly and Tweetmeme. Couldn’t we just store each tweet in our votes table? Perhaps. But this would break our “user has many fails through votes” association. How do I store this data in Redis?

Data Modeling

My first pass at the problem took advantage Redis’s sets. For each fail that was mentioned on Twitter, I created a set containing the users that mention a fail. So, key fail:1:tweets contained the set ['alexanderkahn', 'levjoy', 'TimKarr']. Since elements in sets must be unique, I wouldn’t have to worry about a user making multiple mentions of a fail getting counted more than once. This was an elegant solution: it allowed looking up the number of mentions a fail has received using SLEN (set length) and retrieving all the Twitter usernames for a fail using SMEMBERS (set members).

As I continued work on this feature, I realized that I wanted to link to the specific tweet where a Twitter user mentioned a fail. One way to do that would be to set a string key for each member of a set that corresponds to a member of the tweet:1:tweets set. So, for the above example, I would also set fail:1:tweet:alexanderkahn to “12330508057”, the tweet ID for my mention of the fail. But this would cause a lot of keys to be created and mean that looking up usernames and tweet IDs for a fail would require a great deal of queries for a popular fail. Thanks to Redis’s new hash data structure, there was a cleaner way. In a hash, I can store both the username and the tweet ID under the same fail:1:tweets key. The hash for this key would look like {"alexanderkahn" => "12330508057", "levjoy" => "12330508067", "TimKarr" => "12330518057"}. Since hash keys have to be unique, a Twitter mention can’t be counted twice, just like with a set. Now I can look up how many mentions a fail has received with HLEN (hash length) and fetch both usernames an tweet IDS with one HGETALL (hash get all) query. Groovy.

Redis and Ruby

The Ruby library for Redis, written by Ezra Zygmuntowicz, is a pleasure to use. It has certain touches that make it feel like Redis was born for Ruby. One example of this quality is how it handles Redis’s MULTI/EXEC transactions:

r = Redis.new
r.multi do
  r.set 'foo', 'bar'
  r.incr 'baz'
end

In the above code, if an exception is raised during the execution of the block, none of the operations inside are committed. This is just a basic transaction like ActiveRecord provides when working with a SQL database, but is neatly implemented in idiomatic Ruby. The block returns the result of each Redis operation in an array.

Another nice Ruby touch is the way the library translates a Redis hashes into a Ruby hashes. When retrieving the entire contents of a hash using HGETALL, rather than returning a flat list (1. key, 2.value, 3. secondkey, 4. secondvalue) as the Redis internal protocol does, the library turns this list into a Ruby hash using Hash::[]. I had never run into this way of creating a Ruby hash out of an array, but its use here struck me as clever. Check out ri Hash::[] for more on this.

Another good project for working with Redis in Ruby is redis-namespace by Chris Wanstrath. This library helps you compartmentalize your Redis keys to keep different sets of data separate. We’re using this to keep tweets, described above, separate from our A/B testing data. To use it, you just interact with a Redis::Namespace object rather than a Redis::Client object.

redis = Redis.new
namespaced = Redis::Namespace.new(:my_feature, :redis => redis)

# Interact with namespaced Redis as you would a normal Redis client:
namespaced.set "foo", "bar"
namespaced.get "foo"
# => 'bar'
namespaced.keys "*"
# => ['foo']

# The actual key in Redis is prefixed with our namespace:
redis.keys "*"
# => ['my_feature:foo']

Nice of Chris to take care of that for us, no?

One last note. How am I accessing Redis from within my Rails app? Currently, I’m taking a page out of Nick Quaranto’s book and setting a global variable (in environment.rb):

$redis = Redis.new # Or a namespaced equivalent

This variable provides access to Redis from anywhere in the application. This is not necessarily the best way, but for now it works. I’m happy to hear how I could improve upon this.

If you haven’t had a chance to explore Redis yet, I recommend doing so. I hope this post helps push you in the right direction. Some handy resources will be the Redis command reference, this PDF cheat sheet, and following @antirez, Redis’s author, on Twitter.

A Sinatra App Wrapping Flickr, Plus HTTP Caching and Heroku

2010-02-22T00:00:00-05:00

In the last several months, Esther has produced a large number of monotype prints at a local printmaking studio. She’s been sharing her work online by scanning it and posting it to Flickr. Recently she experimented with using Tumblr to create a stylish gallery of this work. But this process — scanning, adding metadata, uploading to Flickr, followed by posting each piece to Tumblr — proved to be laborious. So we sought to streamline this workflow, hoping to eliminate a step or two.

Titles, descriptions, tags and sets are all easy to manage on Flickr. What if there was a way to take advantage of Flickr’s great management interface, taking images and metadata and putting them on an external site? I took on this problem with a Sinatra application that wraps the Flickr API in a warm embrace. The app makes an API request to Flickr asking for all the images from Esther’s printmaking photoset and renders them as thumbnails, each image linking to a page for that image. On the individual image page, the app requests information on an image – title, description, and the next and previous image – and displays this information along with a larger image of the piece. Now, Flickr itself is the CMS and the gallery is just a pretty shell for information already stored on Flickr. No more managing metadata in multiple places. Have a look at the site.

What about performance? Isn’t it slow to make an API request to Flickr for every visit to the site? This is a definite concern: making an API call to Flickr and then responding takes the application about 500ms – respectable but not exactly speedy. But the content of the site is basically static. Once an image is uploaded and described on Flickr, it is unlikely to change. This means I can cache the entire page with a gateway cache¹. To do this, I set Cache-Control headers that say that my pages are allowed to be cached for long time, and the cache, Varnish, serves up a static representation of the page that the app produced. Once a page has been requested once, subsequent requests for that page are lightning-fast – under 100ms. After six hours, the cached representation of the homepage is considered stale, and a new one will be generated. After two days, the cached representation of an individual page is considered stale. This way, updates to the photoset or changes to images do eventually get served to visitors.

I should note that I’m hosting this site on Heroku. It’s been said elsewhere and should perhaps be obvious by now, but I’ll say it again: Heroku is awesome. Their use of Varnish made it so easy to make this gallery app fast. Deploying code with just a git push is seriously great. I recommend hosting an app on Heroku if you haven’t before. It’s easy and, for a simple app like this one, is free.

1: For more on HTTP caching, see Ryan Tomayko’s excellent article Things Caches Do

FakeWeb and Regular Expressions

2009-12-16T00:00:00-05:00

FakeWeb is a great way to speed up your tests and remove dependencies on external web services. Using FakeWeb is simple: specify a URL for FakeWeb to respond to, and give it the response you expect from the request. For example:

FakeWeb.register_uri(:get, "http://www.google.com", :body => "Stubbed Google!")
HTTParty.get("http://www.google.com") # => "Stubbed Google!"

Now FakeWeb will prevent your code from making an actual HTTP request and instead it will return the response you specified. But what if your application makes requests to URLs with query parameters? It would be tedious to register each URL with FakeWeb:

FakeWeb.register_uri(:get, "http://www.google.com/search?q=fakeweb", :body => "Stubbed Google!")
FakeWeb.register_uri(:get, "http://www.google.com/search?q=excited+kitten", :body => "Stubbed Google!")
FakeWeb.register_uri(:get, "http://www.google.com/search?q=ponies", :body => "Stubbed Google!")

Thankfully, FakeWeb has a (slightly underdocumented) solution for this. Instead of passing a string to FakeWeb.register_uri, pass it a regular expression (using Ruby’s %r|| syntax to avoid having to escape forward slashes):

FakeWeb.register_uri(:get, %r|http://www.google.com/search\?q=|, :body => "A bunch of results")
HTTParty.get("http://www.google.com/search?q=tunisia") # => "A bunch of results"
HTTParty.get("http://www.google.com/search?q=vegan+latkes") # => "A bunch of results"

By registering a regular expression, FakeWeb will respond to any HTTP GET request whose URL matches the pattern you’ve supplied. Your test suite can make requests with all different query parameters without having to register each individual URL. This feature adds a lot of flexibility to FakeWeb and makes it feasible to stub out your application’s HTTP requests, even when complex, changing query parameters are involved.

For a general overview of FakeWeb, as well a some far-less-contrived examples of its use, check out Josh Nichols’ post, Stop Net::HTTP dead in its tracks. Also check out FakeWeb on GitHub.

Pretty Feed Items with Facebook Connect

2009-11-06T00:00:00-05:00

At Free Press, when a user signs a petition or shows support for our issues on one of our sites, we want them to spread the word. We want to make it easy for users to share the story they told, or the petition they signed, and to encourage others to do the same. We usually do this with links to Facebook’s sharer.php and Twitter’s status parameter.

But sharer.php left us frustrated with its awkward interface and lack of branding. What about the pretty updates that Facebook applications produce? We wanted a feed item that looked branded, looked good, and compelled a user’s friends to also take action. This is where Facebook Connect comes in.

Using Facebook Connect for this task turned out to be simple. Surprisingly simple. Here’s the routine:

Create a Facebook application
Create a file to allow cross-domain JavaScript communication
Include Facebook’s JavaScript library
Write some JavaScript

Steps 1-3 are fairly well documented in the links above. Step 4 is the interesting part. Take a look at the code below.

var AK = {
  apiKey:  "de24e7cfd602218871fc8e30b3dd8a5f",
  message: "Facebook Connect is easy!",
  attachment: {
    name: "Updating from akahn.net",
    href: "http://www.akahn.net",
    caption: "So cool!",
    media: [{
      type: "image",
      src:  "http://akahn.net/images/shoes.jpg",
      href: "http://www.akahn.net"
    }]
  },
}
FB.init(AK.apiKey, "/xd_comm.html");
FB.ensureInit(function() {
  FB.Connect.streamPublish(AK.message, AK.attachment);
});

What’s going on here? First, I’m creating an object, AK, to store the various pieces of data I need. This includes an API key, the default message that the user will be posting to their feed, along with some other bits of text, and an image.

I then get the API ready, using FB.init, specifying my API key and the location of my cross-domain communication file.

Then I make sure that the API has finished loading, using FB.ensureInit. I pass in a function, to be executed once the API has finished loading.

In the function, I call FB.Connect.streamPublish. I pass in the message and the attachment object that I defined earlier. If the user is logged in on Facebook, they’re presented with a modal popup inviting them to post to their feed. If not, they’re prompted to log in before they are invited to create a feed item. Try it out below:

Post to my Facebook wall!

And that’s it! It takes a lot of setup just to make one little call to FB.Connect.streamPublish. But the smooth user experience and the customized, pretty feed item make it all worthwhile.