e;feed

Starting a New Chapter and Moving East

Eric Richardson — Mon, 14 May 2012 07:08:26 -0700

The Atlanta skyline as seen from Piedmont Park, a 189-acre green space near midtown. ()

It has been a whirlwind month for Kathy and me, starting with a vacation back east and culminating with my last day at KPCC just over one week ago. Next month we'll be backing our bags and moving to Atlanta after a decade here in L.A.

Yesterday I bought the plane tickets: our time as Angelenos ends at 1:10pm on June 15.

Moving back east is a thought that we had entertained for a while, since we don't have any family out here on the west coast. Once Kathy decided that she wasn't going to be returning to her school next year, the conversation about where we might want to be started in earnest.

We went back to South Carolina for Kathy's spring break, and while there decided that we were serious enough about the idea of Atlanta as a possible destination that I should start looking into companies who might be hiring. I sent off a few resumes, and ended up setting two interviews for a day that we would be passing through before flying back out west. Both turned into offers, and a week later I turned in my notice to the station and we began the process of planning a move.

Last Monday I started my new job with Emcien, a company that has a suite of applications built around really fast pattern matching in large data sets. While I won't be going anywhere near the math itself—I had to admit during my interview that my last math class was my junior year of high school—I'm really intrigued by the challenges that come in building the applications that allow users to make effective use of their data and the results that come out of it.

blogdowntown stays at KPCC, where it hopefully has a long life ahead of it. My day-to-day involvement in the site ended back in January, so our move won't create much of a change there. I'll continue to be involved with the open-source development of AssetHost and StreamMachine, the visual asset handling and audio streaming systems I wrote while there.

There's a lot that we'll miss here in L.A., but we're excited to head out on a new adventure. Unfortunately, that adventure starts with logistics: this week I get to start calling movers.

Making Sense of Numbers

Eric Richardson — Thu, 03 May 2012 22:02:12 -0700

Graphs of listener data powered by Cubism (Eric Richardson)

After several partial-deployment tests, we launched StreamMachine at KPCC on Tuesday evening, putting online listeners in the hands of a Node.JS app that I barely figured out how to start writing back in January.

There have been a few hiccups—I don't think people with Roku boxes are very happy at the moment—but on the whole it's been a very successful launch.

While audio streaming provides a number of interesting challenges—how do you handle deployment of new app versions when your connections are of indefinite length, for instance—I'm really interested to play with ways of visualizing listener behavior and interactions.

For the StreamMachine launch I'm using a pair of really interesting open source packages put out recently by Square to enable real-time analysis of listening patterns. Cube is a datastore built on top of MongoDB that's designed specifically for time-series data. Cubism is a visualization plugin using d3 that then turns that data into interesting visuals.

Traditional logging for a listening stream doesn't make for a very good event. Saying that a listener completed a 45-minute streaming session at 6:55pm is great, but that data point needs to be spread out over the whole listening period to give a true picture.

But what if instead you logged every minute listened as it was listened? You would end up with a series of events taking place at the right time in the timeline, and you could then analyze them as they're still in progress.

The screenshot above shows the results, which gives a snapshot of all listening, a comparison to yesterday's stats, and a view of KPCC app listening vs some in-browser numbers. Refining the options and presentation is a work-in-progress, but already it's a fascinating live view of what's going out of our servers.

Authenticating a Google Service Account

Eric Richardson — Thu, 26 Apr 2012 15:34:34 -0700

I spent a few hours at work today trying to make sense of Google's new OAuth2 service accounts, implementing code to generate the JSON Web Token it wants and then query Google's auth server to get the OAuth token. Google intends the accounts to be used by backend services, which makes a lot more sense than shoe-horning them into access via some user's account as is normally the case.

Only after I had all that and I tried to make an API request did I realize that Google doesn't support service accounts in Analytics yet, so all of that work was in vain. To try to keep it from going to waste, I figured I would post it here. Looking to authenticate a Google service account for a Ruby app? Hopefully this will save you a little time.

The JWT gem can be found here.

require 'jwt'

# -- Create JWT -- #

# load in private key
keydata = nil
File.open(Rails.root.to_s+"/config/google_api_key.p12",:encoding => "binary") {|f| keydata = f.read(); }
pk = OpenSSL::PKCS12.new(keydata,"notasecret")

# generate the JWT
jwt = JWT.encode({
  :iss    => "12345678@developer.gserviceaccount.com",
  :scope  => "https://www.googleapis.com/auth/someservice.readonly",
  :aud    => "https://accounts.google.com/o/oauth2/token",
  :exp    => (Time.now + 3600).to_i,
  :iat    => Time.now.to_i
},pk.key,"RS256")

# -- Convert into an OAUTH2 Token -- #

uri = URI("https://accounts.google.com/o/oauth2/token")
req = Net::HTTP::Post.new uri.path
req.set_form_data(:grant_type => "assertion", :assertion_type => "http://oauth.net/grant_type/jwt/1.0/bearer", :assertion => jwt)

conn = Net::HTTP.new uri.host, uri.port
conn.use_ssl = true

resp = conn.start do |http|
  http.request(req)
end

json = JSON.parse(resp.body)

# token is at json['access_token']
puts json

Scratching your own itch...

Eric Richardson — Wed, 25 Apr 2012 19:04:42 -0700

My proof-of-concept "KPCC To Go" app in front of the Xcode development environment. (Eric Richardson)

It feels nice to be able to build something that scratches your own itch, even if it may be one that no one else has. This morning I commuted to work while listening to some "nearly live" radio via a small iPhone app that I put together to work with StreamMachine, my experiment in next-gen streaming audio server.

While we didn't get picked to receive the grant I wrote about a few weeks ago, I've been saying for several months now that I really wanted an app to let me grab a commute's worth of audio just before I walked out the door and onto the subway.

Last night, I finally got the app to the point where I could do just that. The server side of things has been in place since mid-February, but the app side of things kept running into the fact that I don't know Objective-C and have never put the time into getting up to speed on iOS app development.

That point kept being hammered home to me over the weekend as I again and again ran into variations of the EXC_BAD_ACCESS error as I tried to reference things that I was inadvertently failing to keep in memory.

Last night I finally refactored my way past that and got the app working.

Then this morning I pushed the "Grab New Audio!" button a few minutes before walking out the door and quickly had the last fifty minutes of live radio to listen to as I navigated the Red and Gold lines—a route that has both periods of no data connection and connectivity losses.

It may just be a proof-of-concept, but it worked like a champ for me.

Extra terabytes

Eric Richardson — Wed, 18 Apr 2012 18:58:53 -0700

Six years ago I had just over 1.2 terabytes of hard drive space in my apartment. I have 3.75 terabytes hooked up to my laptop right now. (Eric Richardson)

Six years ago today I added up all the hard drives in my apartment and got ~1,210 gigabytes. The largest single drive was 250 gigabytes.

Today I have a two terabyte drive that I use for backups, and 750 gigabytes inside my laptop. Another terabyte drive (the one pictured) is sitting on the desk, but not being used for anything in particular.

I still marvel at how cheap storage has gotten, and how quick it has gotten there.

A Return to Reading

Eric Richardson — Sun, 15 Apr 2012 10:41:20 -0700

(Eric Richardson)

When the new iPad came out a few weeks ago, I hopped in the car and took a drive over to the Apple store at The Grove to pick one up. I had been intrigued but not seriously tempted by the first two models, but this time the new screen and a growing fascination with mobile devices made it something I couldn't resist.

My first few weeks have produced an interesting result, though: they've made me a reader again.

I've gone through four books since buying the device just a little over a month ago. I started out with Neal Stephenson's most recent release, Reamde, then circled back to some of the books that followed Orson Scott Card's Ender's Game, reading Speaker for the Dead, Ender's Shadow and Shadow of the Hegemon.

In that time no apps have become essential parts of my life, though I will admit to spending a few hours on Real Racing 2 HD (and have spent a few hours more than I will admit). I think the iPad makes an amazing web browser, and have been thoroughly impressed with the battery life.

Above all, though, I've just loved having a device that I can carry to catch up on news, do some emails, and sneak a few pages from a book every time I've got an extra minute. I've started carrying it instead of my laptop on my commute, which means that I can grab a few pages while waiting for the train and then a chapter more as I'm on the Gold Line. Sitting in Starbucks before I head into the office it's web and email, then maybe a few more pages at lunch if I'm headed off on my own.

I don't doubt that eventually other apps will find their way into my day-to-day, and I have a few ideas for ones that I could write to scratch some of my own itches, but in the meantime I'm enjoying this new return to books. Next up: Don Delillo's Underworld, a book that's been sitting on my shelf for the last few years. Here's to hoping this time I'll go ahead and read it through.

Making Django and Rails Play Nice, Part 4: Nginx Conditionals and Passenger

Eric Richardson — Sat, 14 Apr 2012 09:42:01 -0700

I got distracted and never managed to post the last piece of my look at making Django and Rails play nice together for KPCC's new beta website. Part one looked at mapping generic relationships with MySQL views, part two at building interoperable sessions and part three at creating an interwoven caching model.

Once you've got all those pieces in place and have both applications happily running off the same data, though, you still need a way to route incoming requests to one or the other. That could be something that you handle in logic on the load balancer, but our setup instead left this as something to solve as the request came into the web server. Turns out, though, nginx made it all quite easy.

The Requirements

In order to declare our deployment a success, we needed the ability to gradually roll features and users from the existing Django site to the new Ruby on Rails frontend.

Some paths needed to be conditional based on a beta cookie, while others needed to only point to the Django site to pick up things that we hadn't yet implemented in Rails. Still others needed to always point to Rails, to allow ground-up development (such as [the new videos section) to exist only on the new site regardless of cookie status.

Nginx to the rescue!

It turns out that those requirements are exactly the sort of thing nginx is good at handling. While "www.scpr.org" is defined in a "server" block in the nginx config, deeper constructs such as conditionals have all the same rights to define settings related to how the request will be handled.

Early in the process of testing out whether this idea would work, I deployed nginx with Passenger to see whether Passenger's "experimental" wsgi support might work for hosting our Django app. It turned out to run quite well, which allowed me to know that all I needed to do in nginx was set root to either the Django or Rails apps and let Passenger handle the rest.

Defining a map

nginx has a nifty map construct that makes up the bulk of our conditional logic:

map $uri $test {
    ~^/$                1;
    ~^/assets/          2;
    default             0;
}

That takes $uri and tests it against the map, putting the result in $test. The tilde in front of the keys allows regular expressions to be used. Our simple test here will return one (can go either way, based on cookie status) when the user is visiting the homepage, two (Rails-only) if they're looking for something under /assets/, and zero (Django-only) if they're trying to get anything else. Our production map is a lot longer, but you get the idea.

Once we have that, we need to add the cookie test. This gets a little funkier. If $test2 contains the results of your cookie test, typically you would want to say something like:

if ( $test1 == 0 || ($test1 == 1 && $test2 == 0) ) {
    # django
} else {
    # rails
}

nginx, though, doesn't support testing multiple variables in a conditional. That meant we needed to combine the two:

# default to old site
set $site 0;
set $test2 "";

# possible match...  two tests are combined
if ($test = 1) {
   set $test2 A;
}

# map based on existence of a cookie
if ($cookie_scprbeta = "true") {
    # use new site
    set $test2 "${test2}B";
}

if ($test2 = "AB") {
     set $site 1;
}

if ($test = 2) {
     # always map to the new site
    set $site 1;
}

Default to site zero, but use site one if either a) we have a conditional match from the map and have the cookie set or b) we have a map match that is only on Rails.

From there we simply set our root based on the $site value:

if ($site = 0) {
        # run old site
        root /web/django_app/public/;
        passenger_min_instances 4;
}

if ($site = 1) {
        # run the new site
        root /web/rails_app/current/public/;
        rails_env "production";
        passenger_min_instances 4;
}

Ta-da! Seamless handoff in the same visit.

Brookgreen Gardens

Eric Richardson — Fri, 06 Apr 2012 04:58:35 -0700

The sun shines on an azalea in bloom near a towering oak tree. (Eric Richardson)

Kathy and I are in South Carolina visiting family during her Spring Break. Just up the highway from our house at Litchfield are Huntington Beach State Park and Brookgreen Gardens, a pair of properties once owned by Archer and Anna Hyatt Huntington.

My family used to camp at the state park every year, so I've spent much time on that side of the road, but I can only remember once that we had ever visited Brookgreen, the sculpture gardens that the pair opened to the public in 1932. We paid a return visit this week and really enjoyed wandering around the former rice plantation.

A warm spring meant that even in the first week of April, many of the Brookgreen blooms were already past their prime.

Given our normal views in Los Angeles, though, all the green and color was still a sight to behold.

You can see more photos over in the Photos section.