Debugging a data pipeline one line at a time, straight in the browser console

The real advantage JavaScript has over SQL isn't syntax, it's that you're almost certainly already running it. Open the console in any browser tab and you have a live interpreter sitting on top of whatever data you've already loaded. That makes it possible to work out a pipeline the way you'd work out a query — one line, check the output, next line — instead of writing the whole thing blind and debugging it after the fact.

This was a real chart for a slide in a pitch deck: count interactions with Facebook posts, bucketed by day, by gender, out of a dataset already built up in the page. The final version of this pipeline is what's actually drawing the gender chart on the Pitch Perfect page today.

Start from what should already be true

> genderedposts
ReferenceError: genderedposts is not defined

Not defined yet — it gets built inside a function, gender(), that hadn't run:

> gender()
undefined
> genderedposts
Array[4254]
> genderedposts[0].from.gender
"male"

Good, that's the shape expected: an array of posts, each carrying who posted it and their gender.

Bucket by day

> d3.nest().key(d => d.tday).entries(genderedposts)
Array[554]
> d3.nest().key(d => d.tday).entries(genderedposts)[0]
Object {key: "2012/9/12", values: Array[27]}

554 distinct days, each holding its own array of posts. So far this is just GROUP BY tday.

Sum within a bucket

> d3.sum(d.values, countinteractions)
204

Checked against one bucket by hand, looks right. Now the actual goal isn't a sum on one bucket, it's a sum per bucket — which in d3's nest API is rollup, and the first attempt gets the argument wrong:

> d3.nest().key(d => d.tday)
    .rollup(d => d3.sum(d.values, countinteractions))
    .entries(genderedposts)
TypeError: Cannot read property 'length' of undefined

rollup hands the callback the array of values directly — not an object with a .values property. Drop the .values and it works:

> d3.nest().key(d => d.tday)
    .rollup(d => d3.sum(d, countinteractions))
    .entries(genderedposts)
Array[554]

From console to source

Once each step checks out, the whole thing gets pasted into the real source with comments explaining each stage as a SQL clause, since that's the mental model it was built against the whole time:

// bucket by day, rollup total interactions
window.genderedints = d3.nest()
    .key(d => d.tday)                                     // GROUP BY tday
    .rollup(d => d3.sum(d, countinteractions))             // count(interactions(post)) AS values
    .entries(genderedposts)                                // FROM genderedposts
    .map(d => ({t: new Date(d.key), d: d.values}))         // key AS datetime(t), values AS d
    .sort((a, b) => a.t - b.t)                              // ORDER BY t

Nothing here needed a debugger, a test file, or a rebuild-and-refresh loop. Every step ran against the real 4,254-post dataset already sitting in memory, and the type error surfaced itself in the one line that introduced it — before it ever made it into a chart nobody could read.