Wednesday, April 10, 2013

Why History Students Should Love Big Data

It's Spring 1976, Wilson Hall, Brown University.  Professor William McLoughlin has just informed his 85 students in “American Social and Intellectual History” that they are to write their first paper. All he has given us is the title: “The Age of Jefferson and Adams.” We groan. Then he adds: “Keep it to three pages or less. Double-spaced.” We smile. Three pages? How hard can that be?

“If you make the margins too narrow,” McLoughlin adds, “I’ll mark you down a grade.”

Needless to say, nobody got an A on that paper. There may have been a B or two, the good professor informed us.  Not me. It was all I could do to contain my flowery opening paragraph to a single page. Some of us recovered slightly on paper two, in which we committed “The Age of Lincoln and Calhoun” to three, double-spaced pages. Some retreated to organic chemistry and other more reasonable challenges.

Little did I know, but I had just been introduced to Big Data—though it would take another generation to earn that name. Take an endless, insurmountable, seemingly disconnected pile of information, separate the grain from the chaff (or, as my engineering friends might say, signal from noise), and tell a concise, compelling story about what it all means.