One-sample Kolmogorov-Smirnov test

data: scores$score
D = 0.0616, p-value < 2.2e-16
alternative hypothesis: two-sided

scores <- load.data()

m <- mean(scores)
s <- sd(scores)

ks.test(scores, 'pnorm', m, s)

See this page for more info: http://sekhon.berkeley.edu/stats/html/ks.test.html

Huge +1 for GFW, I had no idea.

As far as frequency goes, you have to keep in mind that I’m using a gardenhose connection that was seriously throttled(as was everyone’s). Around the 15th of July, my tweets/day dropped from as many as 8-9 million/day to as low as 2.5 million/day. Oh, such first-world problems!

This is a known and documented issue with Twitter’s gardenhose feed; I have since applied for limited firehose access, my application is still pending.

I’m basically at the whim of twitter’s streaming api rate limits as to how many tweets I can download/day.

As far as storing them, I parse the raw twitter JSON using my handy-dandy twitter parser. It’s lightweight and built for speed. It only extracts a few fields out of the entire tweet (id, date, screenname, and tweet), and writes to STDOUT. I’ve run a few hundred million tweets through this parser and it has not failed me once. I then group the tweets by day and save them to HDFS. I thought about storing them on AWS but I’m crippled by lousy DSL upload speeds. The days when I was grabbing 8-9 million tweets/day produced files that are 1.1gb AFTER parsing.

Ideally, as my cluster, as well as my experience, matures, I will store the tweets in a database, most likely MongoDB. That, in and of itself, is a new project; I’m still trying to work on my pig(and hadoop)-fu in the meantime!

Thanks for the comments, they are great ones. Watch for more data-experiments to come.

Am I missing something?

May I ask which db stack are you using for storing that many tweets?

Thanks for the tutorial!
Matt