I decided to take a closer look at what people are saying about Steve Jobs, as a whole. Looking at how the site updates, it appears to use Ajax to retrieve and display new messages. Using Chrome’s developer tools, I monitored the requests it was making to get the new messages.

Once I found the location of the individual messages, it was trivial to download all of them. The message endpoint URLs are in the format

http://www.apple.com/stevejobs/messages/3679.json?28106802

and a sample message looks like

{
mainText: "This is equivalent to my mom's generation of Elvis dying for me. I am very
sadden and emotionally moved at the moment. He was more influential on my
life than my parents and friends. While my parents loved me and friends
shared fun times. Steve influenced me, motivated me to become the innovated,
creative technologist I have become. I got into computer technology in 1980
and moved to Silicon Valley because of him. I have been one of his biggest
admirers and looked to him as a mentor to push the boundaries of my own
creative abilities to develop technology solutions which I hope made a
difference and impact to the industries I worked in. We've lost a
significant influence and icon in technology. We won't see another person of
his innovation and foresight within my life time. He was the Edison of
technology. He was and is one of my biggest inspirations.

I feel I have lost a close family member"
header: "What Steve Jobs meant to me"
author: "Skip"
location: ""
}

The site makes a request to http://www.apple.com/stevejobs/messages/main.json which returns

 {
 totalMessages: "10975"
 timestamp: "28106802"
 }

So it appears that it cycles through 10975 messages. I didn’t decompose the javascript powering the site to determine this, I just made an assumption. I tried querying values greater than 10975 and they returned 404. I wrote a quick python program to download the messages:

#!/usr/bin/python
import urllib2
import simplejson as json
import time
import codecs

# a page on apple's site shows the # of messages available
# start with 0 and retrieve up to message_range messages
metadata = json.loads(urllib2.urlopen('http://www.apple.com/stevejobs/messages/main.json').read())
message_range = metadata['totalMessages']

# the url for each message. i learned of this url by inspecting
# the network calls to http://www.apple.com/stevejobs
# using chrome's developer tools
url="http://www.apple.com/stevejobs/messages/%d.json"

# create our destination file
# i'm using codecs because it does a better job at handling international characters
output_file = 'stevejobs_tribute.txt'
file_handle = codecs.open(output_file,'w','utf-8')

# helper function to remove tabs and linefeeds
def clean(txt):
  return txt.replace('\n','').replace('\t','')

# iterate from 0 to the max # of messages and download the message text
# for these purposes, I'm ignoring the other fields as they weren't always present
for i in range(0, message_range):
  req = url % i
  data = urllib2.urlopen(req).read()
  data = json.loads(data)
  file_handle.write(clean(data['mainText']) + '\n')
file_handle.close()

So now, we have over ten thousand tribute messages saved to the file stevejobs_tribute.txt. What I was most interested in seeing how many of these messages contain a reference to a certain Apple product.
I came up with a few search terms based on some legendary Apple product names including

• Newton
• Macintosh
• MacBook
• iBook
• Mac
• iPhone
• iPod
• iMac
• iPad
• Apple II family
• OSX
• iMovie
• Apple TV
• iTunes
• LaserWriter (yes, Laserwriter)

products = {'iPhone':{'regex':'iphones?','count':0},
	'iMac':{'regex':'imacs?','count':0},
	'iPad':{'regex':'ipads?','count':0},
	'iTunes':{'regex':'itunes','count':0},
	'iPod':{'regex':'ipods?','count':0},
	'cube':{'regex':'cubes?','count':0},
	'MacBook':{'regex':'macbooks?','count':0},
	'iBook':{'regex':'ibooks?','count':0},
	'Apple TV':{'regex':'apple ?tvs?','count':0},
	'Apple II Family':{'regex':r'(apple )?(2|ii|\]\[|\/\/)([ce\+|]|gs|s)?[^0-9]', 'count':0},
	'LaserWriter':{'regex':'laserwriter?','count':0},
	'PowerBook':{'regex':'powerbook?','count':0},
	'Newton':{'regex':'newton?','count':0},
	'OSX':{'regex':'osx','count':0},
	'iMovie':{'regex':'imovie','count':0},
	'Macintosh':{'regex':'macintosh','count':0},
	'Lisa':{'regex':'lisa','count':0},
	'Mac':{'regex':'mac','count':0},
}

Here’s a screenshot of me testing the Apple II regular expression, using the excellent Regexr.

Overall, out of 10975 messages downloaded(as of now), 2,186, or just under 20% mentioned an apple product by name. Here’s the breakdown of the products mentioned:

LaserWriter        1
iMovie             3
OSX                9
iBook             22
PowerBook         22
Lisa              24
Apple TV          31
Newton            33
iTunes            52
Macintosh        163
iMac             235
MacBook          366
Apple II Family  481
iPad             574
iPod             575
iPhone           875
Mac             1315

More than one out of every ten messages included a reference to a Mac! Nearly one in ten mentioned an iPhone – not bad for a device that’s been out a fraction of the time the Mac has been available.I’m pleased to see so many references to the Apple II including several mentions of the//c, which was my first Apple product.

It’s also interesting to note that out of 33 mentions of Newton, only a handful of those were about the actual Apple product – most were comparing Steve Jobs to Newton himself. Check out my earlier post on NLTK concordance for details on how I did this:

import nltk
import string
f = open('stevejobs_tribute.txt').read()
f = f.translate(string.maketrans("",""), string.punctuation)
foo=nltk.Text(f.split())
print foo.concordance('newton')

result:

op If history misses men like Isaac Newton Graham Bell Galileu Thomas Edison a
mbered though his legacy Now he met Newton Einstein and other geniuses like hi
oday I was one of the few who had a Newton Today I have an iPhone 4 an iPad2 a
oduct that came thereafter from the Newton to the Cube to the iPhone 4S God Bl
with the likes of Edison Garcia and Newton for his impact and vision I wish hi
ntioned in the same breath as Isaac Newton Thomas Edison and Bill Gates The le
 off a tree we are thinking of Adam Newton and Steve Jobs He open new dimensio
Jobs will be missed Da Vinci Mozart Newton Franklin Jobs Nobody is out of plac
ged my life starting with the Apple Newton followed by the iPod and then the i
 sorely missed nbsp Da Vinci Mozart Newton Franklin Jobs Nobody is out of plac
ve dared to Einstein Freud Da Vinci Newton Galileo Darwin among others is prou
embered beside Einstein Pasteur and Newton The world is moving toward his crea
irst Apple Mac I remember the first Newton I willnbspremembernbspSteves creati
e to contact us againnbsp How Isaac Newton and Albert Einstein contributed gre
 world One seduced Eve One awakened Newton and One was in the hands of Steve J
the way you have influenced mine If Newton discovered something as remarkable
rld One seduced Eve second awakened Newton the third one was in the hands of S
lent to Leonardo Da Vinci Sir Issac Newton Albert Einstein and the like He was
t of the caliber of that of DaVinci Newton Pythagorous etc The list can go on
hen people say names like ie Edison Newton and Einstein I guarantee that the n
 Computers” The Apple II Lisa Mac Newton iPod iTunes store iPod Touch iPhone
ember Steve Jobs the way I remember Newton or Einstein I lived with Apple prod
set consultant who bought his first Newton MacBook 170 and all the dozens of o
 br 3 Apples change the world Adán Newton Steve Jobs 19552011 Rest in Peace t
back to the Apple IIGS I also had a Newton Steve Jobs death hurts me personall
ed the world apple to adam apple to newton and apple to steve jobs Steve was a
dam and Eva Second one that wake up newton third one that Steve Jobs create St

Also interesting where the number of mentions to other historical figures in the Steve Job remembrance messages. According to the submitters, Steve Jobs is clearly in some elite company. I don’t know if I’d go so far as to group him with the man who brought automobiles and light bulbs to the masses but hey, we all have our priorities. All counts were determined through a simple grep command piped to wc -l.Here are a few examples:

• Einstein – 70
• Ford – 189
• Edison – 110
• DaVinci – 15
• Bill Gates – 8

Finally, I wanted to see what how people were speaking about Steve Jobs and especially what terms were being used to describe him. There was no point in performing sentiment analysis on this text as all of the texts were not only obviously positive but were also vetted by Apple for content. Using NLTK, I performed part-of-speech tagging on every word in each tribute message and then wrote some code to total the adjectives and adverbs used in the tribute messages.

The most commonly-used adjectives are

('great', 1961)
('steve', 1808)
('many', 1459)
('first', 917)
('sad', 862)
('better', 857)
('such', 727)
('best', 721)
('visionary', 645)
('new', 579)
('more', 556)
('true', 538)
('most', 476)
('creative', 471)
('apple', 435)
('other', 427)
('same', 415)
('good', 412)
('greatest', 376)
('wonderful', 373)
('sorry', 362)
('old', 325)
('brilliant', 283)
('able', 281)
('incredible', 267)
('big', 260)

Humorously, NLTK frequently considered “Steve” to be an adjective. This is likely because it is always followed by the proper noun “Jobs.” A tweet from NLTK expert Jacob Perkins reminded me that machines are dumb and proper nouns should be capitalized. In order to aggregate the counts, I normalized the text by converting to lowercase – I wasn’t interested in nouns, only adjectives so proper nouns didn’t matter to me.
The top adverbs, according to NLTK, were not as interesting, at least to me.

('so', 2220)
('never', 2111)
('not', 1897)
('always', 1798)
('just', 1402)
('now', 1028)
('truly', 989)
('only', 945)
('very', 919)
('much', 908)
('ever', 751)
('even', 743)
('really', 567)
('forever', 508)
('more', 486)
('still', 447)
('well', 398)
('most', 375)
('personally', 352)

And finally, I ran tri-gram analysis, again using NLTK. 

trigrams = defaultdict(int)
nltk_trigrams = nltk.trigrams(text)
for itm in nltk_trigrams:
  trigrams[itm] += 1

As one would expect, the leading trigram was ‘rest in peace‘ with 1838 mentions, 16.7% of all mentions. ‘thank you for‘ was found in 1446 messages, ‘will be missed‘ was found in 827 messages. Other interesting trigrams are ‘thank you steve‘ with 791 mentions and ‘changed the world‘ with 551 mentions.

The full python code and resulting data can be found on github.

#!/usr/bin/python
#nltk.help.upenn_tagset('RB')
from collections import defaultdict
from operator import itemgetter
import re
import urllib2
import string
import simplejson as json

import codecs
import nltk

OUTPUT_FILE = 'data/stevejobs_tribute.txt'

adverbs = defaultdict(int)
adjectives = defaultdict(int)
trigrams = defaultdict(int)

message_has_adjective = False
message_has_adverb = False
message_contains_product_mention = False
messages_with_adjective = 0
messages_with_adverb = 0
messages = 0
messages_with_product_mention = 0

exclude = set(string.punctuation)

products = {'iPhone':{'regex':'iphones?','count':0},
	'iMac':{'regex':'imacs?','count':0},
	'iPad':{'regex':'ipads?','count':0},
	'iTunes':{'regex':'itunes','count':0},
	'iPod':{'regex':'ipods?','count':0},
	'cube':{'regex':'cubes?','count':0},
	'MacBook':{'regex':'macbooks?','count':0},
	'iBook':{'regex':'ibooks?','count':0},
	'Apple TV':{'regex':'apple ?tvs?','count':0},
	'Apple II Family':{'regex':r'(apple )?(2|ii|\]\[|\/\/)([ce\+|]|gs|s)?[^0-9]', 'count':0},
	'LaserWriter':{'regex':'laserwriter?','count':0},
	'PowerBook':{'regex':'powerbook?','count':0},
	'Newton':{'regex':'newton?','count':0},
	'OSX':{'regex':'osx','count':0},
	'iMovie':{'regex':'imovie','count':0},
	'Macintosh':{'regex':'macintosh','count':0},
	'Lisa':{'regex':'lisa','count':0},
	'Mac':{'regex':'mac','count':0},
}

def top_n(dct,n = 10):
	srtd=sorted(dct.iteritems(), key=itemgetter(1), reverse=True)
	for x in srtd[0:n+1]:
		print x

def nltk_concordance(term,text_file):
	f = open(text_file).read()
	# remove punctuation
	f = f.translate(string.maketrans("",""), string.punctuation)
	split_text=nltk.Text(f.split())
	split_text.concordance(term,lines=100)

	# >>> f = f.translate(string.maketrans("",""), string.punctuation)
	# >>> foo=nltk.Text(f.split())
	# >>> print foo.concordance('newton')

def unescape(s):
	"""unescapes html codes"""
	s = s.replace("<", "	s = s.replace(" ", " ")
	# this has to be last:
	s = s.replace("&", "&")
	return s

for line in open(OUTPUT_FILE):
	message_has_adjective = False
	message_has_adverb = False
	message_contains_product_mention = False

	# remove the trailing linefeed and convert to lower-case
	# and remove html control characters
	messages += 1
	data = line.strip()
	data = data.lower()
	data = unescape(data)

	# check for product mentions
	for k,v in products.iteritems():
		if re.search(v['regex'],data):
			products[k]['count'] += 1
			message_contains_product_mention = True

	# if the message contains a product mention
	# increment the product mention counter
	if message_contains_product_mention:
		messages_with_product_mention += 1

# tokenize the sentences using nltk's wordpuncttokenizer
	text = nltk.WordPunctTokenizer().tokenize(data)

# compute trigrams
	nltk_trigrams = nltk.trigrams(text)
	for itm in nltk_trigrams:
		trigrams[itm] += 1

# pos-tag each token. we're interested in adjectives and adverbs
	parts_of_speech = nltk.pos_tag(text)
	# test for adjectives and adverbs, increment the counters
	# when we find one.

	for (word,pos) in parts_of_speech:
		if pos.startswith('JJ'):
			message_has_adjective = True
			adjectives[word] += 1

		if pos.startswith('RB'):
			message_has_adverb = True
			adverbs[word] += 1

	# if the message contains an adverb or an adjective, increment a counter
	if message_has_adjective:
		messages_with_adjective += 1
	if message_has_adverb:
		messages_with_adverb += 1

# output the 25 most frequently-used adjectives and adverbs
n = 25
print "top %s adverbs" % n
top_n(adverbs, n)
print
print "top %s adjectives" % n
top_n(adjectives, n)

print "messages with adjectives: %s" % messages_with_adjective
print "messages with adverbs: %s" % messages_with_adverb
print "total messages with product mentions: %s" % messages_with_product_mention
print "total messages: %s" % messages

# output the top 50 most-common trigrams
n = 50
print "top %s trigrams" % n
top_n(trigrams, n)
srtd=sorted(products.iteritems(),key=itemgetter(1))
for x,y in srtd:
	print "%s\t\t%s" % (x,y['count'])

print
print
# concordance for newton
print "concordance for newton:"
nltk_concordance('newton',OUTPUT_FILE)

I’m admittedly not the biggest Seinfeld fan so I need your help! Please leave some new quotes in the comments. The quotes will make it back into @HelloooooNewman‘s replies. Ideally the quotes are no longer than 125 or so characters in length to allow for a username to be included in the reply. My program will truncate any tweets longer than 140 characters. Full quote database after the break.

I had a dream last night that a hamburger was eating ME!
If you've got a t-shirt with blood stains all over it, maybe laundry isn't your biggest problem right now.
Look, Vanessa, of course the market fluctuates. Everybody knows that. I just got fluctuated out of four thousand dollars!
I don't return fruit. Fruit's a gamble. I know that going in.
Sometimes the road less travelled is less travelled for a reason.
Two hundred seats on a plane, I gotta wind up next to Yukon Jack and his dog Cujo.
Oh, you're as pretty as them. You just need a nose job
No, no, I don't think I'm special. My mother always said I'm not special.
I'm speechless! I have no speech!
I hate asking for change. They always make a face. Like I'm asking them to donate a kidney.
You know a muffin can be very filling.
I don't wanna get a movie hot dog! I want a Papaya King hot dog!
I'm disturbed, I'm depressed, I'm inadequate, I've got it all!
Looking at a cleavage is like looking at the sun. You don't stare at it.Its too risky.You get a sense of it and then you look away.
Everyone thinks we're gay!... Not that there's anything wrong with that.
And one more thing; they're real, and they're spectacular.
You know, sometimes when I think youre the shallowest man I've ever met,you somehow manage to drain a little more out of the pool.
This is beyond B.O. This is B.B.O.
This is our best model: The Cougar 9000. It's the Rolls Royce of wheelchairs. This is like...you're almost glad to be handicapped.
When you control the mail, you control information!
Jerry...this woman hates me so much...I'm starting to like her.
A woman that hates me this much comes along once in a lifetime.
I'm doing a coffee-table book on coffee tables
He's a male bimbo...He's a mimbo!
The sea was angry that day, my friends, like an old man trying to return soup at a deli!
Jerry, you stand on the threshold to the magical world of sensual delights that most men dare not dream of.
You're in the kitchen. You see an eclair in the receptacle. So you think to yourself, "What the hell, I'll just eat some trash."
He recycled this gift. He's a regifter
Jerry, just remember, it's not a lie if you believe it.
Newman, you magnificent bastard, you've done it!
You may know it better as Myanmar, but it'll always be Burma to me.
Why do they call it Ovaltine? The mug is round. The jar is round. They should call it Roundtine. That's gold, Jerry! Gold!
You can stuff your sorries in a sack, mister!
It's Christmas for some, a Festivus for the rest of us!!
Ew, Mr. Apple, you have a brown spot!
Hey, I'm on First and... First. How can the same street intersect with itself? I must be at the nexus of the universe.
My parents didn't want to move to Florida, but they turned sixty, and that's the law.
Sometimes the road less traveled is less traveled for a reason
Why do they call it a "building"? It looks like they're finished. Why isn't it a "built"?
People who read the tabloids deserve to be lied to
Men don't care what's on TV. They only care what else is on TV.
Boy, these pretzels are makin' me thirsty.
Yo Yo Ma.
You know I always wanted to pretend I was an architect
I'm not a lesbian. I hate men, but I'm not a lesbian.
I'm speechless. I have no speech.
God... it's like a sauna in here.
I have a bad feeling that whenever a lesbian looks at me they think "That's why I'm not a heterosexual."
Hi, my name is George, I'm unemployed and I live with my parents.
I've driven women to lesbianism before but never to a mental institution.
Divorce is always hard. Especially on the kids. 'Course I am the result of my parents having stayed together so ya never know.
My father was a quitter, my grandfather was a quitter, I was raised to give up. It's one of the few things I do well.
Boy, a little too much chlorine in that gene pool.
Hoochie Mama.
People don't just bump into each other and have sex. This isn't Cinemax.
Here's to those who wish us well, and those who don't can go to hell.
That's a lotta potatoes.
Yada, Yada, Yada.
You have the rooster, the hen, and the chicken. The rooster goes with the chicken... So who's having sex with the hen?
I don't know if it's possible, but could you people conduct the psychopath convention down the hall?
You're killing independent George.
Helllllooooo.
That... is one magic loogie.
You, my friend, have crossed the line between man and bum.
Why don't you just get a pair of white shoes, move down to Miami Beach and get this whole thing over with?
See... I have two friends. You were up, he was down. Now he's up and you're down. See how it all evens out for me?
I'll be back. We'll make out.
What is this salty discharge?
You ever dream in 3-D? It's like the Boogie Man is coming RIGHT AT YOU.
The carpet sweeper is the biggest scam perpetrated on the American public since One Hour Martinizing.
Is this a gym or a fitness museum?
He fires people like it's a bodily function!
You very bad man, Jerry. Very bad man.
I don't know what it is about that mirror in that bathroom. I love the way I look in it... I feel like Robert Wagner.
The cat - mmrrrooowwwrr - is out of the bag!
This thing is like an onion: the more layers you peel, the more it stinks!
Somewhere in this hospital, the anguished squeal of Pigman cries out!
Produce section. Very provocative area. A lot of melons and shapes. Everyone's squeezing and smelling..
It pains me to say this, but I may be getting too mature for details.
I've always been a stall man.
It's one day. Half a day, really. I mean you subtract showers and meals, it's like twenty minutes.
I can't go to a bad movie by myself. What, am I gonna make sarcastic remarks to strangers?
You should've seen her face. It was the exact same look my father gave me when I told him I wanted to be a ventriloquist.
I hate asking for change. They always make a face. It's like asking them to donate a kidney.
The apples are mealy, the oranges are dry... I don't know what's going on with the papayas!
I can feel his blood inside of me. Borrowing things from my blood.
Women don't respect salad eaters.
Man, it's the nineties... It's Hammer time!
Why is nice bad? What kind of a sick society are we living in when nice is bad?
I'm much more comfortable criticizing people behind their backs.
There's nothing more sophisticated than diddling the maid and then chewing some gum.
I've never heard of a relationship being affected by punctuation.
Moles: Freckle's ugly cousin.
I'm in the unfortunate position of having to consider other people's feelings.
I would drape myself in velvet if it were socially acceptable.
She had man-hands!
You know what they say, 'You don't sell the steak, you sell the sizzle.
It's more like a full-body dry heave set to music.
If you can't say something bad about a relationship you shouldn't say anything at all.
Did you know that the original title for War and Peace was War, What Is It Good For?
This woman is bending my mind into a pretzel.
It's amazing that the amount of news that happens in the world every day always just exactly fits the newspaper.
There is no such thing as fun for the whole family.
Where lipstick is concerned, the important thing is not color, but to accept God's final word on where your lips end
A two-year old is kind of like having a blender, but you don't have a top for it.
Make no mistake about why these babies are here - they are here to replace us.
A bookstore is one of the only pieces of evidence we have that people are still thinking.
You think people will still be using napkins in the year 2000? Or is this mouth vacuum thing for real?
If I had a son, I would name him Isosceles. Isosceles Kramer.
My neck is one gargantuan monkey fist.
Vomitting is not a deal breaker. If Hitler had vomitted on Chamberlain, Chamberlain still would have given him Czechoslovakia.
When you're shopping on Madison Avenue, you don't want to skimp on the swank.
Are you still master of your domain?
Well, I'm Out!
To the idiotmobile.
You're an anti-dentite.
You're killing independent George.
I was in the pool, I was in the pool.
I think I can sum up the show for you with one word; Nothing.
Yeah, I'm a great quitter: It's one of the few things I do well...I come from a long line of quitters.
I come from a long line of quitters. My father was a quitter, my grandfather was a quitter.
Hello Newman.
No soup for you.
You can stuff your sorrys in a sack, mister.
The sea was angry that day my friends, like an old man sending back soup in a deli.
So you're killing yourself, because your dreams of becoming a banker have gone unfulfilled!
Ted Danson makes $800,000 an episode. I can't live knowing that Ted Danson makes that much more than me.
No, I don't have a square to spare. I can't spare a square.
You win: I drop dead. I win: I don't drop dead, and I get 100% anti-drop-dead protection - forever!
You know you're not supposed to drink while you're keeping a secret.
You see, my dear, all certified mail is registered... but registered mail is not necessarily certified.
I'll tell you a little secret about zip codes: They're meaningless.
In any other shoe, I lose two inches; I can't have a drop-down. We were eye-to-eye; I can't go eye-to-chin.
You put the balm on? Who told you to put the balm on? I didn't tell you to put the balm on! Why'd you put the balm on?
Two months ago, I saw a provocative movie on cable TV. It was called 'The Net,' with that girl from the bus.
What kind of a man are you? The guy is unconscious in a coma and you don't have the guts to kiss his girlfriend?
I think that ginger ale at the coffee shop is just Coke and Sprite mixed together. How can I prove it? Ah! Can't, dammit.
Do you ever get down on your knees and thank God you know me and have access to my dementia?
Yeah, you better give me the insurance. Because I'm gonna beat the hell out of this car.
I need to talk to you about my friend, Dr. Tim Whatley. I think he's converted to Judaism just for the jokes!
Here's to those who wish us well, and those who don't can go to hell!
Speaking of ex's, my ex-boyfriend came over late last night, and, yada yada yada, anyway. I'm really tired today.
Too many people got their mail. Close to 80%. Nobody's ever cracked the 50% barrier.
I'm sorry, but I can't be with someone whose protege is a hack.
I don't know if it's possible, but could you people conduct the psychopath convention down the hall?
Did the medical journal mention anything about standing in a pool of somebody else's urine???
Have ya been to the Motor Vehicle Bureau? It's a leper colony there!
Who buys an umbrella anyway? You can get them for free at the coffee shop in those metal cans!
You say you're a dermatologist? Well, I call you Pimple Popper, MD.
Oh my God, I'm having an affair. That's so adult. It's like with stockings and martinis, and William Holden.
I don't know what it is about that mirror in that bathroom. I love the way I look in it... I feel like Robert Wagner.
Cheap fabric, and dim lighting. That's how you move merchandise.
The Dewey Decimal System... What a scam that was!
I don't judge a man by the length of his hair or the kind of music he listens to. Rock was never my bag.
You'd better not screw up again, Seinfeld, because if you do, I'll be all over you like a pit bull on a poodle.
It's Risk. It's a game of world domination being played by two guys who can barely run their own lives.
Let me just finish my coffee, and then we'll go watch them cut the fat bastard up.
Who's gonna turn down a Junior Mint? It's chocolate, it's peppermint -- it's delicious!
I think we really need to be in front of the television set. You take TV out of this relationship, it is just torture.
It's a reverse peephole. Now I can peek in and see if anyone is waiting to jack me with a sock full of pennies.
My boyfriend said I got gonorrhea from riding the tractor in my bathing suit.

Using Python and BeautifulSoup, it simply loops through all of the pages on http://www.chucknorrisfacts.com and reads the items displayed on the page.

#!/usr/bin/python
import urllib2, time
from BeautifulSoup import BeautifulSoup, BeautifulStoneSoup

# 674 pages last time I checked. Oddly enough, their pages seem zero-based. Additionally, if you
# substitute an arbitrary number, outside of the range of pages, you'll get data back instead
# of 404. I'm not sure why they're doing this.
for page_num in range(0,674):
	url = 'http://www.chucknorrisfacts.com/all-chuck-norris-facts?page=%d' % page_num
	html = urllib2.urlopen(url)
	soup = BeautifulSoup(html)

	entries = soup.findAll("li","views-row")
	for entry in entries:

		# use BeautifulStoneSoup to remove any HTML-escaped text that BS returns.
		the_quote = BeautifulStoneSoup(entry.div.text,
		                   convertEntities=BeautifulStoneSoup.HTML_ENTITIES).contents[0]

		# print it to stdout. I just redirect the program's output to a file.
		print the_quote.encode('utf-8')
	# be a good citizen and wait a few seconds before visiting the next page
	time.sleep(6)

Output looks like

if the mountain won't come to Muhammad, Chuck Norris will bring it.
if you watch the ring you die in 7 days,if you look at Ckuck Norris you die instantly
in a real zombie apocalypse, Chuck Norris can roundhouse-kick 53,596 zombies dead.
in space no-one can hear you scream.....except chuck norris!
iphone 4? chuck norris has iphone 8
most kids pee their name into snow... Chuck Norris pisses his in concreate...
never say you can'thurt a fly to chuck norris because he will hurt you
new never-before-seen behind-the-scenes shots from Walker Texas Range shows Chuck Norris carrying his truck home after it broke down
no one has ever found where the smurfs live thats  becuase they live  in chuck norrises beard
...

Visit my github for the full dataset (5500 entries). ]]> http://www.neilkodner.com/2011/03/chuck-norris-doesnt-screen-scrape-the-data-runs-scared-to-his-hard-drive/feed/ 0 http://www.neilkodner.com/2011/02/fun-with-awk-and-dead-people/ http://www.neilkodner.com/2011/02/fun-with-awk-and-dead-people/#comments Thu, 24 Feb 2011 19:36:15 +0000 admin http://www.neilkodner.com/?p=521 Just playing around with some Freebase data in preparation for a ‘who died today’ twitter bot.

Get the data and determine on which date did the most people die?

hadoop3:Downloads nkodner$ curl -O "http://download.freebase.com/datadumps/latest/browse/people/deceased_person.tsv"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 16.3M  100 16.3M    0     0   209k      0  0:01:19  0:01:19 --:--:--  248k
hadoop3:Downloads nkodner$ awk -F'\t' '{print $4}' deceased_person.tsv|grep "-"|sort|uniq -c|sort -n|tail -11|head
  22 2008-01-03
  22 2008-02-21
  22 2008-05-20
  23 1989-06-07
  23 2009-01-13
  24 2009-01-11
  26 2009-04-03
  27 1912-04-15
  63 2001-09-11
  65 1965-11-08

Surprised to see 1965-11-08 listed ahead of 2001-09-11. Why? Lets look at where people died on 1965-11-08:

hadoop3:Downloads nkodner$ grep "1965-11-08" deceased_person.tsv |awk -F'\t' '{print $5}' |sort|uniq -c|sort -n
   1 Kenton County
   1 Latium
   1 Leicester
   1 New York City
   1 Toronto
   3
  57 American Airlines Flight 383 Crash Site

Upon further investigation, it looks as if Freebasers have set up a Victims of AA Flight 383 page, containing info on the deceased. Works for me.

How about which month/year did the most people die on?

hadoop3:Downloads nkodner$ awk -F'\t' '{print $4}' deceased_person.tsv|grep "-"|awk -F'-' '{print $2"-"$3}'|sort|uniq -c|sort -n|tail -11|head
 668 02-08
 668 03-06
 672 01-06
 673 02-11
 676 01-28
 677 01-10
 683 01-04
 692 12-31
 702 01-22
 752 02-02

Method of death?

hadoop3:Downloads nkodner$ awk -F'\t' '{print $3}' deceased_person.tsv|sort|uniq -c|sort -n|tail -11|head
 505 Cardiovascular disease
 603 Tuberculosis
 742 Assassination
 745 Stroke
 799 Pneumonia
 832 Lung cancer
 913 Murder
1618 Suicide
1978 Cancer
2503 Myocardial infarction

And finally, the most common names of the deceased people listed on Freebase

hadoop3:Downloads nkodner$ awk -F '\t' '{print $1}' deceased_person.tsv |sort|uniq -c|sort -n|tail -11|head
  21 William Anderson
  23 John White
  25 John Campbell
  25 John Wilson
  29 George Smith
  30 John Anderson
  32 William Smith
  34 John Williams
  35 John Taylor
  36 John Smith

Nothing too deep today, maybe this data might be worth a closer look in R someday.

Each of these tweets contains a bit of information – The score represented in meters, the method of death (hitting a wall and tumbling to my death) and the device (iPhone). Other useful information can easily be extracted such as the date/time played and information about the user (name, location, friend count, follower count, etc). Over the next few weeks I aim to see what features, if any, has any influence on Canabalt scores.

The first thing I needed to do was capture the tweeted Canabalt scores. I have a process running on an EC2 micro instance that downloads tweets from the Twitter Streaming API based on certain key words, one of them being canabalt. The process loads each matching tweet into a MongoDB instance hosted on MongoHQ.com.

curl -s -u $TWITTER_USERNAME:$TWITTER_PASSWORD -d @/home/ec2-user/trackingkeywords http://stream.twitter.com/1/statuses/filter.json |/home/ec2-user/mongodb/bin/mongoimport &

Where trackingkeywords is a file containing a comma-separated list of keywords that I track on twitter. Additionally, I left connection details out of the mongoimport command. You’ll need to provide a host, port, database, and collection into the mongoimport command.

I then run some python code to query the MongoDB instance and retrieve tweets mentioning Canabalt, based on a simple regular expression. I’m expecting the tweet to begin with ‘I’ and contain the word Canabalt. Pretty naive but it worked fine. If it’s not a true Canabalt score, I’ll be able to determine in no time. From there, I use regular expressions to extract(for now) the score, the method of death, and the device name.

def canabalt_tweets():

	# connect to MongoDB
	tweets = create_connection(False)

	# regular expression to extract components of a canabalt score
	canabalt_regexp = re.compile(r'I ran (\d{3,7})m before (.*) on my ([^.]+)\.')

	# regular expression to match tweets that begin with I ran and mention canabalt
	regexp = re.compile('^I ran .*canabalt')

	# create a MongoDB cursor(query)
	cur = tweets.conftweets.find({'text': regexp}, {'text': 1})

	# iterate through the cursor. If a tweet fits the pattern, print it.
	for item in cur:
		try:
			(score,death,device) = canabalt_regexp.search(item['text']).groups()
			print ','.join([strip_text(score),strip_text(death),strip_text(device)])
		except:
			pass

Function strip_text() is part of my data tools Bat-Utility Belt and cleans text by removing leading/trailing spaces, crlf, tabs and some other junk.

We now have some comma-separated data in this shape

score,death,device
2860,hitting a wall and tumbling to my death,iPhone
3427,hitting a wall and tumbling to my death,iPad
4496,hitting a wall and tumbling to my death,iPad
3635,missing another window,iPhone
2040,colliding with some enormous obstacle,iPhone
6017,somehow hitting the edge of a billboard,iPhone
8374,knocking a building down,iPhone
2939,hitting a wall and tumbling to my death,iPad
2021,turning into a fine mist,iPad

Now for some more fun – visualization and analysis. This is performed in R because, well, R is awesome. That, and I really need some more practice with R.

To date, I’ve collected just over 1200 Canabalt ‘events’. I will likely turn this into a web app if there’s enough interest.

A couple of summaries:

scores by device type:

      device count mean stddev median   max min range
      iPhone   735 4491   3882 3419.0 36332 102 36230
        iPad   284 4723   3884 4041.5 40630 104 40526
  iPod touch   189 3734   3644 2713.0 28024 102 27922>

scores by type of death:

                                            death count mean stddev median   max  min range
          hitting a wall and tumbling to my death   684 4155   3481 3319.5 36332  102 36230
                           missing another window   243 5898   4981 4486.0 40630  409 40221
                         turning into a fine mist    86 3592   2698 2662.5 16441  614 15827
            colliding with some enormous obstacle    40 4768   4247 3256.5 16933  433 16500
                              falling to my death    37 4176   3160 3619.0 13573  567 13006
                       missing a crane completely    22 2950   1774 2923.5  7883  381  7502
                         knocking a building down    21 3399   2267 2849.0  8374  336  8038
                   not quite reaching a billboard    19 3098   1244 2980.0  5772  444  5328
              landing where a building used to be    17 4804   4970 3631.0 22685 1170 21515
          somehow hitting the edge of a billboard    14 5991   3827 5518.5 13547  566 12981
   just barely stumbling out of the first hallway    13  104      1  104.0   104  102     2
              somehow hitting the edge of a crane     7 5497   4835 4942.0 13275  510 12765
       riding a falling building all the way down     4 4278   2162 4195.5  6993 1727  5266
           completely  missing the entire hallway     1 1046     NA 1046.0  1046 1046     0

And now, in the spirit of killing the almighty ink-data ratio, here are some pictures:

What have we learned? So far, while my data set isn’t altogether that large(1200 events), we might have enough to make some basic observations and assumptions(correction please!). Going into this experiment I thought that iPad players would have generally higher scores. This is because of #1 the larger screen size and #2 players wouldn’t necessarily be playing ‘on-the-go’ as they would be (I know I am) on an iPhone or iPod touch. The iPad has higher median and average scores than the other devices. I’d like to revisit this as I collect more data.

The leading cause of Canabalt death, by far, is hitting a wall and tumbling to one’s death. This surprised me as I thought it would be falling to death – that’s how my Canabalt games seem to end.

I’d like to hear your comments suggestions for new analysis, and most of all, your corrections.  You know who you are and this is how I learn. The data and python/R source can be found on github.

The stack: Twitter Streaming API, EC2, MongoDB, Python, Regular Expressions, R

Things I learned working on this: plyr(group-by and aggregation in R), sorting dataframes in R, couple of new ggplot2 tricks.

Someone suggested I create a bot that tweets the US National Debt.  Here’s how I’m retrieving the National Debt amount from the US Treasury site.  I then retrieve the US Population from census.gov to figure out each person’s share of the national debt.

I present to you @usadebtlevel, my debt-tweeting bot.

from BeautifulSoup import BeautifulSoup
import urllib2
debt_url = 'http://www.treasurydirect.gov/NP/BPDLogin?application=np'
page = urllib2.urlopen(debt_url)
soup = BeautifulSoup(page)
national_debt = soup.find('table',{'class':'data1'}).findAll('td')[3].text
as_of = soup.find('table',{'class':'data1'}).findAll('td')[0].text

population_url = 'http://www.census.gov/main/www/popclock.html'
population_page = urllib2.urlopen(population_url)
soup = BeautifulSoup(population_page)
population = soup.find('span',{'id':'usclocknum'}).text
debt_amount = float(national_debt.replace(',',''))
population_amount = int(population.replace(',',''))
per_person = debt_amount / population_amount
print "US National debt as of %s is %s, or %.2f for each person(%s) in the US" % (as_of, debt,per_person,population)

Output:

US National debt as of 01/27/2011 is 14,059,409,159,678.42, or 45064.71 for each person(311,982,692) in the US

158636 gol
126669 goal
31722 Goal
24735 Gol
19610 GOL
14317 GOAL
4178 gool
2981 ggol
2219 goll
1771 goool
1641 GOOOL
1564 Gooool
1498 Goool
1279 GOOOOL
1188 Goooool
1158 GOOOOOOOL
1124 GOOOOOL
1116 gooool
1075 GOOL

A few rough visualizations, they are in need of an update. The first is a scatter between the length of the GOAL and the number of Os contained. You’ll see that a lot of excited soccer fans like to use the full 140 characters in their celebrations, often using 138 Os. Huge thanks to @johnmyleswhite for the inspiration and the R/ggplot2 help. You don’t even want to see my original version!

Inspired by @peteskomoroch, who was inspired by the frequencies of the length of mentions of KAAAAAAHN! here are the frequencies of the length of GOAL! celebrations found on twitter.

A visualization of the hashtag mentions by country through all of the World Cup Tweets.  Click for full-sized version.

And my personal favorite visualization, the number of tweets mentioning vuvuzela, hourly, during the course of the world cup.

My current army of twitter bots and the keyword that each one responds to:

• @HelloooooNewman (seinfeld) Klout score 74
• @TheBotLebowski (lebowski) Klout score 70
• @ACenterForAnts (zoolander) Klout score 70
• @IAmJacksBot (fight club) Klout Score 74
• @AManInAMask (V for Vendetta)
• @WorldOfShit (full metal jacket) Klout score 65
• @SomeGrenades (serenity + firefly) Klout score 56
• @GunShowTickets (Ron Burgundy) No Klout score yet
• @PleaseBe18 (Ricky Bobby)
• @ABakingPowder (schwing) Klout score 54
• @Which_is_nice (caddyshack) Klout score 56
• @MitchHedbot (mitch hedberg) No Klout Score yet
• @dubbbya (gwb) — banned from twitter
• @dreidly (dreidel) — retired

I’ve also built a few programs that scrape air quality data from the State of Utah and tweet the results.

• @UtahAirQuality serving Salt Lake and Davis Counties
• @WeberCountyAir
• @CacheCountyAir
• @UtahCountyAir

@usadebtlevel which tweets the US National Debt and each US Citizen’s share.

And here’s a sneak preview: @SarahEffinPalin was conceived after a friend, Willie Morris (@morewillie) suggested a bot that, lets say, repurposes Sarah Palin’s tweets.  I think @SarahEffinPalin is going to be a hit.

We all know The Big Lebowski is a cult classic and one of the most quoteable movies of all time.  I don’t exactly remember how this started but a long time ago, I thought people who mentioned “Lebowski” in a tweet would appreciate receiving a quote from the movie.  So with nothing but the Twitter API docs and a little bit of python, I built @thebotlebowski, my first auto-responder.  The idea was simple – using urllib2, perform a search for “lebowski”, and iterate through the results.  For each result, retrieve a random entry out of a quotes database and tweet it as a reply to the original tweet.

After a ton of retweets, #ff mentions, replies, and followers, it became pretty obvious that people liked it.  I needed a followup – another infinitely quoteable movie.  Zoolander!  Thus, @ACenterForAnts was born.  Again, my research showed that all mentions of Zoolander on twitter were either references to the movie or Derek Zoolander himself.

Another follow-up was in order.  A friend suggested a Seinfeld one.  Done. Welcome @HelloooooNewman. And then @IAmJacksBot and then the others.  The key was to create bots that use search terms that are not vague – If someone tweets “Full Metal Jacket”, then they’re obviously talking about the movie.  Same with “Fight Club.”

One of the lessons learned was that not everyone who tweets about “GWB” was necessarily referencing the president.  A friend suggested a bot that replies to mentions of GWB with one of George Bush’s self-butchered quotes.  People loved it except for people in New York – hey, I didn’t realize that so many people tweeted about the George Washington Bridge in abbreviated format!  This includes several NYC twitter accounts that automatically post traffic conditions.  The complaints came in quicker than I could add people to the ignore list.  Eventually, the well-loved but polarizing @dubbbya was banned from twitter.  May his quotes live on in infamy.

While the bots have been very well-received, not everyone likes them.  When there were only a handful of bots, I used to monitor their responses.  Not that there are so many, I’ve added an ignore list for just this reason.  To add yourself to the ignore list, either contact me, tweet me, or visit http://neilkodsbots.appspot.com.

Frequently Asked Questions:

Have any celebrities found your bots?

The bots tweet out to celebrities all of the time.  @ACenterForAnts has tweeted Ben Stiller many, many times but Ben has never replied.  Sometimes, the celebrities tweet back.  I don’t actively monitor the mentions and replies to the bots – there are just too many.  My favorite anecdote, so far, is when Adam Baldwin discovered @worldofshit, my Full Metal Jacket bot and immediately triggered it over and over to receive new quotes.  He then started tweeting about the bot to his followers and it quickly picked up steam.  I was thrilled to see that one of the stars of Full Metal Jacket was tweeting so favorably about a program that I wrote that I created @somegrenades in his honor.

If you notice a celebrity or otherwise notable person referencing one of my bots, please let me know.  The mentions that I know about include Q-Tip, Taleb Kweli, and Fred Durst.

But you’re a data geek, not a twitter programmer!  Are you doing anything cool with the data?

Yes!  Every tweet that I find, I log.  For example, since @HelloooooNewman has already sent out over 170,000 replies, I have at least that many incoming tweets mentioning Seinfeld in my logs.  I am able to tell who’s tweeting about Seinfeld, when people are talking about Seinfeld, what they’re saying, and so on and so forth.  I can even tell if certain events, such as the release of a box set or a new event have resulted in an increase of Seinfeld tweets.  For examples of some of the things I’ve done with the twitter data, check out this analysis of Seinfeld Tweets or this word cloud generated from 23,000 tweets about Charlie Sheen.  Please contact me if you’d like to hear more.

Would you create a bot for me/my company/my promotion?

I get asked this all of the time.  The answer is:  It depends.  Lets talk.  Before we go about doing this, we’d need to establish a few ground rules.  I’ve worked very hard to keep the bots entertaining and not spammy.

The tweets don’t include urls or advertisements – Are you making any money off of them?

While the bots don’t generate income directly, they have led to other opportunities and benefits.  For starters, I’ve picked up a ton of quality followers and contacts that I would have never met.  Additionally, through this experience, I’ve learned a great deal about twitter the twitter API, and numerous features of Python that I wouldn’t have normally dived into.  To answer the question, A few companies and web sites have licensed the technology and I’ve created custom bots and twitter searches for them.  I’ve elected to not mention them directly in this post.

I don’t want the bots to reply to my tweets.  Can they ignore me?

Sure, the easiest way to be ignored is to visit http://neilkodsbots.appspot.com and add yourself to the ignore list.  Honor system please!  I didn’t feel it was necessary to ask users to authenticate via twitter just so my application could ignore them.

I’m selling Seinfeld/Zoolander/Lebowski products – will you tweet this link to all of your bots followers?

I also get asked this all of the time.  @HelloooooNewman has over ten thousand followers.  @ACenterForAnts and @TheBotLebowski also combine for another ten thousand followers.  While I won’t send a mention of your product/URL/promotion to their followers, I do have other methods of driving traffic and building awareness to a targeted group of followers.  Lets talk.

May I have the source code?

Since I’ve been using this program for a few not-mentioned commercial purposes, I’m not interested in sharing the secret sauce.  I will, however, let you know it was pretty straightforward to do.  Anyone with a minimum of programming skill should be able to do this.

Will there be more bots?

Always.  I’m always on the lookout for new ideas.  Let me know if you have any.  The next one on my plate will be one for Eastbound and Down.

Visualization of the twitter networks of the top 150 pubcon tweeters

The size of the node indicates their in-degree, which represents the number of pubcon-tweeters that that follow a given user.  The color, from light-to-dark, represents the out-degree – the number of users within the community that they themselves follow.

Note that the degree does not represent the total number of followers/friends pertaining to each node – rather, it’s the connectedness within the group I extracted.  I’d like to run the visualization on a larger group of users, perhaps 300 or so, but due to rate limiting, I have to wait ten seconds between each twitter api query.

None of this would have been possible without the help and inspiration of @psychmedia and his excellent blog, OUseful.info.    His Gephi Tutorials have been a tremendous inspiration – I encourage all of you to follow him on twitter and subscribe to his blog.

Edit, here are the top 300 twitter-ers at pubcon 2010.  I’m going to try and run some numbers on an even larger piece of the pubcon network.  Make sure you use your browser’s zoom feature to view the image in more detail.

The connected-ness of the top 300 pubcon twitter-ers