From CS 585 lecture, Sept 7 2017

In [1]:
x = 5
In [6]:
x ** 100
Out[6]:
7888609052210118054117285652827862296732064351090230047702789306640625L
In [8]:
3/7
Out[8]:
0
In [9]:
from __future__ import division
In [10]:
3/7
Out[10]:
0.42857142857142855
In [11]:
import math
In [12]:
math.log(10)
Out[12]:
2.302585092994046
In [13]:
# data structures
xx = [1,2,5,10,20,50]
In [14]:
xx
Out[14]:
[1, 2, 5, 10, 20, 50]
In [15]:
xx[3]
Out[15]:
10
In [16]:
xx[0:2]
Out[16]:
[1, 2]
In [17]:
[x**2 for x in xx]
Out[17]:
[1, 4, 25, 100, 400, 2500]
In [19]:
text = """
This is an overview of Hurricane Irma, reported by journalists throughout the region. To see their latest dispatches from places hit by the storm, go to Thursday's live Irma updates.

SAN JUAN, P.R. — Hurricane Irma struck the northeast Caribbean with terrifying force Wednesday, its battering rain and winds of up to 185 miles per hour leaving a trail of chaos, wreckage and flooding from Barbuda to Puerto Rico, before taking aim at islands farther west and, beyond them, Florida.

Already one of the most powerful storms ever recorded, Irma could become one of the most destructive as well, depending on its path, and officials from Turks and Caicos to Florida pleaded with people to heed advisories to evacuate to shelters and higher ground. The National Hurricane Center described the hurricane as “potentially catastrophic.”

The storm made direct hits on Barbuda, St. Barthélemy, St. Martin, Anguilla and the British Virgin Islands, and raked the United States Virgin Islands and Puerto Rico with hurricane-force wind and torrential rain. Gaston Browne, prime minister of Antigua and Barbuda, said Irma had destroyed 95 percent of the structures on Barbuda, an island with about 1,600 people.

By Thursday morning, the authorities had confirmed that the hurricane had killed at least one person in Antigua and Barbuda; one on Anguilla, a British possession; and eight in French territory, which includes St. Barthélemy and the northern part of St. Martin. Another died in Puerto Rico while preparing for the storm.

Continue reading the main story
Irma “will bring life-threatening wind, storm surge and rainfall hazards” to the northern coast of Hispaniola, which includes the Dominican Republic and Haiti, on Thursday, the Hurricane Center warned. It will pass directly over — or very near — the low-lying islands of Turks and Caicos, a British possession, and parts of the Bahamas on Thursday and Friday, the center forecast, and push a storm surge of seawater 15 to 20 feet high. The surge could put large parts of the islands under water.

Across the islands that were hit on Wednesday, people posted videos and photos online of the hurricane’s fury: debris flying sideways in near-zero visibility, roofs ripped off structures, waves surging into buildings, downed trees and utility poles, and streets that had turned into raging currents carrying away cars and trucks.

Newsletter Sign UpContinue reading the main story
Sign Up for the Race/Related Newsletter
Join a deep and provocative exploration of race with a diverse group of New York Times journalists.


Enter your email address
 Sign Up

You agree to receive occasional updates and special offers for The New York Times's products and services.

SEE SAMPLE MANAGE EMAIL PREFERENCES PRIVACY POLICY OPT OUT OR CONTACT US ANYTIME
The French interior minister, Gérard Collomb, said the four sturdiest buildings on St. Martin had been destroyed, “which means that in all likelihood the more rustic buildings are probably totally or partially destroyed.”

View image on Twitter
View image on Twitter
 Follow
#eustorm @EUStormMap
Storm surge from #Irma has had a major effect on the Island of St-Martin, this image taken 55min ago from inside the eye wall! #TY Chi-Chi
8:59 AM - Sep 6, 2017
 17 17 Replies   488 488 Retweets   295 295 likes
Twitter Ads info and privacy
With phone lines and electricity cut in many places, and roads impassable, President Emmanuel Macron of France and other officials said it was far too early to assess the true toll, in either lives or property.

The aftermath of the storm will be “harsh and cruel,” Mr. Macron said after a crisis meeting at the Interior Ministry in Paris. “We will have victims to lament, and the material damage on the two islands is considerable.”

The devastating winds left many scrambling for safety.

Carmen Caballero, a 69-year-old retired doctor, was unsure at first whether to vacate her two-story home in San Juan, the capital of Puerto Rico. It is made of concrete, but has a metal roof that she feared might be ripped away. Then the power went out, the pelting rain and howling wind began, and she could hear debris crashing into things around her house and see tree branches falling into the streets.

She packed some water bottles, nuts, medicines, linens and other supplies in her car and drove to the shelter at the Roberto Clemente Coliseum. “All my neighbors left too,” she said over the phone from there. “I wasn’t going to stay alone in my house!”

In all, nearly 70 percent of households were without power and 17 percent of Puerto Rico was without water, Gov. Ricardo Rosselló said on Thursday, but otherwise the island was largely unscathed.

“We would like to start out thanking the Almighty,” Mr. Rosselló said. “Our prayers were answered.”

 
EXTREME WEATHER By CAMILLA SCHICK and ROBIN LINDSAY 1:51
Hurricane Irma Pummels Caribbean Islands
Video
Hurricane Irma Pummels Caribbean Islands
One of the strongest Atlantic storms ever recorded has caused damage on several islands, including Barbuda, Anguilla and St. Martin. By CAMILLA SCHICK and ROBIN LINDSAY on Publish Date September 6, 2017. Photo by Hector Retamal/Agence France-Presse — Getty Images. Watch in Times Video »
 Embed
ShareTweet
On Culebra, a small, rustic Puerto Rican island east of the main island, José Pérez, the municipality’s director of emergency management, took shelter with about 65 other people at a public high school Wednesday afternoon. Like many people there, he said, he lives in a wooden home.

“Right now we are feeling the fury of this hurricane,” Mr. Pérez said in Spanish by phone. “I was 13 and I obviously remember Hurricane Hugo, but this is something incomparable. This is something terrible, an experience out of this world.”

Kelsey Nowakowski and some friends boarded up her house on St. Thomas, part of the United States Virgin Islands, and hunkered down, listening to the howling and thumping outside. “Based on the water we took in we think there is significant damage to the roof but don’t think it blew off yet,” she said.

“We’ve all been in hurricanes before. There are five of us here, but have never felt anything like this before,” she added. “It feels seismic, it feels catastrophic.”

Alex Woolfall, a British public relations consultant who was staying at the Westin resort on the Dutch side of St. Martin, posted a stream of alarming updates on Twitter while taking shelter with other guests in the hotel’s reinforced stairwell. He described “thunderous sonic boom noises” and the “scream of things being hurled against the building.”

“This is like a movie I never want to see,” he wrote.

Throughout the region, communication was spotty at best; calls to people, businesses and government agencies — even the British Virgin Islands Department of Disaster Management — rang without being answered, or did not ring at all.

“Stuff is flying around outside, and the visibility is down to about 20 to 30 feet, and we haven’t hit the peak yet,” Gerry Yandel, executive editor of The Virgin Islands Daily News, said in the early afternoon. Two hours later, calls to the newspaper could not get through.
"""
In [21]:
len(text)
Out[21]:
7260
In [22]:
text[0:140]
Out[22]:
'\nThis is an overview of Hurricane Irma, reported by journalists throughout the region. To see their latest dispatches from places hit by the'
In [26]:
"Hello UMass Amherst how are you".split()
Out[26]:
['Hello', 'UMass', 'Amherst', 'how', 'are', 'you']
In [29]:
# tokenization is hard!
tokens = text.lower().split()
In [30]:
tokens[:50]
Out[30]:
['this',
 'is',
 'an',
 'overview',
 'of',
 'hurricane',
 'irma,',
 'reported',
 'by',
 'journalists',
 'throughout',
 'the',
 'region.',
 'to',
 'see',
 'their',
 'latest',
 'dispatches',
 'from',
 'places',
 'hit',
 'by',
 'the',
 'storm,',
 'go',
 'to',
 "thursday's",
 'live',
 'irma',
 'updates.',
 'san',
 'juan,',
 'p.r.',
 '\xe2\x80\x94',
 'hurricane',
 'irma',
 'struck',
 'the',
 'northeast',
 'caribbean',
 'with',
 'terrifying',
 'force',
 'wednesday,',
 'its',
 'battering',
 'rain',
 'and',
 'winds',
 'of']
In [31]:
wordcounts = {}
In [32]:
wordcounts['abcdef'] = 5
In [33]:
wordcounts
Out[33]:
{'abcdef': 5}
In [34]:
wordcounts = {}
In [36]:
for word in tokens:
    if word not in wordcounts:
        wordcounts[word] = 0
    wordcounts[word] += 1
In [37]:
wordcounts
Out[37]:
{'#eustorm': 1,
 '#irma': 1,
 '#ty': 1,
 '-': 1,
 '1,600': 1,
 '13': 1,
 '15': 1,
 '17': 3,
 '185': 1,
 '1:51': 1,
 '20': 2,
 '2017': 1,
 '2017.': 1,
 '295': 2,
 '30': 1,
 '488': 2,
 '55min': 1,
 '6,': 2,
 '65': 1,
 '69-year-old': 1,
 '70': 1,
 '8:59': 1,
 '95': 1,
 '@eustormmap': 1,
 'a': 16,
 'about': 3,
 'across': 1,
 'added.': 1,
 'address': 1,
 'ads': 1,
 'advisories': 1,
 'after': 1,
 'aftermath': 1,
 'afternoon.': 2,
 'against': 1,
 'agencies': 1,
 'ago': 1,
 'agree': 1,
 'aim': 1,
 'alarming': 1,
 'alex': 1,
 'all': 2,
 'all,': 1,
 'all.': 1,
 'almighty,\xe2\x80\x9d': 1,
 'alone': 1,
 'already': 1,
 'am': 1,
 'an': 3,
 'and': 49,
 'and,': 1,
 'anguilla': 2,
 'anguilla,': 1,
 'another': 1,
 'answered,': 1,
 'answered.\xe2\x80\x9d': 1,
 'antigua': 2,
 'anything': 1,
 'anytime': 1,
 'are': 3,
 'around': 2,
 'as': 2,
 'assess': 1,
 'at': 9,
 'atlantic': 1,
 'authorities': 1,
 'away': 1,
 'away.': 1,
 'bahamas': 1,
 'barbuda': 1,
 'barbuda,': 4,
 'barbuda;': 1,
 'barth\xc3\xa9lemy': 1,
 'barth\xc3\xa9lemy,': 1,
 'battering': 1,
 'be': 2,
 'become': 1,
 'been': 2,
 'before': 1,
 'before,\xe2\x80\x9d': 1,
 'before.': 1,
 'began,': 1,
 'being': 2,
 'best;': 1,
 'beyond': 1,
 'blew': 1,
 'boarded': 1,
 'boom': 1,
 'bottles,': 1,
 'branches': 1,
 'bring': 1,
 'british': 5,
 'browne,': 1,
 'building.\xe2\x80\x9d': 1,
 'buildings': 2,
 'buildings,': 1,
 'businesses': 1,
 'but': 5,
 'by': 7,
 'caballero,': 1,
 'caicos': 1,
 'caicos,': 1,
 'calls': 2,
 'camilla': 2,
 'capital': 1,
 'car': 1,
 'caribbean': 3,
 'carmen': 1,
 'carrying': 1,
 'cars': 1,
 'catastrophic.\xe2\x80\x9d': 2,
 'caused': 1,
 'center': 3,
 'chaos,': 1,
 'chi-chi': 1,
 'clemente': 1,
 'coast': 1,
 'coliseum.': 1,
 'collomb,': 1,
 'communication': 1,
 'concrete,': 1,
 'confirmed': 1,
 'considerable.\xe2\x80\x9d': 1,
 'consultant': 1,
 'contact': 1,
 'continue': 1,
 'could': 4,
 'crashing': 1,
 'crisis': 1,
 'cruel,\xe2\x80\x9d': 1,
 'culebra,': 1,
 'currents': 1,
 'cut': 1,
 'daily': 1,
 'damage': 3,
 'date': 1,
 'debris': 2,
 'deep': 1,
 'department': 1,
 'depending': 1,
 'described': 2,
 'destroyed': 1,
 'destroyed,': 1,
 'destroyed.\xe2\x80\x9d': 1,
 'destructive': 1,
 'devastating': 1,
 'did': 1,
 'died': 1,
 'direct': 1,
 'directly': 1,
 'director': 1,
 'disaster': 1,
 'dispatches': 1,
 'diverse': 1,
 'doctor,': 1,
 'dominican': 1,
 'don\xe2\x80\x99t': 1,
 'down': 1,
 'down,': 1,
 'downed': 1,
 'drove': 1,
 'dutch': 1,
 'early': 2,
 'east': 1,
 'editor': 1,
 'effect': 1,
 'eight': 1,
 'either': 1,
 'electricity': 1,
 'email': 2,
 'embed': 1,
 'emergency': 1,
 'emmanuel': 1,
 'enter': 1,
 'evacuate': 1,
 'even': 1,
 'ever': 2,
 'executive': 1,
 'experience': 1,
 'exploration': 1,
 'extreme': 1,
 'eye': 1,
 'falling': 1,
 'far': 1,
 'farther': 1,
 'feared': 1,
 'feeling': 1,
 'feels': 2,
 'feet': 1,
 'feet,': 1,
 'felt': 1,
 'first': 1,
 'five': 1,
 'flooding': 1,
 'florida': 1,
 'florida.': 1,
 'flying': 2,
 'follow': 1,
 'for': 4,
 'force': 1,
 'forecast,': 1,
 'four': 1,
 'france': 1,
 'france-presse': 1,
 'french': 2,
 'friday,': 1,
 'friends': 1,
 'from': 6,
 'fury': 1,
 'fury:': 1,
 'gaston': 1,
 'gerry': 1,
 'get': 1,
 'getty': 1,
 'go': 1,
 'going': 1,
 'gov.': 1,
 'government': 1,
 'ground.': 1,
 'group': 1,
 'guests': 1,
 'g\xc3\xa9rard': 1,
 'had': 6,
 'haiti,': 1,
 'has': 3,
 'have': 2,
 'haven\xe2\x80\x99t': 1,
 'hazards\xe2\x80\x9d': 1,
 'he': 4,
 'hear': 1,
 'hector': 1,
 'heed': 1,
 'her': 4,
 'here,': 1,
 'high': 1,
 'high.': 1,
 'higher': 1,
 'hispaniola,': 1,
 'hit': 3,
 'hits': 1,
 'home': 1,
 'home.': 1,
 'hotel\xe2\x80\x99s': 1,
 'hour': 1,
 'hours': 1,
 'house': 2,
 'house!\xe2\x80\x9d': 1,
 'households': 1,
 'howling': 2,
 'hugo,': 1,
 'hunkered': 1,
 'hurled': 1,
 'hurricane': 9,
 'hurricane,\xe2\x80\x9d': 1,
 'hurricane-force': 1,
 'hurricanes': 1,
 'hurricane\xe2\x80\x99s': 1,
 'i': 2,
 'image': 3,
 'images.': 1,
 'impassable,': 1,
 'in': 19,
 'includes': 2,
 'including': 1,
 'incomparable.': 1,
 'info': 1,
 'inside': 1,
 'interior': 2,
 'into': 4,
 'irma': 7,
 'irma,': 1,
 'is': 9,
 'island': 4,
 'island,': 1,
 'islands': 10,
 'islands,': 3,
 'it': 5,
 'its': 2,
 'join': 1,
 'jos\xc3\xa9': 1,
 'journalists': 1,
 'journalists.': 1,
 'juan,': 2,
 'kelsey': 1,
 'killed': 1,
 'lament,': 1,
 'large': 1,
 'largely': 1,
 'later,': 1,
 'latest': 1,
 'least': 1,
 'leaving': 1,
 'left': 2,
 'life-threatening': 1,
 'like': 4,
 'likelihood': 1,
 'likes': 1,
 'lindsay': 2,
 'linens': 1,
 'lines': 1,
 'listening': 1,
 'live': 1,
 'lives': 2,
 'low-lying': 1,
 'macron': 2,
 'made': 2,
 'main': 3,
 'major': 1,
 'manage': 1,
 'management': 1,
 'management,': 1,
 'many': 3,
 'martin': 1,
 'martin,': 2,
 'martin.': 2,
 'material': 1,
 'means': 1,
 'medicines,': 1,
 'meeting': 1,
 'metal': 1,
 'might': 1,
 'miles': 1,
 'minister': 1,
 'minister,': 1,
 'ministry': 1,
 'more': 1,
 'morning,': 1,
 'most': 2,
 'movie': 1,
 'mr.': 3,
 'municipality\xe2\x80\x99s': 1,
 'my': 2,
 'national': 1,
 'near': 1,
 'near-zero': 1,
 'nearly': 1,
 'neighbors': 1,
 'never': 2,
 'new': 2,
 'news,': 1,
 'newsletter': 2,
 'newspaper': 1,
 'noises\xe2\x80\x9d': 1,
 'northeast': 1,
 'northern': 2,
 'not': 2,
 'now': 1,
 'nowakowski': 1,
 'nuts,': 1,
 'obviously': 1,
 'occasional': 1,
 'of': 35,
 'off': 2,
 'offers': 1,
 'officials': 2,
 'on': 20,
 'one': 5,
 'online': 1,
 'opt': 1,
 'or': 5,
 'other': 4,
 'otherwise': 1,
 'out': 3,
 'out,': 1,
 'outside,': 1,
 'outside.': 1,
 'over': 2,
 'overview': 1,
 'p.r.': 1,
 'packed': 1,
 'paris.': 1,
 'part': 2,
 'partially': 1,
 'parts': 2,
 'pass': 1,
 'path,': 1,
 'peak': 1,
 'pelting': 1,
 'people': 4,
 'people,': 1,
 'people.': 1,
 'per': 1,
 'percent': 3,
 'person': 1,
 'phone': 2,
 'phone.': 1,
 'photo': 1,
 'photos': 1,
 'places': 1,
 'places,': 1,
 'pleaded': 1,
 'poles,': 1,
 'policy': 1,
 'possession,': 1,
 'possession;': 1,
 'posted': 2,
 'power': 2,
 'powerful': 1,
 'prayers': 1,
 'preferences': 1,
 'preparing': 1,
 'president': 1,
 'prime': 1,
 'privacy': 2,
 'probably': 1,
 'products': 1,
 'property.': 1,
 'provocative': 1,
 'public': 2,
 'publish': 1,
 'puerto': 6,
 'pummels': 2,
 'push': 1,
 'put': 1,
 'p\xc3\xa9rez': 1,
 'p\xc3\xa9rez,': 1,
 'race': 1,
 'race/related': 1,
 'raging': 1,
 'rain': 2,
 'rain.': 1,
 'rainfall': 1,
 'raked': 1,
 'rang': 1,
 'reading': 2,
 'receive': 1,
 'recorded': 1,
 'recorded,': 1,
 'region,': 1,
 'region.': 1,
 'reinforced': 1,
 'relations': 1,
 'remember': 1,
 'replies': 1,
 'reported': 1,
 'republic': 1,
 'resort': 1,
 'retamal/agence': 1,
 'retired': 1,
 'retweets': 1,
 'rican': 1,
 'ricardo': 1,
 'rico': 3,
 'rico,': 1,
 'rico.': 1,
 'ring': 1,
 'ripped': 2,
 'roads': 1,
 'roberto': 1,
 'robin': 2,
 'roof': 2,
 'roofs': 1,
 'rossell\xc3\xb3': 2,
 'rustic': 2,
 'safety.': 1,
 'said': 8,
 'said,': 1,
 'said.': 2,
 'sample': 1,
 'san': 2,
 'schick': 2,
 'school': 1,
 'scrambling': 1,
 'seawater': 1,
 'see': 3,
 'see,\xe2\x80\x9d': 1,
 'seismic,': 1,
 'sep': 1,
 'september': 1,
 'services.': 1,
 'several': 1,
 'sharetweet': 1,
 'she': 6,
 'shelter': 3,
 'shelters': 1,
 'side': 1,
 'sideways': 1,
 'sign': 3,
 'significant': 1,
 'small,': 1,
 'some': 2,
 'something': 2,
 'sonic': 1,
 'spanish': 1,
 'special': 1,
 'spotty': 1,
 'st-martin,': 1,
 'st.': 8,
 'stairwell.': 1,
 'start': 1,
 'states': 2,
 'stay': 1,
 'staying': 1,
 'storm': 5,
 'storm,': 1,
 'storm.': 1,
 'storms': 2,
 'story': 2,
 'stream': 1,
 'streets': 1,
 'streets.': 1,
 'strongest': 1,
 'struck': 1,
 'structures': 1,
 'structures,': 1,
 'sturdiest': 1,
 'supplies': 1,
 'surge': 4,
 'surging': 1,
 'taken': 1,
 'taking': 2,
 'terrible,': 1,
 'terrifying': 1,
 'territory,': 1,
 'thanking': 1,
 'that': 5,
 'the': 70,
 'their': 1,
 'them,': 1,
 'then': 1,
 'there': 2,
 'there,': 1,
 'there.': 1,
 'things': 2,
 'think': 2,
 'this': 7,
 'thomas,': 1,
 'through.': 1,
 'throughout': 2,
 'thumping': 1,
 'thursday': 2,
 "thursday's": 1,
 'thursday,': 2,
 'times': 2,
 "times's": 1,
 'to': 24,
 'toll,': 1,
 'too': 1,
 'too,\xe2\x80\x9d': 1,
 'took': 2,
 'torrential': 1,
 'totally': 1,
 'trail': 1,
 'tree': 1,
 'trees': 1,
 'trucks.': 1,
 'true': 1,
 'turks': 2,
 'turned': 1,
 'twitter': 4,
 'two': 2,
 'two-story': 1,
 'under': 1,
 'united': 2,
 'unscathed.': 1,
 'unsure': 1,
 'up': 4,
 'upcontinue': 1,
 'updates': 2,
 'updates.': 1,
 'us': 2,
 'utility': 1,
 'vacate': 1,
 'very': 1,
 'victims': 1,
 'video': 2,
 'videos': 1,
 'view': 2,
 'virgin': 5,
 'visibility': 1,
 'visibility,': 1,
 'wall!': 1,
 'want': 1,
 'warned.': 1,
 'was': 7,
 'wasn\xe2\x80\x99t': 1,
 'watch': 1,
 'water': 2,
 'water,': 1,
 'water.': 1,
 'waves': 1,
 'we': 4,
 'weather': 1,
 'wednesday': 1,
 'wednesday,': 2,
 'well,': 1,
 'went': 1,
 'were': 3,
 'west': 1,
 'westin': 1,
 'whether': 1,
 'which': 2,
 'while': 2,
 'who': 1,
 'will': 3,
 'wind': 2,
 'wind,': 1,
 'winds': 2,
 'with': 8,
 'without': 3,
 'wooden': 1,
 'woolfall,': 1,
 'world.\xe2\x80\x9d': 1,
 'would': 1,
 'wreckage': 1,
 'wrote.': 1,
 'yandel,': 1,
 'yet,\xe2\x80\x9d': 2,
 'york': 2,
 'you': 1,
 'your': 1,
 '\xc2\xbb': 1,
 '\xe2\x80\x94': 6,
 '\xe2\x80\x9call': 1,
 '\xe2\x80\x9cbased': 1,
 '\xe2\x80\x9charsh': 1,
 '\xe2\x80\x9ci': 2,
 '\xe2\x80\x9cit': 1,
 '\xe2\x80\x9cour': 1,
 '\xe2\x80\x9cpotentially': 1,
 '\xe2\x80\x9cright': 1,
 '\xe2\x80\x9cscream': 1,
 '\xe2\x80\x9cstuff': 1,
 '\xe2\x80\x9cthis': 1,
 '\xe2\x80\x9cthunderous': 1,
 '\xe2\x80\x9cwe': 2,
 '\xe2\x80\x9cwe\xe2\x80\x99ve': 1,
 '\xe2\x80\x9cwhich': 1,
 '\xe2\x80\x9cwill': 1}
In [39]:
wc_pairs = wordcounts.items()
In [40]:
wc_pairs[:5]
Out[40]:
[('neighbors', 1), ('all', 2), ('recorded', 1), ('four', 1), ('debris', 2)]
In [52]:
def get_value_for_item(pair):
#     print pair
#     word,count = pair
#     return -count
    return -pair[1]
wc_pairs.sort(key=get_value_for_item)
In [50]:
# shorthand for the above
wc_pairs.sort(key=lambda (w,c): -c )
In [49]:
print wc_pairs[:500]

[('the', 70), ('and', 49), ('of', 35), ('to', 24), ('on', 20), ('in', 19), ('a', 16), ('islands', 10), ('is', 9), ('hurricane', 9), ('at', 9), ('st.', 8), ('said', 8), ('with', 8), ('this', 7), ('by', 7), ('was', 7), ('irma', 7), ('puerto', 6), ('she', 6), ('had', 6), ('from', 6), ('\xe2\x80\x94', 6), ('british', 5), ('one', 5), ('storm', 5), ('that', 5), ('but', 5), ('it', 5), ('or', 5), ('virgin', 5), ('barbuda,', 4), ('island', 4), ('could', 4), ('he', 4), ('we', 4), ('like', 4), ('people', 4), ('twitter', 4), ('for', 4), ('surge', 4), ('into', 4), ('her', 4), ('up', 4), ('other', 4), ('islands,', 3), ('sign', 3), ('damage', 3), ('main', 3), ('image', 3), ('out', 3), ('17', 3), ('were', 3), ('mr.', 3), ('without', 3), ('rico', 3), ('hit', 3), ('see', 3), ('are', 3), ('many', 3), ('will', 3), ('center', 3), ('has', 3), ('caribbean', 3), ('about', 3), ('an', 3), ('percent', 3), ('shelter', 3), ('all', 2), ('debris', 2), ('winds', 2), ('thursday', 2), ('updates', 2), ('295', 2), ('video', 2), ('juan,', 2), ('while', 2), ('new', 2), ('ever', 2), ('public', 2), ('never', 2), ('french', 2), ('water', 2), ('my', 2), ('lindsay', 2), ('two', 2), ('rustic', 2), ('phone', 2), ('left', 2), ('roof', 2), ('something', 2), ('united', 2), ('camilla', 2), ('said.', 2), ('martin,', 2), ('martin.', 2), ('buildings', 2), ('wind', 2), ('wednesday,', 2), ('its', 2), ('20', 2), ('not', 2), ('house', 2), ('york', 2), ('times', 2), ('think', 2), ('story', 2), ('privacy', 2), ('anguilla', 2), ('took', 2), ('part', 2), ('states', 2), ('catastrophic.\xe2\x80\x9d', 2), ('san', 2), ('have', 2), ('schick', 2), ('which', 2), ('interior', 2), ('afternoon.', 2), ('most', 2), ('northern', 2), ('storms', 2), ('over', 2), ('there', 2), ('feels', 2), ('calls', 2), ('view', 2), ('officials', 2), ('turks', 2), ('macron', 2), ('reading', 2), ('email', 2), ('taking', 2), ('6,', 2), ('throughout', 2), ('described', 2), ('been', 2), ('lives', 2), ('antigua', 2), ('robin', 2), ('things', 2), ('parts', 2), ('rain', 2), ('thursday,', 2), ('early', 2), ('posted', 2), ('off', 2), ('i', 2), ('pummels', 2), ('being', 2), ('around', 2), ('some', 2), ('\xe2\x80\x9cwe', 2), ('yet,\xe2\x80\x9d', 2), ('newsletter', 2), ('rossell\xc3\xb3', 2), ('be', 2), ('ripped', 2), ('power', 2), ('\xe2\x80\x9ci', 2), ('flying', 2), ('made', 2), ('us', 2), ('howling', 2), ('as', 2), ('includes', 2), ('488', 2), ('neighbors', 1), ('recorded', 1), ('four', 1), ('follow', 1), ('getty', 1), ('electricity', 1), ('chi-chi', 1), ('see,\xe2\x80\x9d', 1), ('territory,', 1), ('retweets', 1), ('under', 1), ('destroyed.\xe2\x80\x9d', 1), ('wind,', 1), ('very', 1), ('school', 1), ('went', 1), ('wooden', 1), ('wednesday', 1), ('large', 1), ('race', 1), ('\xe2\x80\x9cstuff', 1), ('poles,', 1), ('barbuda;', 1), ('1,600', 1), ('warned.', 1), ('streets', 1), ('direct', 1), ('go', 1), ('pass', 1), ('even', 1), ('aim', 1), ('preferences', 1), ('culebra,', 1), ('feet,', 1), ('sideways', 1), ('falling', 1), ('safety.', 1), ('unsure', 1), ("thursday's", 1), ('hotel\xe2\x80\x99s', 1), ('raked', 1), ('reported', 1), ('phone.', 1), ('address', 1), ('alone', 1), ('aftermath', 1), ('extreme', 1), ('gerry', 1), ('receive', 1), ('30', 1), ('experience', 1), ('leaving', 1), ('seismic,', 1), ('products', 1), ('added.', 1), ('spotty', 1), ('house!\xe2\x80\x9d', 1), ('retired', 1), ('manage', 1), ('browne,', 1), ('guests', 1), ('rican', 1), ('crisis', 1), ('eye', 1), ('stream', 1), ('would', 1), ('hector', 1), ('dispatches', 1), ('france', 1), ('live', 1), ('taken', 1), ('start', 1), ('more', 1), ('rainfall', 1), ('journalists', 1), ('st-martin,', 1), ('visibility', 1), ('updates.', 1), ('carrying', 1), ('shelters', 1), ('95', 1), ('noises\xe2\x80\x9d', 1), ('and,', 1), ('join', 1), ('hour', 1), ('car', 1), ('began,', 1), ('minister,', 1), ('\xe2\x80\x9cwe\xe2\x80\x99ve', 1), ('hurricane\xe2\x80\x99s', 1), ('carmen', 1), ('gaston', 1), ('life-threatening', 1), ('high.', 1), ('sample', 1), ('caused', 1), ('water,', 1), ('want', 1), ('water.', 1), ('haven\xe2\x80\x99t', 1), ('fury:', 1), ('florida.', 1), ('said,', 1), ('nowakowski', 1), ('barth\xc3\xa9lemy,', 1), ('thanking', 1), ('far', 1), ('after', 1), ('chaos,', 1), ('waves', 1), ('date', 1), ('president', 1), ('pleaded', 1), ('hispaniola,', 1), ('scrambling', 1), ('branches', 1), ('remember', 1), ('lines', 1), ('region.', 1), ('anytime', 1), ('region,', 1), ('westin', 1), ('enter', 1), ('impassable,', 1), ('reinforced', 1), ('nearly', 1), ('wall!', 1), ('caicos,', 1), ('september', 1), ('emmanuel', 1), ('rang', 1), ('concrete,', 1), ('2017.', 1), ('hurricane,\xe2\x80\x9d', 1), ('including', 1), ('down,', 1), ('replies', 1), ('wrote.', 1), ('before', 1), ('hazards\xe2\x80\x9d', 1), ('ricardo', 1), ('destroyed', 1), ('resort', 1), ('\xe2\x80\x9charsh', 1), ('offers', 1), ('retamal/agence', 1), ('policy', 1), ('2017', 1), ('might', 1), ('then', 1), ('material', 1), ('seawater', 1), ('now', 1), ('killed', 1), ('hunkered', 1), ('1:51', 1), ('down', 1), ('alex', 1), ('\xe2\x80\x9cthunderous', 1), ('\xe2\x80\x9cthis', 1), ('all,', 1), ('possession;', 1), ('did', 1), ('possession,', 1), ('barbuda', 1), ('side', 1), ('another', 1), ('staying', 1), ('beyond', 1), ('small,', 1), ('special', 1), ('best;', 1), ('linens', 1), ('publish', 1), ('building.\xe2\x80\x9d', 1), ('storm,', 1), ('dutch', 1), ('get', 1), ('wasn\xe2\x80\x99t', 1), ('roofs', 1), ('cars', 1), ('sturdiest', 1), ('hurricane-force', 1), ('management', 1), ('put', 1), ('likelihood', 1), ('municipality\xe2\x80\x99s', 1), ('jos\xc3\xa9', 1), ('ministry', 1), ('boarded', 1), ('florida', 1), ('haiti,', 1), ('first', 1), ('major', 1), ('already', 1), ('structures,', 1), ('doctor,', 1), ('powerful', 1), ('blew', 1), ('feet', 1), ('cruel,\xe2\x80\x9d', 1), ('directly', 1), ('spanish', 1), ('force', 1), ('ring', 1), ('drove', 1), ('households', 1), ('destructive', 1), ('prayers', 1), ('least', 1), ('bahamas', 1), ('their', 1), ('terrifying', 1), ('185', 1), ('world.\xe2\x80\x9d', 1), ('structures', 1), ('property.', 1), ('too', 1), ('journalists.', 1), ('pelting', 1), ('later,', 1), ('13', 1), ('15', 1), ('tree', 1), ('don\xe2\x80\x99t', 1), ('feeling', 1), ('caicos', 1), ('\xe2\x80\x9cbased', 1), ('turned', 1), ('considerable.\xe2\x80\x9d', 1), ('well,', 1), ('incomparable.', 1), ('obviously', 1), ('depending', 1), ('-', 1), ('woolfall,', 1), ('online', 1), ('terrible,', 1), ('who', 1), ('medicines,', 1), ('eight', 1), ('significant', 1), ('70', 1), ('clemente', 1), ('devastating', 1), ('metal', 1), ('rico.', 1), ('rico,', 1), ('movie', 1), ('atlantic', 1), ('latest', 1), ('path,', 1), ('race/related', 1), ('bring', 1), ('businesses', 1), ('p\xc3\xa9rez', 1), ('crashing', 1), ('northeast', 1), ('true', 1), ('going', 1), ('caballero,', 1), ('low-lying', 1), ('almighty,\xe2\x80\x9d', 1), ('executive', 1), ('means', 1), ('lament,', 1), ('capital', 1), ('watch', 1), ('g\xc3\xa9rard', 1), ('trees', 1), ('coast', 1), ('preparing', 1), ('supplies', 1), ('\xe2\x80\x9cit', 1), ('prime', 1), ('across', 1), ('storm.', 1), ('miles', 1), ('\xe2\x80\x9cscream', 1), ('them,', 1), ('anguilla,', 1), ('streets.', 1), ('disaster', 1), ('sep', 1), ('roberto', 1), ('national', 1), ('55min', 1), ('packed', 1), ('feared', 1), ('ground.', 1), ('alarming', 1), ('away', 1), ('p.r.', 1), ('probably', 1), ('recorded,', 1), ('battering', 1), ('agencies', 1), ('answered,', 1), ('group', 1), ('hours', 1), ('hits', 1), ('people,', 1), ('all.', 1), ('visibility,', 1), ('against', 1), ('toll,', 1), ('torrential', 1), ('otherwise', 1), ('coliseum.', 1), ('overview', 1), ('likes', 1), ('two-story', 1), ('there,', 1), ('there.', 1), ('thomas,', 1), ('west', 1), ('fury', 1), ('through.', 1), ('meeting', 1), ('relations', 1), ('photos', 1), ('minister', 1), ('east', 1), ('consultant', 1), ('near', 1), ('newspaper', 1), ('outside.', 1), ('\xe2\x80\x9cour', 1), ('outside,', 1), ('flooding', 1), ('provocative', 1), ('friday,', 1), ('barth\xc3\xa9lemy', 1)]
In [ ]:
In [57]:
# shorthand for the above
wc_pairs.sort(key=lambda (w,c): w )
In [58]:
wc_pairs[:100]
Out[58]:
[('#eustorm', 1),
 ('#irma', 1),
 ('#ty', 1),
 ('-', 1),
 ('1,600', 1),
 ('13', 1),
 ('15', 1),
 ('17', 3),
 ('185', 1),
 ('1:51', 1),
 ('20', 2),
 ('2017', 1),
 ('2017.', 1),
 ('295', 2),
 ('30', 1),
 ('488', 2),
 ('55min', 1),
 ('6,', 2),
 ('65', 1),
 ('69-year-old', 1),
 ('70', 1),
 ('8:59', 1),
 ('95', 1),
 ('@eustormmap', 1),
 ('a', 16),
 ('about', 3),
 ('across', 1),
 ('added.', 1),
 ('address', 1),
 ('ads', 1),
 ('advisories', 1),
 ('after', 1),
 ('aftermath', 1),
 ('afternoon.', 2),
 ('against', 1),
 ('agencies', 1),
 ('ago', 1),
 ('agree', 1),
 ('aim', 1),
 ('alarming', 1),
 ('alex', 1),
 ('all', 2),
 ('all,', 1),
 ('all.', 1),
 ('almighty,\xe2\x80\x9d', 1),
 ('alone', 1),
 ('already', 1),
 ('am', 1),
 ('an', 3),
 ('and', 49),
 ('and,', 1),
 ('anguilla', 2),
 ('anguilla,', 1),
 ('another', 1),
 ('answered,', 1),
 ('answered.\xe2\x80\x9d', 1),
 ('antigua', 2),
 ('anything', 1),
 ('anytime', 1),
 ('are', 3),
 ('around', 2),
 ('as', 2),
 ('assess', 1),
 ('at', 9),
 ('atlantic', 1),
 ('authorities', 1),
 ('away', 1),
 ('away.', 1),
 ('bahamas', 1),
 ('barbuda', 1),
 ('barbuda,', 4),
 ('barbuda;', 1),
 ('barth\xc3\xa9lemy', 1),
 ('barth\xc3\xa9lemy,', 1),
 ('battering', 1),
 ('be', 2),
 ('become', 1),
 ('been', 2),
 ('before', 1),
 ('before,\xe2\x80\x9d', 1),
 ('before.', 1),
 ('began,', 1),
 ('being', 2),
 ('best;', 1),
 ('beyond', 1),
 ('blew', 1),
 ('boarded', 1),
 ('boom', 1),
 ('bottles,', 1),
 ('branches', 1),
 ('bring', 1),
 ('british', 5),
 ('browne,', 1),
 ('building.\xe2\x80\x9d', 1),
 ('buildings', 2),
 ('buildings,', 1),
 ('businesses', 1),
 ('but', 5),
 ('by', 7),
 ('caballero,', 1)]