On English Words Frequencies From Shakespeare To Modern Soap Operas

Word frequencies are used in many text mining and information retrieval applications. Nowadays, and thanks to the Internet, it is easy to find lists of word frequencies for various languages.

Wikitionary, the sister website of Wikipedia provides such lists for many languages. I have decided to extract two of these lists the Gutenberg list and the TV movies list.

The Gutenberg list was built from the Gutenberg project book collection. This is a collection of classic books that fell into the public domain. The list has been last updated in 2005. All books in the considered collection were published before 1923. The TV movies list was built from TV shows and movies scripts and transcripts.

I extracted both lists by HTML scraping the Wikitionary pages.  After some filtering, I got two dictionaries: The Gutenberg dictionary and the movies dictionary. The first dictionary contains 36500 words, whereas the second dictionar has 37500 words.Both dictionaries will be used in some future text mining projects.

The first question I was interested in was to find out the most stable words in English. That is words who have very close frequencies in the Gutenburg and movies lists. To do so, I have defined the relative frequency change (rfc) metric as follows:

rfc = |f_gut - f_{mov| / f}gut

As can be seen from the equation above, the Gutenberg dictionary was used as a reference as one usually moves from past to present. The table below shows the words with an rfc less than 1%.

I have been expecting this list to be mostly populated by stop words. This is obviously not the case. Anectoditically, the most stable US state (frequency-wise) is Texas and the most stable country is Cuba. Champanzees did not evolve much over time. Also, the only family-related words that made it to this list are motherhood and stepmother. Finally, the only biblical character who managed to get on the list, or should I say who managed to be raised from the dead is Lazarus.