Text ID
This project offers you an opportunity to use Python to develop a statistical model of text that will allow you to “ID” an author or style—with varying degrees of success—from samples of text!
Background
Statistical models are one way to quantify how similar one piece of text is to another. Such models were used as evidence that the book The Cuckoo’s Calling was written by J. K. Rowling (using the name Robert Galbraith) in the summer of 2013. Details on that story and the analysis done appear at this link.
The comparative analysis of those works used simple models—perhaps surprisingly simple—of an “author’s style,” for example, the distribution of word lengths in a document. The five “text features” that we ask you to implement for this project are:
- The distribution of words used by an author.
- The distribution of word lengths (noted above).
- The distribution of word stems used (e.g., “spam” and “spamming” would have the same stem).
- The distribution of sentence lengths used.
- Plus, the project asks you to include one more “text feature” of your own design:
- For example, you could use punctuation in some way.
- Or you could compute something else about the text—it could be the number of words ending in “ness,” something about capitalization, or anything else you can compute or count using Python…
- Here is an article on the punctuation analysis of several works…
Perhaps the most-common and most-successful application of this kind of textual analysis is Spam detection and filtering.
Overview of the TextModel project
To create your Bayesian classifier (which is what this project is!) you will create TextModels, which will be lists of dictionaries. Each of these lists should have at least five dictionaries. Here is some starter code that creates a single model with the following four dictionaries, initially empty:
wordswordlengthsstemssentencelengths- Remember that you’ll need one more dictionary for a text feature of your own design! Punctuation is totally ok, as are many other options.
Feel free to start with this:
We provide more detail below, along with tests to make sure your methods are working.
What methods do we need?
First, we recommend starting with these methods:
Method 1: readTextFromFile(self, filename): should accept a filename (a string) and set self.text to all of the text in that file, represented as a single, very large string.
Method 2: makeSentenceLengths(self): should use the text in self.text to create the self.sentencelengths dictionary.
Method 3: __repr__(self): should generate a string containing all of the dictionaries in the text model, in full—for testing and checking how they are working.
- This one is provided above…
There’s a good start! Next, implement these:
Method 4: cleanString(self, s): should accept a string s and return a revised string with no punctuation and no upper-case letters.
Method 5: makeWordLengths(self): should use the text in self.text to create the self.wordlengths dictionary.
Method 6: makeWords(self): should use the text in self.text to create the self.words dictionary
Method 7: makeStems(self): should use the text in self.text to create the self.stems dictionary
Method 8: makeOtherFeature(s): should use the text in self.text to create the otherfeature dictionary. This one is up to you (I used puctuation—feel free to use that or something else…!)
Note that you should choose a better name than otherfeature!
And, the final ones:
Method 9: normalizeDictionary(self, d): should accept any single one of the model dictionaries d and return a normalized version, i.e., one in which the values add up to 1.0. Remember to return, not print!
- Use
d.values()to obtain a list of all of the values of a dictionaryd. - In your code, you’ll use something like this:
self.normalizedWords = self.normalizedDictionary(self.words)
- (although you might be able to assign the result to
self.wordsitself, depending on how your code works).
Method 10: smallestValue(self, nd1, nd2): should accept any two model dictionaries nd1 and nd2 and return the smallest positive (non-zero) value across them both. When used, the two input dictionaries are likely to be normalized, so we’ve named them nd1 and nd2, but this method should work regardless of whether they’re normalized or not. Remember to return, not print!
Method 11: compareDictionaries(self, d, nd1, nd2): should compute
- The log-probability that the dictionary
darose from the distribution of data in the normalized dictionarynd1and. - The log-probability that dictionary
darose from the distribution of data in normalized dictionarynd2 and. - Both of those log-probabilities should be returned in a list, e.g.,
[lp1, lp2], which stands for log-prob1 and log-prob2. - Both
nd1andnd2should be normalized first, before computing the log-probabilities. ddoes not have to be normalized (in fact, it shouldn’t be).- For any keys in
dthat do not appear ind1ord2, you should use a value of half the smallest value across both normalized model dictionaries,d1andd2. UsesmallestValueto help you here!
Method 12: compareTextWithTwoModels(self, model1, model2): Each of self, model1, and model2 should be a TextModel with at least five dictionaries. This function should use the compareDictionaries method for each of those five dictionaries in self against the corresponding (normalized!) dictionaries in model1 and model2. Then, this method should
- Print out the comparative results of the log-probabilities for all five dictionaries.
- Implement some system for determining which of the two models
selfis closer to, e.g., majority vote from the five dictionaries or weighted sum of the log-probabilities… - Declare a “winner”—namely the model with which
selfis more similar.
Below, there are examples you can try to ensure things are working as they should…
In addition, this project asks you to use your text-analysis engine! Be sure to:
- Create at least two contrasting text models. These are the “trained” models. You will need to find one or more texts to train each model. If you have multiple texts, you can simply place them all into a single file for processing. For instance, Shakespeare vs. J. K. Rowling was our slides’ running example; lots of others are possible! Your two (or more) trained models should use a moderate-or-greater amount of text. (Models based on too small a source text tend to be “brittle”—they depend too much on the idiosyncrasies of that source.)
- Choose at least two other “test texts” that will be compared to your two trained models. Here, you might choose your own writing, or other writing from one or both of your trained models’ sources, or something else entirely. The “test” texts or “unknown” texts can be shorter than the trained models and still work well.
- Run the comparison of your two “test texts” against the two trained models (for a total of at least four comparisons), and write a short paragraph reporting the results, how you obtained them (including how you decided which model would be declared the winner if there were differing underlying dictionary-comparisons), and whether you expected those results…!
Method-by-method details
This section has some method-by-method details on the above pieces of your TextModel project.
readTextFromFile(self, filename)
This method should take in a filename (a string) and should set self.text to the text in that file represented as a single, possibly very large string.
Here is an example to try—place this text into a file named test.txt:
Then, run these commands at the python command prompt:
TM = TextModel()
# be sure this is in a file named test.txt
test_text = """This is a small sentence. This isn't a small sentence, because
this sentence contains more than 10 words and a number! This isn't
a question, is it?"""
TM.readTextFromFile("test.txt")
print(TM.text == test_text)It should print True!
makeSentenceLengths(self)
This method should use the text in self.text to create the Python dictionary self.sentencelengths.
You’ll want to build from the previous assignment’s starting point on this—it reuses the way you detected sentence endings in the Markov problem.
Here is a test to check your results:
which should yield this dictionary:
because there is one sentence of length 16, one of length 5, and one of length 6.
Note that the key:value pairs in a dictionary may be in any order—yours may differ (this is OK!)
cleanString(self, s)
This method should accept a string s and return a string with no punctuation and no upper-case letters.
- You can use
s = s.lower()to get rid of upper-case letters. - Removing punctuation is up to you…
- One hint: use
import string - And then create a loop such as
for p in string.punctuation - Which will loop the variable
pover each possible item ofpunctuation… - Then you can use
s = s.replace(p, '')to replace each item ofpunctuationwith the empty string! - Do try it on a small example first, if you decide to try this path…
- One hint: use
- However, don’t use recursion—use loops (recursion will run out of memory on big strings!)
Here is a test to check your results:
which should yield this string:
You probably won’t want to strip numbers away!
makeWordLengths(self)
This method is similar to makeSentenceLengths—except that it makes a dictionary of the word-length features (which are what Sarah Pavis used to identify J. K. Rowling :-))
Be sure to use the cleaned string for this one—no punctuation!
Here is a test to check your results:
which should yield this dictionary:
Again, the key:value pairs in your dictionary may be in another order—but they should be the same key:value pairs!
makeWords(self)
This method is similar to makeWordLengths—except that it makes a dictionary of words themselves (cleaned!) So, be sure to use the cleaned string for this one, too.
Here is a test to check your results:
which should yield this dictionary:
As always, the key:value pairs in your dictionary may be in another order—but they should be the same key:value pairs!
makeStems(self)
This method is similar to makeWords—except that it makes a dictionary of the stems of the words themselves (cleaned!) Again, be sure to use the cleaned string for this one, too.
Here is a test to check your results:
which, for me, yielded this dictionary:
Yours may be different, depending on your stemmer! This is totally OK—as long as it is reasonable (not every word has the same stem!), your stemmer is completely OK. (Try adding some different forms of the same words to the file, as well…)
To implement makeStems, you will need a function that accepts words and outputs stems. You’re invited to:
- Write your own (it should include at least ten stemming rules of the sorts we discussed in class. Don’t feel you need to implement all possible rules—in fact, it would be impossible to do so, since there are so many exceptions!
- Or, find a stemming library online and incorporate it into your project.
- Or, use this Porter Stemmer: PorterStemmerForTextID , which is one of the original stemming algorithms for English—it’s not perfect either, but it is one of the early standards use in the linguistics community.
- To use it, you would place that code into a file named
porter.pyand then includefrom porter import create_stemat the top of your file. From there, callcreate_stem(w), wherewis a word whose stem you’d like!
makeOtherFeature(self)
This function is similar to makeWords, but it’s for the feature you choose and create. I named mine makePunctuation, but others have used very different features, depending on the texts they’re interested in. Do be sure there are at least two items in the dictionary (for texts of a reasonable size)—otherwise it won’t be a good feature for comparisons!
Here’s my test—please edit, adapt, and adjust it to suit your analysis!
which, for me, yielded this dictionary:
__repr__(self)
This method should return a representation of all the TM’s dictionaries in full, for testing and checking how they are working.
This is provided in the starter code, above.
Testing your TextModel’s model-building
For the milestone, here is a small set of tests that will check whether your TextModel is working. As above, you can:
- Run these from the terminal.
- Or, paste them at the bottom of your file and run them from there—easier!
Note that you don’t need to create a punctuation dictionary! You should, however, create some dictionary other than the basic four: stems, words, word lengths, and sentence lengths. (And, if you like punctuation, it’s OK to use it!)
Results
Our results were the following—check that at least the sentence lengths, word lengths, and words are the same!
Small variations in printing are certainly ok!
Additional method-by-method details
These next methods (functions) are more algorithmically challenging, so we describe them in more detail.
normalizeDictionary(self, d)
This method should accept any single one of the model dictionaries d and return a normalized version, i.e., one in which the values add up to 1.0. Remember to return, not print!
Hint: You can use d.values() to obtain a list of all of the values of a dictionary, d.
Even more important hint: You can loop through all of the keys in a dictionary d with
Here is an example to test your normalizeDictionary method:
smallestValue(self, nd1, nd2)
This method should accept any two model dictionaries nd1 and nd2 and should return the smallest positive (that is, non-zero) value across them both.
When used, the two input dictionaries are likely to be normalized, so we’ve named them nd1 and nd2, but this method should work regardless of whether they’re normalized or not. Remember to return, not print!
Here is an example to test your smallestValue method:
and here are the output results we obtained:
compareDictionaries(self, d, nd1, nd2)
This method is the fundamental workhorse of the project. The compareDictionaries method should compute:
- The log-probability that the dictionary
darose from the distribution of data in the normalized dictionarynd1and - The log-probability that dictionary
darose from the distribution of data in normalized dictionarynd2
In addition, it should return both of these log-probability values:
- Use
math.log, which is the natural logarithm (log-base-e). - Both of those log-probabilities should be returned together in a list, e.g.,
[lp1, lp2](which stands for log-prob1 and log-prob2, where the first is the probabilities with respect tond1and the second is with respect tond2). - You should be sure that both
nd1andnd2are normalized first, before computing the log-probabilities. - On the other hand,
ddoes not have to be normalized (in fact, it shouldn’t be). - For any keys in
dthat do not appear ind1ord2, you should use a value of half the smallest value across both normalized model dictionaries,d1andd2. UsesmallestValueto help you here!
For a detailed example, if you were comparing the unnormalized dictionary d = {'a':2, 'b':1, 'c':1, 'd':1, 'e':1} against the normalized dictionary nd1 = {'a':0.625, 'b':0.125, 'c':0.25} and the normalized dictionary nd2 = {'a':0.9375, 'd':0.0625}, you would compute, first for the comparison with nd1,
- Start with a
total_log_prob = 0.0 - Perhaps define a small value, epsilon, to be half the smallest value in both
nd1andnd2(use the function above…), then use a loop so that:- (For the two
'a's), you would addtotal += 2*math.log(0.625) - (For the one
'b'), you would addtotal += 1*math.log(0.125) - (For the one
'c'), you would addtotal += 1*math.log(0.25) - (For the one
'd'), you would addtotal += 1*math.log(epsilon), using that small value (half of it) - (For the one
'e'), you would addtotal += 1*math.log(epsilon), again using that small value (half of it) - Which should yield
-11.337… (see below)
- (For the two
Notes!
- When a key
kis ind, but is is not innd1, be sure to add the probability that ismath.log(epsilon) - Behave similarly when
kis not innd2… - The same procedure will be used for comparing
dwithnd2: you should obtain-13.29…
Here is an example to test your compareDictionaries method:
and here are the output results we obtained:
(These have been checked by hand, too, so I’m doubly hopeful they’re right!)
A note on interpreting those negative values
The log-probability is perhaps best thought of as a “shorthand” for very small probabilities. The two values above, about -11.3 and -13.2, mean that:
- The probability of
darising from the modelnd1ise^-11.3, and - The probability of
darising from the modelnd2ise^-13.2.
These numbers are both very small, but the former is more than seven times as likely as the latter.
Put another way, even though the values are negative the larger probabilities still “win”—in the sense that they are (often much) more likely.
createAllDictionaries(self)
This is a small, useful function that simply creates all of the dictionaries with the (raw) input string self.text. We provide a version of it for you to use here.
You may need to change it a bit, especially if your fifth feature isn’t punctuation!
Here are three examples to test this—run these now, since you’ll use these three models in the next, final example too!
To test this, create three files:
- Create a file named
train1.txtcontaining this little bit of text:
- Create a file named
train2.txtcontaining this little bit of text:
- Create a file named unknown.txt containing this little bit of text:
Then, run or paste into the bottom of your textmodel.py file (or whatever you’ve named it!) these example TextModel-building commands:
and check if your results are similar to these. They don’t have to be exact—small differences in implementing stems or sentence-counting or string-cleaning will change the values somewhat:
Next, you’ll compare the unknown_tm TextModel with both the trained_tm1 TextModel and the trained_tm2 TextModel.
compareTextWithTwoModels(self, model1, model2)
This method should run the compareDictionaries method, described above, for each of the feature dictionaries in newmodel against the corresponding (normalized!) dictionaries in model1 and model2. Note that all of those three objects (self, model1, model2) will be TextModels.
This method compareTextWithTwoModels should:
- Print out the comparative results of the log-probabilities for all five dictionaries—the precise details of how to format and arrange these data are up to you (see below for one example).
- Implement some system for determining which of the two models
selfis closest to, e.g., majority vote from the five dictionaries or weighted sum of the log-probabilities… Again, this part of the algorithm is up to you (below a majority-vote approach is used.). - Declare a “winner”—namely, it should indicate the model with which
selfis more similar.
Here, finally, you’re welcome to print the results, since there are no other functions that will use them downstream! (Though you’re welcome to also return things and create additional analysis, if you’d like!)
Here is an example—it uses the three models defined in the previous example:
TM1(to be used asmodel1)TM2(to be used asmodel2)TM_Unk(this will be self because it will callcompareTextWithTwoModels)
The example compares all five of the dictionaries of TM_Unk against the five (normalized!) dictionaries of TM1 and TM2. Admittedly, our examples have far too little text for a real run—but they have the right amount for debugging and making sure things are computing correctly!
Here is the example to try:
and here are our results—your formatting, overall model-choosing, and some other details may certainly be different:
Your own texts and analysis
Finally, this project asks you to use your text-analysis engine! Be sure to:
- Create at least two contrasting text models. These are the “trained” models. You will need to find one or more texts to train each model. If you have multiple texts from the same author, you can simply place them all into a single file for processing. For instance, Shakespeare vs. J. K. Rowling was our slides’ running example; lots of others are possible! Your two (or more) trained models should use a moderate-or-greater amount of text. (Models based on too small a source text tend to be “brittle”—they depend too much on the idiosyncrasies of that source.)
- Choose at least two other “test texts” that will be compared to your two trained models. Here, you might choose your own writing, or other writing from one or both of your trained models’ sources, or something else entirely. The “test” texts or “unknown” texts can be shorter than the trained models and still work well.
- Run the comparison of your two “test texts” against the two trained models (for a total of at least four comparisons).
If you try to open a file and get an error similar to this: codec can't encode character ... in ... position,
try using different encodings in your readTextFromFile method, e.g.,
open(filename, encoding = 'latin1')oropen(filename, encoding = 'utf-8')
Extras are encouraged:
- Using more trained models or more test texts.
- Using more features or features carefully tuned to the texts you’re investigating.
- Other creative extras that are specific to your particular analysis…