Home

NLTK in Action: Authorship Attribution

This program demonstrates the use of statistical natural language processing to determine the identity of the author of an unknown text. There are many methods of achieving this, one of which is the use of the Bayesian method.

Two things are required for this program
•The test corpus i.e. the text, whose author is to be determined
•Sample works of two or more authors (at least four per author). One of them should be the correct author.

This program uses the unigram language model for each author. The method of solving the problem is as follows:

Let A1, A2, A3 ......., An be n authors, one of which is the true author.

We need to calculate P(T | Ai), where i ϵ [1, n]. (Probability of the test corpus T given author Ai.) by the following formula.

Let w1, w2, w3.... wm be words which occur only once in the test corpus and let P(w1), P(w2), P(w3)... P(wm) be the probability of these words occurring in the sample works of author Ai. We have:

In plain language the procedure is as follows.

•Taking the language model from the corpus of Ai
•Multiplying the probabilities of all the unigrams which occur only once in the test corpus.
•Taking the geometric mean of these probabilities.

The same process is used on all authors. The author with the highest P(T | Ai ) will be the most likely author for the test corpus.

This is the first version of our Authorship Attribution program. It does not include statistical enhancement techniques such as smoothing, discounting etc. Thus it may not be highly accurate. Future versions will include these features and as a result, achieve greater accuracy and efficiency.

Code:

Importing Required Modules from NLTK and the standard Python Library

  1. from nltk.corpus import PlaintextCorpusReader
  2. from nltk import FreqDist
  3. from operator import itemgetter
  4. from math import log10
  5. import os

Here we read textual data of all corpora

  1. def readCorpus(filePathList):
  2.     corpusList = []
  3.     for filePath in filePathList:
  4.         corpusList.append(PlaintextCorpusReader(filePath, '.*'))
  5.     return corpusList

This method counts the number of words in each corpus and returns a list of values.

  1. def corporaWordCount(corpusList):
  2.     corporaWordCountList=[]
  3.     for corpus in corpusList:
  4.         corporaWordCountList.append(len(corpus.words()))
  5.     return corporaWordCountList

This method uses the FreqDist class of NLTK to calculate the frequency distribution of words in each corpus. It returns a list of FreqDist objects.

  1. def calculateFreqDist(corpusList):
  2.     corporaFreqDistList=[]
  3.     for corpus in corpusList:
  4.         corporaFreqDistList.append(FreqDist(w.lower() for w in corpus.words()))
  5.     return corporaFreqDistList

This method returns a list of words which occur once in the test corpus. Note that corpusList[-1] refers to the unknown corpus which is the last element in the corpus list.

  1. def getOnceOccuringWords(corpusList,corporaFreqDistList):
  2.     OnceOccuringWords = []
  3.     for word in corpusList[-1].words():
  4.         if corporaFreqDistList[-1][word] == 1:
  5.             NoOccurenceInCorpusFlag=0
  6.             for corpusFreqDist in corporaFreqDistList[:-1]:
  7.                 if corpusFreqDist[word] ==0:
  8.                     NoOccurenceInCorpusFlag=1
  9.             if not NoOccurenceInCorpusFlag:
  10.                  OnceOccuringWords.append(word)
  11.     return OnceOccuringWords

This method calculates the probability of occurrence of all the words which occur once in the unknown corpus, in each author's corpus.

  1. def getProbabilityList(OnceOccuringWords,corporaFreqDistList,corporaWordCountList):
  2.     probabilityList = []
  3.     for corpusWordCount in corporaWordCountList[:-1]:
  4.         probabilityList.append(1)
  5.     for word in OnceOccuringWords:
  6.         for index in range(len(corporaWordCountList)-1):
  7.             probabilityList[index] = probabilityList[index] * log10(-1*log10((float(corporaFreqDistList[index][word])/corporaWordCountList[index])))
  8.     return probabilityList

This method calculates the geometric means of probabilities of once occurring words for each author.

  1. def getGeometricMeanList(probabilityList,CountOfOnceOccuringWords):
  2.     return [probability ** (1.0/CountOfOnceOccuringWords) for probability in probabilityList]

This method determines the most likely author by finding the author with highest geometric mean. It presents results in the appropriate manner.

  1. def getMostLikelyAuthor(authorList,GeometricMeanList):
  2.     maxMean = max(GeometricMeanList)
  3.     authorIndex=GeometricMeanList.index(maxMean)
  4.     maxProbableAuthor = authorList[authorIndex]
  5.     return maxProbableAuthor

The main function. Execution begins here.

  1. if __name__ == "__main__":
  2.     authorList = ['Austen','Twain','Doyle','Unknown']
  3.     presentWorkingDirectory = os.getcwd()
  4.     fileList = [presentWorkingDirectory + os.sep + author for author in authorList]
  5.     print "Reading Corpora..."    
  6.     corpusList = readCorpus(fileList)
  7.     print "Counting Words in Corpora..."    
  8.     corporaWordCountList = corporaWordCount(corpusList)
  9.     print "Calculating Frequency Distributions..."
  10.     corporaFreqDistList = calculateFreqDist(corpusList)
  11.     print "Finding Words Which Occur Once..."
  12.     OnceOccuringWords = getOnceOccuringWords(corpusList,corporaFreqDistList)
  13.     CountOfOnceOccuringWords =  len(OnceOccuringWords)
  14.     print "Calculating Probabilities of Words which Occur Once..."
  15.     probabilityList = getProbabilityList(OnceOccuringWords,corporaFreqDistList,corporaWordCountList)
  16.     print "Finalizing Results..."
  17.     GeometricMeanList = getGeometricMeanList(probabilityList,CountOfOnceOccuringWords)
  18.     maxProbableAuthor = getMostLikelyAuthor(authorList,GeometricMeanList)
  19.     print "Max probable Author is :" + maxProbableAuthor

AttachmentSize
AuthAttr.zip1.8 MB