Python

Using scikit-Learn on your own text data

Scikit-learn’s Working with Text Data provides a superb starting point for learning how to harness the power and ease of the sklearn framework for the construction of really powerful and accurate predictive models over text data. The only problem is that scikit-learn’s extensive documentation (and, be in no doubt, the documentation is phenomenal) doesn’t help much if you want to apply a cross-validated model on your own text data

At some point, you’re going to want to move away from experimenting with one of the built-in datasets (e.g. twentynewsgroups) and start doing data science on textual material you understand and care about. 

The purpose of this tutorial is to demonstrate the basic scaffold you need to build to apply the power of scikit-learn to your own text data. I’d recommend methodically working your way through the Working with Text Data tutorial before diving in here, but if you really want to get cracking, read on.

If you can't be bothered reading on and just want to see the code, it's in a repo on GitHub, here.

Objectives

Before we start, let’s be clear about what we’re trying to do. We have a great big collection of text documents (ideally as plain text from the offing). Our documents are, to use the twentynewsgroups example, all news articles. The news articles have been grouped together, in directories, by their subject matter. We might have one subdirectory consisting of technology articles, called Technology. We might have another subdirectory consisting of articles about tennis, called Tennis

Our project directory might look like this (assume each subdirectory has 100 text documents inside):

news_articles \
    art
    business
    culture
    design
    food
    technology
    tennis
    war

The aim of the game is to use this data to train a classifier that is capable analysing a new, unlabelled article and determining which bucket to put it in (this is an article about food, this is an article about business, etc). 

What our code is going to do

We’re going to write some code, using scikit-learn, that does the following:

  • Loads our dataset of news articles and categorises those articles according to the name of the folder they live in (e.g. art, food, tennis)
  • Splits the dataset into two chunks: a chunk we’re going to use to train our classifier and another chunk that we’re going to use to test how good the classifier is
  • Converts the training data into a form the classifier can work with
  • Converts the test data into a form the classifier can work with
  • Builds a classifier 
  • Applies that classifier to our training data
  • Fires the test data into our trained classifier
  • Tells us how well the classifier did at predicting the right label (art, food, tennis etc) of the each document in the test dataset

1. Get the environment ready

The first job is to bring in everything we need from scikit-learn:

import sklearn
import numpy as np
from glob import glob
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import SGDClassifier
from sklearn import metrics
from sklearn.pipeline import Pipeline 

That’s the stage set in terms of bringing in our dependencies.

2. Set our categories

The next job is to state the names of the categories (our folders of grouped news articles) in a list. These need to exactly match the names of the subdirectories acting as the categorical buckets in your project directory.

categories = [‘art’, ‘business’, ‘culture’, ‘design’, ‘food’, ‘technology’, ‘tennis’, ‘war’]

This approach of manually setting the folder names works well if you only have a few categories or you’re just using a small sample of a larger set of categories. However, if you’ve got lots of category folders, manually entering them as list items is going to be a bore and will make your code very, very ugly (I'll write a separate blog post on a better way of dealing with this, or look at the repo on GitHub, which incorporates the solution to this problem).

3. Load the data

We’re now ready to load our data in:

docs_to_train = sklearn.datasets.load_files(“/path/to/the/project/folder/“, 
    description=None, categories=categories, 
    load_content=True, encoding='utf-8', shuffle=True, random_state=42)

All we’re doing here is saying that our dataset, docs_to_train, consist of the files contained within all of the subdirectories to the path specified inside the .load_files function and that the categories are the categories set out in our categories list (see above). Forget about the other stuff in there for now.

4. Split the dataset we’ve just loaded into a training set and a test set

This is where the real work begins. We’re going to use the entire dataset, docs_to_train, to both train and test our classifier. For this reason, we’ve got to split the dataset into two chunks: one chunk for training and another chunk (that the classifier won’t get to look at in training) for testing. We’re going to “hold out” 40% of the the dataset for testing:

X_train, X_test, y_train, y_test = train_test_split(docs_to_train.data,
    docs_to_train.target, test_size=0.4)

It’s really important to understand what this line of code is doing. 

First, we’re creating four new objects, X_train, X_test, y_train and y_test. The X objects are going to hold our data, the content of the text files. We’ve got one X object, X_train, and that will hold the text file data we’ll use to train the classifier. We have another X object, X_test, and that will hold the text file data we’ll use to test the classifier. The Xs are the data.

The we have the Ys. The Y objects hold the category names (art, culture, war etc). y_train will hold the category names that correspond to the text data in X_train. y_test will hold the category names category names that correspond to the text data in X_test. The y value are the targets. 

Finally, we’re using test_size=0.4 to say that out of all the data in docs_to_train we want 40% to be held out for the test data in X_test and y_test.

5. Transform the training data into a form the classifier can work with

Our classifier uses mathematics to determine whether Document X belongs in bucket A, B, or C. The classifier therefore expects numeric data rather than text data. This means we’ve got to take our text training data, stored in X_train, and transform it into a form our classifier can work with. 

count_vect = CountVectorizer(stop_words='english')

X_train_counts = count_vect.fit_transform(raw_documents=X_train)

These two lines are doing are a lot of heavy lifting and I would strongly urge you to go back to the Working with Text Data tutorial to fully understand what’s going on here.

The first thing we’re doing is setting up a vectoriser, CountVectorizer(). This is a function that will count the number of times each word in the dataset occurs and project that count into a vector. 

Then, we take that vector and apply it to the training data stored in X_train. We store those occurrence vectors in X_train_counts.

Once that’s done we move on to the clever transformation bit. We’re going to take the occurrence counts, stored in X_train_counts, and transform them into a term frequency inverse document frequency value. 

tfidf_transformer = TfidfTransformer(use_idf=True)

X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

Why are we doing this? Well, if you think about it, the documents in your training set will naturally vary in word length; some are going to be long, others are going to be short. Longer documents have more words in them and that’s going to result in a higher word count for each word. That’s going to skew your results. What we really want to do is get get a sense of the count of each word proportionate to the number of words in the document. Tf-idf (term frequency inverse document frequency achieves this). 

6. Transform the test data into a form the classifier can work with

Since we’ve gone to the trouble of splitting the dataset into a training set and a test set, we also need to transform our test data in exactly the same way as we just did with the training set. All we’re doing here is mirroring the transformation process we just applied to X_train onto X_test. 

count_vect = CountVectorizer(stop_words='english')
X_test_counts = count_vect.fit_transform(raw_documents=X_test)

tfidf_transformer = TfidfTransformer(use_idf=True)
X_test_tfidf = tfidf_transformer.fit_transform(X_test_counts)

7. Scikit-learn gives us a far better way to deal with these transformations: pipelines!

It was worth reading about the transformation process, because if you’re working with text data and trying to do science with it you really do need to at least see why and how that text is transformed into a numerical form a predictive classifier can deal with. 

However, scikit-learn actually gives us a far more efficient way (in terms of lines of code) to deal with the transformations — it’s called a pipeline. The pipeline is this example has three phases. The first creates the vectoriser — the machine used to turn our text into numbers — a count of occurrences. The second phase deals with transforming the crude vectorisation handled in the first into a frequency-based representation of the data — the term frequency inverse document frequency. Finally, and most excitingly, the third phase of the pipeline sets up the classifier — the machine that’s going to train the model. 

Here’s the pipeline code:

text_clf = Pipeline([('vect', CountVectorizer(stop_words='english')),
    ('tfidf', TfidfTransformer(use_idf=True)),
    ('clf', SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, random_state=42, 
    verbose=1)),])

For now, don’t worry about the parameters set out in the classifier, just appreciate the structure and content of the pipeline. 

8. Deploy the pipeline and train the model

Now it’s time to train our model by applying the pipeline we’ve just built to our training data. All we’re doing here is taking our training data (X_train) and the corresponding training labels (y_train) and passing them into the fit function that comes built into the pipeline. 

text_clf.fit(X_train, y_train)

Depending on how big your dataset is, this could take a few minutes or a bit longer. 

As a sidenote, on this point, you might have noticed that I set the verbose parameter in the classifier as 1. This is purely so I can see that the classifier is running and that the script isn't hanging because I’m chewing through memory. 

9. Test the model we’ve just trained using the test data

We’ve now trained our model on the training data. It’s time to see how well trained the model really is by letting it loose on our test data. What we’re going to do is take the test data, X_test, let the model evaluate it based on what it learned from being fed the training data (X_train) and the training labels (y_train) and see what categories the model predicts the test data belongs to. 

predicted = text_clf.predict(X_test)

We can measure the model’s accuracy by taking the mean of the classifier’s predictive accuracy like so:

print (np.mean(predicted == y_test))

Better yet, we can use scikit-learn’s built in metrics library to give us some detailed performance statistics by category (i.e. how well did the classifier at predicting that an article about “design” is an article about “design”.

print(metrics.classification_report(y_test, predicted, 
    target_names=docs_to_train.target_names))

The metrics will provide you with a precision score for each category and an overall average of the performance of the model between 0 and 1. 

The closer that average score gets to 1, the better the model will perform. Mind you, if your model is averaging a score of 1 on the nose, something has gone wrong!   

Learning Python: a handful of tips

I've been learning Python for the past year or so and have just reached the stage where I've moved from being a beginner to an intermediate level user of the language. Out of all of the programming languages I've dabbled in (C++, Delphi, Java, R and Javascript), Python is my firm favourite and the only language I've settled down with. 

Python, in my view at least, is an extraordinarily approachable language because:

  1. The syntax is extremely readable, descriptive and easy to follow
  2. It's interpreted, meaning you don't have to go through the bother of compiling your code every time you want to run it. 
  3. The community and documentation is phenomenal.

It didn't take too long for me to start writing code that just about did what I wanted it to do. In the main, I'd have an idea (say, to write some code to extract information from a group of text documents), sketch out roughly how I thought the code should be structured (e.g. do this, then do that, then do the other), and hunt through StackOverflow in search of examples of how to accomplish each step. 

I'm now at the point where I'm able to do 80% of the code for any particular project without having to resort to StackOverflow and I'm more confident in starting to write my code unassisted.  

Looking back, several habits I've started to practise have helped me move my knowledge of Python forward at a faster rate and I think that had I have started to practise these habits a bit sooner, I'd have reached the level I'm at a little quicker.

So, what follows are some general tips that may help you if you're a beginner who wants to make progress with Python a bit faster. 

TIP #1: USE PYCHARM CE

For the first six months, I did the vast bulk of my coding in the interpreter at the command line, a la:

$ python

> print "hello, world"
hello, world

This worked really well early on when I was trying trivial examples, because I was immediately seeing the result of the code I was writing. However, as soon as I wanted to produce something with more than 20 lines of code, I got into a horrible mess. 

I soon realised that I needed to start using a proper Python IDE and after a bit of digging around, I settled on PyCharm CE. It's free and it's excellent, because:

  • You can run your code in the IDE whenever you like, so the instant feedback of the command line interpreter is available at the click of the button, but your code is still sitting there in the editor helping you keep track on what's going on where.
  • PyCharm automatically deals with whitespace indents, so you won't run into errors because you've forgotten to indent the first statement in a for loop.
  • You can easily flip between the 2.7 and 3.6 interpreter
  • The editor provides feedback line by line on errors and style
  • The editor provides inline information that helps you see, for example, what arguments a particular function takes
  • The editor auto-completes your code

TIP #2: MAKE LIBERAL USE OF THE PRINT FUNCTION

Whether I'm writing my own program from scratch or playing with a complex example, I initially pepper my code with print. It really helps to be able to see what's going on in the code at various stages. So, my advice is as follows: if it moves, print it to the console and take a look. 

TIP #3: LOOK AFTER YOUR CODE

You're going to find that as you begin to grow more competent you'll start referring back to your own code to remind yourself how to perform certain types of operation. Your own code, in a sense, will become a valuable source of reference.

Keep your code organised in a way that makes sense to you. I recommend these simple rules to help you keep your code accessible:

  • Store each distinct project in its own folder and give the folder a sensible name
  • Give your individual Python files sensible names, too
  • At the top of your scripts, add a short comment block explaining what the code sets out to achieve (this comes in handy when you're looking back over your older code)

TIP #4: FOCUS ON GETTING THE CODE TO WORK

Personally, I'm of the view that as beginners our first focus should be getting the code to work rather than ensuring it looks beautiful or runs blisteringly fast. Writing optimised, beautiful code comes later - the priority is first to get your code to do what you want it to do.

TIP #5: keep notes

Note-taking forms an important part of our general process for learning anything (whether it's courses on a law degree, learning quadratics in secondary school or learning Python), so take notes as you progress through the fundamentals in Python.

Notes serve at least two really useful functions:

  • They reinforce your learning through the process of writing the notes themselves
  • They act as an invaluable reference source as you start to accumulate a larger base of knowledge. 

Your notes can be as basic or elaborate as you like, but it's a good idea to get a note of the fundamentals, such as the structure of different loops, iterating over files, reading and writing files etc.

TIP #6: DON'T BE AFRAID TO BREAK YOUR CODE

As a beginner, there's nothing like slogging over a coding project and finally seeing it run and work as expected. Once this point has been reached, there may be a strong reluctance to touch the code again for fear of breaking it. This is understandable, but it will prevent you from  reviewing your code and making improvements to it.

If you're really worried that you're going to break the code, copy and paste it into a new file and work on it away from the original. Asking how you can make the intention of your code more obvious and finding ways to reflect that in your code is a really important part of the Python learning curve.

TIP #7: IMMERSE YOURSELF IN ERRORS

Errors that cause your code to halt in its tracks are inevitable. I run into at least half a dozen whenever I'm attempting something new or complex. As beginners, error messages can be a bit bamboozling, but it's important that you consciously engage with them and pursue a way to fix them rather than being tempted to abandon the project.

To begin with, errors will look pretty alien. However, one of the things I've come to love about Python is how descriptive its error handling is once you've got used to it. My advice is to make an effort to read the entire error trace, which will tell you exactly where in your code the interpreter is running into problems. If the error doesn't speak for itself, copy and paste it into Google: chances are there will be a StackOverflow question and answer about the error you're hitting.

Persevere with errors and work hard to fix them.

TIP #8: READ DOCUMENTATION

As with any new technical subject, the surrounding literature will initially be difficult to penetrate. However, the more you make a habit of reading documentation, the more sense it will start to make over time. 

Python benefits from an excellent community of developers who put an incredible amount of thought into the documentation they produce for their code, take advantage of it.

TIP #9: THINK ABOUT CODING WHEN YOU'RE NOT CODING

A significant amount of the cerebral heavy lifting for a coding project occurs when you're nowhere near your machine. More often that not, an idea forms in my mind when I'm walking over Waterloo Bridge or sitting on a train. When those moments arrive, I start planning the structure of the code in my mind and identifying obstacles and dependencies (am I going to need to restructure or wrangle of a dataset? what packages will I need to do that? what processing do I need to do before I get to the heart of the code? what do I think I'm going to need to learn before I can do what I want to do?). 

Allowing ideas to have to chance to marinate before you even start writing the code will get you off to a far better start than blindly jumping straight in.

TIP #10: LISTEN TO TALK PYTHON TO ME

The TalkPythonToMe podcast is fabulous. The podcast's host, Michael Kennedy, has curated what is in all probability the definitive Python podcast and he knows his onions. 

There's definitely something about listening to experts talk about a subject you're trying to get a grip on, even if for some or most of the time you have no idea what they're going on about. 

There's your starter for ten. Good luck and have fun!

 

A first run with Hierarchical Dirichlet Process

I've been experimenting with Latent Dirichlet Allocation for a while in R, but was looking for a topic model algorithm that did not require the number of topics (k) to be defined apriori the application of the algorithm to the text data I wanted to work with. 

As a relative newcomer to topic modelling, I hadn't even heard of David Blei or Chong Wang, both of whom I now know to be pioneers in modern topic modelling. Quite by chance, I stumbled on Wang and Blei's implementation of Hierarchical Direchlet Processing in C++, a topic model where the data determine the number of topics. 

Getting HDP to work required quite a bit of wrangling and I couldn't find any walkthroughs suitable for novices like me, so I thought it would be worth noting up how I managed to get it to work with a small sample set of text data. 

What follows is far from a perfect (or even good) representation of how to apply HDP to text data, but the steps that follow did work for me. Here goes...


Sample Data

It goes without saying that the very first step is to assemble a corpus of text data against which we'll apply the HDP algorithm. In my case, as is usual, I used ten English judgments (all of which are recent decisions from the Criminal Division of the Court of Appeal) in .txt format. Save these into a folder.


Getting the Sample Data ready for HDP

Before we even go near Wang & Blei's algorithm, we need to prepare the sample data in a particular way. 

The algorithm requires the data to be in LDA-C format, which looks like this:

 [M] [term_1]:[count] [term_2]:[count] ...[term_N]:[count]
where [M] is the number of unique terms in the document, and the [count] associated with each term is how many times that term appeared in the document.

This presents the first problem, because our data appear as words in .txt format files. Fortunately, there's an excellent Python program called text2ldac that comes to our rescue. Text2ldac takes the data in .txt format and outputs the files we need, in the form in which we need them.

Clone text2ldac from the git repo here

Once you've pulled down text2ldac, you're ready to take your text files and process them. To do this, go to the command line and run the following command (make adjustments to the example that follows to suit your own directories and filenames:

$ text2ldac danielhoadley$ python text2ldac.py --stopwords stopwords.txt /Users/danielhoadley/Documents/Topic_Model/text2ldac/input

All that's happening here is we're running text2ldac.py, using the --stopwords flag to pass in our stopwords (which, in my case, are in a file named stopword.txt) and then passing in the directory that contains our .txt files.

This will output three files: a .dat file (e.g. input.dat) which is the all import LDA-C formatted input for the HDP algorithm, a .vocab file, which contains all of the words in the corpus (one word per line) and a .dmap file, which lists the input .txt documents. 


Time to run HDP

Now that we have our data in the format required by the HDP algorithm, we're ready to apply the algorithm. 

For convenience, I recommend copying the three files generated by text2ldac into the folder you're going to run HDP from, but you can leave them wherever you like. 

Go to the folder containing the HDP program files and run the following command (again, adjust to your own folder and filenames:

$ ./hdp --algorithm train --data /Users/danielhoadley/Documents/Topic_Model/hdp/hdp/Second_run/input.dat --directory train_dir

Let's unpack this a bit: 

1. ./hdp invokes the HDP program

2. The --algorithm flag sets the algorithm to be applied, namely train

3. The path that follows the algorithm flag points to the .dat file produced by text2ldac

4. --directory train_dir is telling HDP to place the output files in a directory called train_dir

You'll know you've successfully executed the program if the prompt begins printing something that looks like this:

Program starts with following parameters:
algorithm:= train
data_path:= /Users/danielhoadley/Documents/Topic_Model/hdp/hdp/Second_run/input.dat
directory:= trainer
max_iter= 1000
save_lag= 100
init_topics = 0
random_seed = 1488746763
gamma_a = 1.00
gamma_b = 1.00
alpha_a = 1.00
alpha_b = 1.00
eta = 0.50
#restricted_scans = 5
split-merge = no
sampling hyperparam = no

reading data from /Users/danielhoadley/Documents/Topic_Model/hdp/hdp/Second_run/input.dat

number of docs: 9
number of terms : 5795
number of total words : 35865

starting with 7 topics 

iter = 00000, #topics = 0008, #tables = 0076, gamma = 1.00000, alpha = 1.00000, likelihood = -305223.54210
iter = 00001, #topics = 0008, #tables = 0079, gamma = 1.00000, alpha = 1.00000, likelihood = -301582.68017
iter = 00002, #topics = 0008, #tables = 0079, gamma = 1.00000, alpha = 1.00000, likelihood = -300273.98808

I'm not going to go into all of these parameters here, but the main one to note is the max_iter, which sets the number of time the algorithm walks over the test data. 

Note also that the algorithm has decided by itself how many topics it's going to work with (in the above example, 7)

The algorithm will produce a bunch of .dat files. The one we're really interested in should have a name like mode-topics.dat and not like this 00300-topics.dat (which was produced on the 300th iteration of the algorithm's walk). 


Printing the topics determined by HDP

Wang & Blei very helpfully provided a R script, print.topics.r, which you can use to turn the results of the algorithm into a human-readable form. This is helpful because the output generated by the algorithm will look like this:

00001 00001 00007 00007 00007
00011 00000 00000 00000 00000

The key thing at this stage is to remember two key files you'll need as input for the R script: the mode-topics.dat file (or similar name) generated by HDP and the .vocab file generated by text2ldac. 

Go back to the command line and navigate to the folder that contains print.topics.r. First, you'll need to make the R script executable, so run:

$ sudo chmod +x print.topics.R

Then run

$ ./print.topics.r /Users/danielhoadley/Documents/Topic_Model/hdp/hdp/trainer/mode-topics.dat /Users/danielhoadley/Documents/Topic_Model/hdp/hdp/Second_run/input.vocab topics.dat 4

1. ./print.topics.r runs the R script

2. The first argument is the path to the mode-topics.dat file produced by the HDP algorithm

3. The second argument is the path to the .vocab file produced by text2ldac

4. The third argument is the name of the file you want to output the human-readable result to, e.g. topics.dat

5. Finally, the fourth argument, which isn't mandatory, is the number of terms per topic you wish to output - the default is 5.


The output

If everything has worked as it ought to have done, you'll get an output like this if you look at the topic.dat file in RStudio:

The output is by no means perfect the first time around. Better results will probably depend on hitting the source data with a raft of stop words and tweaking HDP's many parameters, but it's a good start.

 

 

 

 

Legislation.gov.uk Statute Scraper

The following Python script can be used to scrape the full text of Public General Acts from legislation.gov.uk.

Prerequisites

The script takes a list of URLs to individual pieces of legislation from a text file. The script processes each URL one by one. The text file needs to look something like this, with each target URL on a new line:

url.txt

http://www.legislation.gov.uk/ukpga/2016/4 
http://www.legislation.gov.uk/ukpga/2016/3 
http://www.legislation.gov.uk/ukpga/2016/2 
http://www.legislation.gov.uk/ukpga/2016/1 
http://www.legislation.gov.uk/ukpga/2017/1 
http://www.legislation.gov.uk/ukpga/2017/2

The Python script is simple enough. 

  • First, it opens url.txt and reads the target URLs line by line
  • For each target URL, the title of the legislation is captured (this is used to name the output files)
  • Each url is cycled through sequentially and the contents of the relevant part of the HTML markup is scraped
  • The scraped material is written to a text file and an prettified HTML file.

Scraper.py

# Environment

import requests
import time
import io
from bs4 import BeautifulSoup
from urllib import urlopen

# Get the text file with the URLs to be scraped and scrape the target section in each page

print "\n\nScraping URLs in urls.txt...\n\n"

with open('urls.txt') as inf:
    
    # Get each url on each line in urls.txt
    urls = (line.strip() for line in inf)
    for url in urls:
        site = urlopen(url)
        soup = BeautifulSoup(site, "lxml")
        
        # Scrape the name of the legislation in each target url for use when saving the output to a file
        
        for legName in soup.find_all("h1", {"class": "pageTitle"}):
            
            actTitle = legName.text
                
                print 'Scraping ' + actTitle + ' ...\n'
    
        # Scrape stuff in <div id="viewLegContents"></div>
        
        for item in soup.find_all("div", {"id": "viewLegContents"}):
            
                # Write what we've scraped, with UTF-8 encoding, as text to a new text file - one file per url
                
                with io.open (actTitle + '.txt', 'w', encoding='utf-8') as g:
                    g.write(item.text)
                    
                    # Write what we've scraped to an html file - one file per url
                    
                    with open (actTitle + '.html', 'w') as g:
                        g.write(item.prettify('utf-8'))

print "\n\nDone! Files created.\n"

Respect the source of the data you're scraping

The people behind legislation.gov.uk have done everyone a big favour in making their information so accessible. The least we can do is to be respectful of their servers when performing scraping tasks like this. If you plan on running this, I'd strongly urge you to break your url.txt input into small chunks and, if you go on to reuse the data, remember to acknowledge the source of that data.

Rapid Keyword Extraction of Donoghue v Stevenson

Sometimes it would be really handy to be able to quickly and accurately extract keywords from a large corpus of documents. It is quite easy to foresee such a use-case arising in legal publishing, for example. 

RAKE (Rapid Keyword Extraction), is a Python natural language processing module that goes a long way in dealing with this use-case. 

I was interested in putting RAKE to the test and thought I'd pit the algorithm against what is perhaps to most well known piece of case law in the common law world: Donoghue v Stevenson (of snail and ginger beer fame). 

What follows is the basic "working out" of the code and the results of the first pass. For anyone interested in replicating this experiment or doing some keyword extraction of their own, see this excellent tutorial - you'll see that my own code follows it closely.

IMPORT THE RELEVANT LIBRARIES

import rake 
import operator

INITIALISE RAKE

rake_object = rake.Rake("smartstoplist.txt", 5, 5, 7)

This line of code does the following:

  • Creates a RAKE object that extracts keywords where (i) each word has at least 5 characters; (ii) each phrase must have at least 5 words; and (iii) each keyword must appear in the text at least 7 times
  • Hits the text file with a list of stop words to remove textual noise

GET THE TEXT

Now we open the text file (in this test, I've saved the judgment in Donoghue as a text file) and save it in a variable:

judgment = open("dono.txt","r") 
text = judgment.read()

RUN RAKE AND PRINT THE KEYWORDS

Now we're ready to run RAKE over the text to get the keywords:

keywords = rake_object.run(text) 
print (keywords)

THE OUTPUT

The following keywords (along with their scores) were returned:

[('give rise', 4.300000000000001), ('common law', 4.184313725490196), ('duty owed', 4.154061624649859), ('ordinary care', 4.115278543849972), ('reasonable care', 4.093482554312047), ('skivington lr 5', 4.050000000000001), ('lake & elliot', 4.0), ('pender 11 qb', 3.966666666666667), ('present case', 3.7993197278911564), ('defective', 1.7619047619047619), ('present', 1.7380952380952381), ('principles', 1.7333333333333334), ('dangerous', 1.6491228070175439), ('exercise', 1.588235294117647), ('cases', 1.5875), ('bottles', 1.5833333333333333), ('liability', 1.5789473684210527), ('relationship', 1.5555555555555556), ('court', 1.5365853658536586), ('supplying', 1.5), ('appears', 1.4761904761904763), ('principle', 1.4736842105263157), ('allowed', 1.4545454545454546), ('party', 1.4375), ('nature', 1.4210526315789473), ('warranty', 1.4166666666666667), ('goods', 1.4090909090909092), ('thing', 1.4090909090909092), ('articles', 1.4), ('condition', 1.4), ('appellant', 1.3953488372093024), ('injured', 1.3863636363636365), ('alleged', 1.375), ('bought', 1.3636363636363635), ('stated', 1.3636363636363635), ('examination', 1.3636363636363635), ('opportunity', 1.3636363636363635), ('appeal', 1.3333333333333333), ('support', 1.3333333333333333), ('defect', 1.3333333333333333), ('decided', 1.3333333333333333), ('relation', 1.3333333333333333), ('bottle', 1.3225806451612903), ('matter', 1.3125), ('authorities', 1.3125), ('injury', 1.3076923076923077), ('carelessness', 1.3076923076923077), ('judgment', 1.3055555555555556), ('proposition', 1.3043478260869565), ('recover', 1.3), ('referred', 1.3), ('circumstances', 1.2972972972972974), ('supplied', 1.2857142857142858), ('found', 1.2857142857142858), ('based', 1.2777777777777777), ('defendant', 1.2666666666666666), ('liable', 1.263157894736842), ('article', 1.26), ('manufactured', 1.25), ('lordships', 1.25), ('danger', 1.25), ('means', 1.25), ('poison', 1.25), ('inspection', 1.2307692307692308), ('purchaser', 1.2272727272727273), ('george', 1.2272727272727273), ('person', 1.2222222222222223), ('courts', 1.2222222222222223), ('house', 1.2105263157894737), ('plaintiff', 1.2096774193548387), ('chattel', 1.2), ('decision', 1.1935483870967742), ('entitled', 1.1818181818181819), ('authority', 1.1666666666666667), ('vendor', 1.1666666666666667), ('dicta', 1.1666666666666667), ('premises', 1.1538461538461537), ('repair', 1.1538461538461537), ('question', 1.1515151515151516), ('pursuer', 1.1428571428571428), ('manufacturer', 1.1384615384615384), ('facts', 1.1333333333333333), ('persons', 1.1333333333333333), ('subject', 1.125), ('class', 1.125), ('scotland', 1.125), ('evidence', 1.125), ('manufacturers', 1.125), ('defender', 1.125), ('contents', 1.1176470588235294), ('words', 1.1), ('longmeid', 1.1), ('holliday 6', 1.1), ('exist', 1.1), ('consequence', 1.1), ('negligence', 1.0985915492957747), ('contract', 1.0918367346938775), ('difficult', 1.0833333333333333), ('proved', 1.0833333333333333), ('respect', 1.0833333333333333), ('respondent', 1.08), ('consumer', 1.0789473684210527), ('proof', 1.0714285714285714), ('regard', 1.0714285714285714), ('manufacture', 1.0666666666666667), ('knowledge', 1.0666666666666667), ('england', 1.0588235294117647), ('langridge', 1.0555555555555556), ('action', 1.0476190476190477), ('opinion', 1.0357142857142858), ('lords', 1.0), ('ginger', 1.0), ('retailer', 1.0), ('result', 1.0), ('neglect', 1.0), ('division', 1.0), ('ground', 1.0), ('fraud', 1.0), ('judgments', 1.0), ('parke', 1.0), ('levy 2', 1.0), ('winterbottom', 1.0), ('wright 10', 1.0), ('stranger', 1.0), ('coach', 1.0), ('reason', 1.0), ('blacker', 1.0), ('breach', 1.0), ('skill', 1.0), ('parties', 1.0), ('brett', 1.0), ('heaven', 1.0), ('point', 1.0), ('treated', 1.0), ('property', 1.0), ('purpose', 1.0), ('thought', 1.0), ('existence', 1.0), ('pointed', 1.0), ('argument', 1.0), ('defendants', 1.0), ('hamilton', 1.0), ('contention', 1.0), ('mullen', 1.0), ('barr &', 1.0), ('defenders', 1.0), ('members', 1.0), ('remote', 1.0), ('bridge', 1.0)]

I was fairly chuffed with these results given it was the first attempt. The key seems to be getting the right balance of parameters when setting the object up. But, it's good to see terms like duty owed and reasonable care appearing at the top of the results. 

It definitely needs some fine tuning and probably an expansion of the stop list, but it's a good start.