In the previous NLTK tutorial, you learned what frequency distribution is. Now, you will learn how what a corpus is and how to use it with NLTK.
What is a Corpus?
Corpus is a collection of written texts and corpora is the plural of corpus. In NLTK, you have some corpora included like Gutenberg Corpus, Web and Chat Text and so on.
Using Corpora in NLTK
In this example, you are going to use Gutenberg Corpus. To import it, create a new file and type:
|
1 2 |
from nltk.corpus import gutenberg as gt |
So this corpus has different txt txt files which contain different texts. If you want to see all the texts that this corpus has, you can say
|
1 2 |
print(gt.fileids()) |
and it will return a list like in the image below.
So you can see that this corpus has texts like Hamlet, Macbeth and a novel of Milton.
Let’s say that you want to access the file shakespeare-macbeth.txt and see what words the text have. To do this, you can use the words method. So in your code type:
|
1 2 3 |
shakespeare_macbeth = gt.words("shakespeare-macbeth.txt") print(shakespeare_macbeth) |
As you can see, the words method receives the file id as its parameter. So if you want to access Milton’s novel, for example, you can type gt.words("milton-paradise.txt") .
If you run this code now, you will get a list of all the words of the text as your output like in the image below.
Another important function is the raw function. What it does is it returns the whole text without doing any linguistic processing. If you type
|
1 2 3 |
raw = gt.raw("shakespeare-macbeth.txt") print(raw) |
and execute your code, you can see that it returns you the raw text.
Let’s say that now you want to see the sentences your text has. You can use the sents function. So in your code type
|
1 2 3 |
sents = gt.sents("shakespeare-macbeth.txt") print(sents) |
and this will return you a list of all the sentences your text has.
You can use those functions to do more elaborate things. If you want for example see the number of words and sentences in all of the texts present in your corpus, you can say:
|
1 2 3 4 5 6 7 |
for fileid in gt.fileids(): num_words = len(gt.words(fileid)) num_sents = len(gt.sents(fileid)) print("Data for file:", fileid) print("Number of words:", num_words) print("Number of sentences:", num_sents, end="\n\n\n") |
and it will give you an output like in the image below.
Loading your own corpus
Now that you learned what is a corpus, you will learn how to load your own corpus.
To do this, you need a corpus reader so create a new file named loading-your-own-corpus.py with the following lines.
|
1 2 3 |
from nltk.corpus import PlaintextCorpusReader import os |
The first import statement is for the PlainTextCorpusReader class, that will be your corpus reader, and the second is for the os module. The os module will give the PlainTextCorpusReader the path of the files you want to load.
To continue, download the play Taming of the Shrew in this link and place it in the same directory of your Python file.
After you download the play, create an object of PlainTextCorpusReader with the following lines:
|
1 2 3 4 |
corpus_root = os.getcwd() + "/" file_ids = ".*.txt" corpus = PlaintextCorpusReader(corpus_root, file_ids) |
As you can see, PlainTextCorpusReader expects two inputs in its constructor. The first one is corpus_root and the second one is the file_ids . The corpus_root is the path of your files and the file_ids are the name of the files.
To get the path of your files, you can use the getcwd method of os module. Note that we add a / in the path. In the file_id , we use a RegEx expression to fetch all the files that you want. In our example, we want all files that have the .txt extension.
As this object returns you a corpus object, you can use the same functions you used in the previous section. So if you want to see the words in the text, for example, you can use:
|
1 2 |
print(corpus.words("shakespeare-taming-2.txt")) |
This is all for this tutorial. If you have any question, feel free to leave it in the comments below.
NLTK Course
Join our NLTK comprehensive course and learn how to create sophisticated applications using NLTK, including Gender Predictor, and Document Classifier, Spelling Checker, Plagiarism Detector, and Translation Memory system.
https://www.udemy.com/natural-language-processing-python-nltk/?couponCode=NLTK-BLOGS