自然语言处理(2)之文本资料库

自然语言处理(2)之文本资料库

1.获取文本资料库

本章首先给出了一个文本资料库的实例：nltk.corpus.gutenberg，通过gutenberg实例来学习文本资料库。我们用help来查看它的类型

>>> import nltk
>>> help(nltk.corpus.gutenberg)
Help on PlaintextCorpusReader in module nltk.corpus.reader.plaintext object:

class PlaintextCorpusReader(nltk.corpus.reader.api.CorpusReader)
 |  Reader for corpora that consist of plaintext documents.  Paragraphs
 |  are assumed to be split using blank lines.  Sentences and words can
 |  be tokenized using the default tokenizers, or by custom tokenizers
 |  specificed as parameters to the constructor.
 |  
 |  This corpus reader can be customized (e.g., to skip preface
 |  sections of specific document formats) by creating a subclass and
 |  overriding the ``CorpusView`` class variable.
 |  
 |  Method resolution order:
 |      PlaintextCorpusReader
 |      nltk.corpus.reader.api.CorpusReader
 |      __builtin__.object
 |  
 |  Methods defined here:
 |  
 |  __init__(self, root, fileids, word_tokenizer=WordPunctTokenizer(pattern='\\w+|[^\\w\\s]+', gaps=False, discard_empty=True, flags=56), sent_tokenizer=<nltk.tokenize.punkt.Punkt
SentenceTokenizer object>, para_block_reader=<function read_blankline_block>, encoding=None)
 |      Construct a new plaintext corpus reader for a set of documents
 |      located at the given root directory.  Example usage:
 |      
 |          >>> root = '/usr/local/share/nltk_data/corpora/webtext/'
 |          >>> reader = PlaintextCorpusReader(root, '.*\.txt')
 |      
 |      :param root: The root directory for this corpus.
 |      :param fileids: A list or regexp specifying the fileids in this corpus.
 |      :param word_tokenizer: Tokenizer for breaking sentences or
 |          paragraphs into words.
 |      :param sent_tokenizer: Tokenizer for breaking paragraphs
 |          into words.
 |      :param para_block_reader: The block reader used to divide the
 |          corpus into paragraph blocks.
 |  
 |  paras(self, fileids=None, sourced=False)
 |      :return: the given file(s) as a list of
 |          paragraphs, each encoded as a list of sentences, which are
 |          in turn encoded as lists of word strings.
 |      :rtype: list(list(list(str)))
 |  
 |  raw(self, fileids=None, sourced=False)
 |      :return: the given file(s) as a single string.
 |      :rtype: str
 |  
 |  sents(self, fileids=None, sourced=False)
 |      :return: the given file(s) as a list of
 |          sentences or utterances, each encoded as a list of word
 |          strings.
 |      :rtype: list(list(str))
 |  
 |  words(self, fileids=None, sourced=False)
 |      :return: the given file(s) as a list of words
|          and punctuation symbols.
 |      :rtype: list(str)
 |  
 |  ----------------------------------------------------------------------
 |  Data and other attributes defined here:
 |  
 |  CorpusView = <class 'nltk.corpus.reader.util.StreamBackedCorpusView'>
 |      A 'view' of a corpus file, which acts like a sequence of tokens:
 |      it can be accessed by index, iterated over, etc.  However, the
 |      tokens are only constructed as-needed -- the entire corpus is
 |      never stored in memory at once.
 |      
 |      The constructor to ``StreamBackedCorpusView`` takes two arguments:
 |      a corpus fileid (specified as a string or as a ``PathPointer``);
 |      and a block reader.  A "block reader" is a function that reads
 |      zero or more tokens from a stream, and returns them as a list.  A
 |      very simple example of a block reader is:
 |      
 |          >>> def simple_block_reader(stream):
 |          ...     return stream.readline().split()
 |      
 |      This simple block reader reads a single line at a time, and
 |      returns a single token (consisting of a string) for each
 |      whitespace-separated substring on the line.
 |      
 |      When deciding how to define the block reader for a given
 |      corpus, careful consideration should be given to the size of
 |      blocks handled by the block reader.  Smaller block sizes will
 |      increase the memory requirements of the corpus view's internal
 |      data structures (by 2 integers per block).  On the other hand,
 |      larger block sizes may decrease performance for random access to
 |      the corpus.  (But note that larger block sizes will *not*
 |      decrease performance for iteration.)
 |      
 |      Internally, ``CorpusView`` maintains a partial mapping from token
 |      index to file position, with one entry per block.  When a token
 |      with a given index *i* is requested, the ``CorpusView`` constructs
 |      it as follows:
 |      
 |        1. First, it searches the toknum/filepos mapping for the token
 |           index closest to (but less than or equal to) *i*.
 |      
 |        2. Then, starting at the file position corresponding to that
 |           index, it reads one block at a time using the block reader
 |           until it reaches the requested token.
 |      
 |      The toknum/filepos mapping is created lazily: it is initially
 |      empty, but every time a new block is read, the block's
 |      initial token is added to the mapping.  (Thus, the toknum/filepos
 |      map has one entry per block.)
 |      
 |      In order to increase efficiency for random access patterns that
 |      have high degrees of locality, the corpus view may cache one or
|      have high degrees of locality, the corpus view may cache one or
 |      more blocks.
 |      
 |      :note: Each ``CorpusView`` object internally maintains an open file
 |          object for its underlying corpus file.  This file should be
 |          automatically closed when the ``CorpusView`` is garbage collected,
 |          but if you wish to close it manually, use the ``close()``
 |          method.  If you access a ``CorpusView``'s items after it has been
 |          closed, the file object will be automatically re-opened.
 |      
 |      :warning: If the contents of the file are modified during the
 |          lifetime of the ``CorpusView``, then the ``CorpusView``'s behavior
 |          is undefined.
 |      
 |      :warning: If a unicode encoding is specified when constructing a
 |          ``CorpusView``, then the block reader may only call
 |          ``stream.seek()`` with offsets that have been returned by
 |          ``stream.tell()``; in particular, calling ``stream.seek()`` with
 |          relative offsets, or with offsets based on string lengths, may
 |          lead to incorrect behavior.
 |      
 |      :ivar _block_reader: The function used to read
 |          a single block from the underlying file stream.
 |      :ivar _toknum: A list containing the token index of each block
 |          that has been processed.  In particular, ``_toknum[i]`` is the
 |          token index of the first token in block ``i``.  Together
 |          with ``_filepos``, this forms a partial mapping between token
 |          indices and file positions.
 |      :ivar _filepos: A list containing the file position of each block
 |          that has been processed.  In particular, ``_toknum[i]`` is the
 |          file position of the first character in block ``i``.  Together
 |          with ``_toknum``, this forms a partial mapping between token
 |          indices and file positions.
 |      :ivar _stream: The stream used to access the underlying corpus file.
 |      :ivar _len: The total number of tokens in the corpus, if known;
 |          or None, if the number of tokens is not yet known.
 |      :ivar _eofpos: The character position of the last character in the
 |          file.  This is calculated when the corpus view is initialized,
 |          and is used to decide when the end of file has been reached.
 |      :ivar _cache: A cache of the most recently read block.  It
 |         is encoded as a tuple (start_toknum, end_toknum, tokens), where
 |         start_toknum is the token index of the first token in the block;
 |         end_toknum is the token index of the first token not in the
 |         block; and tokens is a list of the tokens in the block.
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from nltk.corpus.reader.api.CorpusReader:
 |  
 |  __repr__(self)
 |  
 |  abspath(self, fileid)
 |      Return the absolute path for the given file.
 |      
 |      :type file: str

|      :param file: The file identifier for the file whose path
 |          should be returned.
 |      :rtype: PathPointer
 |  
 |  abspaths(self, fileids=None, include_encoding=False, include_fileid=False)
 |      Return a list of the absolute paths for all fileids in this corpus;
 |      or for the given list of fileids, if specified.
 |      
 |      :type fileids: None or str or list
 |      :param fileids: Specifies the set of fileids for which paths should
 |          be returned.  Can be None, for all fileids; a list of
 |          file identifiers, for a specified set of fileids; or a single
 |          file identifier, for a single file.  Note that the return
 |          value is always a list of paths, even if ``fileids`` is a
 |          single file identifier.
 |      
 |      :param include_encoding: If true, then return a list of
 |          ``(path_pointer, encoding)`` tuples.
 |      
 |      :rtype: list(PathPointer)
 |  
 |  encoding(self, file)
 |      Return the unicode encoding for the given corpus file, if known.
 |      If the encoding is unknown, or if the given file should be
 |      processed using byte strings (str), then return None.
 |  
 |  fileids(self)
 |      Return a list of file identifiers for the fileids that make up
 |      this corpus.
 |  
 |  open(self, file, sourced=False)
 |      Return an open stream that can be used to read the given file.
 |      If the file's encoding is not None, then the stream will
 |      automatically decode the file's contents into unicode.
 |      
 |      :param file: The file identifier of the file to read.
 |  
 |  readme(self)
 |      Return the contents of the corpus README file, if it exists.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from nltk.corpus.reader.api.CorpusReader:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)
 |  
 |  root
 |      The directory where this corpus is stored.
 |      
 |      :type: PathPointer

在PlaintextCorpusReader中可以看到很多本文例子中方法，比如fileids()，words()等等。

1.1 fileids()返回语料库的文件标识符

>>> from nltk.corpus import gutenberg
>>> gutenberg.fileids()
['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']

1.2 words()返回文件的单词列表

>>> from nltk.corpus import gutenberg
>>> gutenberg.fileids()
['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']
>>> gutenberg.words('austen-emma.txt')
['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ']', ...]
>>> len(gutenberg.words('austen-emma.txt'))
192427

用concordance()来搜索文本里的单词

>>> emma = nltk.Text(gutenberg.words('austen-emma.txt'))
>>> emma
<Text: Emma by Jane Austen 1816>
>>> emma.concordance('surperize')
Building index...
No matches
>>> emma.concordance('surprize')
Displaying 25 of 37 matches:
er father , was sometimes taken by surprize at his being still able to pity ` 
hem do the other any good ." " You surprize me ! Emma must do Harriet good : a
Knightley actually looked red with surprize and displeasure , as he stood up ,
r . Elton , and found to his great surprize , that Mr . Elton was actually on 
d aid ." Emma saw Mrs . Weston ' s surprize , and felt that it must be great ,
father was quite taken up with the surprize of so sudden a journey , and his f
y , in all the favouring warmth of surprize and conjecture . She was , moreove
he appeared , to have her share of surprize , introduction , and pleasure . Th
ir plans ; and it was an agreeable surprize to her , therefore , to perceive t
talking aunt had taken me quite by surprize , it must have been the death of m
f all the dialogue which ensued of surprize , and inquiry , and congratulation
 the present . They might chuse to surprize her ." Mrs . Cole had many to agre
the mode of it , the mystery , the surprize , is more like a young woman ' s s
 to her song took her agreeably by surprize -- a second , slightly but correct
" " Oh ! no -- there is nothing to surprize one at all .-- A pretty fortune ; 
t to be considered . Emma ' s only surprize was that Jane Fairfax should accep
of your admiration may take you by surprize some day or other ." Mr . Knightle
ation for her will ever take me by surprize .-- I never had a thought of her i
 expected by the best judges , for surprize -- but there was great joy . Mr . 
 sound of at first , without great surprize . " So unreasonably early !" she w
d Frank Churchill , with a look of surprize and displeasure .-- " That is easy
; and Emma could imagine with what surprize and mortification she must be retu
tled that Jane should go . Quite a surprize to me ! I had not the least idea !
 . It is impossible to express our surprize . He came to speak to his father o
g engaged !" Emma even jumped with surprize ;-- and , horror - struck , exclai

这里用到了nltk.Text类，再次通过help查看这个类，通过method的查看发现这个类非常有用。

class Text(__builtin__.object)
 |  A wrapper around a sequence of simple (string) tokens, which is
 |  intended to support initial exploration of texts (via the
 |  interactive console).  Its methods perform a variety of analyses
 |  on the text's contexts (e.g., counting, concordancing, collocation
 |  discovery), and display the results.  If you wish to write a
 |  program which makes use of these analyses, then you should bypass
 |  the ``Text`` class, and use the appropriate analysis function or
 |  class directly instead.
 |  
 |  A ``Text`` is typically initialized from a given document or
 |  corpus.  E.g.:
 |  
 |  >>> import nltk.corpus
 |  >>> from nltk.text import Text
 |  >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt'))
 |  
 |  Methods defined here:
 |  
 |  __getitem__(self, i)
 |  
 |  __init__(self, tokens, name=None)
 |      Create a Text object.
 |      
 |      :param tokens: The source text.
 |      :type tokens: sequence of str
 |  
 |  __len__(self)
 |  
 |  __repr__(self)
 |      :return: A string representation of this FreqDist.
 |      :rtype: string
 |  
 |  collocations(self, num=20, window_size=2)
 |      Print collocations derived from the text, ignoring stopwords.
 |      
 |      :seealso: find_collocations
 |      :param num: The maximum number of collocations to print.
 |      :type num: int
 |      :param window_size: The number of tokens spanned by a collocation (default=2)
 |      :type window_size: int
 |  
 |  common_contexts(self, words, num=20)
 |      Find contexts where the specified words appear; list
 |      most frequent common contexts first.
 |      
 |      :param word: The word used to seed the similarity search
 |      :type word: str
 |      :param num: The number of words to generate (default=20)
 |      :type num: int
 |      :seealso: ContextIndex.common_contexts()
 |  
|  concordance(self, word, width=79, lines=25)
 |      Print a concordance for ``word`` with the specified context window.
 |      Word matching is not case-sensitive.
 |      :seealso: ``ConcordanceIndex``
 |  
 |  count(self, word)
 |      Count the number of times this word appears in the text.
 |  
 |  dispersion_plot(self, words)
 |      Produce a plot showing the distribution of the words through the text.
 |      Requires pylab to be installed.
 |      
 |      :param words: The words to be plotted
 |      :type word: str
 |      :seealso: nltk.draw.dispersion_plot()
 |  
 |  findall(self, regexp)
 |      Find instances of the regular expression in the text.
 |      The text is a list of tokens, and a regexp pattern to match
 |      a single token must be surrounded by angle brackets.  E.g.
 |      
 |      >>> from nltk.book import text1, text5, text9
 |      >>> text5.findall("<.*><.*><bro>")
 |      you rule bro; telling you bro; u twizted bro
 |      >>> text1.findall("<a>(<.*>)<man>")
 |      monied; nervous; dangerous; white; white; white; pious; queer; good;
 |      mature; white; Cape; great; wise; wise; butterless; white; fiendish;
 |      pale; furious; better; certain; complete; dismasted; younger; brave;
 |      brave; brave; brave
 |      >>> text9.findall("<th.*>{3,}")
 |      thread through those; the thought that; that the thing; the thing
 |      that; that that thing; through these than through; them that the;
 |      through the thick; them that they; thought that the
 |      
 |      :param regexp: A regular expression
 |      :type regexp: str
 |  
 |  generate(self, length=100)
 |      Print random text, generated using a trigram language model.
 |      
 |      :param length: The length of text to generate (default=100)
 |      :type length: int
 |      :seealso: NgramModel
 |  
 |  index(self, word)
 |      Find the index of the first occurrence of the word in the text.
 |  
 |  plot(self, *args)
 |      See documentation for FreqDist.plot()
 |      :seealso: nltk.prob.FreqDist.plot()
 |  
 |  readability(self, method)
 |  
 |  similar(self, word, num=20)
 |      Distributional similarity: find other words which appear in the
 |      same contexts as the specified word; list most similar words first.
 |      
 |      :param word: The word used to seed the similarity search
 |      :type word: str
 |      :param num: The number of words to generate (default=20)
 |      :type num: int
 |      :seealso: ContextIndex.similar_words()
 |  
 |  vocab(self)
 |      :seealso: nltk.prob.FreqDist
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)

1.3 raw,sent，words的区别

我们通过以下例子来查看raw，sent，words的区别：

  1 #!/bin/envs python
  2 from nltk.corpus import gutenberg
  3 for fileid in gutenberg.fileids():
  4     num_chars = len(gutenberg.raw(fileid))                                  // 字母的个数
  5     num_words = len(gutenberg.words(fileid))                                // 单词的个数
  6     num_sents = len(gutenberg.sents(fileid))                                // 句子的个数
  7     num_vocab = len(set([w.lower() for w in gutenberg.words(fileid)]))      // 不相同的单词的个数
  8     print int(num_chars/num_words),int(num_words/num_sents),int(num_words/num_vocab),fileid
  
4 21 26 austen-emma.txt  //平均单词长度   平均每句单词个数   平均单词的重复个数
4 23 16 austen-persuasion.txt
4 23 22 austen-sense.txt
4 33 79 bible-kjv.txt
4 18 5 blake-poems.txt
4 17 14 bryant-stories.txt
4 17 12 burgess-busterbrown.txt
4 16 12 carroll-alice.txt
4 17 11 chesterton-ball.txt
4 19 11 chesterton-brown.txt
4 16 10 chesterton-thursday.txt
4 17 24 edgeworth-parents.txt
4 24 15 melville-moby_dick.txt
4 52 10 milton-paradise.txt
4 11 8 shakespeare-caesar.txt
4 12 7 shakespeare-hamlet.txt
4 12 6 shakespeare-macbeth.txt
4 35 12 whitman-leaves.txt

获取并查看shakespeare-macbeth.txt文本最长的一个句子

  1 #!/bin/envs python
  2 from nltk.corpus import gutenberg
  3 macbenth_sentences = gutenberg.sents('shakespeare-macbeth.txt') # 获取句子的list
  4 print macbenth_sentences
  5 print macbenth_sentences[1037]
  6 longtest_len=max([len(s) for s in macbenth_sentences])         # 获取最长句子的长度
  7 [ s for s in macbenth_sentences if longtest_len == len(s)]     # 获取最长句子的内容

[['[', 'The', 'Tragedie', 'of', 'Macbeth', 'by', 'William', 'Shakespeare', '1603', ']'], ['Actus', 'Primus', '.'], ...]

['Good', 'night', ',', 'and', 'better', 'health', 'Attend', 'his', 'Maiesty']

[['Doubtfull', 'it', 'stood', ',', 'As', 'two', 'spent', 'Swimmers', ',', 'that', 'doe', 'cling', 'together', ',', 'And', 'choake', 'their', 'Art', ':', 'The', 'mercilesse', 'Macdonwald', '(', 'Worthie', 'to', 'be', 'a', 'Rebell', ',', 'for', 'to', 'that', 'The', 'multiplying', 'Villanies', 'of', 'Nature', 'Doe', 'swarme', 'vpon', 'him', ')', 'from', 'the', 'Westerne', 'Isles', 'Of', 'Kernes', 'and', 'Gallowgrosses', 'is', 'supply', "'", 'd', ',', 'And', 'Fortune', 'on', 'his', 'damned', 'Quarry', 'smiling', ',', 'Shew', "'", 'd', 'like', 'a', 'Rebells', 'Whore', ':', 'but', 'all', "'", 's', 'too', 'weake', ':', 'For', 'braue', 'Macbeth', '(', 'well', 'hee', 'deserues', 'that', 'Name', ')', 'Disdayning', 'Fortune', ',', 'with', 'his', 'brandisht', 'Steele', ',', 'Which', 'smoak', "'", 'd', 'with', 'bloody', 'execution', '(', 'Like', 'Valours', 'Minion', ')', 'caru', "'", 'd', 'out', 'his', 'passage', ',', 'Till', 'hee', 'fac', "'", 'd', 'the', 'Slaue', ':', 'Which', 'neu', "'", 'r', 'shooke', 'hands', ',', 'nor', 'bad', 'farwell', 'to', 'him', ',', 'Till', 'he', 'vnseam', "'", 'd', 'him', 'from', 'the', 'Naue', 'toth', "'", 'Chops', ',', 'And', 'fix', "'", 'd', 'his', 'Head', 'vpon', 'our', 'Battlements']]

1.4 NPSChatCorpusReader类

接下来学习下新的一个reader类，nltk给出另一个实例类nltk.corpus.nps_chat，同样用help来查看下该类的信息。可以初步看出该类与xml格式的文件有关。

nps_chat = class NPSChatCorpusReader(nltk.corpus.reader.xmldocs.XMLCorpusReader)
 |  Method resolution order:
 |      NPSChatCorpusReader
 |      nltk.corpus.reader.xmldocs.XMLCorpusReader
 |      nltk.corpus.reader.api.CorpusReader
 |      __builtin__.object
 |  
 |  Methods defined here:
...

>>> from nltk.corpus import nps_chat
>>> nps_chat.fileids()
['10-19-20s_706posts.xml', '10-19-30s_705posts.xml', '10-19-40s_686posts.xml', '10-19-adults_706posts.xml', '10-24-40s_706posts.xml', '10-26-teens_706posts.xml', '11-06-adults_706posts.xml', '11-08-20s_705posts.xml', '11-08-40s_706posts.xml', '11-08-adults_705posts.xml', '11-08-teens_706posts.xml', '11-09-20s_706posts.xml', '11-09-40s_706posts.xml', '11-09-adults_706posts.xml', '11-09-teens_706posts.xml']
>>> chartoom=nps_chat.posts('10-19-20s_706posts.xml')
>>> chartoom[123]
['i', 'do', "n't", 'want', 'hot', 'pics', 'of', 'a', 'female', ',', 'I', 'can', 'look', 'in', 'a', 'mirror', '.']

 1.5 CategorizedTaggedCorpusReader类

本文以brown类为实例介绍了CategorizedTaggedCorpusReader类。

>>> from nltk.corpus import brown
>>> help(brown)
class CategorizedTaggedCorpusReader(nltk.corpus.reader.api.CategorizedCorpusReader, TaggedCorpusReader)
 |  A reader for part-of-speech tagged corpora whose documents are
 |  divided into categories based on their file identifiers.
 |  
 |  Method resolution order:
 |      CategorizedTaggedCorpusReader
 |      nltk.corpus.reader.api.CategorizedCorpusReader
 |      TaggedCorpusReader
 |      nltk.corpus.reader.api.CorpusReader
 |      __builtin__.object
 |  
 |  Methods defined here:
 |  
 |  __init__(self, *args, **kwargs)
 |      Initialize the corpus reader.  Categorization arguments
 |      (``cat_pattern``, ``cat_map``, and ``cat_file``) are passed to
 |      the ``CategorizedCorpusReader`` constructor.  The remaining arguments
 |      are passed to the ``TaggedCorpusReader``.
 |  
 |  paras(self, fileids=None, categories=None)
 |  
 |  raw(self, fileids=None, categories=None)
 |  
 |  sents(self, fileids=None, categories=None)
 |  
 |  tagged_paras(self, fileids=None, categories=None, simplify_tags=False)
 |  
 |  tagged_sents(self, fileids=None, categories=None, simplify_tags=False)
 |  
 |  tagged_words(self, fileids=None, categories=None, simplify_tags=False)
 |  
 |  words(self, fileids=None, categories=None)
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from nltk.corpus.reader.api.CategorizedCorpusReader:
 |  
 |  categories(self, fileids=None)
 |      Return a list of the categories that are defined for this corpus,
 |      or for the file(s) if it is given.
 |  
 |  fileids(self, categories=None)
 |      Return a list of file identifiers for the files that make up
 |      this corpus, or that make up the given category(s) if specified.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from nltk.corpus.reader.api.CategorizedCorpusReader:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from nltk.corpus.reader.api.CorpusReader:
 |  
 |  __repr__(self)
 |  
 |  abspath(self, fileid)
 |      Return the absolute path for the given file.
 |      
 |      :type file: str
 |      :param file: The file identifier for the file whose path
 |          should be returned.
 |      :rtype: PathPointer
 |  
 |  abspaths(self, fileids=None, include_encoding=False, include_fileid=False)
 |      Return a list of the absolute paths for all fileids in this corpus;
 |      or for the given list of fileids, if specified.
 |      
 |      :type fileids: None or str or list
 |      :param fileids: Specifies the set of fileids for which paths should
 |          be returned.  Can be None, for all fileids; a list of
 |          file identifiers, for a specified set of fileids; or a single
 |          file identifier, for a single file.  Note that the return
 |          value is always a list of paths, even if ``fileids`` is a
 |          single file identifier.
 |      
 |      :param include_encoding: If true, then return a list of
 |          ``(path_pointer, encoding)`` tuples.
 |      
 |      :rtype: list(PathPointer)
 |  
 |  encoding(self, file)
 |      Return the unicode encoding for the given corpus file, if known.
 |      If the encoding is unknown, or if the given file should be
 |      processed using byte strings (str), then return None.
 |  
 |  open(self, file, sourced=False)
 |      Return an open stream that can be used to read the given file.
 |      If the file's encoding is not None, then the stream will
 |      automatically decode the file's contents into unicode.
 |      
 |      :param file: The file identifier of the file to read.
 |  
 |  readme(self)
 |      Return the contents of the corpus README file, if it exists.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from nltk.corpus.reader.api.CorpusReader:
 |  
 |  root
 |      The directory where this corpus is stored.
 |      
 |      :type: PathPointer

看下 brown的内容，如果获取brown资料库的主题和文件

>>> from nltk.corpus import brown
>>> brown.categories()   //返回brown资料库的主题种类
['adventure', 'belles_lettres', 'editori', 'editorial', 'fiction', 'government', 'hobbies', 'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']
>>> brown.fileids()[1:10] //返回brown资料库内的文件
['ca02', 'ca03', 'ca04', 'ca05', 'ca06', 'ca07', 'ca08', 'ca09', 'ca10']
>>> brown.words(categories='news') //返回brown资料库内类别名为news的类别，并按次进行切分
['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...]
>>> brown.words(fileids=['cg22'])  //返回brown资料库内的文件名为cg22的文件，并按词进行切分
['Does', 'our', 'society', 'have', 'a', 'runaway', ',', ...]
>>> brown.sents(categories=['news','editori','reviews'])//返回多个类别，并按句进行切分
[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of', "Atlanta's", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence', "''", 'that', 'any', 'irregularities', 'took', 'place', '.'], ['The', 'jury', 'further', 'said', 'in', 'term-end', 'presentments', 'that', 'the', 'City', 'Executive', 'Committee', ',', 'which', 'had', 'over-all', 'charge', 'of', 'the', 'election', ',', '``', 'deserves', 'the', 'praise', 'and', 'thanks', 'of', 'the', 'City', 'of', 'Atlanta', "''", 'for', 'the', 'manner', 'in', 'which', 'the', 'election', 'was', 'conducted', '.'], ...]

对brown内的特定的文体进行计数：

1 from nltk.corpus import brown
2 import nltk
3 news_text = brown.words(categories='news')   //返回brown资料库内类别名为news的类别，并按次进行切分
4 fdist = nltk.FreqDist([w.lower() for w in news_text])   //获取news的频率分布
5 modals = ['can','could','may','might','must','will']    
6 for m in modals : 
7     print m + ':',fdist[m],                  //获取modals的计数

输出

  can: 94 could: 87 may: 93 might: 38 must: 53 will: 389

计算多个特定类别的多个文体进行统计

  1 from nltk.corpus import brown
  2 import nltk
  3 cfd = nltk.ConditionalFreqDist(
  4         (genre,word)
  5         for genre in brown.categories()
  6         for word in brown.words(categories=genre))
  7 genres=['new','religion','hobbies','science_fiction','romance','humor']
  8 modals = ['can','could','may','might','must','will']
  9 cfd.tabulate(conditions=genres,samples=modals)

                 can could  may might must will
            new    0    0    0    0    0    0
       religion   82   59   78   12   54   71
        hobbies  268   58  131   22   83  264
science_fiction   16   49    4   12    8   16
        romance   74  193   11   51   45   43
          humor   16   30    8    8    9   13

1.6  CategorizedPlaintextCorpusReader类

相比与brown(CategorizedTaggedCorpusReader)，retuters(CategorizedPlaintextCorpusReader)的区别在于，retuters可以查找一个或者多个文档涵盖的主题，也可以查找包含在一个或多个类别的文档。

>>> from nltk.corpus import reuters
>>> reuters.fileids()[1:10]
['test/14828', 'test/14829', 'test/14832', 'test/14833', 'test/14839', 'test/14840', 'test/14841', 'test/14842', 'test/14843']
>>> reuters.categories()
['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa', 'coconut', 'coconut-oil', 'coffee', 'copper', 'copra-cake', 'corn', 'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl', 'dlr', 'dmk', 'earn', 'fuel', 'gas', 'gnp', 'gold', 'grain', 'groundnut', 'groundnut-oil', 'heat', 'hog', 'housing', 'income', 'instal-debt', 'interest', 'ipi', 'iron-steel', 'jet', 'jobs', 'l-cattle', 'lead', 'lei', 'lin-oil', 'livestock', 'lumber', 'meal-feed', 'money-fx', 'money-supply', 'naphtha', 'nat-gas', 'nickel', 'nkr', 'nzdlr', 'oat', 'oilseed', 'orange', 'palladium', 'palm-oil', 'palmkernel', 'pet-chem', 'platinum', 'potato', 'propane', 'rand', 'rape-oil', 'rapeseed', 'reserves', 'retail', 'rice', 'rubber', 'rye', 'ship', 'silver', 'sorghum', 'soy-meal', 'soy-oil', 'soybean', 'strategic-metal', 'sugar', 'sun-meal', 'sun-oil', 'sunseed', 'tea', 'tin', 'trade', 'veg-oil', 'wheat', 'wpi', 'yen', 'zinc']
>>> reuters.categories('training/9865')
['barley', 'corn', 'grain', 'wheat']
>>> reuters.categories(['training/9865','training/9880'])
['barley', 'corn', 'grain', 'money-fx', 'wheat']
>>> reuters.categories('training/9880')
['money-fx']

对比brown：

>>> from nltk.corpus import brown
>>> brown.categories(['news','reviews'])   //不能对多个主题进行查找
[]
4 >>> brown.fileids(['cr05','cr06'])
5 []

1.7 基本语料库函数

示例	描述
fileids()	语料库的文件
fileids([categories])	分类对应的语料库中的文件
categories()	语料库中的分类
categoried([fileids])	文件对应的语料库中的分类
raw()	语料库的原始内容
raw(fileids=[f1,f2,f3])	指定文件的原始内容
raw(categories=[c1,c2])	制定分类的原始内容
words()	整个语料库中的词汇
words(fileids=[f1,f2,f3])	指定文件的词汇
words(categories=[c1,c2])	指定分类的词汇
sents()	指定分类的句子
sents(fileids=[f1,f2,f3])	指定文件的句子
sents(categories=[c1,c2])	指定分类的句子
abspath(fileid)	制定文件在磁盘的位置
encoding(fileid)	文件的编码（如果知道的话）
open(fileid)	打开指定语料库文件的文件流
root()	到本地安装的语料库根目录的路径
readme()	语料库的README文件的内容

1.8 载入自己的语料库

>>> from nltk.corpus import PlaintextCorpusReader
>>> corpus_root='/Users/rcf/workspace/python/python_test/NLP_WITH_PYTHON/chapter_2'
>>> wordlist=PlaintextCorpusReader(corpus_root,'.*')   //corpus_root 资料库路径，'.*'文件类型
>>> wordlist.fileids() 
['1.py', '2.py', '3.py', '4.py'] 
>>> wordlist.words('3.py') 
['from', 'nltk', '.', 'corpus', 'import', 'brown', ...]