11.28

jieba分词讲解2

 

3. 关键词提取

基于 TF-IDF 算法的关键词抽取

import jieba.analyse

• jieba.analyse.extract_tags(sentence, topK=20, withWeight=False, allowPOS=())
  • sentence 为待提取的文本
  • topK 为返回几个 TF/IDF 权重最大的关键词，默认值为 20
  • withWeight 为是否一并返回关键词权重值，默认值为 False
  • allowPOS 仅包括指定词性的词，默认值为空，即不筛选
• jieba.analyse.TFIDF(idf_path=None) 新建 TFIDF 实例，idf_path 为 IDF 频率文件

代码示例 （关键词提取）:https://github.com/fxsjy/jieba/blob/master/test/extract_tags.py(代码如下)

import sys
sys.path.append('../')

import jieba
import jieba.analyse
from optparse import OptionParser

USAGE = "usage:    python extract_tags.py [file name] -k [top k]"

parser = OptionParser(USAGE)
parser.add_option("-k", dest="topK")
opt, args = parser.parse_args()


if len(args) < 1:
    print(USAGE)
    sys.exit(1)

file_name = args[0]

if opt.topK is None:
    topK = 10
else:
    topK = int(opt.topK)

content = open(file_name, 'rb').read()

tags = jieba.analyse.extract_tags(content, topK=topK)

print(",".join(tags))

关键词提取所使用逆向文件频率（IDF）文本语料库可以切换成自定义语料库的路径

• 用法： jieba.analyse.set_idf_path(file_name) # file_name为自定义语料库的路径
• 自定义语料库示例：https://github.com/fxsjy/jieba/blob/master/extra_dict/idf.txt.big(内容如下)

• 劳动防护 13.900677652
  勞動防護 13.900677652
  生化学 13.900677652
  生化學 13.900677652
  奥萨贝尔 13.900677652
  奧薩貝爾 13.900677652
  考察队员 13.900677652
  考察隊員 13.900677652
  ......

• 用法示例：https://github.com/fxsjy/jieba/blob/master/test/extract_tags_idfpath.py

import sys
sys.path.append('../')

import jieba
import jieba.analyse
from optparse import OptionParser

USAGE = "usage:    python extract_tags_idfpath.py [file name] -k [top k]"

parser = OptionParser(USAGE)
parser.add_option("-k", dest="topK")
opt, args = parser.parse_args()


if len(args) < 1:
    print(USAGE)
    sys.exit(1)

file_name = args[0]

if opt.topK is None:
    topK = 10
else:
    topK = int(opt.topK)

content = open(file_name, 'rb').read()

jieba.analyse.set_idf_path("../extra_dict/idf.txt.big");#与extract_tags相比多了这一句

tags = jieba.analyse.extract_tags(content, topK=topK)

print(",".join(tags))

关键词提取所使用停止词（Stop Words）文本语料库可以切换成自定义语料库的路径

• 用法： jieba.analyse.set_stop_words(file_name) # file_name为自定义语料库的路径
• 自定义语料库示例：https://github.com/fxsjy/jieba/blob/master/extra_dict/stop_words.txt(内容如下)

• the
  of
  is
  and
  to
  in
  that
  we
  for
  an
  are
  by
  be
  as
  on
  with
  can
  if
  from
  which
  you
  it
  this
  then
  at
  have
  all
  not
  one
  has
  or
  that
  的
  了
  和
  是
  就
  都
  而
  及
  與
  著
  或
  一個
  沒有
  我們
  你們
  妳們
  他們
  她們
  是否

• 用法示例：https://github.com/fxsjy/jieba/blob/master/test/extract_tags_stop_words.py(代码如下)

import sys
sys.path.append('../')

import jieba
import jieba.analyse
from optparse import OptionParser

USAGE = "usage:    python extract_tags_stop_words.py [file name] -k [top k]"

parser = OptionParser(USAGE)
parser.add_option("-k", dest="topK")
opt, args = parser.parse_args()


if len(args) < 1:
    print(USAGE)
    sys.exit(1)

file_name = args[0]

if opt.topK is None:
    topK = 10
else:
    topK = int(opt.topK)

content = open(file_name, 'rb').read()

jieba.analyse.set_stop_words("../extra_dict/stop_words.txt")#停用词
jieba.analyse.set_idf_path("../extra_dict/idf.txt.big");#idf词频

tags = jieba.analyse.extract_tags(content, topK=topK)

print(",".join(tags))

关键词一并返回关键词权重值示例

• 用法示例：https://github.com/fxsjy/jieba/blob/master/test/extract_tags_with_weight.py(代码如下)

import sys
sys.path.append('../')

import jieba
import jieba.analyse
from optparse import OptionParser

USAGE = "usage:    python extract_tags_with_weight.py [file name] -k [top k] -w [with weight=1 or 0]"

parser = OptionParser(USAGE)
parser.add_option("-k", dest="topK")
parser.add_option("-w", dest="withWeight")
opt, args = parser.parse_args()


if len(args) < 1:
    print(USAGE)
    sys.exit(1)

file_name = args[0]

if opt.topK is None:
    topK = 10
else:
    topK = int(opt.topK)

if opt.withWeight is None:
    withWeight = False
else:
    if int(opt.withWeight) is 1:
        withWeight = True
    else:
        withWeight = False

content = open(file_name, 'rb').read()

tags = jieba.analyse.extract_tags(content, topK=topK, withWeight=withWeight)

if withWeight is True:
    for tag in tags:
        print("tag: %s\t\t weight: %f" % (tag[0],tag[1]))
else:
    print(",".join(tags))