TF-IDF算法

TF-IDF（term frequency–inverse document frequency，词频-逆向文件频率）

TF-IDF本质上是一种统计方法，用来评估一个词/token在整个语料库中当前文档中的重要程度，字词的重要性随着它在当前文档中出现的频率成正比增加，随着它在整个语料库中出现的频率成反比降低。

主要思想：某个单词在当前文章中出现的次数越多，并且在其他文章中出现的次数越少，则说明该词是当前文档的一个关键词，对于当前文档具有较好的类别区分能力，适合用来分类。

1）TF(Term Frequency)  词频

这个数字通常会被归一化(一般是词频除以文章总词数，也可以是这篇文章中出现最多的词的出现次数),以防止它偏向长的文件。

2）IDF(Inverse Document Frequency) 逆向文件频率

IDF是针对某一特定词语进行计算，可以由总文件数目除以包含该词语的文件的数目，再将得到的商取对数得到。

如果包含该特定词语的文档越少, IDF越大，则说明词条具有很好的类别区分能力。

 

3）计算TF-IDF

4）TF-IDF的用法

• 文档关键词提取

• 文档信息检索

 5）TF-IDF的优缺点

优点：简单快速，结果比较符合实际情况

缺点：1.没有考虑关键词的位置信息，词语出现在文档不同位置，贡献度是不一样的（首行和末行权重应更大）；2.生僻词的idf会高；不够全面，有些关键的人名、地名提取效果不好

import math
# 建立一个语料库
corpus = [
    "what is the weather like today",    
    "what is for dinner tonight",    
    "this is a question worth pondering",    
    "it is a beautiful day today"
]
# 进行分词,去停用词
# 如果有自定义的停用词典，我们可以用下列方法来分词并去掉停用词
words = []
f = ["is", "the"]
for i in corpus:
    all_words = i.split()
    new_words = []
    for j in all_words:
        if j not in f:
            new_words.append(j)
    words.append(new_words)
print(words)

# 计算词频TF
TF_list = []
def counter_Tf(words):
  for i in words:
    temp = []
    all_words = len(i)
    for word in i:
        temp.append(i.count(word)/all_words)
    TF_list.append(temp)
  return TF_list
counter_Tf(words)
print(TF_list)



# 计算IDF
word_dict = {}
def counter_Word_IDF(words):
  for i in words:
    for word in i:
      if word in word_dict:
        word_dict[word] += 1
      else:
        word_dict[word] = 0
  return word_dict

IDF_list = []
def counter_Tf(words):
  all_words = len(words)
  for i in words:
    temp = []
    for word in i:
        temp.append(math.log(all_words/(1+word_dict[word])))
    IDF_list.append(temp)
  return IDF_list
counter_Word_IDF(words)
counter_Tf(words)
print(IDF_list)
# 计算TF-IDF
TF_IDF = TF*IDF

 

 

import math

# 建立一个语料库

corpus = [

    "what is the weather like today",    

    "what is for dinner tonight",    

    "this is a question worth pondering",    

    "it is a beautiful day today"

]

# 进行分词,去停用词

# 如果有自定义的停用词典，我们可以用下列方法来分词并去掉停用词

words = []

f = ["is", "the"]

for i in corpus:

    all_words = i.split()

    new_words = []

    for j in all_words:

        if j not in f:

            new_words.append(j)

    words.append(new_words)

print(words)

# 计算词频TF

TF_list = []

def counter_Tf(words):

  for i in words:

    temp = []

    all_words = len(i)

    for word in i:

        temp.append(i.count(word)/all_words)

    TF_list.append(temp)

  return TF_list

counter_Tf(words)

print(TF_list)

# 计算IDF

word_dict = {}

def counter_Word_IDF(words):

  for i in words:

    for word in i:

      if word in word_dict:

        word_dict[word] += 1

      else:

        word_dict[word] = 0

  return word_dict

IDF_list = []

def counter_Tf(words):

  all_words = len(words)

  for i in words:

    temp = []

    for word in i:

        temp.append(math.log(all_words/(1+word_dict[word])))

    IDF_list.append(temp)

  return IDF_list

counter_Word_IDF(words)

counter_Tf(words)

print(IDF_list)

# 计算TF-IDF

TF_IDF = TF*IDF