综合练习：词频统计

下载一首英文的歌词或文章

将所有,.？！’:等分隔符全部替换为空格

将所有大写转换为小写

生成单词列表

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20

将分析对象存为utf-8编码的文件，通过文件读取的方式获得词频分析内容。

f=open('new.txt','r')
str0 = f.read();
f.close();
str1 = ''',.'?!"''';
for i in str1:
    list1 = str0.replace(i,' ');
list1 = str0.lower().split();

gath2={'in','to','your','you','and','the','for'};

gath=set(list1)-gath2;
print(gath)
#字典
dict={}
for w in gath:
    dict[w]=list1.count(w)

list1 = list(dict.items())
list1.sort(key=lambda x:x[1],reverse=True)
print(list1)

f=open('newscount.txt','a')
for i in range(25):
    f.write(list1[i][0]+' '+str(list1[i][1])+'\n')
f.close()

运行结果：

[('be', 8), ('one', 8), ('a', 8), ('wanna', 6), ('not', 6), ('or', 5), ('can', 5), ('just', 5), ('that', 4), ('president', 3), ('know', 3), ('actor', 3), ('we', 3), ('lawyer', 3), ('all', 3), ('singerwhy', 3), ('dreameryou', 3), ('could', 2), ('manyou', 2), ('bepolice', 2), ('manwhy', 2), ('old', 2), ('fighter', 2), ('like', 2), ('this', 2), ('got', 2), ('something', 2), ('what', 2), ('really', 2), ('man', 2), ('life', 2), ('post', 2), ('fire', 2), ('on', 2), ('foryou', 1), ('it', 1), ('matterwe', 1), ('matterluxury', 1), ('live', 1), ('real', 1), ('cars', 1), ('caught', 1), ('ways', 1), ('reach', 1), ('lifefocus', 1), ('team', 1), ('nice', 1), ('does', 1), ("that's", 1), ("thingthat's", 1), ('of', 1), ('little', 1), ('play', 1), ("doesn't", 1), ('medoctor', 1), ('topmake', 1), ('dream', 1), ('fori', 1), ('may', 1), ("bring'cause", 1), ('foreverjust', 1), ('nothing', 1), ('every', 1), ('steam', 1), ('lasts', 1), ('bedoctor', 1), ('up', 1), ('bei', 1), ('hold', 1), ('bewe', 1), ('doctor', 1), ('never', 1), ('thingthat', 1), ('different', 1), ('have', 1), ('stopbe', 1), ('sure', 1), ('benow', 1), ('share', 1), ('thinkbut', 1), ('with', 1), ('bling', 1), ("won't", 1), ('sing', 1), ('togetherwe', 1)]

2.中文词频统计：

下载一长篇中文文章。

从文件读取待分析文本。

news = open('gzccnews.txt','r',encoding = 'utf-8')

安装与使用jieba进行中文分词。

pip install jieba

import jieba

list(jieba.lcut(news))

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20（或把结果存放到文件里）

import jieba
f=open('xiyouji.txt','r',encoding='utf-8')
result = f.read()
f.close();
str1 = '''，。‘’“”：；（）！？、 '''
dele = {'我', '道', '不', '一', '了', '那', '是', '来', '他', '个', '行', '你', '的',
     '者','有','\n','-','出'}
jieba.add_word('三藏')
jieba.add_word('孙行者')
jieba.add_word('猴王')

for i in str1:
    result = result.replace(i, '')

tempwords = list(jieba.cut(result))
count = {}
words = list(set(tempwords) - dele)

for i in range(0, len(words)):
    count[words[i]] = result.count(str(words[i]))

countList = list(count.items())
countList.sort(key=lambda x: x[1], reverse=True)
print(countList)

f = open('baocun.txt', 'a',encoding='utf-8')
for i in range(30):
    f.write(countList[i][0] + ':' + str(countList[i][1]) + '\n')
f.close()

运行结果：