综合练习：词频统计

1.英文词频统计：

下载一首英文的歌词或文章

news = ''' A Chinese Foreign Ministry spokesperson said here Monday that the United States should know that in the 21st century, 
global trade requires rules rather than power.
Foreign Ministry spokesperson Hua Chunying made the remarks at a daily press briefing, when commenting on U.S. 
Vice President Mike Pence's remarks that recent U.S. trade moves meant the "era of economic surrender is over."
Instead of saying the "era of economic surrender is over," 
it is better for the U.S. side to say that the U.S. 
economic intimidation and hegemony should be eliminated, Hua said.
She said that China has always complied with WTO rules and maintained a multilateral 
trading system with the WTO as its core and rules as the basis. 
It has always advocated that in a spirit of mutual respect, equality and mutual benefit, 
disputes including economic and trade friction should be settled through consultation.
Hua said that both China and the United States had been negotiating on economic and trade issues. 
And China has the confidence and ability to safeguard its legitimate interests under any circumstances.
"China hopes that the United States will make rational and cautious decisions and choices," she said.
U.S. President Donald Trump on Thursday signed a memorandum that could impose tariffs on up to 60 billion U.S. 
dollars of imports from China and restrictions on Chinese investment in the United States.
"China is willing to have consultations with the U.S. side guided by the principle of mutual respect and benefit," 
Hua said, noting that the the door to dialogue and consultation between the two sides has always been open.
China's Vice Foreign Minister Zheng Zeguang said on Monday that the economic and trade relations 
between China and the United States were essentially mutually beneficial and brought a lot of tangible benefits to the two peoples.
"We don't want to have a trade war, but we are never afraid of it," Zheng said, noting that China was fully 
determined and prepared to deal with such a situation to safeguard the national interest.
Zheng expressed his hope that the U.S. side would calm down and find a solution.'''

将所有,.？！’:等分隔符全部替换为空格，将所有大写转换为小写，生成单词列表，生成词频统计

sep = ''',.?';'"'''
for i in sep:
    news.replace(i," ")
newsList =  news.lower().split()
countdict = {}
newsset = set(newsList)

for i in newsset:
    countdict[i] = newsList.count(i)
for i in countdict:
    print(i,countdict[i])

排序，排除语法型词汇，代词、冠词、连词，输出词频最大TOP20

dictList = list(countdict.items())
dictList.sort(key = lambda x:x[1],reverse = True)
delList = {"the","a"，"an"}
newsset = set(newsList) - delList
for i in range(20):
    print(dictList[i])

将分析对象存为utf-8编码的文件，通过文件读取的方式获得词频分析内容。

读取歌词：

f = open("D:/news.txt","r")
news= f.read();
f.close()

保存分析结果：

f = open("D:/result.txt","a")
for i in range(20):
    f.write('\n'+dictList[i][0]+" "+str(dictList[i][1]))
f.close()

实验结果：

 

2.中文词频统计：

下载一长篇中文文章。

从文件读取待分析文本。

news = open('gzccnews.txt','r',encoding = 'utf-8')

安装与使用jieba进行中文分词。

pip install jieba

import jieba

list(jieba.lcut(news))

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20（或把结果存放到文件里）

 

import jieba
f = open("D:\新闻.txt","r")
str1 = f.read()
stringList =list(jieba.cut(str1))

delset = {"，","。","：","“","”","？"," ","；","！","、"}
stringset = set(stringList) - delset

countdict = {}
for i in stringset:
    countdict[i] = stringList.count(i)

dictList = list(countdict.items())
dictList.sort(key = lambda x:x[1],reverse = True)

f = open("D:/result.txt", "a")
for i in range(20):
 f.write('\n' + dictList[i][0] + " " + str(dictList[i][1]))
f.close()