2.安装spark与python练习

二：Python编程练习：英文文本的词频统计

1.准备文本文件

 

 

2.读文件　　预处理：大小写，标点符号，停用词，分词，统计每个单词出现的次数 ，按词频大小排序

def get_text(file_name):
  with open(file_name,'r',encoding='utf-8') as fr:
    text = fr.read()
    text = text.lower()#注意会有大小写之分，这里不区分大小写
    delete_ch = ['\n',':','!','?',',','.']#要删除的标点
    for ch in delete_ch:
      text = text.replace(ch,' ')
  return text

file_name = 'test.txt'#要处理的文件
text = get_text(file_name)
text = text.split()#注意！不能有split(' ')
count_dict = {}
#统计每个单词的个数
for i in text:
  count_dict[i] = count_dict.get(i,0) + 1
#转化成列表的形式
count_dict = list(count_dict.items())
count_dict.sort(key = lambda x:x[1], reverse = True)

for i in range(10):
  word,count = count_dict[i]
  print(word,count)

展示结果