一、安装spark
1.基础环境——环境准备检查
2. 配置环境变量
3.配置相关文件
4.运行pyspark
5.在pyspark中运行代码
Python实现英文文本的词频统计
1.准备文本文件
2.读文件
txt = open("fit.txt", "r",encoding='UTF-8').read()
3.预处理:大小写、标点符号、停用词
txt = txt.lower() for ch in '!"@#$%^&*()+,-./:;<=>?@[\\]_`~{|}': txt=txt.replace(ch," ") words = txt.split() stop_words = ['so','out','all','for','of','to','on','in','if','by','under','it','at','into','with','about'] lenwords=len(words) afterwords=[] for i in range(lenwords): z=1 for j in range(len(stop_words)): if words[i]==stop_words[j]: continue else: if z==len(stop_words): afterwords.append(words[i]) break z=z+1 continue
4.统计每个单词出现的次数
counts = {} for word in afterwords: counts[word] = counts.get(word,0) + 1 items = list(counts.items()) items.sort(key=lambda x:x[1],reverse=True)
5.按词频大小排序
i=1 while i<=len(items): word,count = items[i-1] print("{0:<20}{1}".format(word,count)) i=i+1
6.结果写文件
txt= open("research.txt", "w",encoding='UTF-8') txt.write(str(items)) print("文件写入成功")
research.txt为输出结果生成文本文件
research.txt内容