04 RDD编程练习
一、filter,map,flatmap练习:
1.读文本文件生成RDD lines
lines=sc.textFile("file:///usr/local/spark/mycode/rdd/word.txt")
lines.collect()
2.将一行一行的文本分割成单词 words
words=lines.map(lambda line:line.split())
words.collect()
3.全部转换为小写
wordsxx=lines.map(lambda word:word.lower())
wordsxx.collect()
4.去掉长度小于3的单词
word=lines.flatmap(lambda line:line.split()).filter(lambda words:len(words)>2)
word.collect()
5.去掉停用词
lines=textFile("file///usr/local/spark/mycode/rdd/word.txt")
with open("/usr/lcaol/spark/mycode/rdd/stopwords.txt") as f:
stops=f.read().split()
lines.flatMap(lambda line:line.split()).filter(lambda word:word not in stops).collect()
二、groupByKey练习
6.练习一的生成单词键值对
words=lines.flatMap(lambda line:line.split()).map(lambda word:(word,1))
words.collect()
7.对单词进行分组
words1=words.groupByKey()
8.查看分组结果
words1.collect()
学生科目成绩文件练习:
0.数据文件上传
1.读大学计算机系的成绩数据集生成RDD
lines=textFile("file///usr/local/spark/mycode/rdd/xs.txt")
lines.take(3)
2.按学生汇总全部科目的成绩
groupName=lines.flatMap(lambda line:line.split(',')).map(lambda line:(line[0],(line[1],line[2]))).groupByKey()
groupName.take(3)
for i in groupName.first()[1]:
print(i)
3.按科目汇总学生的成绩
groupKm=lines.flatMap(lambda line:line.split(',')).Map(word:(word[1],(word[0],word[2]))).groupByKey()
groupKm.take(3)
for j in groupKm.first()[1]
print(j)