RDD 编程练习

RDD 编程练习

一、filter, map, flatmap 练习：

1. 读文本文件生成 RDD lines

>>> lines=sc.textFile("file:///usr/local/spark/mycode/rdd/word.txt")

2. 将一行一行的文本分割成单词 words

>>> words = lines.flatMap(lambda line:line.split()).collect()

3. 全部转换为小写

>>> wordsL = sc.parallelize(words)
>>> sc.parallelize(words).pipe("tr 'A-Z' 'a-z'").collect()

4. 去掉长度小于3的单词

>>> wordsL = sc.parallelize(words)
>>> wordsL.collect()
>>> wordsL.filter(lambda word:len(word)>3).collect()

5. 去掉停用词

>>> with open('/usr/local/spark/mycode/rdd/stopwords.txt')as f:
...     stops=f.read().split()

>>> wordsL.filter(lambda word:word not in stops).collect()

二、groupByKey 练习：

6. 练习一的生成单词键值对

>>> words = sc.parallelize([("Hadoop",1),("is",1),("good",1),("Spark",1),\
... ("is"),("fast",1),("Spark",1),("is",1),("better",1)])

7. 对单词进行分组

>>> wordsL = words.groupByKey()

8. 查看分组结果

>>> wordsL.foreach(print)

三、学生科目成绩文件练习：

0. 数据文件上传

>>> lines = sc.textFile('file:///usr/local/spark/mycode/rdd/chapter4-data01.txt')

1. 读大学计算机系的成绩数据集生成 RDD

>>> lines.take(5)

2. 按学生汇总全部科目的成绩

>>> groupByName=lines.map(lambda line:line.split(',')).\
... map(lambda line:(line[0],(line[1],line[2]))).groupByKey()
>>> groupByName.take(5)
>>> groupByName.first()
>>> for i in groupByName.first()[1]:
...     print(i)

3. 按科目汇总学生的成绩

>>> groupByCourse=lines.map(lambda line:line.split(',')).\
... map(lambda line:(line[1],(line[0],line[2]))).groupByKey()
>>> groupByCourse.first()
>>> for i in groupByCourse.first()[1]:
...     print(i)