|NO.Z.00020|——————————|^^ 案例 ^^|——|Hadoop&Spark.V08|——|Spark.v08|sparkcore|算子综合案例&wordcount-scala|

一、算子综合应用案例：wordCount-scala

### --- WordCount - scala

package cn.yanqi.sparkcore

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object ScalaWordCount {
  def main(args: Array[String]): Unit = {
    // 1、创建SparkContext
    val conf = new SparkConf().setAppName("WordCount")
    val sc = new SparkContext(conf)
    // 设置日志的显示级别：WARN为空
    sc.setLogLevel("WARN")

    // 2、读本地文件(集群运行：输入参数)
    val lines: RDD[String] = sc.textFile(args(0))

    // 3、RDD转换
    val words: RDD[String] = lines.flatMap(line => line.split("\\s+"))
    val wordsMap: RDD[(String, Int)] = words.map(x => (x, 1))
    val result: RDD[(String, Int)] = wordsMap.reduceByKey(_ + _)

    // 4、输出
    result.foreach(println)

    // 5、关闭SparkContext
    sc.stop()

    // 6、打包，使用spark-submit提交集群运行
    // spark-submit --master local[*] --class cn.yanqi.sparkcore.WordCount \
    // original-YanqiBigData-1.0-SNAPSHOT.jar /wcinput/*

    // spark-submit --master yarn --class cn.yanqi.sparkcore.WordCount \
    // original-YanqiBigData-1.0-SNAPSHOT.jar /wcinput/*
  }
}

### --- 编译打印

~~~     # 定义为本地文件地址：
    // 1、创建SparkContext
    val conf = new SparkConf().setMaster("local[*]").setAppName("WordCount")
    val sc = new SparkContext(conf)
    // 设置日志的显示级别为空
    sc.setLogLevel("WARN")

    // 2、读本地文件(集群运行：输入参数)
    val lines: RDD[String] = sc.textFile( path = "file:///E:\\NO.Z.10000——javaproject\\NO.Z.00002.Hadoop\\SparkBigData\\data\\wc.txt")

~~~     # 打印输出

(hadoop,2)
(hdfs,1)
(yanqi,3)
(mapreduce,5)
(yarn,2)

二、编译打包并上传服务器

### --- 编译打包

~~~     # 编程代码参数
    // 1、创建SparkContext
    val conf = new SparkConf().setAppName("WordCount")
    val sc = new SparkContext(conf)
    // 设置日志的显示级别：WARN为空
    sc.setLogLevel("WARN")

    // 2、读本地文件(集群运行：输入参数)
    val lines: RDD[String] = sc.textFile(args(0))

### --- 编译打包上传服务器

~~~     打包——>生成jar包：original-com.yanqi.sparkbigdata-1.0-SNAPSHOT.jar
~~~     重命名jar文件为：scalawordcoun.jar

 

~~~     # 将打包好的scalawordcount.jar包上传到服务器下

[root@hadoop02 ~]# ls
data  scalawordcoun.jar

三、在spark-standalone模式下运行

### --- 在local下运行

~~~     # 在standalone模式下运行自定义的jar包
[root@hadoop02 ~]# spark-submit --master local[*] --class cn.yanqi.sparkcore.WordCount \
scalawordcoun.jar /wcinput/*
~~~ 输出参数
(hadoop,2)
(yanqi,3)
(mapreduce,3)
(yarn,2)
(hdfs,1)

### --- 提交到yarn下运行

~~~     # 启动yarn服务
[root@hadoop01 ~]# start-yarn.sh 

~~~     # 在yarn模式下验证程序

[root@hadoop02 ~]# spark-submit --master yarn --class cn.yanqi.sparkcore.ScalaWordCount \
original-YanqiBigData-1.0-SNAPSHOT.jar /wcinput/*
~~~ 输出参数
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.1.123
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1634629619872
     final status: UNDEFINED
     tracking URL: http://hadoop01:8088/proxy/application_1634629602495_0001/
     user: root

---

 

 

 

 

 

 

 

 

 

---

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

                                                                                                                                                   ——W.S.Landor

---