sparkStreaming 知识点

 

 

hadoop仓库： https://repositories.cloudera.com/artifactory

spark streaming定义：

   将不同的数据源数据经过spark streaming处理之后将结果输出到外部文件系统

 

特点

     低延时

     能从错误中高效的恢复

    能够运行在成百上千的节点

    能够将批处理、机器学习、图计算等综合使用

 

./spark-submit --master local[2]  --class  org.apache.spark.examples.streaming.NetworkWordCount  \

--name NetworkWordCount  \

/home/hadoop/app/spark-2.4.4-bin-2.6.0-cdh5.16.1/examples/jars/spark-examples_2.11-2.4.4.jar  yqdata000  9999

 

工作原理 ：

  spark streaming 接受到实时数据流，把数据按照指定的时间切成一片片小的数据块，

然后把小数据块交给spark engine处理

 

 

 

spark streaming 核心

 一、streaming context

def this(conf: SparkConf, batchDuration: Duration) = {
  this(StreamingContext.createNewSparkContext(conf), null, batchDuration)
}

def this(sparkContext: SparkContext, batchDuration: Duration) = {
  this(sparkContext, null, batchDuration)
}

batch interval 可以根据你的应用程序需求的延迟要求以及集群可用的资源情况来设置。

一旦StreamingContext定义好后。。。

1. Define the input sources by creating input DStreams.
2. Define the streaming computations by applying transformation and output operations to DStreams.
3. Start receiving data and processing it using streamingContext.start().
4. Wait for the processing to be stopped (manually or due to any error) using streamingContext.awaitTermination().
5. The processing can be manually stopped using streamingContext.stop().

Points to remember:

• Once a context has been started, no new streaming computations can be set up or added to it.
• Once a context has been stopped, it cannot be restarted.
• Only one StreamingContext can be active in a JVM at the same time.
• stop() on StreamingContext also stops the SparkContext. To stop only the StreamingContext, set the optional parameter of stop() called stopSparkContext to false.
• A SparkContext can be re-used to create multiple StreamingContexts, as long as the previous StreamingContext is stopped (without stopping the SparkContext) before the next StreamingContext is created.

二、DStreams（Discretized Streams）

   Internally, a DStream is represented by a continuous series of RDDs

   Each RDD in a DStream contains data from a certain interval,

  对DStream操作算子，比如map/flatMap ，其实底层会被翻译为对DStream中的每个RDD都做相同操作，

  因为一个DStream是由不同批次的RDD所构成的。

textFileStream ，
数据文件创建、修改时间要晚于服务启动时间数据才会被加载。数据必须以原子性方式添加，已添加的文件不能再进行修改

三、Input DStreams and Receivers

 Every input DStream (except file stream, discussed later in this section) is associated with a Receiver object

  which receives the data from a source and stores it in Spark’s memory for processing.

四、Transformations

五、Output Operations