Spark入门教程收集

Spark教程

https://www.yiibai.com/spark/apache-spark-filter-function.html

Apache Spark是一个开源集群计算框架。其主要目的是处理实时生成的数据。

Spark建立在Hadoop MapReduce的顶部。它被优化为在内存中运行，而Hadoop的MapReduce等替代方法将数据写入计算机硬盘驱动器或从计算机硬盘驱动器写入数据。因此，Spark比其他替代方案更快地处理数据。

Spark架构依赖于两个抽象：

弹性分布式数据集(RDD)
有向无环图(DAG)

scala> val data = sc.parallelize(List(10,20,30))
data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[2] at parallelize at <console>:24

scala> data.collect
res3: Array[Int] = Array(10, 20, 30)

scala> val abc = data.filter(x => x!=30)

Spark 编程指南

https://www.w3cschool.cn/spark/

Spark SQL教程

https://www.w3cschool.cn/spark_sql/

Spark Overview

https://spark.apache.org/docs/3.0.0/index.html

Programming Guides:

Quick Start: a quick introduction to the Spark API; start here!
RDD Programming Guide: overview of Spark basics - RDDs (core but old API), accumulators, and broadcast variables
Spark SQL, Datasets, and DataFrames: processing structured data with relational queries (newer API than RDDs)
Structured Streaming: processing structured data streams with relation queries (using Datasets and DataFrames, newer API than DStreams)
Spark Streaming: processing data streams using DStreams (old API)
MLlib: applying machine learning algorithms
GraphX: processing graphs