Flink - [03] API
使用scala编写flink api从不同的数据源（源端）读取数据，并进行无界流/有界流的数据处理，最终将处理好的数据sink到对应的目标端
一、maven配置
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>flinkapi</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.11</artifactId>
<version>1.7.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-scala -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.11</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka-0.11_2.11</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.44</version>
</dependency>
<dependency>
<groupId>org.apache.bahir</groupId>
<artifactId>flink-connector-redis_2.11</artifactId>
<version>1.0</version>
</dependency>
<!--<dependency>-->
<!--<groupId>org.apache.flink</groupId>-->
<!--<artifactId>flink-connector-elasticsearch6_2.11</artifactId>-->
<!--<version>1.7.2</version>-->
<!--</dependency>-->
<!--<dependency>-->
<!--<groupId>org.apache.flink</groupId>-->
<!--<artifactId>flink-statebackend-rocksdb_2.11</artifactId>-->
<!--<version>1.7.2</version>-->
<!--</dependency>-->
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<!--<arg>-make:transitive</arg>-->
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>WordCount</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
二、WordCount
（1）准备好wordcount.txt文本
hello world
dog fish
hadoop
spark
hello world
dog fish
hadoop
spark
hello world
dog fish
hadoop
spark（2）有界数据流有状态计算（scala语言）
package com.harley.test
import org.apache.flink.api.scala._
object Demo01_ExecutionEnvironment {
def main(args: Array[String]): Unit = {
val env = ExecutionEnvironment.getExecutionEnvironment
val ds: DataSet[String] = env.readTextFile("src/main/resources/wordcount.txt")
val res = ds.flatMap(_.split(" "))
.map((_,1))
.groupBy(0) //元组的第一个元素
.sum(1)
//输出结果
res.print()
}
}
（3）无界数据流有状态计算
package com.harley.test
import org.apache.flink.streaming.api.scala._
object Demo02_StreamExecutionEnvironment {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val ds: DataStream[String] = env.socketTextStream("localhost", 9999)
val res = ds.flatMap(_.split(" "))
.filter(_ != "a")
.map((_, 1))
.keyBy(0)
.sum(1)
// 设置并行度
res.print("Stream print").setParallelism(2)
env.execute("Stream job")
}
}
三、整合不同的数据源
3.1、将集合作为数据源
import org.apache.flink.streaming.api.scala._
object Demo01_Source{
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
val ds = env.fromCollection(List(
SensorReader("sensor_1", 1547718199, 35.80018327300259),
SensorReader("sensor_6", 1547718201, 15.402984393403084),
SensorReader("sensor_7", 1547718202, 6.720945201171228),
SensorReader("sensor_10", 1547718205, 38.101067604893444)
))
ds.print("collection source")
env.execute("source job")
}
case class SensorReader(id: String,timestamp: Long,temperature: Double)
}
3.2、Kafka作为数据源
（1）启动Kafka服务，创建topic，并启动一个producer，向producer中发送数据
nohup bin/kafka-server-config.sh config/server.properties &
bin/kafka-console-producer.sh --bootstrap-server node7-2:9092 --topic test（2）通过Flink API服务topic中的数据
import java.util.Properties
import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.connectors.kafka.{FlinkKafkaConsumer010}
object Demo02_KafkaSource {
def main(args: Array[String]): Unit = {
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
// 设置全局并行度
env.setParallelism(1)
val props = new Properties()
props.setProperty("bootstrap.servers","node7-2:9092")
props.setProperty("group.id","kafkasource")
props.setProperty("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer")
props.setProperty("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer")
props.setProperty("auto.offset.reset","latest")
val stream = env.addSource(new FlinkKafkaConsumer010[String](
"test", // topic
new SimpleStringSchema(),
props
))
stream.print("kafka source")
env.execute("job")
}
}
3.3、自定义Source
import org.apache.flink.streaming.api.functions.source.SourceFunction
import org.apache.flink.streaming.api.scala._
import scala.util.Random
object Demo03_DefinedSource {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
val stream = env.addSource(new SensorSource())
stream.print("defined source")
env.execute("defined job")
}
class SensorSource extends SourceFunction[Sensor1]{
var running: Boolean = true
override def run(sourceContext: SourceFunction.SourceContext[Sensor1]): Unit = {
val random = new Random()
var currentTemp = 1.to(10).map(
i => ("sensor_"+i,random.nextGaussian()*20+60)
)
while(running){
currentTemp = currentTemp.map(tuple=>{
(tuple._1,tuple._2+random.nextGaussian())
})
val ts = System.currentTimeMillis()
currentTemp.foreach(tuple=>{
sourceContext.collect(Sensor1(tuple._1,ts,tuple._2))
})
Thread.sleep(1000)
}
}
override def cancel(): Unit = running = false
}
case class Sensor1(id:String,timestamp:Long,tempreture: Double)
}
四、Flink的方法
4.1、Transform
（1）准备文本文件sensor.txt
sensor_1,1600828704738,62.00978962180007
sensor_2,1600828704738,88.07800632412795
sensor_3,1600828704738,63.85113916269769
sensor_4,1600828704738,88.11328700513668
sensor_5,1600828704738,104.80491942566778
sensor_6,1600828704738,57.27152286624301
sensor_7,1600828704738,42.542439944867574
sensor_8,1600828704738,59.3964933103558
sensor_9,1600828704738,59.967837312594796
sensor_10,1600828704738,77.23695484678282
sensor_1,1600828705745,62.5
sensor_1,1600828705745,88.86940686134874
sensor_3,1600828705745,64.9
sensor_1,1600828705745,87.8
sensor_5,1600828705745,104.33176752272263
sensor_6,1600828705745,56.14405735923403（2）
import org.apache.flink.api.java.tuple.Tuple
import org.apache.flink.streaming.api.scala._
object Demo04_Transform {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
val stream: DataStream[String] = env.readTextFile("src/main/resources/sensor.txt")
val value = stream.map(line => {
val splitedArrs = line.split(",")
Sensor1(splitedArrs(0), splitedArrs(1).trim.toLong, splitedArrs(2).trim.toDouble)
})
val value1 = stream.flatMap(_.split(","))
stream.filter(_.split(",")(0)=="sensor_1")
value.keyBy("id")
.reduce((s1,s2)=>Sensor1(s1.id,s1.timestamp,s2.tempreture+100))
.print("keyby")
val value2: KeyedStream[Sensor1, Tuple] = value.keyBy(0)
env.execute("job")
}
case class Sensor1(id:String,timestamp: Long,tempreture: Double)
}
4.2、Split
import com.jinghang.day13.Demo04_Transform.Sensor1
import org.apache.flink.streaming.api.scala._
object Demo05_Split {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
val stream: DataStream[String] = env.readTextFile("src/main/resources/sensor.txt")
val ds: DataStream[Sensor1] = stream.map(line => {
val splited = line.split(",")
Sensor1(splited(0), splited(1).trim.toLong, splited(2).trim.toDouble)
})
val splitStream: SplitStream[Sensor1] = ds.split(
sensor => {
if (sensor.tempreture > 60)
Seq("high")
else
Seq("low")
})
val highStream: DataStream[Sensor1] = splitStream.select("high")
val lowStream: DataStream[Sensor1] = splitStream.select("low")
val allStream: DataStream[Sensor1] = splitStream.select("high","low")
val connStream: ConnectedStreams[Sensor1, Sensor1] = highStream.connect(lowStream)
val unionStream: DataStream[Sensor1] = highStream.union(lowStream).union(allStream)
highStream.print("high")
lowStream.print("low")
allStream.print("all")
env.execute("job")
}
}
4.3、UDF
import com.jinghang.day13.Demo04_Transform.Sensor1
import org.apache.flink.api.common.functions.{FilterFunction, RichFilterFunction, RichFlatMapFunction}
import org.apache.flink.configuration.Configuration
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector
object Demo06_UDF {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
val stream: DataStream[String] = env.readTextFile("src/main/resources/sensor.txt")
val ds: DataStream[Sensor1] = stream.map(line => {
val splitedArrs = line.split(",")
Sensor1(splitedArrs(0), splitedArrs(1).trim.toLong, splitedArrs(2).trim.toDouble)
})
val dataStream = ds.filter(new UDFFilter())
ds.filter(new RichFilterFunction[Sensor1] {
override def filter(value: Sensor1): Boolean = {
value.id == "sensor_1"
}
//可以做一些预操作
override def open(parameters: Configuration): Unit = super.open(parameters)
})
dataStream.print("filter")
env.execute("filter job")
}
class UDFFilter() extends FilterFunction[Sensor1]{
override def filter(value: Sensor1): Boolean = {
value.id == "sensor_1"
}
}
class UDFFlatMap extends RichFlatMapFunction[String,String]{
override def flatMap(value: String, out: Collector[String]): Unit = {
}
}
}
4.4、JDBC Sink
import java.sql.{Connection, DriverManager, PreparedStatement}
import org.apache.flink.configuration.Configuration
import org.apache.flink.streaming.api.functions.sink.{RichSinkFunction, SinkFunction}
import org.apache.flink.streaming.api.scala._
object Demo01_JdbcSink {
def main(args: Array[String]): Unit = {
// 创建流处理的环境对象
val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
// 设置全局并行度
env.setParallelism(1)
val stream: DataStream[String] = env.readTextFile("src/main/resources/sensor.txt")
val dataStream: DataStream[Sensor] = stream.map(line => {
val splitedArrs = line.split(",")
Sensor(splitedArrs(0), splitedArrs(1).trim.toLong, splitedArrs(2).trim.toDouble)
})
dataStream.addSink(new JDBCSink())
env.execute("job")
}
case class Sensor(id: String,timestamp: Long,temperature: Double)
class JDBCSink extends RichSinkFunction[Sensor]{
var connection: Connection = _
var insertStatement: PreparedStatement = _
var updateStatement: PreparedStatement = _
/**
* 初始化操作
* @param parameters
*/
override def open(parameters: Configuration): Unit = {
Class.forName("com.mysql.cj.jdbc.Driver")
connection = DriverManager.getConnection("jdbc:mysql:///test?serverTimezone=UTC","root","123456")
insertStatement = connection.prepareStatement("insert into temperatures (sensor,temps) values (?,?)")
println(insertStatement)
updateStatement = connection.prepareCall("update temperatures set temps = ? where sensor = ?")
println(updateStatement)
}
override def invoke(value: Sensor, context: SinkFunction.Context[_]): Unit = {
updateStatement.setDouble(1,value.temperature)
updateStatement.setString(2,value.id)
updateStatement.execute()
if (updateStatement.getUpdateCount == 0){
insertStatement.setString(1,value.id)
insertStatement.setDouble(2,value.temperature)
insertStatement.execute()
}
}
override def close(): Unit = {
insertStatement.close()
updateStatement.close()
connection.close()
}
}
}
— 业精于勤荒于嬉，行成于思毁于随 —
