Apache Sedona 流数据处理入门
Apache Flink介绍
Apache Flink是由Apache软件基金会开发的开源流处理框架，其核心是用Java和Scala编写的分布式流数据流引擎。Flink以数据并行和流水线方式执行任意流数据程序，Flink的流水线运行时系统可以执行批处理和流处理程序。
Apache Sedona介绍
Apache Sedona（孵化中）是一个用于处理大规模空间数据的集群计算系统。Sedona用一套开箱即用的分布式空间数据集和空间SQL扩展了Apache Spark和Apache Flink，可以在机器间有效地加载、处理和分析大规模空间数据。
处理流程
java代码
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().build();
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings);
SedonaFlinkRegistrator.registerType(env);
SedonaFlinkRegistrator.registerFunc(tableEnv);
DataStream<String> socketTextStream = env.socketTextStream("localhost", 8888);
DataStream<Row> rowDataStream = socketTextStream.map(new MapFunction<String, Row>() {
private static final long serialVersionUID = -3351062125994879777L;
@Override
public Row map(String line) throws Exception {
//System.out.println(line);
//System.out.println("11");
String[] fields = line.split(",");
String pointWkt = fields[1] ;
int id = Integer.parseInt(fields[0]);
return Row.of(pointWkt, id);
}
}).returns(Types.ROW(Types.STRING, Types.INT));
Table pointTable = tableEnv.fromDataStream(rowDataStream);
tableEnv.createTemporaryView("myTable", pointTable);
Table geomTbl = tableEnv.sqlQuery("SELECT ST_GeomFromWKT(f0) as geom_polygon, f1 FROM myTable");
tableEnv.createTemporaryView("geoTable", geomTbl);
geomTbl = tableEnv.sqlQuery("SELECT f1, geom_polygon FROM geoTable where ST_Contains (ST_GeomFromWKT('MultiPolygon (((110.499997 20.010307, 110.499995 20.010759, 110.500473 20.01076, 110.500475 20.010308, 110.499997 20.010307)))'),geom_polygon)");
geomTbl.execute().print();
pom.xml配置
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cn.hwang</groupId>
<artifactId>geospark-dev</artifactId>
<version>1.0</version>
<properties>
<scala.version>2.12</scala.version>
<scala.compat.version>2.12</scala.compat.version>
<geospark.version>1.2.0</geospark.version>
<spark.compatible.verison>3.0</spark.compatible.verison>
<spark.version>3.1.2</spark.version>
<hadoop.version>3.2.0</hadoop.version>
<geotools.version>24.0</geotools.version>
<flink.version>1.14.3</flink.version>
<kafka.version>2.8.1</kafka.version>
<dependency.scope>compile</dependency.scope>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.sedona</groupId>
<artifactId>sedona-core-3.0_2.12</artifactId>
<version>1.2.0-incubating</version>
</dependency>
<dependency>
<groupId>org.apache.sedona</groupId>
<artifactId>sedona-viz-3.0_2.12</artifactId>
<version>1.2.0-incubating</version>
</dependency>
<dependency>
<groupId>org.apache.sedona</groupId>
<artifactId>sedona-sql-3.0_2.12</artifactId>
<version>1.2.0-incubating</version>
</dependency>
<dependency>
<groupId>org.apache.sedona</groupId>
<artifactId>sedona-flink_2.12</artifactId>
<version>1.2.0-incubating</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.12.13</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>${spark.version}</version>
<scope>${dependency.scope}</scope>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>${spark.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.5</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.geotools</groupId>-->
<!-- <artifactId>gt-shapefile</artifactId>-->
<!-- <version>22-RC</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.datasyslab</groupId>
<artifactId>geotools-wrapper</artifactId>
<version>1.1.0-25.2</version>
</dependency>
<dependency>
<groupId>org.locationtech.jts</groupId>
<artifactId>jts-core</artifactId>
<version>1.18.0</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.wololo</groupId>-->
<!-- <artifactId>jts2geojson</artifactId>-->
<!-- <version>0.16.1</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.wololo</groupId>
<artifactId>jts2geojson</artifactId>
<version>0.16.1</version>
<exclusions>
<exclusion>
<groupId>org.locationtech.jts</groupId>
<artifactId>jts-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-main</artifactId>
<version>24.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.geotools/gt-referencing -->
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-referencing</artifactId>
<version>24.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.geotools/gt-epsg-hsql -->
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-epsg-hsql</artifactId>
<version>24.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<!-- For Flink DataStream API-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.compat.version}</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<!-- Kafka -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
</dependency>
<!-- Flink Kafka connector-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_${scala.compat.version}</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<!-- For playing flink in IDE-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_${scala.compat.version}</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<!-- For Flink flink api, planner, udf/udt, csv-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge_${scala.compat.version}</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<!-- Starting Flink 14, Blink planner has been renamed to the official Flink planner-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_${scala.compat.version}</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-common</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-csv</artifactId>
<version>${flink.version}</version>
<scope>${dependency.scope}</scope>
</dependency>
<!-- For Flink Web Ui in test-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime-web_${scala.compat.version}</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>central</id>
<name>maven.aliyun.com</name>
<url>https://maven.aliyun.com/repository/public</url>
</repository>
<repository>
<id>maven2-repository.dev.java.net</id>
<name>Java.net repository</name>
<url>https://download.java.net/maven/2</url>
</repository>
<repository>
<id>osgeo</id>
<name>OSGeo Release Repository</name>
<url>https://repo.osgeo.org/repository/release/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
<repository>
<id>Central</id>
<name>Central Repository</name>
<url>https://repo1.maven.org/maven2/</url>
</repository>
</repositories>
</project>
下载netcat解压到本地，使用cmd运行nc程序，模拟流数据输入。下面就是测试的数据模板。
1,Point (110.500235 20.0105335)
2,Point (110.18832409 20.06088375)
3,Point (110.18784591 20.06088125)
4,Point (109.4116775 18.2997045)
5,Point (109.55539791 18.30762275)
6,Point (109.5483405 19.778523)
粘贴测试数据到nc程序下，开始运行代码。
Apache Sedona已经可以成功运行一些空间流数据，十分感谢JupiterChow同学对我的帮助，给我提供了示例代码和配置，还有耐心讲解。
附（ 花了好几天安装但是没用上的,flink1.14 部署到ubuntu）
安装JDK
sudo apt update
sudo apt install openjdk-11-jdk
下载解压Flink
wget https://dlcdn.apache.org/flink/flink-1.14.4/flink-1.14.4-bin-scala_2.11.tgz
tar -xzf flink-1.14.4-bin-scala_2.11.tgz
cd flink-1.14.4
启动集群
./bin/start-cluster.sh
测试自带的例子
./bin/flink run ./examples/batch/WordCount.jar
打开自带的UI界面
wget https://github.com/glink-incubator/glink/releases/download/release-1.0.0/glink-1.0.0-bin.tar.gz
tar -zxvf glink-1.0.0-bin.tar.gz
./flink-1.14.4/bin/flink run ./examples/batch/WordCount.jar
参考资料
https://zhuanlan.zhihu.com/p/447743903
https://www.cnblogs.com/liufei1983/p/15661322.html
https://blog.csdn.net/weixin_46684578/article/details/122803180
https://juejin.cn/post/7023210394894204936
https://cloud.tencent.com/developer/article/1626610
https://baike.baidu.com/item/Apache Flink/59924858
https://eternallybored.org/misc/netcat/
https://sedona.apache.org/
