【Flink系列二十】Flink Example AsyncIOExample long running 应用程序的应用

1.【Flink系列二十】Flink Example AsyncIOExample long running 应用程序的应用2023-10-26

Flink AsyncIOExample

Flink 的源码内有这么一个AsyncIOExample程序，非常好用，可以用来无限跑流。
用英文说就是long running或者runs forever。

同时可以用来测试checkpoints，metricReporter。

如何找到这个程序

这个程序在源码内，但很不幸，官方可能忘记加到maven ant插件的构建过程中。

以Flink-1.13.0为例：

flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOExample.java

使用方法

private static void printUsage() {
        System.out.println(
                "To customize example, use: AsyncIOExample [--fsStatePath <path to fs state>] "
                        + "[--checkpointMode <exactly_once or at_least_once>] "
                        + "[--maxCount <max number of input from source, -1 for infinite input>] "
                        + "[--sleepFactor <interval to sleep for each stream element>] [--failRatio <possibility to throw exception>] "
                        + "[--waitMode <ordered or unordered>] [--waitOperatorParallelism <parallelism for async wait operator>] "
                        + "[--eventType <EventTime or IngestionTime>] [--shutdownWaitTS <milli sec to wait for thread pool>]"
                        + "[--timeout <Timeout for the asynchronous operations>]");
}

为什么下载的flink examples文件夹内没有AsyncIO.jar

flink-examples/flink-examples-streaming/pom.xml

仅仅因为下方的代码忘记写一行copy了。

<!--simplify the name of example JARs for build-target/examples -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-antrun-plugin</artifactId>
				<executions>
					<execution>
						<id>rename</id>
						<configuration>
							<target>
								<copy file="${project.basedir}/target/flink-examples-streaming_${scala.binary.version}-${project.version}-Iteration.jar" tofile="${project.basedir}/target/Iteration.jar" />
								<copy file="${project.basedir}/target/flink-examples-streaming_${scala.binary.version}-${project.version}-SessionWindowing.jar" tofile="${project.basedir}/target/SessionWindowing.jar" />
								<copy file="${project.basedir}/target/flink-examples-streaming_${scala.binary.version}-${project.version}-TopSpeedWindowing.jar" tofile="${project.basedir}/target/TopSpeedWindowing.jar" />
								<copy file="${project.basedir}/target/flink-examples-streaming_${scala.binary.version}-${project.version}-WindowJoin.jar" tofile="${project.basedir}/target/WindowJoin.jar" />
								<copy file="${project.basedir}/target/flink-examples-streaming_${scala.binary.version}-${project.version}-WordCount.jar" tofile="${project.basedir}/target/WordCount.jar" />
								<copy file="${project.basedir}/target/flink-examples-streaming_${scala.binary.version}-${project.version}-SocketWindowWordCount.jar" tofile="${project.basedir}/target/SocketWindowWordCount.jar" />
							</target>
						</configuration>
					</execution>
				</executions>
			</plugin>

如何运行

可以直接复制源码，进行编译，无需compile任何依赖（可能需要rocksdb的依赖）

编译完成以后复制到任何位置，例如本应该存在于下方的路径：

./bin/flink run-application examples/streaming/AsyncIO.jar \
   -D "state.checkpoints.num-retained=5"
   --class org.apache.flink.streaming.examples.async.AsyncIOExample \
    --fsStatePath viewfs://slankka_hdfs/user/slankka/flink_checkpoints \
   --checkpointMode exactly_once \
   --maxCount \
   -1