Flink Deployment

Architecture

 

Standalone

https://nightlies.apache.org/flink/flink-docs-release-2.0/zh/docs/try-flink/local_installation

curl -LOJ https://mirrors.cloud.tencent.com/apache/flink/flink-2.0.0/flink-2.0.0-bin-scala_2.12.tgz
tar -zxf flink-2.0.0-bin-scala_2.12.tgz
sudo mv flink-2.0.0* /opt/
cd /opt/flink-2.0.0

vim conf/config.yaml
env.java.home: /opt/jdk21
vim conf/workers
vim conf/masters

bin/start-cluster.sh
./bin/flink run examples/streaming/WordCount.jar
tail log/flink-*-taskexecutor-*.out
./bin/stop-cluster.sh

Apache Flink Dashboard：http://localhost:8081

 

YARN

注意主机名：https://github.com/apache/flink/blob/release-2.0.0/flink-core/src/main/java/org/apache/flink/util/NetUtils.java#L220，检查 /etc/hosts 中不要加 . 结尾

/opt/hadoop-3.4.1/bin/hdfs namenode -format
/opt/hadoop-3.4.1/sbin/start-dfs.sh
/opt/hadoop-3.4.1/sbin/start-yarn.sh

HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` ./bin/yarn-session.sh --detached

nc -lp 12345 # ss -alp | grep nc
HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` ./bin/flink run -d examples/streaming/SocketWindowWordCount.jar --port 12345 --hostname 127.0.0.1
HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` \
./bin/flink run -d -t yarn-session -Dyarn.application.id=application_1745114892818_0001 examples/streaming/SocketWindowWordCount.jar --port 12345 --hostname 127.0.0.1

HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` ./bin/yarn-session.sh -id application_1745114892818_0001
HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` echo "stop" | ./bin/yarn-session.sh -id application_1745114892818_0001
yarn application -kill application_1745114892818_0001
yarn logs -applicationId application_1745114892818_0001

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java

HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` \
./bin/flink run -d -t yarn-application -c org.apache.flink.streaming.examples.socket.SocketWindowWordCount examples/streaming/SocketWindowWordCount.jar --port 12345

HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` ./bin/flink list -t yarn-session -Dyarn.application.id=application_1745114892818_0001
HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` ./bin/flink cancel -t yarn-session -Dyarn.application.id=application_1745114892818_0001 bf7a7db1909cb08fc050509642420755

yarn.provided.lib.dirs

hadoop fs -mkdir /flink-dist
hadoop fs -put lib/ /flink-dist
hadoop fs -put plugins/ /flink-dist
hadoop fs -mkdir /flink-apps
hadoop fs -put examples/streaming/SocketWindowWordCount.jar /flink-apps

HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` \
./bin/flink run -d -t yarn-application -Dyarn.provided.lib.dirs="hdfs://localhost:9000/flink-dist" hdfs://localhost:9000/flink-apps/SocketWindowWordCount.jar --port 12345

https://nightlies.apache.org/flink/flink-docs-stable/zh/docs/deployment/advanced/historyserver

hadoop fs -mkdir /flink-history
vim config/config.yaml
jobmanager.archive.fs.dir: hdfs://localhost:9000/flink-history
#historyserver.web.address: 0.0.0.0
#historyserver.web.port: 8082
historyserver.archive.fs.dir: hdfs://localhost:9000/flink-history
#historyserver.archive.fs.refresh-interval: 10000

bin/historyserver.sh stop && HADOOP_CLASSPATH=`/opt/hadoop-3.4.1/bin/hadoop classpath` bin/historyserver.sh start
cat log/flink-user-historyserver-1-hostname.log

Apache Flink Dashboard WebUI：http://hostname:8082，可以看到 Cancel Job 的历史

 

---

https://nightlies.apache.org/flink/flink-docs-stable/zh/docs/deployment/overview