Flink-10 Flink Java 3分钟上手 Docker容器化部署 JobManager TaskManager Kafka Redis Dockerfile docker-compose 原创
代码仓库
会同步代码到 GitHub
https://github.com/turbo-duck/flink-demo
本节内容
在学习和测试阶段,需要快速启动一套Flink
供我们学习测试。使用常规的安装的方式太费劲了,利用Docker
可以快速启动一套完整
的系统,方便我们的进行测试!
这里放一些我平常常用的 docker-compose
的文件。供大家学习和测试!
这些都是我们之前要用到的内容:
- Flink 部署
- Kafka 部署
- Redis 部署
Flink 单节点
- 1个 job manager
- 1个 task manager
version: '2'
services:
jobmanager:
image: 10.10.52.8/flink/flink
expose:
- "6123"
ports:
- "8081:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager:
image: 10.10.52.8/flink/flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
启动方式
docker-compose up -d
查看日志
docker-compose logs -f
运行效果
Flink高可用
通过 zookeeper
做的高可用
- 2个 job manager
- 3个 task manager
- 1个 zookeeper
version: '2'
services:
zookeeper:
image: zookeeper:3.5
ports:
- "2181:2181"
jobmanager1:
image: 10.10.52.8/flink/flink
expose:
- "6123"
ports:
- "8081:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager1
- ZOOKEEPER_SERVERS=zookeeper:2181
- HIGH_AVAILABILITY=zookeeper
- HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
- HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/
jobmanager2:
image: 10.10.52.8/flink/flink
expose:
- "6123"
ports:
- "8082:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager2
- ZOOKEEPER_SERVERS=zookeeper:2181
- HIGH_AVAILABILITY=zookeeper
- HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
- HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/
taskmanager1:
image: 10.10.52.8/flink/flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager1
- jobmanager2
command: taskmanager
links:
- "jobmanager1:jobmanager1"
- "jobmanager2:jobmanager2"
- "zookeeper:zookeeper"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager1
- ZOOKEEPER_SERVERS=zookeeper:2181
- HIGH_AVAILABILITY=zookeeper
- HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
taskmanager2:
image: 10.10.52.8/flink/flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager1
- jobmanager2
command: taskmanager
links:
- "jobmanager1:jobmanager1"
- "jobmanager2:jobmanager2"
- "zookeeper:zookeeper"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager1
- ZOOKEEPER_SERVERS=zookeeper:2181
- HIGH_AVAILABILITY=zookeeper
- HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
taskmanager3:
image: 10.10.52.8/flink/flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager1
- jobmanager2
command: taskmanager
links:
- "jobmanager1:jobmanager1"
- "jobmanager2:jobmanager2"
- "zookeeper:zookeeper"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager1
- ZOOKEEPER_SERVERS=zookeeper:2181
- HIGH_AVAILABILITY=zookeeper
- HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
启动方式
docker-compose up -d
查看日志
docker-compose logs -f
Flink多节点
可以不做高可用,直接 1个 JOB 对应 3个 TASK 节点。
但是如果JOB节点挂掉了,TASK节点将无法工作!!!
- 1个 job manager
- 3个 task manager
version: '2'
services:
jobmanager:
image: 10.10.52.8/flink/flink
expose:
- "6123"
ports:
- "8081:8081"
command: jobmanager
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager1:
image: 10.10.52.8/flink/flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager2:
image: 10.10.52.8/flink/flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
taskmanager3:
image: 10.10.52.8/flink/flink
expose:
- "6121"
- "6122"
depends_on:
- jobmanager
command: taskmanager
links:
- "jobmanager:jobmanager"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
Kafka 部署
注意!KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092
要修改为自己的IP!
version: "3"
services:
zookeeper:
image: 'bitnami/zookeeper:latest'
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'bitnami/kafka:2.8.0'
ports:
- '9092:9092'
- '9999:9999'
environment:
- KAFKA_BROKER_ID=1
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
# 注意这里的IP 改成自己的!!!
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
- KAFKA_CFG_LOG_RETENTION_HOURS=6
- JMX_PORT=9999
depends_on:
- zookeeper
Redis 部署
version: "3"
services:
redis:
image: "redis:latest"
container_name: "redis_container"
ports:
- "6379:6379"
· 分享4款.NET开源、免费、实用的商城系统
· 全程不用写代码,我用AI程序员写了一个飞机大战
· MongoDB 8.0这个新功能碉堡了,比商业数据库还牛
· 白话解读 Dapr 1.15:你的「微服务管家」又秀新绝活了
· 上周热点回顾(2.24-3.2)