Flink-10 Flink Java 3分钟上手 Docker容器化部署 JobManager TaskManager Kafka Redis Dockerfile docker-compose 原创

代码仓库

会同步代码到 GitHub
https://github.com/turbo-duck/flink-demo

本节内容

在学习和测试阶段，需要快速启动一套Flink供我们学习测试。使用常规的安装的方式太费劲了，利用Docker可以快速启动一套完整的系统，方便我们的进行测试！
这里放一些我平常常用的 docker-compose 的文件。供大家学习和测试！

这些都是我们之前要用到的内容：

• Flink 部署
• Kafka 部署
• Redis 部署

Flink 单节点

• 1个 job manager
• 1个 task manager

version: '2'

services:
  jobmanager:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

启动方式

docker-compose up -d

查看日志

docker-compose logs -f

运行效果

Flink高可用

通过 zookeeper 做的高可用

• 2个 job manager
• 3个 task manager
• 1个 zookeeper

version: '2'

services:
  zookeeper:
    image: zookeeper:3.5
    ports:
      - "2181:2181"

  jobmanager1:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
      - HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/

  jobmanager2:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8082:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager2
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
      - HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/

  taskmanager1:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager1
      - jobmanager2
    command: taskmanager
    links:
      - "jobmanager1:jobmanager1"
      - "jobmanager2:jobmanager2"
      - "zookeeper:zookeeper"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181

  taskmanager2:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager1
      - jobmanager2
    command: taskmanager
    links:
      - "jobmanager1:jobmanager1"
      - "jobmanager2:jobmanager2"
      - "zookeeper:zookeeper"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181

  taskmanager3:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager1
      - jobmanager2
    command: taskmanager
    links:
      - "jobmanager1:jobmanager1"
      - "jobmanager2:jobmanager2"
      - "zookeeper:zookeeper"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181

启动方式

docker-compose up -d

查看日志

docker-compose logs -f

Flink多节点

可以不做高可用，直接 1个 JOB 对应 3个 TASK 节点。
但是如果JOB节点挂掉了，TASK节点将无法工作！！！

• 1个 job manager
• 3个 task manager

version: '2'

services:
  jobmanager:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager1:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager2:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager3:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

Kafka 部署

注意！KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092 要修改为自己的IP！

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:2.8.0'
    ports:
      - '9092:9092'
      - '9999:9999'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      # 注意这里的IP 改成自己的！！！
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_LOG_RETENTION_HOURS=6
      - JMX_PORT=9999
    depends_on:
      - zookeeper

Redis 部署

version: "3"

services:
  redis:
    image: "redis:latest"
    container_name: "redis_container"
    ports:
      - "6379:6379"