Flink-10 Flink Java 3分钟上手 Docker容器化部署 JobManager TaskManager Kafka Redis Dockerfile docker-compose 原创

代码仓库

会同步代码到 GitHub
https://github.com/turbo-duck/flink-demo
在这里插入图片描述

本节内容

在学习和测试阶段,需要快速启动一套Flink供我们学习测试。使用常规的安装的方式太费劲了,利用Docker可以快速启动一套完整的系统,方便我们的进行测试!
这里放一些我平常常用的 docker-compose 的文件。供大家学习和测试!

这些都是我们之前要用到的内容:

  • Flink 部署
  • Kafka 部署
  • Redis 部署

Flink 单节点

  • 1个 job manager
  • 1个 task manager
version: '2'

services:
  jobmanager:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

启动方式

docker-compose up -d

查看日志

docker-compose logs -f

请添加图片描述

运行效果

请添加图片描述

Flink高可用

通过 zookeeper 做的高可用

  • 2个 job manager
  • 3个 task manager
  • 1个 zookeeper
version: '2'

services:
  zookeeper:
    image: zookeeper:3.5
    ports:
      - "2181:2181"

  jobmanager1:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
      - HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/

  jobmanager2:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8082:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager2
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
      - HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/

  taskmanager1:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager1
      - jobmanager2
    command: taskmanager
    links:
      - "jobmanager1:jobmanager1"
      - "jobmanager2:jobmanager2"
      - "zookeeper:zookeeper"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181

  taskmanager2:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager1
      - jobmanager2
    command: taskmanager
    links:
      - "jobmanager1:jobmanager1"
      - "jobmanager2:jobmanager2"
      - "zookeeper:zookeeper"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181

  taskmanager3:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager1
      - jobmanager2
    command: taskmanager
    links:
      - "jobmanager1:jobmanager1"
      - "jobmanager2:jobmanager2"
      - "zookeeper:zookeeper"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager1
      - ZOOKEEPER_SERVERS=zookeeper:2181
      - HIGH_AVAILABILITY=zookeeper
      - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181

启动方式

docker-compose up -d

查看日志

docker-compose logs -f

Flink多节点

可以不做高可用,直接 1个 JOB 对应 3个 TASK 节点。
但是如果JOB节点挂掉了,TASK节点将无法工作!!!

  • 1个 job manager
  • 3个 task manager
version: '2'

services:
  jobmanager:
    image: 10.10.52.8/flink/flink
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager1:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager2:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager3:
    image: 10.10.52.8/flink/flink
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

Kafka 部署

注意!KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092 要修改为自己的IP!

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:2.8.0'
    ports:
      - '9092:9092'
      - '9999:9999'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      # 注意这里的IP 改成自己的!!!
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_LOG_RETENTION_HOURS=6
      - JMX_PORT=9999
    depends_on:
      - zookeeper

请添加图片描述

Redis 部署

version: "3"

services:
  redis:
    image: "redis:latest"
    container_name: "redis_container"
    ports:
      - "6379:6379"

请添加图片描述

posted @   武子康  阅读(0)  评论(0编辑  收藏  举报  
相关博文:
阅读排行:
· 分享4款.NET开源、免费、实用的商城系统
· 全程不用写代码,我用AI程序员写了一个飞机大战
· MongoDB 8.0这个新功能碉堡了,比商业数据库还牛
· 白话解读 Dapr 1.15:你的「微服务管家」又秀新绝活了
· 上周热点回顾(2.24-3.2)
点击右上角即可分享
微信分享提示