kubebetes 搭建rocketmq集群

k8s 部署rocketmq集群

RocketMQ主要有四大组成部分：NameServer、Broker、Producer、Consumer。

Nameserver作用：

NameServer 可以说是 Broker 的注册中心，Broker 在启动的时候，会根据配置信息向所有的 NameServer 进行注册，NameServer 会和每次前来注册的 Broker 保持长连接，并每 30s 检查 Broker 是否还存活，对于宕机的 Broker，NameServer 会将其从列表中剔除。当生产者需要向 Broker 发送消息的时候，就会先从 NameServer 里面获取 Broker 的地址列表，然后负载均衡，选择一台消息服务器进行发送。

RocketMQ的部署方式有多种：

  • 2m-noslave： 多Master模式，无Slave。[双主模式]
  • 2m-2s-sync： 多Master多Slave模式，同步双写 [双主双从+同步模式]
  • 2m-2s-async：多Master多Slave模式，异步复制 [双主双从+异步模式]

RocketMQ 提供了三种方式发送消息：同步、异步和单向：

  • 同步发送: 指消息发送方发出数据后会在收到接收方发回响应之后才发下一个数据包。
  • 异步发送: 指发送方发出数据后，不等接收方发回响应，接着发送下个数据包, 异步方式也需要Broker返回确认信息。
  • 单向发送: 指只负责发送消息而不等待服务器回应且没有回调函数触发。

RocketMQ 三种消息发送模式的使用场景：

具体使用哪种模式，这主要是看应用场景。

  • 同步发送：主要运用在比较重要一点消息传递/通知等业务：
  • 异步发送：通常用于对发送消息响应时间要求更高/更快的场景：
  • 单向发送：适用于某些耗时非常短，但对可靠性要求并不高的场景，例如日志收集。只发送消息，不等待服务器响应，只发送请求不等待应答。此方式发送消息的过程耗时非常短，一般在微秒级别。

RocketMQ端口：

  • rocketmq 默认端口：9876（即nameserver端口）
  • 非vip通道端口：10911
  • vip通道端口：10909
  • 10909是VIP通道对应的端口，在JAVA中的消费者对象或者是生产者对象中关闭VIP通道即可无需开放10909端口

本案例部署的是RocketMQ的"双主双从+同步模式"，涉及6个pod：

  • broker-a 主1
  • broker-b 主2
  • broker-a-s 从1
  • broker-a-s 从2
  • NameServer（注册中心）
  • mq-console（可视化web界面）

部署说明

​ 本次部署主双从集群

​ rocketmq根据官方部署制作镜像参考：https://www.cnblogs.com/pgy674477230/p/15601744.html

准备集群broker配置文件

配置文件说明参考文章：https://www.cnblogs.com/pgy674477230/p/16287735.html

mq-broker-a.conf

brokerClusterName=k8s-mq-cluster
brokerName=broker-a
brokerId=1
deleteWhen=04
fileReservedTime=120
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
listenPort=10911

mq-broker-a-s.conf

brokerClusterName=k8s-mq-cluster
brokerName=broker-a
brokerId=2
deleteWhen=04
fileReservedTime=120
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
listenPort=10911

mq-broker-b.conf

brokerClusterName=k8s-mq-cluster
brokerName=broker-b
brokerId=1
deleteWhen=04
fileReservedTime=120
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
listenPort=11011

mq-broker-b-s.conf

brokerClusterName=k8s-mq-cluster
brokerName=broker-b
brokerId=2
deleteWhen=04
fileReservedTime=120
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
listenPort=11011

创建k8s cm配置管理

命令说明：kubectl create cm cmname --from-file=filename -n namespace

kubectl create cm mq-broker-a --from-file=mq-broker-a.conf
kubectl create cm mq-broker-a-s --from-file=mq-broker-a-s.conf
kubectl create cm mq-broker-b --from-file=mq-broker-b.conf
kubectl create cm mq-broker-b-s --from-file=mq-broker-b-s.conf

查看创建结果：

kubectl -n namespace get cm cmname

[root@k8s-master-1 ~]# kubectl -n test get cm mq-broker-a
NAME          DATA   AGE
mq-broker-a   1      20h

查看具体内容：

[root@k8s-master-1 ~]# kubectl -n test describe cm mq-broker-a   
Name:         mq-broker-a
Namespace:    test03
Labels:       <none>
Annotations:  <none>

Data
====
broker-a.conf:
----
brokerClusterName=k8s-mq-cluster
brokerName=broker-a
brokerId=1
deleteWhen=04
fileReservedTime=120
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
listenPort=10911
Events:  <none>

部署nameserver

部署nameserver节点1

部署StatefulSets或者是Deployments

cat nameserver-01.yml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: nameserver-01
  namespace: test
  labels:
    app: nameserver-01
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nameserver-01
  template:
    metadata:
      labels:
        app: nameserver-01
      annotations:
        kubectl.kubernetes.io/restartedAt: '2022-06-01T03:36:04.740Z'
    spec:
      containers:
        - name: nameserver-01
          image: testregister/public/rocketmq:4.9.3
          command:
            - sh
          args:
            - mqnamesrv
          ports:
            - name: ports
              containerPort: 9876
              protocol: TCP
          resources: {}
          imagePullPolicy: IfNotPresent

      restartPolicy: Always

创建service关联pod对外提供服务

cat nameserver-01-svc.yml

kind: Service
apiVersion: v1
metadata:
  name: nameserver-01
  namespace: test
spec:
  ports:
    - name: ports
      protocol: TCP
      port: 9876
      targetPort: 9876
  selector:
    app: nameserver-01
  clusterIP: 
  type: ClusterIP

部署nameserver节点2

部署StatefulSets或者是Deployments

cat nameserver-02.yml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: nameserver-02
  namespace: test
  labels:
    app: nameserver-02
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nameserver-02
  template:
    metadata:
      labels:
        app: nameserver-02
      annotations:
        kubectl.kubernetes.io/restartedAt: '2022-06-01T03:36:04.740Z'
    spec:
      containers:
        - name: nameserver-02
          image: testregister/public/rocketmq:4.9.3
          command:
            - sh
          args:
            - mqnamesrv
          ports:
            - name: ports
              containerPort: 9876
              protocol: TCP
          resources: {}
          imagePullPolicy: IfNotPresent

      restartPolicy: Always

创建service关联pod对外提供服务

cat nameserver-02-svc.yml

kind: Service
apiVersion: v1
metadata:
  name: nameserver-02
  namespace: test
spec:
  ports:
    - name: ports
      protocol: TCP
      port: 9876
      targetPort: 9876
  selector:
    app: nameserver-02
  clusterIP: 
  type: ClusterIP

部署mq-console 可视化界面

为了后续部署broker调试方便优先部署mq-console

cat mq-console.yml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: mq-console
  namespace: test
  labels:
    app: mq-console
    kubepi.org/name: mq-console
  annotations:
    deployment.kubernetes.io/revision: '4'
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mq-console
      kubepi.org/name: mq-console
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: mq-console
        kubepi.org/name: mq-console
    spec:
      containers:
        - name: container-0
          image: apacherocketmq/rocketmq-dashboard:1.0.0
          ports:
            - name: ports
              containerPort: 8080
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: >-
                -Drocketmq.namesrv.addr=nameserver-01:9876;nameserver-02:9876;nameserver-03:9876
                -Dcom.rocketmq.sendMessageWithVIPChannel=false
          resources: {}
          imagePullPolicy: IfNotPresent
      restartPolicy: Always

image-20220601142519803

部署broker节点

部署broker-a

部署StatefulSets或者是Deployments

cat mq-broker-a.yml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mq-broker-a
  namespace: test
  labels:
    app: rockermq-broker
    kubepi.org/name: mq-broker-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rockermq-broker
      kubepi.org/name: mq-broker-a
  template:
    metadata:
      labels:
        app: rockermq-broker
        kubepi.org/name: mq-broker-a
    spec:
      volumes:
        - name: mq-broker-a  # 挂载broker的cm配置
          configMap:
            name: mq-broker-a
            defaultMode: 493
      containers:
        - name: container-0
          image: testregister/public/rocketmq:4.9.3
          command:
            - sh
          args:
            - mqbroker
            - '-c'
            - /opt/broker-a.conf  #指定启动配置文件
          ports:
            - name: listenport
              containerPort: 10911
              protocol: TCP
            - name: fastlistenport
              containerPort: 10909
              protocol: TCP
            - name: halistenport
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR #指定nameserver的serivce名称和端口
              value: nameserver-01:9876;nameserver-02:9876;nameserver-03:9876 
            - name: JAVA_OPT_EXT # 默认主机内存>16G 使用8G内存，需指定，否则无法启动
              value: '-server -Xms256m -Xmx256m -Xmn256m'
          resources: {}
          volumeMounts:
            - name: data
              mountPath: /home/rocketmq/store
            - name: mq-broker-a
              mountPath: /opt
            - name: data
              mountPath: /home/rocketmq/logs
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: data
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 10Gi
        storageClassName: managed-nfs-storage
        volumeMode: Filesystem

部署broker-a-s

部署StatefulSets或者是Deployments

cat mq-broker-a-s.yml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mq-broker-a-s
  namespace: test
  labels:
    app: rockermq-broker
    kubepi.org/name: mq-broker-a-s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rockermq-broker
      kubepi.org/name: mq-broker-a-s
  template:
    metadata:
      labels:
        app: rockermq-broker
        kubepi.org/name: mq-broker-a-s
    spec:
      volumes:
        - name: mq-broker-a-s  # 挂载broker的cm配置
          configMap:
            name: mq-broker-a-s
            defaultMode: 493
      containers:
        - name: container-0
          image: testregister/public/rocketmq:4.9.3
          command:
            - sh
          args:
            - mqbroker
            - '-c'
            - /opt/broker-a-s.conf  #指定启动配置文件
          ports:
            - name: listenport
              containerPort: 10911
              protocol: TCP
            - name: fastlistenport
              containerPort: 10909
              protocol: TCP
            - name: halistenport
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR #指定nameserver的serivce名称和端口
              value: nameserver-01:9876;nameserver-02:9876;nameserver-03:9876 
            - name: JAVA_OPT_EXT # 默认主机内存>16G 使用8G内存，需指定，否则无法启动
              value: '-server -Xms256m -Xmx256m -Xmn256m'
          resources: {}
          volumeMounts:
            - name: data
              mountPath: /home/rocketmq/store
            - name: mq-broker-a-s
              mountPath: /opt
            - name: data
              mountPath: /home/rocketmq/logs
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: data
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 10Gi
        storageClassName: managed-nfs-storage
        volumeMode: Filesystem

部署broker-b

部署StatefulSets或者是Deployments

cat mq-broker-b.yml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mq-broker-b
  namespace: test
  labels:
    app: rockermq-broker
    kubepi.org/name: mq-broker-b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rockermq-broker
      kubepi.org/name: mq-broker-b
  template:
    metadata:
      labels:
        app: rockermq-broker
        kubepi.org/name: mq-broker-b
    spec:
      volumes:
        - name: mq-broker-b  # 挂载broker的cm配置
          configMap:
            name: mq-broker-b
            defaultMode: 493
      containers:
        - name: container-0
          image: testregister/public/rocketmq:4.9.3
          command:
            - sh
          args:
            - mqbroker
            - '-c'
            - /opt/broker-b.conf  #指定启动配置文件
          ports:
            - name: listenport
              containerPort: 10911
              protocol: TCP
            - name: fastlistenport
              containerPort: 10909
              protocol: TCP
            - name: halistenport
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR #指定nameserver的serivce名称和端口
              value: nameserver-01:9876;nameserver-02:9876;nameserver-03:9876 
            - name: JAVA_OPT_EXT # 默认主机内存>16G 使用8G内存，需指定，否则无法启动
              value: '-server -Xms256m -Xmx256m -Xmn256m'
          resources: {}
          volumeMounts:
            - name: data
              mountPath: /home/rocketmq/store
            - name: mq-broker-b
              mountPath: /opt
            - name: data
              mountPath: /home/rocketmq/logs
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: data
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 10Gi
        storageClassName: managed-nfs-storage
        volumeMode: Filesystem

部署broker-b

部署StatefulSets或者是Deployments

cat mq-broker-b-s.yml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mq-broker-b-s
  namespace: test
  labels:
    app: rockermq-broker
    kubepi.org/name: mq-broker-b-s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rockermq-broker
      kubepi.org/name: mq-broker-b-s
  template:
    metadata:
      labels:
        app: rockermq-broker
        kubepi.org/name: mq-broker-b-s
    spec:
      volumes:
        - name: mq-broker-b-s  # 挂载broker的cm配置
          configMap:
            name: mq-broker-b-s
            defaultMode: 493
      containers:
        - name: container-0
          image: testregister/public/rocketmq:4.9.3
          command:
            - sh
          args:
            - mqbroker
            - '-c'
            - /opt/broker-b-s.conf  #指定启动配置文件
          ports:
            - name: listenport
              containerPort: 10911
              protocol: TCP
            - name: fastlistenport
              containerPort: 10909
              protocol: TCP
            - name: halistenport
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR #指定nameserver的serivce名称和端口
              value: nameserver-01:9876;nameserver-02:9876;nameserver-03:9876 
            - name: JAVA_OPT_EXT # 默认主机内存>16G 使用8G内存，需指定，否则无法启动
              value: '-server -Xms256m -Xmx256m -Xmn256m'
          resources: {}
          volumeMounts:
            - name: data
              mountPath: /home/rocketmq/store
            - name: mq-broker-b-s
              mountPath: /opt
            - name: data
              mountPath: /home/rocketmq/logs
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: data
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 10Gi
        storageClassName: managed-nfs-storage
        volumeMode: Filesystem
posted @ 2022-03-11 16:51  蒲公英PGY  阅读(510)  评论(0)  编辑  收藏  举报