kubernetes statefulset kafka 部署后， 外部访问超时问题解决

k8s 内部的kafka要映射到外网，直接把 kafka 通过 expose 把pod 映射成服务，使用nodeport 连接，出现超时问题，

 

解决思路：

　　　　1.  查看zk中，kafka的注册信息，PLAINTEXT://kafka-0.kafka-hs.sy-platform-demo.svc.cluster.local.:9093"， 而应用代码中，必须使用这个域名+端口进行连接，由于k8s使用30000-32000，端口对外映射，所以9093端口必须修改

 

 

 

　　2.  kafka.yaml文件修改端口

apiVersion: v1
kind: Service
metadata:
  name: kafka-hs
  namespace: sy-platform-demo
  labels:
    app: kafka
spec:
  ports:
  - port: 31902
    name: server
  clusterIP: None
  selector:
    app: kafka


---
apiVersion: v1
kind: Service
metadata:
  namespace: sy-platform-demo
  name: kafka-cs
  labels:
    app: kafka
spec:
  ports:
  - port: 31902
    name: client
  selector:
    app: kafka

---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  namespace: sy-platform-demo
  name: kafka-pdb
spec:
  selector:
    matchLabels:
      app: kafka
  maxUnavailable: 1
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  namespace: sy-platform-demo
  name: kafka
spec:
  serviceName: kafka-hs
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
      type: RollingUpdate
  template:
    metadata:
      labels:
        app: kafka
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values: 
                    - kafka
              topologyKey: "kubernetes.io/hostname"
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
             - weight: 1
               podAffinityTerm:
                 labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values: 
                        - zk
                 topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 300
      containers:
      - name: k8skafka
        imagePullPolicy: Always
        image: 192.168.200.10/fengjian/kafka_2.11-0.10.1.1:20190326
        resources:
          requests:
            memory: "3Gi"
            cpu: 2
        ports:
        - containerPort: 31902
          name: server
        command:
        - sh
        - -c
        - "exec /data/kafka/bin/kafka-server-start.sh /data/kafka/config/server.properties --override broker.id=${HOSTNAME##*-} \
          --override listeners=PLAINTEXT://:31902 \
          --override zookeeper.connect=zk-cs.sy-platform-demo.svc.cluster.local:2181 \
          --override request.timeout.ms=60000 \
          --override session.timeout.ms=150000 \
          --override heartbeat.interval.ms=50000"
        env:
        - name: KAFKA_HEAP_OPTS
          value : "-Xmx1G -Xms1G"
        - name: KAFKA_OPTS
          value: "-Dlogging.level=INFO"
        volumeMounts:
        - name: datadir
          mountPath: /export
        readinessProbe:
          tcpSocket:
            port: 31902
          initialDelaySeconds: 30
          periodSeconds: 10
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.beta.kubernetes.io/storage-class: "ceph-rbd-provisioner"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 30Gi

 

 

　　3. 对pod打标签，然后映射成服务

kubectl label pod kafka-0 kafkaInst=0 -n sy-platform-demo
kubectl expose pod kafka-0 --port=31902 --target-port=31902 --name=kafka-0 --selector=kafkaInst=0 --type=NodePort -n sy-platform-demo

 

　　4. 修改kafka-0 的nodeport 端口号

kubectl edit  service kafka-0 -n sy-platform-demo

 

 　　5. 外部测试机修改host，写入对应关系

[root@node2 logs]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.20.171        master1    etcd1
192.168.20.172        master2 etcd2
192.168.20.173        master3    etcd3
192.168.20.174        node1
192.168.20.175        node2
192.168.20.176        node3
192.168.20.177        node4
192.168.20.178        node5
192.168.20.174        kafka-0.kafka-hs.senyint.svc.cluster.local.
192.168.20.174        zk-cs

 

　　6. 修改代码文件，写入域名

abcd.queue.internal.notify.kafka.producer.bootstrap.servers=kafka-0.kafka-hs.senyint.svc.cluster.local.:30010
abcd..queue.zkAddress=zk-cs:31097

 

　　7. 遗留问题

目前只映射了一个kafka pod， 如果全部映射，那么需要修改端口后，保持kafka-0 ---kafka-2 的端口号不同