K8S部署Apollo配置中心

K8S部署Apollo配置中心

参考文档: https://github.com/apolloconfig/apollo/tree/v1.8.0

	[K8S部署apollo配置中心](https://www.cnblogs.com/Fengyinyong/p/14903725.html)

	[apollo官网文档](https://www.apolloconfig.com/#/zh/README)

1、错误问题记录

在k8s里面部署时也遇到了同样的一些问题，在此记录下：

• 提示exec user process caused "no such file or directory"

  • 由于将Windows文件拷贝到CentOS里面导致的换行符不对，解决方案是在CentOS虚机内git clone 当前仓储

• 构建的docker镜像启动失败

  • 同上一个问题，从Windows拷贝过去的文件权限不正确导致的shell脚本执行失败，解决方案同上或者chmod +x

• 一个环境的admin server由于健康检查不过导致不断重启

  • 数据库帐号填错了导致无法登录数据库引发的问题

2、介绍

Apollo（阿波罗）是携程框架部门研发的分布式配置中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性，适用于微服务配置管理场景。

服务端基于Spring Boot和Spring Cloud开发，打包后可以直接运行，不需要额外安装Tomcat等应用容器。

Java客户端不依赖任何框架，能够运行于所有Java运行时环境，同时对Spring/Spring Boot环境也有较好的支持。

.Net客户端不依赖任何框架，能够运行于所有.Net运行时环境。

‍

说明：最近在用K8S部署微服务，而微服务的配置文件众多，需要一个配置中心来处理配置文件。于是采用apollo来作为配置中心。本实例介绍了如何采用K8S部署高可用的apollo集群。

‍

3、环境配置

mysql5.7.39

注意：mysql的版本一定要大于5.7，不然导入数据库sql文件会报错的

因为实验方便，给root远程登录权限（切忌，生产环境，禁用root，单独数据库配置单独账号）

数据库名称	IP	作用	远程账号
DevApolloConfigDB	192.168.102.20	DEV环境的配置	root
ProdApolloConfigDB	192.168.102.20	PRO环境的配置	root
ApolloPortalDB	192.168.102.20	WEB界面管理	root

‍

安装Apollo版本：1.8.0（注意：sql文件和jar版本一定要对应）

（目前官方给4套环境部署安装，我这边采用PRO和DEV环境）

已经构建成功的Kubernetes集群，

[root@bt nginx]# kubectl get node 
NAME       STATUS   ROLES                  AGE     VERSION
master01   Ready    control-plane,master   2d16h   v1.23.0
node01     Ready    <none>                 2d16h   v1.23.0
node02     Ready    <none>                 2d16h   v1.23.0

4、Apollo安装

‍

1、克隆Apollo代码

官方地址：https://github.com/apolloconfig/apollo

在Centos上克隆（因为字符原因，所以要在Linux克隆，千万不要win上下载上传，容易出问题）

git clone https://github.com/apolloconfig/apollo.git

cd apollo 

#查看版本，
git tag
#切换分支
git checkout v1.8.0
#创建新的分支，切换
git checkout -b heian

2、初始化Mysql数据库

[root@bt db]# pwd
/opt/apollo/scripts/apollo-on-kubernetes/db
[root@bt db]# ls
config-db-dev  config-db-prod  config-db-test-alpha  config-db-test-beta  portal-db
[root@bt db]# tree .
.
├── config-db-dev
│?? └── apolloconfigdb.sql
├── config-db-prod
│?? └── apolloconfigdb.sql
├── config-db-test-alpha
│?? └── apolloconfigdb.sql
├── config-db-test-beta
│?? └── apolloconfigdb.sql
└── portal-db
    └── apolloportaldb.sql

大胆执行，数据库名称官方定义过，不会冲突
执行dev环境
 cd config-db-dev/
[root@bt config-db-dev]# mysql -uroot -p < apolloconfigdb.sql 
Enter password: 
执行pro环境
 cd config-db-pro/
[root@bt config-db-dev]# mysql -uroot -p < apolloconfigdb.sql 
Enter password: 

执行portal环境
cd portal-db/
[root@bt config-db-dev]# mysql -uroot -p < apolloportaldb.sql

3、下载jar版本

‍

官方下载地址：https://github.com/apolloconfig/apollo/releases/tag/v1.8.0

cd /opt/apollo
wget https://github.com/apolloconfig/apollo/releases/download/v1.8.0/apollo-adminservice-1.8.0-github.zip
wget https://github.com/apolloconfig/apollo/releases/download/v1.8.0/apollo-configservice-1.8.0-github.zip
wget https://github.com/apolloconfig/apollo/releases/download/v1.8.0/apollo-portal-1.8.0-github.zip

进行解压，需要其中的jar

​    解压 apollo-portal-1.8.0-github.zip
​  获取 apollo-portal-1.8.0.jar, 重命名为 apollo-portal.jar, 放到 scripts/apollo-on-kubernetes/apollo-portal-server

​    解压 apollo-adminservice-1.8.0-github.zip
​  获取 apollo-adminservice-1.8.0.jar, 重命名为 apollo-adminservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-admin-server

​    解压 apollo-configservice-1.8.0-github.zip
​  获取 apollo-configservice-1.8.0.jar, 重命名为 apollo-configservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-config-se

4、构建Docker镜像

1、采用K8S部署apollo时，需要用到多个镜像。这些镜像，需要自己构建

2、如果麻烦，可以去官方镜像 https://hub.docker.com/u/apolloconfig

docker pull apolloconfig/apollo-configservice:1.8.0
docker pull apolloconfig/apollo-adminservice:1.8.0
docker pull apolloconfig/apollo-portal:1.8.0
还需要一个alpine-bash-3.8 做初始化镜像操作
docker pull zgadocker/alpine-bash-3.8-image:latest
测试这个不行，就按照下方自己打包

alpine-bash-3.8-image

[root@bt apollo-on-kubernetes]# cd alpine-bash-3.8-image
[root@bt alpine-bash-3.8-image]# docker build -t harbor.ownit.top/ownit/alpine-bash:3.8 .
[root@bt alpine-bash-3.8-image]# docker push harbor.ownit.top/ownit/alpine-bash:3.8

apollo-config-server

[root@bt apollo-on-kubernetes]# cd apollo-config-server
[root@bt apollo-config-server]# docker build -t harbor.ownit.top/ownit/apollo-configservice:1.8.0 .
[root@bt apollo-config-server]# docker push harbor.ownit.top/ownit/apollo-configservice:1.8.0

apollo-admin-server

[root@bt apollo-on-kubernetes]# cd apollo-admin-server
[root@bt apollo-admin-server]# docker build -t harbor.ownit.top/ownit/apollo-adminservice:1.8.0 .
[root@bt apollo-admin-server]# docker  push harbor.ownit.top/ownit/apollo-adminservice:1.8.0

apollo-portal-server

[root@bt apollo-on-kubernetes]# cd apollo-portal-server
[root@bt apollo-portal-server]# docker build -t harbor.ownit.top/ownit/apollo-portal:1.8.0 .
[root@bt apollo-portal-server]# docker push harbor.ownit.top/ownit/apollo-portal:1.8.0

验证

5、配置Yaml文件

环境	文件名称	执行顺序
dev	service-apollo-config-server-dev.yaml	1
dev	service-apollo-admin-server-dev.yaml	2
pro	service-apollo-config-server-prod.yaml	3
pro	service-apollo-admin-server-prod.yaml	4
portal	service-apollo-portal-server.yaml	5

1、service-apollo-config-server-dev.yaml

---
# configmap for apollo-config-server-dev
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-config-server-dev
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/DevApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-meta-server-dev
  labels:
    app: service-apollo-meta-server-dev
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: pod-apollo-config-server-dev
  type: ClusterIP
  clusterIP: None
  sessionAffinity: ClientIP

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-config-server-dev
  labels:
    app: service-apollo-config-server-dev
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30002
  selector:
    app: pod-apollo-config-server-dev 
  type: NodePort
  sessionAffinity: ClientIP

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  namespace: sre
  name: statefulset-apollo-config-server-dev
  labels:
    app: statefulset-apollo-config-server-dev
spec:
  serviceName: service-apollo-meta-server-dev
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-config-server-dev
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-config-server-dev
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-config-server-dev
              topologyKey: kubernetes.io/hostname

      volumes:
        - name: volume-configmap-apollo-config-server-dev
          configMap:
            name: configmap-apollo-config-server-dev
            items:
              - key: application-github.properties
                path: application-github.properties
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-configservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-config-server-dev
          ports:
            - protocol: TCP
              containerPort: 8080
        
          volumeMounts:
            - name: volume-configmap-apollo-config-server-dev
              mountPath: /apollo-config-server/config/application-github.properties
              subPath: application-github.properties
        
          env:
            - name: APOLLO_CONFIG_SERVICE_NAME
              value: "service-apollo-config-server-dev.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds:  120
            periodSeconds: 10
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

2、service-apollo-admin-server-dev.yaml

---
# configmap for apollo-admin-server-dev
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-admin-server-dev
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/DevApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-admin-server-dev
  labels:
    app: service-apollo-admin-server-dev
spec:
  ports:
    - protocol: TCP
      port: 8090
      targetPort: 8090
  selector:
    app: pod-apollo-admin-server-dev
  type: ClusterIP
  sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: sre
  name: deployment-apollo-admin-server-dev
  labels:
    app: deployment-apollo-admin-server-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-admin-server-dev
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-admin-server-dev
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-admin-server-dev
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-admin-server-dev
          configMap:
            name: configmap-apollo-admin-server-dev
            items:
              - key: application-github.properties
                path: application-github.properties
    
      initContainers:
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-config-server-dev
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-dev.sre:8080"]
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-adminservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-admin-server-dev
          ports:
            - protocol: TCP
              containerPort: 8090
        
          volumeMounts:
            - name: volume-configmap-apollo-admin-server-dev
              mountPath: /apollo-admin-server/config/application-github.properties
              subPath: application-github.properties
        
          env:
            - name: APOLLO_ADMIN_SERVICE_NAME
              value: "service-apollo-admin-server-dev.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 120
            periodSeconds: 10
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

3、service-apollo-config-server-prod.yaml

---
# configmap for apollo-config-server-prod
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-config-server-prod
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/ProdApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-prod-0.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-1.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-2.service-apollo-meta-server-prod:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-meta-server-prod
  labels:
    app: service-apollo-meta-server-prod
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: pod-apollo-config-server-prod
  type: ClusterIP
  clusterIP: None
  sessionAffinity: ClientIP

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-config-server-prod
  labels:
    app: service-apollo-config-server-prod
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30005
  selector:
    app: pod-apollo-config-server-prod
  type: NodePort
  sessionAffinity: ClientIP

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  namespace: sre
  name: statefulset-apollo-config-server-prod
  labels:
    app: statefulset-apollo-config-server-prod
spec:
  serviceName: service-apollo-meta-server-prod
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-config-server-prod
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-config-server-prod
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-config-server-prod
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-config-server-prod
          configMap:
            name: configmap-apollo-config-server-prod
            items:
              - key: application-github.properties
                path: application-github.properties
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-configservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: Always
          name: container-apollo-config-server-prod
          ports:
            - protocol: TCP
              containerPort: 8080

          volumeMounts:
            - name: volume-configmap-apollo-config-server-prod
              mountPath: /apollo-config-server/config/application-github.properties
              subPath: application-github.properties
          env:
            - name: APOLLO_CONFIG_SERVICE_NAME
              value: "service-apollo-config-server-prod.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 120
            periodSeconds: 10
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

3、service-apollo-admin-server-prod.yaml

---
# configmap for apollo-admin-server-prod
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-admin-server-prod
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/ProdApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-prod-0.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-1.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-2.service-apollo-meta-server-prod:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-admin-server-prod
  labels:
    app: service-apollo-admin-server-prod
spec:
  ports:
    - protocol: TCP
      port: 8090
      targetPort: 8090
  selector:
    app: pod-apollo-admin-server-prod  
  type: ClusterIP
  sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: sre
  name: deployment-apollo-admin-server-prod
  labels:
    app: deployment-apollo-admin-server-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-admin-server-prod
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-admin-server-prod
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-admin-server-prod
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-admin-server-prod
          configMap:
            name: configmap-apollo-admin-server-prod
            items:
              - key: application-github.properties
                path: application-github.properties
    
      initContainers:
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-config-server-prod
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 50 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-prod.sre:8080"]
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-adminservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-admin-server-prod
          ports:
            - protocol: TCP
              containerPort: 8090
        
          volumeMounts:
            - name: volume-configmap-apollo-admin-server-prod
              mountPath: /apollo-admin-server/config/application-github.properties
              subPath: application-github.properties
        
          env:
            - name: APOLLO_ADMIN_SERVICE_NAME
              value: "service-apollo-admin-server-prod.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 120
            periodSeconds: 10

      dnsPolicy: ClusterFirst
      restartPolicy: Always

5、service-apollo-portal-server.yaml

---
# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-mysql-for-portal-server
  labels:
    app: service-mysql-for-portal-server
spec:
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
  type: ClusterIP
  sessionAffinity: None
---
kind: Endpoints
apiVersion: v1
metadata:
  namespace: sre
  name: service-mysql-for-portal-server
subsets:
  - addresses:
      # 更改为你的 mysql addresses, 例如 1.1.1.1
      - ip: 192.168.102.20
    ports:
      - protocol: TCP
        port: 3306

---
# configmap for apollo-portal-server
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-portal-server
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/ApolloPortalDB?characterEncoding=utf8
    # mysql username
    spring.datasource.username = root
    # mysql password
    spring.datasource.password = 123456
  apollo-env.properties: |
    dev.meta=http://service-apollo-config-server-dev.sre:8080
    fat.meta=http://service-apollo-config-server-test-alpha.sre:8080
    uat.meta=http://service-apollo-config-server-test-beta.sre:8080
    pro.meta=http://service-apollo-config-server-prod.sre:8080

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-portal-server
  labels:
    app: service-apollo-portal-server
spec:
  ports:
    - protocol: TCP
      port: 8070
      targetPort: 8070
      nodePort: 30001
  selector:
    app: pod-apollo-portal-server
  type: NodePort
  # portal session 保持
  sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: sre
  name: deployment-apollo-portal-server
  labels:
    app: deployment-apollo-portal-server
spec:
  # 3 个实例
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-portal-server
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-portal-server
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-portal-server
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-portal-server
          configMap:
            name: configmap-apollo-portal-server
            items:
              - key: application-github.properties
                path: application-github.properties
              - key: apollo-env.properties
                path: apollo-env.properties
    
      initContainers:
        # 确保 admin-service 正常提供服务
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-admin-server-dev
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-dev.sre:8090"]
        # - image: harbor.ownit.top/ownit/alpine-bash:3.8
        #   name: check-service-apollo-admin-server-alpha
        #   command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-test-alpha.sre:8090"]
        # - image: harbor.ownit.top/ownit/alpine-bash:3.8
        #   name: check-service-apollo-admin-server-beta
        #   command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-test-beta.sre:8090"]
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-admin-server-prod
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-prod.sre:8090"]  
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-portal:1.8.0    # 更改为你的 docker registry 下的 image
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-portal-server
          ports:
            - protocol: TCP
              containerPort: 8070
        
          volumeMounts:
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal-server/config/application-github.properties
              subPath: application-github.properties
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal-server/config/apollo-env.properties
              subPath: apollo-env.properties
        
          env:
            - name: APOLLO_PORTAL_SERVICE_NAME
              value: "service-apollo-portal-server.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8070
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8070
            # 120s 内, server 未启动则重启 container
            initialDelaySeconds: 120
            periodSeconds: 15
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

6、执行Yaml文件

# create namespace
kubectl create namespace sre

dev环境
kubectl apply -f apollo-env-dev/service-apollo-config-server-dev.yaml --record 
kubectl apply -f apollo-env-dev/service-apollo-admin-server-dev.yaml --record

pro环境
kubectl apply -f apollo-env-prod/service-apollo-config-server-prod.yaml --record
kubectl apply -f apollo-env-prod/service-apollo-admin-server-prod.yaml --record

7、配置ingress域名

apollo-ingress.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: sre
  name: service-apollo-portal-server
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: "example-issuer"
  labels:
    app: service-apollo-portal-server
spec:
  rules:
  - host: apollo.ownit.top
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: service-apollo-portal-server
            port:
              number: 8070

8、配置nginx代理

‍

nginx配置

apollo.ownit.top.conf

server {
        listen       80;
        server_name  apollo.ownit.top;
        rewrite ^/(.*)$ https://$host/$1 permanent;
        # IP白名单
        include /opt/nginx/whitelist/corporation.conf;
}
server {
        listen       443 ssl;
        server_name  apollo.ownit.top;
        # IP白名单
        include /opt/nginx/whitelist/corporation.conf;

        #ssl                   on;
        ssl_certificate      /opt/nginx/ssl/ownit.top.crt;
        ssl_certificate_key  /opt/nginx/ssl/ownit.top.key;
        include ssl.conf;

        location / {
            proxy_pass  http://kubernetes-cluster; #转发到k8s的ingress 80端口
            include https_proxy.conf;
        }
	access_log  /www/wwwlogs/apollo.ownit.top.log;
    error_log  /www/wwwlogs/apollo.ownit.top.error.log;

}

[root@bt nginx]# nginx  -t 
nginx: the configuration file /www/server/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /www/server/nginx/conf/nginx.conf test is successful
[root@bt nginx]# nginx  -s reload

kubernetes-cluster.conf 转发

[root@bt nginx]# cat kubernetes-cluster.conf 
upstream kubernetes-cluster {
  server 192.168.102.40 weight=5;
  keepalive 16;
}

9、配置powerdns域名解析

10、测试验证

https://apollo.ownit.top/

默认账号/密码：apollo / admin

如果没有ingress，则可使用nodeport访问

k8s节点ip+30001 端口即可访问

http://k8s节点ip:30001/

5、Apollo配置使用

1、添加超级管理员用户

1、管理员用户选择 管理员工具->系统参数

2、填入系统内置Key：superAdmin 并点击查询，在value部分添加要加入人的用户id(LDAP用户ID为准，可通过wiki后台用户列表查询)后保存即可

2、新增部门：organizations

[{"orgId":"kaifa","orgName":"开发部门"},{"orgId":"yunwei","orgName":"运维部门"},{"orgId":"ceshi","orgName":"测试部门"}]

3、新增环境：apollo.portal.envs

apollo.portal.envs

6、namespace管理

私有namespace

1、添加namespace

Namespace作为配置的分类，可当成一个配置文件。
以添加rocketmq配置为例，添加"spring-rocketmq”Namespace配置rocketmq相关信息。

1、添加项目私有Namespace:spring-rocketmq
进入项目首页，点击左下脚的”添加Namespace”，共包括两项:关联公共Namespace和创建Namespace ,这里选择"创建Namespace"

2、填写详细的信息
有两种选择，一种是public（所有项目的）；另一种是private（当前项目的）；如下图所示

3、提交过后会自动跳转到下面的页面（操控权限）

4、不操作返回首页

2、为namespace操作配置

可以按之前的方法进行操作，也可以通过下面的内容进行批量操作
如：

rocketmq.name-server = 127.0.0.0:9876
rocketmq.producer.group = PID_ACCOUNT

发布

‍

3、获取namespace里面的数据

修改代码

	public class GetConfigTest {
	    public static void main(String[] args) {
	        while (true){
	            try {
	                Thread.sleep(1000);
	            } catch (InterruptedException e) {
	                e.printStackTrace();
	            }

	//            Config appConfig = ConfigService.getAppConfig();
	            //读取指定的namespace下的配置信息
	            Config config = ConfigService.getConfig("spring-rocketmq");
	            //获取配置信息,第一个参数:配置的key，第二个参数:默认值
	            String value = config.getProperty("rocketmq.producer.group", null);
	            System.out.printf("现在：%s, sms.enable: %s%n", new Date().toString(),value);
	        }


	    }
	}

运行结果

公共配置

1、添加公共的namespace

在项目开发中，有一些配置可能是通用的，我们可以通过把这些通用的配置放到公共的Namespace中，这样其他项目要使用时可以直接添加需要的Namespace

1. 新建common-template项目

2、新建namespace

3、添加配置信息

2、前往其他的项目去关联第一步的namespace

‍

3、修改公共配置

修改server.servlet.context-path为：/account-service

4、使用代码读取配置信息

注意：如果项目不一样的话，就需要去将之前在运行里面的环境进行修改，如下

修改代码：

public class GetConfigTest {
    public static void main(String[] args) {
        while (true){
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }

//            Config appConfig = ConfigService.getAppConfig();
            //读取指定的namespace下的配置信息
            Config config = ConfigService.getConfig("TEST1.spring-boot-http");
            //获取配置信息,第一个参数:配置的key，第二个参数:默认值
            String value = config.getProperty("server.servlet.context-path", null);
            System.out.printf("现在：%s, sms.enable: %s%n", new Date().toString(),value);
        }


    }
}