Apollo on K8S
Apollo(阿波罗)是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景。
本文将介绍如何在K8S平台上部署Apollo。
Apollo介绍
Apollo(阿波罗)是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景。
服务端基于 Spring Boot 和 Spring Cloud 开发,打包后可以直接运行,不需要额外安装Tomcat等应用容器。且 Java 客户端不依赖任何框架,能够运行于所有 Java 运行时环境,同时对 Spring/Spring Boot 环境也有较好的支持。
依赖的组件有:存储-Mysql,服务注册中心-Eureka。
官网
https://github.com/ctripcorp/apollo
官方wiki
https://github.com/ctripcorp/apollo/wiki
https://ctripcorp.github.io/apollo/
分布式配置中心选型
微服务架构中配置中心的选择
分布式配置中心选型
Apollo VS Nacos
部署与使用
Apollo 提供了四套环境配置:
- DEV:
Development environment
开发环境
,用于开发人员调试使用,版本变化较大。 - FAT:
Feature Acceptance Test environment
功能验收测试环境
,用于软件测试人员测试使用。 - UAT:
User Acceptance Test environment
用户验收测试环境,用于生产环境下的功能验证,可作为预发布环境
。 - PRO:
Production environment
生产环境
,即正式线上环境。
每个环境都需要安装相应组件,这里为了方便演示,只列出 DEV 与 PRO 环境的组件部署。
可以提前看下部署架构图便于更好的理解下述步骤。
准备
- 下载apollo组件包
从 https://github.com/ctripcorp/apollo/releases 下载官方打好的组件包:
#下载 Apollo 源码
$ wget https://github.com/ctripcorp/apollo/archive/v1.7.1.zip
#下载 Apollo Admin Server
$ wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-adminservice-1.7.1-github.zip
#下载 Apollo Config Service
$ wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-configservice-1.7.1-github.zip
#下载 Apollo Portal
$ wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-portal-1.7.1-github.zip
将源码包中部署文件复制到自定义目录:
#解压源码包
$ unzip v1.7.1.zip
#创建用于部署的自定义文件夹
$ mkdir ./deploy
#将部署文件和脚本复制到自定义 deploy 目录
$ cp -R ./apollo-1.7.1/scripts/apollo-on-kubernetes/* ./deploy
解压组件包并放在 deploy 目录下对应的文件夹下:
#解压各个组件压缩包
$ unzip -o apollo-configservice-1.7.1-github.zip -d ./deploy/apollo-config-server/
$ unzip -o apollo-adminservice-1.7.1-github.zip -d ./deploy/apollo-admin-server/
$ unzip -o apollo-portal-1.7.1-github.zip -d ./deploy/apollo-portal-server/
#应用 Jar 重命名
$ mv ./deploy/apollo-config-server/apollo-configservice-1.7.1.jar ./deploy/apollo-config-server/apollo-configservice.jar
$ mv ./deploy/apollo-admin-server/apollo-adminservice-1.7.1.jar ./deploy/apollo-admin-server/apollo-adminservice.jar
$ mv ./deploy/apollo-portal-server/apollo-portal-1.7.1.jar ./deploy/apollo-portal-server/apollo-portal.jar
- 构建并推送组件镜像
以腾讯云镜像仓库为例:
(请将{{YOUR_NAMESPACE}}替换成你的镜像仓库命名空间名称,腾讯云镜像仓库操作文档。)
#alpine 镜像
docker build -t ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/alpine-bash:3.8 ./deploy/alpine-bash-3.8-image
docker push ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/alpine-bash:3.8
#config-service 镜像
docker build -t ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/apollo-config-server:v1.7.1 ./deploy/apollo-config-server
docker push ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/apollo-config-server:v1.7.1
#admin-server 镜像
docker build -t ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/apollo-admin-server:v1.7.1 ./deploy/apollo-admin-server
docker push ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/apollo-admin-server:v1.7.1
#portal 镜像
docker build -t ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/apollo-portal-server:v1.7.1 ./deploy/apollo-portal-server
docker push ccr.ccs.tencentyun.com/{{YOUR_NAMESPACE}}/apollo-portal-server:v1.7.1
- 数据库准备
这里建议使用腾讯云Mysql。
实际情况下DEV环境和PRO环境建议使用两个不同的数据库实例,特别是PRO环境的数据库应该独立。
为每个环境的config-server和portal-server各创建一个db:
create database apollo_config_dev;
create database apollo_config_fat;
create database apollo_config_uat;
create database apollo_config_pro;
create database apollo_portal;
将config-db-dev,config-db-prod和portal-db下面的sql导入到上面创建的数据库中:
# tree ./deploy/db/
.
├── config-db-dev
│ └── apolloconfigdb.sql
├── config-db-prod
│ └── apolloconfigdb.sql
├── config-db-test-alpha #同fat
│ └── apolloconfigdb.sql
├── config-db-test-beta #同uat
│ └── apolloconfigdb.sql
└── portal-db
└── apolloportaldb.sql
然后准备好3份数据库配置:
# dev环境config数据库
datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_config_dev?characterEncoding=utf8
datasource.username = {{USERNAME}}
datasource.password = {{PASSWORD}}
# pro环境config数据库
datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_config_pro?characterEncoding=utf8
datasource.username = {{USERNAME}}
datasource.password = {{PASSWORD}}
# portal数据库
datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_portal?characterEncoding=utf8
datasource.username = {{USERNAME}}
datasource.password = {{PASSWORD}}
K8S资源创建
以下使用的镜像是本人已经打好的,你可以构建自己的镜像使用。
注意修改mysql实际配置以及需要的副本数配置。
- 创建namespace
kubectl create ns apollo
- 部署DEV环境apollo组件
- apollo-config-server-dev 部署文件
apollo-config-server-dev.yaml:
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-apollo-config-server-dev
namespace: apollo
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_config_dev?characterEncoding=utf8
spring.datasource.username = {{USERNAME}}
spring.datasource.password = {{PASSWORD}}
eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/
---
kind: Service
apiVersion: v1
metadata:
name: service-apollo-meta-server-dev
namespace: apollo
labels:
app: service-apollo-meta-server-dev
spec:
type: ClusterIP
clusterIP: None
sessionAffinity: ClientIP
ports:
- port: 8080
targetPort: 8080
selector:
app: pod-apollo-config-server-dev
---
kind: Service
apiVersion: v1
metadata:
name: service-apollo-config-server-dev
namespace: apollo
labels:
app: service-apollo-config-server-dev
spec:
type: NodePort
sessionAffinity: ClientIP
ports:
- port: 8080
targetPort: 8080
nodePort: 30002 #NodePort方式暴露配置服务端地址,用于Kubernetes集群外部访问、测试
selector:
app: pod-apollo-config-server-dev
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: statefulset-apollo-config-server-dev
namespace: apollo
labels:
app: statefulset-apollo-config-server-dev
spec:
serviceName: service-apollo-meta-server-dev
replicas: 1
selector:
matchLabels:
app: pod-apollo-config-server-dev
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-config-server-dev
spec:
containers:
- image: ccr.ccs.tencentyun.com/awesome/apollo-config-server:v1.7.1
name: container-apollo-config-server-dev
ports:
- containerPort: 8080
env:
- name: APOLLO_CONFIG_SERVICE_NAME
value: "service-apollo-config-server-dev.apollo" #修改namespace名称
resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 500m
memory: 512Mi
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 120
periodSeconds: 10
volumeMounts:
- name: volume-configmap-apollo-config-server-dev
mountPath: /apollo-config-server/config/application-github.properties
subPath: application-github.properties
volumes:
- name: volume-configmap-apollo-config-server-dev
configMap:
name: configmap-apollo-config-server-dev
items:
- key: application-github.properties
path: application-github.properties
- apollo-admin-server-dev 部署文件
apollo-admin-server-dev.yaml:
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-apollo-admin-server-dev
namespace: apollo
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_config_dev?characterEncoding=utf8
spring.datasource.username = {{USERNAME}}
spring.datasource.password = {{PASSWORD}}
eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/
---
kind: Service
apiVersion: v1
metadata:
name: service-apollo-admin-server-dev
namespace: apollo
labels:
app: service-apollo-admin-server-dev
spec:
type: ClusterIP
sessionAffinity: ClientIP
ports:
- port: 8090
targetPort: 8090
selector:
app: pod-apollo-admin-server-dev
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: deployment-apollo-admin-server-dev
namespace: apollo
labels:
app: deployment-apollo-admin-server-dev
spec:
replicas: 1
selector:
matchLabels:
app: pod-apollo-admin-server-dev
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: pod-apollo-admin-server-dev
spec:
initContainers:
- image: ccr.ccs.tencentyun.com/awesome/alpine-bash:3.8
name: check-service-apollo-config-server-dev
command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-dev.apollo:8080"]
containers:
- image: ccr.ccs.tencentyun.com/awesome/apollo-admin-server:v1.7.1
name: container-apollo-admin-server-dev
ports:
- containerPort: 8090
env:
- name: APOLLO_ADMIN_SERVICE_NAME
value: "service-apollo-admin-server-dev.apollo" # 注意修改 Namespace
resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 500m
memory: 512Mi
readinessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 120
periodSeconds: 10
volumeMounts:
- name: volume-configmap-apollo-admin-server-dev
mountPath: /apollo-admin-server/config/application-github.properties
subPath: application-github.properties
volumes:
- name: volume-configmap-apollo-admin-server-dev
configMap:
name: configmap-apollo-admin-server-dev
items:
- key: application-github.properties
path: application-github.properties
- k8s部署dev环境的config和admin组件
$ kubectl apply -f apollo-config-server-dev.yaml -n apollo
$ kubectl apply -f appllo-admin-server-dev.yaml -n apollo
或者在k8s控制台上通过yaml手动创建对应的资源。
-
部署PRO环境apollo组件
- apollo-config-server-pro 部署文件
apollo-config-server-pro.yaml:
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-apollo-config-server-pro
namespace: apollo
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_config_pro?characterEncoding=utf8
spring.datasource.username = {{USERNAME}}
spring.datasource.password = {{PASSWORD}}
eureka.service.url = http://statefulset-apollo-config-server-pro-0.service-apollo-meta-server-pro:8080/eureka/,http://statefulset-apollo-config-server-pro-1.service-apollo-meta-server-pro:8080/eureka/,http://statefulset-apollo-config-server-pro-2.service-apollo-meta-server-pro:8080/eureka/
---
kind: Service
apiVersion: v1
metadata:
name: service-apollo-meta-server-pro
namespace: apollo
labels:
app: service-apollo-meta-server-pro
spec:
type: ClusterIP
clusterIP: None
sessionAffinity: ClientIP
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: pod-apollo-config-server-pro
---
kind: Service
apiVersion: v1
metadata:
name: service-apollo-config-server-pro
namespace: apollo
labels:
app: service-apollo-config-server-pro
spec:
type: NodePort
sessionAffinity: ClientIP
ports:
- port: 8080
targetPort: 8080
nodePort: 30005
selector:
app: pod-apollo-config-server-pro
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: statefulset-apollo-config-server-pro
namespace: apollo
labels:
app: statefulset-apollo-config-server-pro
spec:
serviceName: service-apollo-meta-server-pro
replicas: 3
selector:
matchLabels:
app: pod-apollo-config-server-pro
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-config-server-pro
spec:
containers:
- image: ccr.ccs.tencentyun.com/awesome/apollo-config-server:v1.7.1.x
name: container-apollo-config-server-pro
ports:
- containerPort: 8080
env:
- name: APOLLO_CONFIG_SERVICE_NAME
value: "service-apollo-config-server-pro.apollo" # 注意修改 Namespace
resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 500m
memory: 512Mi
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 120
periodSeconds: 10
volumeMounts:
- name: volume-configmap-apollo-config-server-pro
mountPath: /apollo-config-server/config/application-github.properties
subPath: application-github.properties
volumes:
- name: volume-configmap-apollo-config-server-pro
configMap:
name: configmap-apollo-config-server-pro
items:
- key: application-github.properties
path: application-github.properties
- apollo-admin-server-pro 部署文件
apollo-admin-server-pro.yaml:
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-apollo-admin-server-pro
namespace: apollo
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_config_pro?characterEncoding=utf8
spring.datasource.username = {{USERNAME}}
spring.datasource.password = {{PASSWORD}}
eureka.service.url = http://statefulset-apollo-config-server-pro-0.service-apollo-meta-server-pro:8080/eureka/,http://statefulset-apollo-config-server-pro-1.service-apollo-meta-server-pro:8080/eureka/,http://statefulset-apollo-config-server-pro-2.service-apollo-meta-server-pro:8080/eureka/
---
kind: Service
apiVersion: v1
metadata:
name: service-apollo-admin-server-pro
namespace: apollo
labels:
app: service-apollo-admin-server-pro
spec:
ports:
- protocol: TCP
port: 8090
targetPort: 8090
selector:
app: pod-apollo-admin-server-pro
type: ClusterIP
sessionAffinity: ClientIP
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: deployment-apollo-admin-server-pro
namespace: apollo
labels:
app: deployment-apollo-admin-server-pro
spec:
replicas: 3
selector:
matchLabels:
app: pod-apollo-admin-server-pro
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: pod-apollo-admin-server-pro
spec:
initContainers:
- image: ccr.ccs.tencentyun.com/awesome/alpine-bash:3.8
name: check-service-apollo-config-server-pro
# 注意修改 Namespace
command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 50 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-pro.apollo:8080"]
containers:
- image: ccr.ccs.tencentyun.com/awesome/apollo-admin-server:v1.7.1
name: container-apollo-admin-server-pro
ports:
- containerPort: 8090
env:
- name: APOLLO_ADMIN_SERVICE_NAME
value: "service-apollo-admin-server-pro.apollo" # 注意修改 Namespace
resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 500m
memory: 512Mi
readinessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 120
periodSeconds: 10
volumeMounts:
- name: volume-configmap-apollo-admin-server-pro
mountPath: /apollo-admin-server/config/application-github.properties
subPath: application-github.properties
volumes:
- name: volume-configmap-apollo-admin-server-pro
configMap:
name: configmap-apollo-admin-server-pro
items:
- key: application-github.properties
path: application-github.properties
- k8s部署pro环境的config和admin组件
$ kubectl apply -f apollo-config-server-pro.yaml -n apollo
$ kubectl apply -f appllo-admin-server-pro.yaml -n apollo
或者在k8s控制台上通过yaml手动创建对应的资源。
- 部署apollo-portal组件
下面 Portal 的部署文件中有个初始化镜像用于进行检测 Config Server和Admin Server 是否正常启动。要注意下,如果上面只部署了 DEV 环境,请删除 Pro 环境的检测配置,否则服务将不能正常启动。
service-apollo-portal-server.yaml:
# Configmap 中配置要使用的环境,这里只用 DEV 与 PRO 两个环境,
kind: ConfigMap
apiVersion: v1
metadata:
name: configmap-apollo-portal-server
namespace: apollo
data:
application-github.properties: |
spring.spring.spring.datasource.url = jdbc:mysql://{{HOST}}:3306/apollo_portal?characterEncoding=utf8
spring.spring.datasource.username = {{USERNAME}}
spring.datasource.password = {{PASSWORD}}
apollo-env.properties: |
dev.meta=http://service-apollo-config-server-dev.apollo:8080
#fat.meta=http://service-apollo-config-server-fat.apollo:8080
#uat.meta=http://service-apollo-config-server-uat.apollo:8080
pro.meta=http://service-apollo-config-server-pro.apollo:8080
---
kind: Service
apiVersion: v1
metadata:
name: service-apollo-portal-server
namespace: apollo
labels:
app: service-apollo-portal-server
spec:
type: NodePort
sessionAffinity: ClientIP # portal session 保持
ports:
- protocol: TCP
port: 8070
targetPort: 8070
nodePort: 30011
selector:
app: pod-apollo-portal-server
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: deployment-apollo-portal-server
namespace: apollo
labels:
app: deployment-apollo-portal-server
spec:
replicas: 1
selector:
matchLabels:
app: pod-apollo-portal-server
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: pod-apollo-portal-server
spec:
initContainers:
# 测试 admin-service 服务是否能正常提供服务,如果执行失败,Portal 服务将不能正常启动,注意修改下面的 Namespace
#### ----- 测 Dev 环境的配置 -------
- image: ccr.ccs.tencentyun.com/awesome/alpine-bash:3.8
name: check-service-apollo-admin-server-dev
command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-dev.apollo:8090"]
#### ------- 检测 Pro 环境的配置,如果你没有创建 Prod 环境,请去掉下面这段配置 -------
- image: ccr.ccs.tencentyun.com/awesome/alpine-bash:3.8
name: check-service-apollo-admin-server-pro
command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-pro.apollo:8090"]
### -----------------------------------------------------------------------------
containers:
- image: ccr.ccs.tencentyun.com/awesome/apollo-portal-server:v1.8.0.teic
name: container-apollo-portal-server
ports:
- containerPort: 8070
env:
- name: APOLLO_PORTAL_SERVICE_NAME
value: "service-apollo-portal-server.apollo" # 注意修改 Namespace
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 250m
memory: 512Mi
readinessProbe:
tcpSocket:
port: 8070
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8070
initialDelaySeconds: 120
periodSeconds: 15
volumeMounts:
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal-server/config/application-github.properties
subPath: application-github.properties
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal-server/config/apollo-env.properties
subPath: apollo-env.properties
volumes:
- name: volume-configmap-apollo-portal-server
configMap:
name: configmap-apollo-portal-server
items:
- key: application-github.properties
path: application-github.properties
- key: apollo-env.properties
path: apollo-env.properties
- k8s部署portal组件
$ kubectl apply -f service-apollo-portal-server -n apollo
或者在k8s控制台上通过yaml手动创建对应的资源。
验证
-
K8S资源检查
上述步骤操作完成后,可依次检查下Pod、Service和ConfigMap资源是否ready:
-
Pod:
-
Service:
-
ConfigMap:
-
-
控制台
根据刚才创建portal时候配置的Service设置的NodePort端口为30011,便可通过
http://{{PORTAL_SERVICE_IP}}:30011
来访问Portal进入Apollo配置中心的控制台。Portal默认账户:
- 用户名:apollo
- 密码:admin
-
使用
https://ctripcorp.github.io/apollo/#/zh/usage/apollo-user-guide
部署架构
其他
-
meta服务中的Eureka组件控制台,维护了对应环境中的admin和config组件的实例监控信息:
http://{{SERVICE_META_IP}}:8080
-
获取注册到Eureka中的应用信息:
http://{{META_SERVICE_IP}}/eureka/apps/{{APPLICATION}}
APPLICATION可以从Eureka控制台中获取,如:
http://{{META_SERVICE_IP}}/eureka/apps/APOLLO-CONFIGSERVICE
-
获取当前环境下指定appId下的所有配置:
http://{{META_SERVICE_IP}}/configfiles/json/{{APP_ID}}/default/application
http://{{META_SERVICE_IP}}/configs/json/{{APP_ID}}/default/application