Deploy Apollo on Kubernetes
Deploy Apollo on Kubernetes
环境说明
在kubernetes上搭建Apollo分布式集群，仅在DEV环境中部署。
| 组件名称 | 副本数 | 所属环境 |
|---|---|---|
| apollo-configservice | 3 | DEV |
| apollo-adminservice | 2 | DEV |
| apollo-portalservice | 1 | DEV |
1、获取yaml文件&和源码包文件
官网说明：https://github.com/ctripcorp/apollo/tree/master/scripts/apollo-on-kubernetes
### 克隆项目，获取yaml文件
git clone https://github.com/ctripcorp/apollo.git
cd apollo/scripts/apollo-on-kubernetes/
ls
alpine-bash-3.8-image/ apollo-admin-server/ apollo-config-server/ apollo-portal-server/ db/ kubernetes/ README.md
tree -L 2
├── alpine-bash-3.8-image # adminserver和portalserver的yaml文件中initContainer引用的镜像
│ └── Dockerfile # 需自己构建
├── apollo-admin-server # adminserver的Dockerfile目录
│ ├── apollo-adminservice.conf
│ ├── apollo-adminservice.jar # jar包需从官网下载zip包，解压后重命名为此名称（必须，否则启动失败）
│ ├── config
│ ├── Dockerfile
│ └── scripts
├── apollo-config-server # configserver的Dockerfile目录
│ ├── apollo-configservice.conf
│ ├── apollo-configservice.jar # jar包需从官网下载zip包，解压后重命名为此名称（必须，否则启动失败）
│ ├── config
│ ├── Dockerfile
│ └── scripts
├── apollo-portal-server # portalserver的Dockerfile目录
│ ├── apollo-portal.conf
│ ├── apollo-portal.jar # jar包需从官网下载zip包，解压后重命名为此名称（必须，否则启动失败）
│ ├── config
│ ├── Dockerfile
│ └── scripts
├── db # 各环境的sql文件，需提前导入到数据库中
│ ├── config-db-dev
│ ├── config-db-prod
│ ├── config-db-test-alpha
│ ├── config-db-test-beta
│ └── portal-db
├── kubernetes # 各环境的yaml文件目录
│ ├── apollo-env-dev
│ ├── apollo-env-prod
│ ├── apollo-env-test-alpha
│ ├── apollo-env-test-beta
│ ├── kubectl-apply.sh
│ └── service-apollo-portal-server.yaml
└── README.md
### 下载源码包，获取jar包
wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-adminservice-1.7.1-github.zip
wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-configservice-1.7.1-github.zip
wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-portal-1.7.1-github.zip
### 解压
mkdir -pv /opt/apollo/{admin,config,portal}
unzip apollo-adminservice-1.7.1-github.zip -d /opt/apollo/admin
unzip apollo-configservice-1.7.1-github.zip -d /opt/apollo/config
unzip apollo-portal-1.7.1-github.zip -d /opt/apollo/config
[root@localhost apollo]# tree -L 2
├── admin
│ ├── apollo-adminservice-1.7.0.jar # 拷贝到adminserver的Dockerfile所在目录，并重命名
│ ├── apollo-adminservice-1.7.0-sources.jar
│ ├── apollo-adminservice.conf
│ ├── apollo-adminservice.jar
│ ├── apollo-adminservice_rootapolloadmin.pid
│ ├── config
│ └── scripts
├── config
│ ├── apollo-configservice-1.7.0.jar # 拷贝到configserver的Dockerfile所在目录，并重命名
│ ├── apollo-configservice-1.7.0-sources.jar
│ ├── apollo-configservice.conf
│ ├── apollo-configservice.jar
│ ├── apollo-configservice_rootapolloconfig.pid
│ ├── config
│ └── scripts
├── package
│ ├── apollo-adminservice-1.7.0-github.zip
│ ├── apollo-configservice-1.7.0-github.zip
│ └── apollo-portal-1.7.0-github.zip
└── portal
# 将各个组件的文件夹下的jar包，拷贝到对应的Dockerfile文件所在目录下并且重命名，如上⬆
2、构建docker镜像
### 构建镜像，然后push到阿里云镜像仓库
# 构建alpine-bash镜像，用于初始化容器
cd apollo/scripts/apollo-on-kubernetes/alpine-bash-3.8-image/
docker build -t registry.cn-hangzhou.aliyuncs.com/smbands/alpine-bash:3.8 .
# 构建apollo-admin镜像
cd apollo/scripts/apollo-on-kubernetes/apollo-admin-server/
docker build -t registry.cn-hangzhou.aliyuncs.com/smbands/apollo-admin:v1 .
# 构建apollo-config镜像
cd apollo/scripts/apollo-on-kubernetes/apollo-config-server/
docker build -t registry.cn-hangzhou.aliyuncs.com/smbands/apollo-config:v1 .
# 构建apollo-portal镜像
cd apollo/scripts/apollo-on-kubernetes/apollo-portal-server/
docker build -t registry.cn-hangzhou.aliyuncs.com/smbands/apollo-portal:v1 .
# 将镜像上传到镜像仓库
docker push registry.cn-hangzhou.aliyuncs.com/smbands/alpine-bash:3.8
docker push registry.cn-hangzhou.aliyuncs.com/smbands/apollo-admin:v1
docker push registry.cn-hangzhou.aliyuncs.com/smbands/apollo-config:v1
docker push registry.cn-hangzhou.aliyuncs.com/smbands/apollo-portal:v1
3、部署
① 部署前准备数据库：
将【apollo/scripts/apollo-on-kubernetes/db】路径下的
【config-db-dev/apolloconfigdb.sql】和【portal-db/apolloportaldb.sql】sql文件导入到MySQL数据库中，
会创建ApolloPortalDB、DevApolloConfigDB两个库，创建后新建一个MySQL用户，并授予权限和远程登录。数据库可以部署在任何与k8s通信的主机上。
mysql -uroot -p(你的密码)< apolloconfigdb.sql
# mysql: [Warning] Using a password on the command line interface can be insecure.
mysql -uroot -p(你的密码)< apolloportaldb.sql
# mysql: [Warning] Using a password on the command line interface can be insecure.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| ApolloPortalDB |
| DevApolloConfigDB |
| mysql |
| performance_schema |
| sys |
+--------------------+
6 rows in set (0.07 sec)
# 创建用户并授权
mysql> create user apollo identified by '密码';
Query OK, 0 rows affected (0.09 sec)
mysql> grant all privileges on *.* to 'apollo'@'%' identified by '密码';
Query OK, 0 rows affected (0.09 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.09 sec)
② 将数据库与k8s关联
通过在k8s上部署svc和endpoint，endpoint指向实际MySQL服务，svc则是引用此endpoint，这样就实现了MySQL和k8s的关联，相关文件在：
【apollo/scripts/apollo-on-kubernetes/kubernetes/apollo-env-dev/service-mysql-for-apollo-dev-env.yaml】
修改一下MySQL主机IP即可
---
# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
namespace: sre
name: service-mysql-for-apollo-dev-env
labels:
app: service-mysql-for-apollo-dev-env
spec:
ports:
- protocol: TCP
port: 3306
targetPort: 3306
type: ClusterIP
sessionAffinity: None
---
kind: Endpoints
apiVersion: v1
metadata:
namespace: sre
name: service-mysql-for-apollo-dev-env
subsets:
- addresses:
- ip: MySQL主机IP地址 # 填写MySQL主机IP
ports:
- protocol: TCP
port: 3306 # 根据实际MySQL端口填写
---
# kubectl create ns sre
# kubectl apply -f service-mysql-for-apollo-dev-env.yaml
# 注意这些资源定义的名称空间是sre，所以，应用yaml时应先创建sre名称空间
③ 部署configserver组件
更改组件对应的yaml文件中的image字段的镜像名称地址
注意yaml文件中包含configMap，这个是配置连接数据库的，一般只需改一下用户名和密码，其他最好别动确认configMap中的数据库连接信息和下面statefulset的镜像地址就行，其他可以保持默认
资源文件：apollo/scripts/apollo-on-kubernetes/kubernetes/apollo-env-dev/service-apollo-config-server-dev.yaml
# configmap for apollo-config-server-dev
kind: ConfigMap
apiVersion: v1
metadata:
namespace: sre
name: configmap-apollo-config-server-dev
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-dev-env.sre:3306/DevApolloConfigDB?characterEncoding=utf8&serverTimezone=GMT
spring.datasource.username = apollo
spring.datasource.password = 你的密码
# 参数说明：
# SPRING_DATASOURCE_URL: 对应环境DevApolloConfigDB的地址
# SPRING_DATASOURCE_USERNAME: 对应环境DevApolloConfigDB的用户名
# SPRING_DATASOURCE_PASSWORD: 对应环境DevApolloConfigDB的密码
---
kind: Service
apiVersion: v1
metadata:
namespace: sre
name: service-apollo-meta-server-dev
labels:
app: service-apollo-meta-server-dev
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: pod-apollo-config-server-dev
type: ClusterIP
clusterIP: None
sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
namespace: sre
name: service-apollo-config-server-dev
labels:
app: service-apollo-config-server-dev
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
nodePort: 30002
selector:
app: pod-apollo-config-server-dev
type: NodePort
sessionAffinity: ClientIP
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
namespace: sre
name: statefulset-apollo-config-server-dev
labels:
app: statefulset-apollo-config-server-dev
spec:
serviceName: service-apollo-meta-server-dev
replicas: 3
selector:
matchLabels:
app: pod-apollo-config-server-dev
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-config-server-dev
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-config-server-dev
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-config-server-dev
configMap:
name: configmap-apollo-config-server-dev
items:
- key: application-github.properties
path: application-github.properties
containers:
- image: registry.cn-hangzhou.aliyuncs.com/smbands/apollo-config:v1
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-config-server-dev
ports:
- protocol: TCP
containerPort: 8080
volumeMounts:
- name: volume-configmap-apollo-config-server-dev
mountPath: /apollo-config-server/config/application-github.properties
subPath: application-github.properties
env:
- name: APOLLO_CONFIG_SERVICE_NAME
value: "service-apollo-config-server-dev.sre"
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
---
# kubectl apply -f service-apollo-config-server-dev.yaml
④ 部署adminserver组件
和部署configserver修改项一样，也是数据库连接信息和镜像地址，但是此yaml中使用了初始化容器，不要忘记修改initContainer的镜像。
资源文件：apollo/scripts/apollo-on-kubernetes/kubernetes/apollo-env-dev/service-apollo-admin-server-dev.yaml
---
# configmap for apollo-admin-server-dev
kind: ConfigMap
apiVersion: v1
metadata:
namespace: sre
name: configmap-apollo-admin-server-dev
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-dev-env.sre:3306/DevApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apollo
spring.datasource.password = 你的密码
---
kind: Service
apiVersion: v1
metadata:
namespace: sre
name: service-apollo-admin-server-dev
labels:
app: service-apollo-admin-server-dev
spec:
ports:
- protocol: TCP
port: 8090
targetPort: 8090
selector:
app: pod-apollo-admin-server-dev
type: ClusterIP
sessionAffinity: ClientIP
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: sre
name: deployment-apollo-admin-server-dev
labels:
app: deployment-apollo-admin-server-dev
spec:
replicas: 3
selector:
matchLabels:
app: pod-apollo-admin-server-dev
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
metadata:
labels:
app: pod-apollo-admin-server-dev
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-admin-server-dev
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-admin-server-dev
configMap:
name: configmap-apollo-admin-server-dev
items:
- key: application-github.properties
path: application-github.properties
initContainers:
- image: registry.cn-hangzhou.aliyuncs.com/smbands/alpine-bash:3.8
imagePullPolicy: IfNotPresent
name: check-service-apollo-config-server-dev
containers:
- image: registry.cn-hangzhou.aliyuncs.com/smbands/apollo-admin:v1
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-admin-server-dev
ports:
- protocol: TCP
containerPort: 8090
volumeMounts:
- name: volume-configmap-apollo-admin-server-dev
mountPath: /apollo-admin-server/config/application-github.properties
subPath: application-github.properties
env:
- name: APOLLO_ADMIN_SERVICE_NAME
value: "service-apollo-admin-server-dev.sre"
readinessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8090
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
---
# kubectl apply -f service-apollo-admin-server-dev.yaml
⑤ 部署portal
此资源中单独定义了连接MySQL的svc和endpoint，不要取消，填写MySQL地址。值得注意的是，在没有配置（alpha、beta、prod）这些环境时，需在configmap-apollo-portal-server注释掉对应环境，同时注释initContainer中对应环境的初始化容器即可，若没有这些环境且不注释，则会在pod初始化时启动失败。
资源文件：apollo/scripts/apollo-on-kubernetes/kubernetes/service-apollo-portal-server.yaml
---
# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
namespace: sre
name: service-mysql-for-portal-server
labels:
app: service-mysql-for-portal-server
spec:
ports:
- protocol: TCP
port: 3306
targetPort: 3306
type: ClusterIP
sessionAffinity: None
---
kind: Endpoints
apiVersion: v1
metadata:
namespace: sre
name: service-mysql-for-portal-server
subsets:
- addresses:
# # 更改为你的 mysql addresses, 例如 1.1.1.1
- ip: MySQL主机IP地址
ports:
- protocol: TCP
port: 3306
---
# configmap for apollo-portal-server
kind: ConfigMap
apiVersion: v1
metadata:
namespace: sre
name: configmap-apollo-portal-server
data:
application-github.properties: |
spring.datasource.url = jdbc:mysql://service-mysql-for-portal-server.sre:3306/ApolloPortalDB?characterEncoding=utf8
# mysql username
spring.datasource.username = apollo
# mysql password
spring.datasource.password = 你的密码
apollo-env.properties: |
dev.meta=http://service-apollo-config-server-dev.sre:8080
# fat.meta=http://service-apollo-config-server-test-alpha.sre:8080
# uat.meta=http://service-apollo-config-server-test-beta.sre:8080
# pro.meta=http://service-apollo-config-server-prod.sre:8080
---
kind: Service
apiVersion: v1
metadata:
namespace: sre
name: service-apollo-portal-server
labels:
app: service-apollo-portal-server
spec:
ports:
- protocol: TCP
port: 8070
targetPort: 8070
nodePort: 30001
selector:
app: pod-apollo-portal-server
type: NodePort
# portal session 保持
sessionAffinity: ClientIP
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: sre
name: deployment-apollo-portal-server
labels:
app: deployment-apollo-portal-server
spec:
# 3 个实例
replicas: 3
selector:
matchLabels:
app: pod-apollo-portal-server
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
app: pod-apollo-portal-server
spec:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
matchExpressions:
- key: app
- pod-apollo-portal-server
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-portal-server
configMap:
name: configmap-apollo-portal-server
items:
- key: application-github.properties
path: application-github.properties
- key: apollo-env.properties
path: apollo-env.properties
initContainers:
# 确保 admin-service 正常提供服务
- image: registry.cn-hangzhou.aliyuncs.com/smbands/alpine-bash:3.8
name: check-service-apollo-admin-server-dev
# - image: registry.cn-hangzhou.aliyuncs.com/smbands/alpine-bash:3.8
# name: check-service-apollo-admin-server-alpha
# - image: registry.cn-hangzhou.aliyuncs.com/smbands/alpine-bash:3.8
# name: check-service-apollo-admin-server-beta
# - image: registry.cn-hangzhou.aliyuncs.com/smbands/alpine-bash:3.8
# name: check-service-apollo-admin-server-prod
containers:
- image: registry.cn-hangzhou.aliyuncs.com/smbands/apollo-portal:v1
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-portal-server
ports:
- protocol: TCP
containerPort: 8070
volumeMounts:
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal-server/config/application-github.properties
subPath: application-github.properties
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal-server/config/apollo-env.properties
subPath: apollo-env.properties
env:
- name: APOLLO_PORTAL_SERVICE_NAME
value: "service-apollo-portal-server.sre"
readinessProbe:
tcpSocket:
port: 8070
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 8070
# 120s 内, server 未启动则重启 container
initialDelaySeconds: 120
periodSeconds: 15
dnsPolicy: ClusterFirst
restartPolicy: Always
---
# kubectl apply -f service-apollo-portal-server.yaml
⑥ 查看部署资源
由于电脑资源有限，启动的副本数和上面的略有出入，不过其他配置基本相同。
kubectl get all -n sre
NAME READY STATUS RESTARTS AGE
pod/deployment-apollo-admin-server-dev-54b8df6cf9-24dgr 1/1 Running 0 80m
pod/deployment-apollo-admin-server-dev-54b8df6cf9-jxcrr 1/1 Running 0 80m
pod/deployment-apollo-portal-server-5d5fffd5b9-6d99p 1/1 Running 0 79m
pod/statefulset-apollo-config-server-dev-0 1/1 Running 0 120m
pod/statefulset-apollo-config-server-dev-1 1/1 Running 0 119m
pod/statefulset-apollo-config-server-dev-2 1/1 Running 0 119m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/service-apollo-admin-server-dev ClusterIP 10.98.14.21 <none> 8090/TCP 80m
service/service-apollo-config-server-dev NodePort 10.109.142.255 <none> 8080:30002/TCP 120m
service/service-apollo-meta-server-dev ClusterIP None <none> 8080/TCP 120m
service/service-apollo-portal-server NodePort 10.101.188.69 <none> 8070:30001/TCP 79m
service/service-mysql-for-apollo-dev-env ClusterIP 10.100.210.26 <none> 3306/TCP 120m
service/service-mysql-for-portal-server ClusterIP 10.100.13.142 <none> 3306/TCP 79m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/deployment-apollo-admin-server-dev 2/2 2 2 80m
deployment.apps/deployment-apollo-portal-server 1/1 1 1 79m
NAME DESIRED CURRENT READY AGE
replicaset.apps/deployment-apollo-admin-server-dev-54b8df6cf9 2 2 2 80m
replicaset.apps/deployment-apollo-portal-server-5d5fffd5b9 1 1 1 79m
NAME READY AGE
statefulset.apps/statefulset-apollo-config-server-dev 3/3 120m
⑦ 访问portal的service
