在kubernetes集群里集成Apollo配置中心(1)之交付Apollo-configservice至Kubernetes集群
1.Apollo简介
Apollo（阿波罗）是携程框架部门研发的分布式配置中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性，适用于微服务配置管理场景。
2.Apollo地址
Apollo官方地址：https://github.com/ctripcorp/apollo
官方release包地址：https://github.com/ctripcorp/apollo/releases
基础架构
简化模型
3.准备apollo-configservice软件包
apollo-configservice软件包下载地址：https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-configservice-1.5.1-github.zip
在运维主机上执行
[root@mfyxw50 ~]# cd /opt/src
[root@mfyxw50 src]# wget https://github.com/ctripcorp/apollo/releases/download/v1.5.1/apollo-configservice-1.5.1-github.zip
[root@mfyxw50 src]# mkdir -p /data/dockerfile/apollo-configservice
[root@mfyxw50 src]# unzip apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice/
[root@mfyxw50 src]# rm -fr /data/dockerfile/apollo-configservice/apollo-configservice-1.5.1-sources.jar #apollo-configservice-1.5.1-sources.jar源码包用不到
4.安装MariaDB数据库
在mfyxw10.mfyxw.com主机上操作
注意：mysql的版本要5.6以上，mariadb要用10.1以上
(1)添加MariaDB源
[root@mfyxw10 ~]# cat > /etc/yum.repos.d/MariaDB.repo << EOF
[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1
EOF
(2)导入MariaDB证书
[root@mfyxw10 ~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
(3)生成缓存
[root@mfyxw10 ~]# yum makecache
(4)查看可用的MariaDB数据库版本
[root@mfyxw10 ~]# yum list MariaDB-server --show-duplicates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* extras: mirrors.ustc.edu.cn
* updates: mirrors.aliyun.com
Available Packages
MariaDB-server.x86_64 10.1.43-1.el7.centos mariadb
MariaDB-server.x86_64 10.1.44-1.el7.centos mariadb
MariaDB-server.x86_64 10.1.45-1.el7.centos mariadb
mariadb-server.x86_64 1:5.5.65-1.el7 base
(5)安装MariaDB-Server 10.1.45版本
[root@mfyxw10 ~]# yum -y install MariaDB-server
(6)设置MariaDB配置文件
/etc/my.cnf.d/server.cnf文件内容如下
[root@mfyxw10 ~]# cat > /etc/my.cnf.d/server.cnf << EOF
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"
#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.1 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.1]
EOF
/etc/my.cnf.d/mysql-clients.cnf文件内容如下
[root@mfyxw10 ~]# cat > /etc/my.cnf.d/mysql-clients.cnf << EOF
#
# These groups are read by MariaDB command-line tools
# Use it for options that affect only one utility
#
[mysql]
default-character-set = utf8mb4
[mysql_upgrade]
[mysqladmin]
[mysqlbinlog]
[mysqlcheck]
[mysqldump]
[mysqlimport]
[mysqlshow]
[mysqlslap]
EOF
(7)启动MariaDB数据库并添加至开机自启
[root@mfyxw10 ~]# systemctl enable --now mariadb
[root@mfyxw10 ~]# systmctl status mysql
[root@mfyxw10 ~]# netstat -tanlp | grep mysql #查看MariaDB的启动端口
(8)设置MariaDB数据库密码
[root@mfyxw10 ~]# mysqladmin -uroot password #设置密码：H@o123456
[root@mfyxw10 ~]# mysql -uroot -p #输入数据库密码登录
MariaDB [(none)]> drop database test;
MariaDB [(none)]> use mysql;
MariaDB [mysql]> delete from user where user='';
或都使用如下命令对MariaDB数据库初始化设置
[root@mfyxw10 ~]# mysql_secure_installation
(9)登录MariaDB数据库查看编码是否都是UTF-8
[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> \s
(10)下载并导入apollo数据库初始化脚本
apollo的初始化数据脚本：https://raw.githubusercontent.com/ctripcorp/apollo/v1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql
[root@mfyxw10 ~]# wget https://raw.githubusercontent.com/ctripcorp/apollo/v1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O apolloconfig.sql
[root@mfyxw10 ~]# mysql -uroot -p < apolloconfig.sql
(11)查看apollo的数据库及表
[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| ApolloConfigDB |
| information_schema |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]> use ApolloConfigDB;
MariaDB [ApolloConfigDB]> show tables;
+--------------------------+
| Tables_in_ApolloConfigDB |
+--------------------------+
| App |
| AppNamespace |
| Audit |
| Cluster |
| Commit |
| GrayReleaseRule |
| Instance |
| InstanceConfig |
| Item |
| Namespace |
| NamespaceLock |
| Release |
| ReleaseHistory |
| ReleaseMessage |
| ServerConfig |
+--------------------------+
15 rows in set (0.00 sec)
(12)给数据库用户授予权限
[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> grant SELECT,DELETE,UPDATE,INSERT on ApolloConfigDB.* to "apolloconfig"@"192.168.80.%" identified by "123456";
Query OK, 0 rows affected (0.00 sec)
MariaDB [mysql]> select user,host from mysql.user;
+--------------+-------------------+
| user | host |
+--------------+-------------------+
| root | 127.0.0.1 |
| apolloconfig | 192.168.80.% |
| root | ::1 |
| root | localhost |
| root | mfyxw10.mfyxw.com |
+--------------+-------------------+
5 rows in set (0.00 sec)
(13)修改ApolloConfigDB数据库的值
先查看默认的值：
[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> use ApolloConfigDB;
MariaDB [ApolloConfigDB]> select * from ServerConfig\G;
*************************** 1. row ***************************
Id: 1
Key: eureka.service.url
Cluster: default
Value: http://localhost:8080/eureka/
Comment: Eureka服务Url，多个service以英文逗号分隔
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
*************************** 2. row ***************************
Id: 2
Key: namespace.lock.switch
Cluster: default
Value: false
Comment: 一次发布只能有一个人修改开关
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
*************************** 3. row ***************************
Id: 3
Key: item.key.length.limit
Cluster: default
Value: 128
Comment: item key 最大长度限制
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
*************************** 4. row ***************************
Id: 4
Key: item.value.length.limit
Cluster: default
Value: 20000
Comment: item value最大长度限制
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
*************************** 5. row ***************************
Id: 5
Key: config-service.cache.enabled
Cluster: default
Value: false
Comment: ConfigService是否开启缓存，开启后能提高性能，但是会增大内存消耗！
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
5 rows in set (0.00 sec)
ERROR: No query specified
MariaDB [ApolloConfigDB]>
修改ServerConig表的Value值
[root@mfyxw10 ~]# mysql -uroot -p
MariaDB [(none)]> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
MariaDB [ApolloConfigDB]> select * from ServerConfig\G;
*************************** 1. row ***************************
Id: 1
Key: eureka.service.url
Cluster: default
Value: http://config.od.com/eureka
Comment: Eureka服务Url，多个service以英文逗号分隔
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 04:20:52
*************************** 2. row ***************************
Id: 2
Key: namespace.lock.switch
Cluster: default
Value: false
Comment: 一次发布只能有一个人修改开关
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
*************************** 3. row ***************************
Id: 3
Key: item.key.length.limit
Cluster: default
Value: 128
Comment: item key 最大长度限制
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
*************************** 4. row ***************************
Id: 4
Key: item.value.length.limit
Cluster: default
Value: 20000
Comment: item value最大长度限制
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
*************************** 5. row ***************************
Id: 5
Key: config-service.cache.enabled
Cluster: default
Value: false
Comment: ConfigService是否开启缓存，开启后能提高性能，但是会增大内存消耗！
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2020-07-03 03:20:18
DataChange_LastModifiedBy:
DataChange_LastTime: 2020-07-03 03:20:18
5 rows in set (0.00 sec)
ERROR: No query specified
MariaDB [ApolloConfigDB]>
5.解析域名
在mfyxw10.mfyxw.com主机上操作
(1)在od.com域名的配置文件中添加mysql.od.com和config.od.com域名
[root@mfyxw10 ~]# cat > /var/named/od.com.zone << EOF
\$ORIGIN od.com.
\$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
;序号请加1，表示比之前版本要新
2020031311 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
\$TTL 60 ; 1 minute
dns A 192.168.80.10
harbor A 192.168.80.50 ;添加harbor记录
k8s-yaml A 192.168.80.50
traefik A 192.168.80.100
dashboard A 192.168.80.100
zk1 A 192.168.80.10
zk2 A 192.168.80.20
zk3 A 192.168.80.30
jenkins A 192.168.80.100
dubbo-monitor A 192.168.80.100
demo A 192.168.80.100
mysql A 192.168.80.10
config A 192.168.80.100
EOF
(2)重启DNS服务器服务
[root@mfyxw10 ~]# systemctl restart named
(3)测试域名解析
[root@mfyxw10 ~]# dig -t A mysql.od.com @192.168.80.10 +short
192.168.80.11
[root@mfyxw10 ~]# dig -t A config.od.com @192.168.80.10 +short
192.168.80.100
6.制作apolloconfig的docker镜像
在运维主机(mfyxw50.mfyxw.com)上执行
(1)更新application-github.properties
[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-configservice/config/application-github.properties << EOF
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
#apollo.eureka.server.enabled=true
#apollo.eureka.client.enabled=true
EOF
(2)更新startup.sh文件
[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-configservice/scripts/startup.sh << EOF
#!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=\$(hostname -i)
SERVER_URL="http://\${APOLLO_CONFIG_SERVICE_NAME}:\${SERVER_PORT}"
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="\$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="\$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="\$JAVA_OPTS -Dserver.port=\$SERVER_PORT -Dlogging.file=\$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=\$LOG_DIR/HeapDumpOnOutOfMemoryError/"
# Find Java
if [[ -n "\$JAVA_HOME" ]] && [[ -x "\$JAVA_HOME/bin/java" ]]; then
javaexe="\$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=\$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi
if [[ "\$javaexe" ]]; then
version=\$("\$javaexe" -version 2>&1 | awk -F '"' '/version/ {print \$2}')
version=\$(echo "\$version" | awk -F. '{printf("%03d%03d",\$1,\$2);}')
# now version is of format 009003 (9.3.x)
if [ \$version -ge 011000 ]; then
JAVA_OPTS="\$JAVA_OPTS -Xlog:gc*:\$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ \$version -ge 010000 ]; then
JAVA_OPTS="\$JAVA_OPTS -Xlog:gc*:\$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ \$version -ge 009000 ]; then
JAVA_OPTS="\$JAVA_OPTS -Xlog:gc*:\$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="\$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="\$JAVA_OPTS -Xloggc:\$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="\$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf "\$(date) ==== Starting ==== \n"
cd \`dirname \$0\`/..
chmod 755 \$SERVICE_NAME".jar"
./\$SERVICE_NAME".jar" start
rc=\$?;
if [[ \$rc != 0 ]];
then
echo "\$(date) Failed to start \$SERVICE_NAME.jar, return code: \$rc"
exit \$rc;
fi
tail -f /dev/null
EOF
(3)编写Dockefile文件
[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-configservice/Dockerfile << EOF
FROM harbor.od.com/base/jre8:8u112
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo "Asia/Shanghai" > /etc/timezone
ADD apollo-configservice-\${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts
CMD ["/apollo-configservice/scripts/startup.sh"]
EOF
(4)制作docker镜像
[root@mfyxw50 ~]# cd /data/dockerfile/apollo-configservice
[root@mfyxw50 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
(5)将制作好的docker镜像上传至私有仓库
[root@mfyxw50 ~]# docker login harbor.od.com
[root@mfyxw50 ~]# docker push harbor.od.com/infra/apollo-configservice:v1.5.1
(6)查看私有仓库中infra是否已经有apollo-configservice:v1.5.1
7.提供apolloconfig配置资源清单
在运维主机(mfyxw50.mfyxw.com)上执行
(1)创建存储apolloconfig配置资源清单的目录
[root@mfyxw50 ~]# mkdir -p /data/k8s-yaml/apollo-configservice
(2)添加配置资源清单
deployment.yaml文件内容如下：
[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-configservice/deployment.yaml << EOF
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.od.com/infra/apollo-configservice:v1.5.1
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
EOF
service.yaml文件内容如下：
[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-configservice/service.yaml << EOF
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
clusterIP: None
type: ClusterIP
sessionAffinity: None
EOF
Ingress.yaml文件内容如下：
[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-configservice/Ingress.yaml << EOF
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
spec:
rules:
- host: config.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080
EOF
configmap.yaml文件内容如下：
[root@mfyxw50 ~]# cat > /data/k8s-yaml/apollo-configservice/configmap.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config.od.com/eureka
app.properties: |
appId=100003171
EOF
8.应用apolloconfig配置资源清单
在master节点(mfyxw30.mfyxw.com或mfyxw40.mfyxw.com)任意一台执行
(1)应用apolloconfig配置资源清单
[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/configmap.yaml
configmap/apollo-configservice-cm created
[root@mfyxw30 ~]#
[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/deployment.yaml
deployment.extensions/apollo-configservice created
[root@mfyxw30 ~]#
[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/service.yaml
service/apollo-configservice created
[root@mfyxw30 ~]#
[root@mfyxw30 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/Ingress.yaml
ingress.extensions/apollo-configservice created
[root@mfyxw30 ~]#
(2)查看apolloconfig的pod是否运行起来
[root@mfyxw30 ~]# kubectl get pod -n infra
NAME READY STATUS RESTARTS AGE
apollo-configservice-5f6555448-wssq5 1/1 Running 0 51s
dubbo-monitor-6676dd74cc-9hghb 1/1 Running 7 14d
dubbo-monitor-6676dd74cc-rd86g 1/1 Running 6 14d
jenkins-b99776c69-p6skp 1/1 Running 14 36d
[root@mfyxw30 ~]#
9.浏览器访问config.od.com
