kubernetes三： 交付dubbo服务到K8S集群
一、Dubbo介绍
1.Dubbo是什么？
- Dubbo是阿里巴巴SOA服务化治理方案的核心框架，每天为2000+个服务提供30亿+次访问支持，并被广泛应用于阿里巴巴集团的各成员站点。
- Dubbo是一个分布式服务框架，致力于提供高性能和透明化的PRC远程服务调用方案，以及SOA服务治理方案
- 简单的说，Dubbo就是个服务框架，如果没有分布式的需求，其实是不需要用的，只有在分布式的时候，才有dubbo这样的的分布式服务框架的需求，并且本质上是个服务调用的东东，说白了就是个远程服务调用的分布式框架
2.Dubbo能做什么？
- 透明化的远程方法调用，就像调用本地方法一样调用远程方法，只需要简单的配置，没有任何API侵入
- 软负载均衡及容错机制，可在内网替代F5等硬件负载均衡器，降低成本，减少单点。
- 服务自动注册与发现，不再需要写死服务提供方地址，注册中心基于接口名查询服务提供者的IP地址，并且能够平滑添加或删除服务提供者。
3.Dubbo工作原理
- 简单的说，Dubbo 是 基于 Java 的RPC 框架。Dubbo 工作分为 4 个角色，分别是服务提供者、服务消费者、注册中心、和监控中心。
- 按照工作阶段又分为部署阶段和运行阶段。
- 其中部署阶段在图中以蓝色的线来表示，代表服务注册、服务订阅的过程，而运行阶段在图中以红色的线来表示，代表一次 RPC 的完整调用。
- 部署阶段中服务提供方在启动时在指定的端口上暴露服务，并向注册中心汇报自己的地址。
- 服务调用方启动时向注册中心订阅自己感兴趣的服务。
- 运行阶段注册中心先将地址列表推送给服务消费者，服务消费者选取一个地址向对端发起调用。
- 在这个过程中，服务消费者和服务提供者的运行状态会上报给监控中心。
二、实战交付一套dubbo微服务到kubernetes集群
1.实验拓扑图
- 第一层代表部署在k8s之外的，第二层部署在k8s中，第三层部署在7-200中
2.基础架构
| 主机名 | 角色 | ip |
|---|---|---|
| kjdow7-11.host.com | k8s代理节点1，zk1 | 10.4.7.11 |
| kjdow7-12.host.com | k8s代理节点2，zk2 | 10.4.7.12 |
| kjdow7-21.host.com | k8s代理节点3，zk3 | 10.4.7.21 |
| kjdow7-22.host.com | k8s运算节点2，jenkins | 10.4.7.22 |
| kjdow7-200.host.com | k8s运维节点(docker仓库) | 10.4.7.200 |
3.部署zookeeper
3.1 安装jdk
在kjdow7-11、kjdow7-12、kjdow7-21三台主机上部署
~]# mkdir /usr/java
~]# tar xf jdk-8u221-linux-x64.tar.gz -C /usr/java
~]# ln -s /usr/java/jdk1.8.0_221 /usr/java/jdk
~]# vim /etc/profile
export JAVA_HOME=/usr/java/jdk
export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
~]# source /etc/profile
~]# java -version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
3.2 安装zookeeper（3台zk角色主机）
在kjdow7-11、kjdow7-12、kjdow7-21三台主机上部署
#解压、配置
~]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
~]# tar xf zookeeper-3.4.14.tar.gz -C /opt
~]# ln -s /opt/zookeeper-3.4.14 /opt/zookeeper
~]# mkdir -p /data/zookeeper/data /data/zookeeper/logs
~]# vi /opt/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zk1.phc-dow.com:2888:3888
server.2=zk2.phc-dow.com:2888:3888
server.3=zk3.phc-dow.com:2888:3888
注意：各节点zk配置相同。
在kjdow7-11主机上部署
[root@kjdow7-11 ~]# cat /data/zookeeper/data/myid
1
在kjdow7-12主机上部署
[root@kjdow7-12 ~]# cat /data/zookeeper/data/myid
2
在kjdow7-21主机上部署
[root@kjdow7-12 ~]# cat /data/zookeeper/data/myid
3
3.3 做dns解析
在kjdow7-11主机上部署
[root@kjdow7-11 ~]# cat /var/named/phc-dow.com.zone
2020010206 ; serial #serial值加一
zk1 60 IN A 10.4.7.11 #末尾添加此三行
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
[root@kjdow7-11 ~]# systemctl restart named
[root@kjdow7-11 ~]# dig -t A zk1.phc-dow.com @10.4.7.11 +short
10.4.7.11
[root@kjdow7-11 ~]# dig -t A zk2.phc-dow.com @10.4.7.11 +short
10.4.7.12
[root@kjdow7-11 ~]# dig -t A zk3.phc-dow.com @10.4.7.11 +short
10.4.7.21
3.4 依次启动zk
[root@kjdow7-11 ~]# /opt/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@kjdow7-11 ~]# netstat -lntup | grep 19333
tcp6 0 0 10.4.7.11:3888 :::* LISTEN 19333/java
tcp6 0 0 :::36989 :::* LISTEN 19333/java
tcp6 0 0 :::2181 :::* LISTEN 19333/java
[root@kjdow7-21 ~]# netstat -lntup | grep 3675
tcp6 0 0 10.4.7.21:2888 :::* LISTEN 3675/java
tcp6 0 0 10.4.7.21:3888 :::* LISTEN 3675/java
tcp6 0 0 :::2181 :::* LISTEN 3675/java
tcp6 0 0 :::39301 :::* LISTEN 3675/java
[root@kjdow7-12 ~]# netstat -lntup | grep 11949
tcp6 0 0 10.4.7.12:3888 :::* LISTEN 11949/java
tcp6 0 0 :::46303 :::* LISTEN 11949/java
tcp6 0 0 :::2181 :::* LISTEN 11949/java
[root@kjdow7-11 ~]# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@kjdow7-12 ~]# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@kjdow7-21 ~]# /opt/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: leader
4.安装部署jenkins准备工作
4.1 准备镜像
[root@kjdow7-200 ~]# docker pull jenkins/jenkins:2.190.3
[root@kjdow7-200 ~]# docker images | grep jenkins
jenkins/jenkins 2.190.3 22b8b9a84dbe 2 months ago 568MB
[root@kjdow7-200 ~]# docker tag 22b8b9a84dbe harbor.phc-dow.com/public/jenkins:v2.190.3
[root@kjdow7-200 ~]# docker push harbor.phc-dow.com/public/jenkins:v2.190.3
4.2 自定义Dockerfile
官网拉取的镜像需要做些自定义操作，才能在k8s集群中部署
在运维主机kjdow-200.host.com`上编辑自定义dockerfile
mkdir -p /data/dockerfile/jenkins
cd /data/dockerfile/jenkins
vim Dockerfile
FROM harbor.phc-dow.com/public/jenkins:v2.190.3
USER root
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo 'Asia/Shanghai' >/etc/timezone
ADD id_rsa /root/.ssh/id_rsa
ADD config.json /root/.docker/config.json
ADD get-docker.sh /get-docker.sh
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config &&\
/get-docker.sh
这个Dockerfile里我们主要做了以下几件事
- 设置容器用户为root
- 设置容器内的时区
- 将ssh私钥加入（使用git拉代码时要用到，配对的公钥应配置在gitlab中）
- 加入了登录自建harbor仓库的config文件
- 修改了ssh客户端的
- 安装一个docker的客户端
- 如果因为网络原因构建失败，可以在最后“ /get-docker.sh --mirror Aliyun”
1) 生成ssh密钥对：
[root@kjdow7-200 jenkins]# ssh-keygen -t rsa -b 2048 -C "897307140@qq.com" -N "" -f /root/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:bIajghsF/BqJouTeNvZXvQWvolAKWvhVSuZ3uVWoVXU 897307140@qq.com
The key's randomart image is:
+---[RSA 2048]----+
| ...E|
|. o .|
|.. o . o . |
|..+ + oo +.. |
|o=.+ +ooS+..o |
|=o* o.++..o. o |
|++...o .. + |
|.o.= .. . o |
|..o.o.... . |
+----[SHA256]-----+
[root@kjdow7-200 jenkins]# cp /root/.ssh/id_rsa .
2) 准备其他文件
[root@kjdow7-200 jenkins]# cp /root/.docker/config.json .
[root@kjdow7-200 jenkins]# curl -fsSL get.docker.com -o get-docker.sh
[root@kjdow7-200 jenkins]# chmod +x get-docker.sh
[root@kjdow7-200 jenkins]# ll
total 28
-rw------- 1 root root 160 Jan 28 23:41 config.json
-rw-r--r-- 1 root root 355 Jan 28 23:38 Dockerfile
-rwxr-xr-x 1 root root 13216 Jan 28 23:42 get-docker.sh
-rw------- 1 root root 1675 Jan 28 23:38 id_rsa
3) 登录harbor仓库页面,创建infra
创建infra的project，access level 为Private
4）生成镜像
[root@kjdow7-200 jenkins]# docker build -t harbor.phc-dow.com/infra/jenkins:v2.190.3 .
Sending build context to Docker daemon 19.46kB
Step 1/7 : FROM harbor.phc-dow.com/public/jenkins:v2.190.3
---> 22b8b9a84dbe
Step 2/7 : USER root
---> Running in 7604d600a620
Removing intermediate container 7604d600a620
---> c8d326bfe8b7
Step 3/7 : RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone
---> Running in 1b72c3d69eea
Removing intermediate container 1b72c3d69eea
---> f839ab1701d0
Step 4/7 : ADD id_rsa /root/.ssh/id_rsa
---> 840bac71419f
Step 5/7 : ADD config.json /root/.docker/config.json
---> 2dcd61ef1c90
Step 6/7 : ADD get-docker.sh /get-docker.sh
---> 9430aa0cb5ad
Step 7/7 : RUN echo " StrictHostKeyChecking no" >> /etc/ssh/sshd_config && /get-docker.sh
---> Running in ff19d96b70da
# Executing docker install script, commit: f45d7c11389849ff46a6b4d94e0dd1ffebca32c1
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
debconf: delaying package configuration, since apt-utils is not installed
+ sh -c curl -fsSL "https://download.docker.com/linux/debian/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sh -c echo "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ [ -n ]
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
debconf: delaying package configuration, since apt-utils is not installed
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:
sudo usermod -aG docker your-user
Remember that you will have to log out and back in for this to take effect!
WARNING: Adding a user to the "docker" group will grant the ability to run
containers which can be used to obtain root privileges on the
docker host.
Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
for more information.
Removing intermediate container ff19d96b70da
---> 637a6cbc288d
Successfully built 637a6cbc288d
Successfully tagged harbor.phc-dow.com/infra/jenkins:v2.190.3
5) 推送镜像到仓库
[root@kjdow7-200 jenkins]# docker push harbor.phc-dow.com/infra/jenkins:v2.190.3
4.3 准备共享存储
jenkins的/var/lib/jenkins_home里面有jenkins的配置等需要挂载到宿主机，这样，无论在哪个运算节点起pod，无论pod是否运行，新运行的pod也会有之前的配置内容不会丢失
1) 在所有主机上运行
yum install nfs-utils -y
2) 配置NFS服务
[root@kjdow7-200 ~]# vim /etc/exports
/data/nfs-volume 10.4.7.0/24(rw,no_root_squash)
###启动NFS服务
[root@kjdow7-200 ~]# mkdir -p /data/nfs-volume
[root@kjdow7-200 ~]# systemctl start nfs
[root@kjdow7-200 ~]# systemctl enable nfs
4.4 准备资源配置清单
运维主机kjdow-200.host.com上：
[root@kjdow7-200 ~]# mkdir /data/k8s-yaml/jenkins && mkdir -p /data/nfs-volume/jenkins_home && cd /data/k8s-yaml/jenkins
[root@kjdow7-200 ~]# vi dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: jenkins
namespace: infra
labels:
name: jenkins
spec:
replicas: 1
selector:
matchLabels:
name: jenkins
template:
metadata:
labels:
app: jenkins
name: jenkins
spec:
volumes:
- name: data
nfs:
server: kjdow7-200
path: /data/nfs-volume/jenkins_home
- name: docker
hostPath:
path: /run/docker.sock
type: ''
containers:
- name: jenkins
image: harbor.phc-dow.com/infra/jenkins:v2.190.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
protocol: TCP
env:
- name: JAVA_OPTS
value: -Xmx512m -Xms512m
volumeMounts:
- name: data
mountPath: /var/jenkins_home
- name: docker
mountPath: /run/docker.sock
imagePullSecrets:
- name: harbor
securityContext:
runAsUser: 0
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
注：imagePullSecrets:中name在创建secret时指定过的
将宿主机/run/docker.sock挂载到pod中，那么pod就可以与宿主机的docker的server端进行通信了
[root@kjdow7-200 ~]# vim service.yaml
kind: Service
apiVersion: v1
metadata:
name: jenkins
namespace: infra
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: jenkins
注： targetport指定镜像端口，jenkins默认打开页面端口是8080
port指定暴露给service中的cluster ip使用80端口
[root@kjdow7-200 ~]# vim ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: jenkins
namespace: infra
spec:
rules:
- host: jenkins.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: jenkins
servicePort: 80
4.5 运算节点创建必须的资源
[root@kjdow7-21 ~]# kubectl create ns infra
namespace/infra created
[root@kjdow7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.phc-dow.com --docker-username=admin --docker-password=Harbor_kjdow1! -n infra
secret/harbor created
###创建一个名字叫harbor的secret
注： 创建infra的命名空间，所有的运维pod都在此空间中
创建secret资源用于从infra的私有仓库中拉取镜像时提供用户名和密码，在上面的dp.yaml里面指定使用此secret
4.6 应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/jenkins/dp.yaml
deployment.extensions/jenkins created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/jenkins/service.yaml
service/jenkins created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/jenkins/ingress.yaml
ingress.extensions/jenkins created
4.7 打开页面访问
[root@kjdow7-200 ~]# cat /data/nfs-volume/jenkins_home/secrets/initialAdminPassword
112f082a79ce4e389be1cf884cc652e8
访问jenkins.phc-dow.com并进行简单的配置，设置用户名admin密码admin123
在页面进行简单的配置
给jenkins添加插件blue ocean
4.8 验证jenkins搭建完成
- 验证用户是否是root
- 验证时间是否对
- 验证docker ps -a是否跟宿主机显示一样
- 验证sshi是否不用输入yes、no
- 验证是否已经登录成功harbor仓库
- 使用私钥验证git是否能连接成功
5.安装部署maven
###查看jenkins的pod中java版本
[root@kjdow7-22 ~]# kubectl get pod -n infra -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
jenkins-67d4b48b54-gd9g7 1/1 Running 0 33m 172.7.22.7 kjdow7-22.host.com <none> <none>
[root@kjdow7-22 ~]# kubectl exec jenkins-67d4b48b54-gd9g7 -it /bin/bash -n infra
root@jenkins-67d4b48b54-gd9g7:/# java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
###下载软件
[root@kjdow7-200 ~]# wget https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz
[root@kjdow7-200 ~]# tar xf apache-maven-3.6.1-bin.tar.gz -C /data/nfs-volume/jenkins_home/
[root@kjdow7-200 ~]# cd /data/nfs-volume/jenkins_home/
[root@kjdow7-200 jenkins_home]# mv apache-maven-3.6.1 maven-3.6.1-8u232
[root@kjdow7-200 ~]# vi /data/nfs-volume/jenkins_home/maven-3.6.1-8u232/conf/settings.xml
<mirrors>
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<!-- mirror
| Specifies a repository mirror site to use instead of a given repository. The repository that
| this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
| for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
|
-->
</mirrors>
###添加到文件的相应位置，jenkins的pod自动就会同步更改
6.dubbo微服务底包镜像制作
6.1 自定义dockerfile
注：我们需要一个java运行时环境的底包
[root@kjdow7-200 ~]# docker pull docker.io/stanleyws/jre8:8u112
[root@kjdow7-200 ~]# docker images | grep jre8
stanleyws/jre8 8u112 fa3a085d6ef1 2 years ago 363MB
[root@kjdow7-200 ~]# docker tag fa3a085d6ef1 harbor.phc-dow.com/public/jre8:8u112
[root@kjdow7-200 ~]# docker push harbor.phc-dow.com/public/jre8:8u112
[root@kjdow7-200 ~]# mkdir /data/dockerfile/jre8
[root@kjdow7-200 ~]# cd /data/dockerfile/jre8
[root@kjdow7-200 jre8]# vim Dockerfile
FROM docker.io/stanleyws/jre8:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo 'Asia/Shanghai' >/etc/timezone
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]
注： 第三行主要是普罗米修斯监控的配置文件
第四行是普罗米修斯使用这个jar包来监控jvm
###准备其他必须的文件
[root@kjdow7-200 jre8]# wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar
————————————————————————————————————————————————————————————————————————
[root@kjdow7-200 jre8]# vi config.yml
---
rules:
- pattern: '.*'
————————————————————————————————————————————————————————————————————————
[root@kjdow7-200 jre8]# vi entrypoint.sh
#!/bin/sh
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
JAR_BALL=${JAR_BALL}
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}
[root@kjdow7-200 jre8]# chmod +x entrypoint.sh
[root@kjdow7-200 jre8]# ll
total 372
-rw-r--r-- 1 root root 29 Jan 29 23:11 config.yml
-rw-r--r-- 1 root root 297 Jan 29 22:54 Dockerfile
-rwxr-xr-x 1 root root 234 Jan 29 23:11 entrypoint.sh
-rw-r--r-- 1 root root 367417 May 10 2018 jmx_javaagent-0.3.1.jar
注：entrypoint.sh文件中
C_OPTS=${C_OPTS}表示将资源配置清单中的变量值赋值给它
${M_PORT:-"12346"}表示如果没有给它赋值，则默认值是12346
最后一行前面加exec是因为这个shell执行完，这个容器就死了，exec作用就是把这个shell 的pid交给 exec后面的命令继续使用，这样java不死，这个pod就能一直存活
shell的内建命令exec将并不启动新的shell，而是用要被执行命令替换当前的shell进程，并且将老进程的环境清理掉，而且exec命令后的其它命令将不再执行。
6.2 harbor页面创建object
在harbor中创建base的object，用来存放所有业务基础镜像.权限为公开
6.3 创建镜像
[root@kjdow7-200 jre8]# docker build -t harbor.phc-dow.com/base/jre8:8u112 .
[root@kjdow7-200 jre8]# docker push harbor.phc-dow.com/base/jre8:8u112
7.使用Jenkins进行持续构建交付dubo服务的提供者
7.1 新建新项目
新建名为dubbo-demo的pipeline项目
7.2 设置丢弃旧的构建，保存三天的，最多30个
参数化构建
7.3 jenkins流水线配置的十个参数
- app_name -->项目名
- image_name --> 镜像名
- git_repo --> 项目的git地址
- git_ver --> 项目的git版本号或分支
- add_tag --> 镜像标签，日期时间戳（20200130_1421）
- mvn_dir --> 编译项目的目录
- target_dir --> 项目编译完成后，禅城的jar、war包所在的目录
- mvn_cmd --> 编译项目的命令
- base_image --> 项目的docker底包镜像
- maven --> maven软件的版本
7.4 pipeline流水线代码
pipeline {
agent any
stages {
stage('pull') {
steps {
sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
}
}
stage('build') {
steps {
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
}
}
stage('package') {
steps {
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
}
}
stage('image') {
steps {
writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.phc-dow.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/project_dir"""
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.phc-dow.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.phc-dow.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
}
}
}
}
7.5 构建前准备工作
harbor仓库创建私有projects名字为app
7.6 开始构建
打开jenkins页面开始构建，填写参数值
依次填入/选择：
app_name: dubbo-demo-service
image_name: app/dubbo-demo-service
git_repo: https://github.com/zizhufanqing/dubbo-demo-service.git
git_ver: master
add_tag: 202001311655
mvn_dir: ./
target_dir: ./dubbo-server/target
mvn_cmd: mvn clean package -Dmaven.test.skip=true
base_image: base/jre8:8u112
maven: 3.6.0-8u181
点击Build进行构建，等待构建完成。
注： 在github上已经添加了公钥
- 构建完成后在harbor的app中查看自动提交的镜像
7.7 准备资源配置清单
在kjdow7-200上部署
[root@kjdow7-200 ~]# mkdir /data/k8s-yaml/dubbo-demo-service
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-demo-service/dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: app
labels:
name: dubbo-demo-service
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
spec:
containers:
- name: dubbo-demo-service
image: harbor.phc-dow.com/app/dubbo-demo-service:master_202001311655
ports:
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-server.jar
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
注意：这里给JAR_BALL进行赋值，使用上面构建的镜像，进行创建pod，由于harbor里app是私有仓库，因此需要在k8s中先创建指定的namespace和secret
7.8 应用配置清单前准备工作
[root@kjdow7-21 ~]# kubectl create ns app
namespace/app created
[root@kjdow7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.phc-dow.com --docker-username=admin --docker-password=Harbor_kjdow1! -n app
secret/harbor created
注意：这里secret的名字要跟上面的dp.yaml中imagePullSecrets的name的值一样，secret的名字可以自定义，但是要引用对应的名字
7.9 应用资源配置清单
- 应用前
[root@kjdow7-11 zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@kjdow7-11 zookeeper]# bin/zkCli.sh -server localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
注：此时里面只有zookeeper
- 应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-demo-service/dp.yaml
deployment.extensions/dubbo-demo-service created
- 应用后
[zk: localhost:2181(CONNECTED) 0] ls /
[dubbo, zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /dubbo
[com.od.dubbotest.api.HelloService]
注：这里可以看到已经自动注册进来了。代码里面写死了注册地址是zk1.od.com，而我们的域名是phc-dow.com，因此可以在bind中添加一个od.com的配置文件，或者修改源码
8.交付dubbo-monitor到K8S集群
8.1 下载源码包
[root@kjdow7-200 ~]# wget https://github.com/Jeromefromcn/dubbo-monitor/archive/master.zip
[root@kjdow7-200 ~]# unzip master.zip
[root@kjdow7-200 ~]# mv dubbo-monitor-master /opt/src/dubbo-monitor
8.2 修改源码包
[root@kjdow7-200 ~]# vim /opt/src/dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties
dubbo.application.name=kjdow-monitor
dubbo.application.owner=kjdow
dubbo.registry.address=zookeeper://zk1.phc-dow.com:2181?backup=zk2.phc-dow.com:2181,zk3.phc-dow.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
8.3 制作配置文件
[root@kjdow7-200 ~]# mkdir /data/dockerfile/dubbo-monitor
[root@kjdow7-200 ~]# cp -r /opt/src/dubbo-monitor/* /data/dockerfile/dubbo-monitor/
[root@kjdow7-200 ~]# cd /data/dockerfile/dubbo-monitor/
[root@kjdow7-200 dubbo-monitor]# ls
Dockerfile dubbo-monitor-simple README.md
[root@kjdow7-200 dubbo-monitor]# vim ./dubbo-monitor-simple/bin/start.sh
if [ -n "$BITS" ]; then
JAVA_MEM_OPTS=" -server -Xmx128m -Xms128m -Xmn32m -XX:PermSize=16m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
JAVA_MEM_OPTS=" -server -Xms128m -Xmx128m -XX:PermSize=16m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi
echo -e "Starting the $SERVER_NAME ...\c"
exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1
注：脚本的59行和61行jvm进行调优
64行java启动脚本改成exec开头，并删除最后的&，让java前台执行，并接管这个shell的进程pid，并删除此行以下的所有内容
[root@kjdow7-200 dubbo-monitor]# docker build -t harbor.phc-dow.com/infra/dubbo-monitor:latest .
[root@kjdow7-200 ~]# docker push harbor.phc-dow.com/infra/dubbo-monitor:latest
8.4 准备k8s资源配置清单
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-monitor/dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
labels:
name: dubbo-monitor
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-monitor
template:
metadata:
labels:
app: dubbo-monitor
name: dubbo-monitor
spec:
containers:
- name: dubbo-monitor
image: harbor.phc-dow.com/infra/dubbo-monitor:latest
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-monitor/svc.yaml
kind: Service
apiVersion: v1
metadata:
name: dubbo-monitor
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: dubbo-monitor
clusterIP: None
type: ClusterIP
sessionAffinity: None
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-monitor/ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
spec:
rules:
- host: dubbo-monitor.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: dubbo-monitor
servicePort: 8080
8.5 应用资源配置清单前准备工作-解析域名
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010207 ; serial #serial值加一
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10 #添加此行配置
[root@kjdow7-11 ~]# systemctl restart named
[root@kjdow7-11 ~]# dig -t A dubbo-monitor.phc-dow.com @10.4.7.11 +short
10.4.7.10
8.6 应用k8s资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-monitor/dp.yaml
deployment.extensions/dubbo-monitor created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-monitor/svc.yaml
service/dubbo-monitor created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-monitor/ingress.yaml
ingress.extensions/dubbo-monitor created
8.7 打开网页进行访问
http://dubbo-monitor.phc-dow.com
9.交付dubbo服务的消费者集群到K8S
9.1 使用jenkins进行持续构建dubbo消费者镜像
依次填入/选择：
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: git@github.com:zizhufanqing/dubbo-demo-web.git
git_ver: master
add_tag: 202002011530
mvn_dir: ./
target_dir: ./dubbo-client/target
mvn_cmd: mvn clean package -Dmaven.test.skip=true
base_image: base/jre8:8u112
maven: 3.6.0-8u181
点击Build进行构建，等待构建完成。
注： 构建完成后在harbor的app中查看自动提交的镜像
9.2 准备资源配置清单
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-demo-consumer/dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
labels:
name: dubbo-demo-consumer
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-consumer
template:
metadata:
labels:
app: dubbo-demo-consumer
name: dubbo-demo-consumer
spec:
containers:
- name: dubbo-demo-consumer
image: harbor.phc.com/app/dubbo-demo-consumer:master_202002011530
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
value: dubbo-client.jar
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-demo-consumer/svc.yaml
kind: Service
apiVersion: v1
metadata:
name: dubbo-demo-consumer
namespace: app
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: dubbo-demo-consumer
clusterIP: None
type: ClusterIP
sessionAffinity: None
[root@kjdow7-200 ~]# vi /data/k8s-yaml/dubbo-demo-consumer/ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
spec:
rules:
- host: demo.phc-dow.com
http:
paths:
- path: /
backend:
serviceName: dubbo-demo-consumer
servicePort: 8080
9.3 应用资源配置清单前准备工作-解析域名
[root@kjdow7-11 ~]# vim /var/named/phc-dow.com.zone
$ORIGIN phc-dow.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.phc-dow.com. dnsadmin.phc-dow.com. (
2020010208 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.phc-dow.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
zk1 60 IN A 10.4.7.11
zk2 60 IN A 10.4.7.12
zk3 60 IN A 10.4.7.21
dubbo-monitor A 10.4.7.10
demo A 10.4.7.10
[root@kjdow7-11 ~]# systemctl restart named
[root@kjdow7-11 ~]# dig -t A demo.phc-dow.com @10.4.7.11 +short
10.4.7.10
9.4 应用资源配置清单
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-demo-consumer/dp.yaml
deployment.extensions/dubbo-demo-consumer created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-demo-consumer/svc.yaml
service/dubbo-demo-consumer created
[root@kjdow7-21 ~]# kubectl apply -f http://k8s-yaml.phc-dow.com/dubbo-demo-consumer/ingress.yaml
ingress.extensions/dubbo-demo-consumer created
9.5 验证
- 登录dubbo-monitor页面查看
http://dubbo-monitor.phc-dow.com/applications.html
在Applications已经能看到部署的三个
- 打开页面进行访问
http://demo.phc-dow.com/hello?name=wanglei
注：这里通过客户端调用hello的方法，客户端通过rpc协议调用服务端的hello方法，返回结果
三 、实战dubbo集群的日常维护
1.jenkins持续集成与持续部署
- 1.jenkins从git上拉取新代码，并按照上述方式进行构建
- 2.jenkins自动集成生成新的app的镜像
- 3.在k8s中修改对应的服务所使用的镜像，k8s自动进行滚动更新
2.服务的扩容与缩容
- 1.修改deployment中声明的pod的个数
- 2.应用新的配置清单
- 3.k8s自动进行扩容与缩容
