交付dubbo 微服务到k8s 集群
dubbo 微服务结构简介
上图中,蓝色的表示与业务有交互,绿色的表示只对Dubbo内部交互。上述图所描述的调用流程如下:
- 服务提供方发布服务到服务注册中心;
- 服务消费方从服务注册中心订阅服务;
- 服务消费方调用已经注册的可用服务
1. 部署maven 并改成国内源
在 7.200 上: tar xf apache-maven-3.6.0-bin.tar.gz -C /data/nfs-volume/jenkins_home/maven-3.6.3
修改maven 仓库为阿里云仓库
<mirrors> <mirror> <id>nexus-aliyun</id> <mirrorOf>*</mirrorOf> <name>Nexus aliyun</name> <url>http://maven.aliyun.com/nexus/content/groups/public</url> </mirror> </mirrors> #修改成阿里云镜像源
2. 准备 dubbo 底层镜像
docker pull docker.io/stanleyws/jre8:8u112 #这个是jdk8的原始镜像 docker tag docker.io/stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112 docker push harbor.od.com/public/jre:8u112 cat <<EOF >/data/dockerfile/jre8/Dockerfile FROM harbor.od.com/public/jre:8u112 RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo 'Asia/Shanghai' >/etc/timezone ADD config.yml /opt/prom/config.yml ADD jmx_javaagent-0.3.1.jar /opt/prom/ WORKDIR /opt/project_dir ADD entrypoint.sh /entrypoint.sh CMD ["/entrypoint.sh"] EOF cat <<EOF>config.yml --- rules: - pattern: '.*' EOF #下载java-agent 用来给普罗米修斯监控 wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar vim entrypoint.sh #!/bin/sh M_OPTS="-Duser.timezone=Asia/Shanghai-javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname-i):${M_PORT:-"12346"}:/opt/prom/config.yml" C_OPTS=${C_OPTS} JAR_BALL=${JAR_BALL} exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL} chmod +x entrypoint.sh #登录harbor 创建base 仓库,权限是public ]# docker build . -t harbor.od.com/base/jre8:8u112 && docker push harbor.od.com/base/jre8:8u112
3.交付dubbo server 到k8s 集群
在 jenkins 中新建 一个pipline的项目,名字自定义,选择参数构建
#然后到Jenkins 新建pipline 工程 ,在general , 选择 this project isparameterized。
选择string parameter 。勾选 trim is string ,分别是:
app_name image_name git_repo git_ver add_tag target_dir (./target) mvn_dir (./)
mvn_cmd (mvn clean package -Dmaven.test.skip=true)
# Choice parameter
base_image (base/jre8:8u112)
maven (3.6.3) #执行编译的软件包版本,要根据实际mvn 写。
以此填入 自定义的参数,我这里放个填好后的图,就是 app_name image_name 那一列:
然后在jenkins pipeline 下面填入pipeline 脚本
pipeline { agent any stages { stage('pull') { //get project code from repo steps { sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}" } } stage('build') { //exec mvn cmd steps { sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}" } } stage('package') { //move jar file into project_dir steps { sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir" } } stage('image') { //build image and push to registry steps { writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image} ADD ${params.target_dir}/project_dir /opt/project_dir"""#这一步是把 FROM harbor.od.com/${params.base_image} #ADD ${params.target_dir}/project_dir /opt/project_dir 具体内容写到jenkins 里面dockerfile 里面 然后在jenkins 里面构建有了新的jar 包的镜像 并推送到 #habor 上,镜像的名字后缀是刚才的add_tag 也就是时间 sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}" } } } }
保存,之后再项目项目里面选择参数构建:
具体的参数如下:
git_repo 要填dubbo-server 的git 仓库地址 ,git_ver 就是代码分支,这里拉取的 apolo 分支的代码。
add_tag 建议用时间日期 base_image 就是上一步构建的dockerfile 基础镜像 maven 就是之前部署的,名字要和 7.200 /data/nfs-volume/jenkins_home/ 里面 maven 的目录名字一致,要不jenkins 构建时候会找不到,类似如下报错
+ /var/jenkins_home/maven-3.6.3/bin/mvn clean package -Dmaven.test.skip=true /var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: 1: /var/jenkins_home/workspace/dubboproject1@tmp/durable-700da7c2/script.sh: /var/jenkins_home/maven-3.6.3/bin/mvn: not found
然后点击构建,成功后结果如下,这时候登陆harbor 就会发现多了 一个
app/dubbo-demo-service:apollo_20200517_1600 的镜像了!
4. 准备dubbo server 的yaml 文件
[root@hdss7-200 dubbo-server]# cat dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-demo-service namespace: app labels: name: dubbo-demo-service spec: replicas: 1 selector: matchLabels: name: dubbo-demo-service template: metadata: labels: app: dubbo-demo-service name: dubbo-demo-service spec: containers: - name: dubbo-demo-service image: harbor.od.com/app/dubbo-demo-service:apollo_20200517_1600 #就是刚才新构建的镜像 ports: - containerPort: 20880 protocol: TCP env: - name: JAR_BALL value: dubbo-server.jar - name: C_OPTS value: -Denv=dev -Dapollo.meta=http://config.od.com #这里结合apolo 配置中心来做的 imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600
5 交付dubbo-monitor 到k8s 集群
#先下载dubbo monitor 包
https://github.com/Jeromefromcn/dubbo-monitor#dubbo-monitor ,下载后解压,
修改配置
cat /opt/src/dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties dubbo.container=log4j,spring,registry,jetty dubbo.application.name=simple-monitor dubbo.application.owner= dubbo.registry.address=zookeeper://10.4.7.11:2181 # 改成实际zk 地址 dubbo.protocol.port=20880 dubbo.jetty.port=8080 dubbo.jetty.directory=/dubbo-monitor-simple/monitor dubbo.statistics.directory=/dubbo-monitor-simple/statistics dubbo.charts.directory=/dubbo-monitor-simple/charts dubbo.log4j.file=logs/dubbo-monitor.log
准备dockerfile
vim /data/dockerfile/dubbo-monitor/Dockerfile FROM jeromefromcn/docker-alpine-java-bash MAINTAINER Jerome Jiang COPY dubbo-monitor-simple/ /dubbo-monitor-simple/ #dubbo-monitor-simple/ 就是dubbo-monitor 包里面的文件夹 CMD /dubbo-monitor-simple/bin/start.sh
docker build . -t harbor.od.com/infra/dubbo-monitor:latest #构建镜像
因为github 上 start.sh 脚本在容器环境中启动有问题 需要修改,下面是修改后的。
[root@hdss7-200 dubbo-monitor]# cat dubbo-monitor-simple/bin/start.sh #!/bin/bash sed -e "s/{ZOOKEEPER_ADDRESS}/$ZOOKEEPER_ADDRESS/g" /dubbo-monitor-simple/conf/dubbo_origin.properties > /dubbo-monitor-simple/conf/dubbo.properties cd `dirname $0` BIN_DIR=`pwd` cd .. DEPLOY_DIR=`pwd` CONF_DIR=$DEPLOY_DIR/conf SERVER_NAME=`sed '/dubbo.application.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` SERVER_PROTOCOL=`sed '/dubbo.protocol.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` SERVER_PORT=`sed '/dubbo.protocol.port/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` LOGS_FILE=`sed '/dubbo.log4j.file/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'` if [ -z "$SERVER_NAME" ]; then SERVER_NAME=`hostname` fi PIDS=`ps -f | grep java | grep "$CONF_DIR" |awk '{print $2}'` if [ -n "$PIDS" ]; then echo "ERROR: The $SERVER_NAME already started!" echo "PID: $PIDS" exit 1 fi if [ -n "$SERVER_PORT" ]; then SERVER_PORT_COUNT=`netstat -tln | grep $SERVER_PORT | wc -l` if [ $SERVER_PORT_COUNT -gt 0 ]; then echo "ERROR: The $SERVER_NAME port $SERVER_PORT already used!" exit 1 fi fi LOGS_DIR="" if [ -n "$LOGS_FILE" ]; then LOGS_DIR=`dirname $LOGS_FILE` else LOGS_DIR=$DEPLOY_DIR/logs fi if [ ! -d $LOGS_DIR ]; then mkdir $LOGS_DIR fi STDOUT_FILE=$LOGS_DIR/stdout.log LIB_DIR=$DEPLOY_DIR/lib LIB_JARS=`ls $LIB_DIR|grep .jar|awk '{print "'$LIB_DIR'/"$0}'|tr "\n" ":"` JAVA_OPTS=" -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true " JAVA_DEBUG_OPTS="" if [ "$1" = "debug" ]; then JAVA_DEBUG_OPTS=" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n " fi JAVA_JMX_OPTS="" if [ "$1" = "jmx" ]; then JAVA_JMX_OPTS=" -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false " fi JAVA_MEM_OPTS="" BITS=`java -version 2>&1 | grep -i 64-bit` if [ -n "$BITS" ]; then JAVA_MEM_OPTS=" -server -Xmx128m -Xms128m -Xmn32m -XX:PermSize=32m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 " else JAVA_MEM_OPTS=" -server -Xms128m -Xmx128m -XX:PermSize=16m -XX:SurvivorRatio=2 -XX:+UseParallelGC " fi echo -e "Starting the $SERVER_NAME ...\c" exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1
6. 准备dobbo monitor 文件
cat deployment.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-monitor namespace: infra labels: name: dubbo-monitor spec: replicas: 1 selector: matchLabels: name: dubbo-monitor template: metadata: labels: app: dubbo-monitor name: dubbo-monitor spec: containers: - name: dubbo-monitor image: harbor.od.com/infra/dubbo-monitor:latest ports: - containerPort: 8080 protocol: TCP - containerPort: 20880 protocol: TCP imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600
[root@hdss7-200 dubbo-monitor]# cat cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: dubbo-monitor-cm namespace: infra data: dubbo.properties: | dubbo.container=log4j,spring,registry,jetty dubbo.application.name=simple-monitor dubbo.application.owner=simplemonitor dubbo.registry.address=zookeeper://10.4.7.11:2181 #zk 的地址一定要和代码里面保持一致! dubbo.protocol.port=20880 dubbo.jetty.port=8080 dubbo.jetty.directory=/dubbo-monitor-simple/monitor dubbo.charts.directory=/dubbo-monitor-simple/charts dubbo.statistics.directory=/dubbo-monitor-simple/statistics dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log dubbo.log4j.level=WARN [root@hdss7-200 dubbo-monitor]# cat ingress.yaml kind: Ingress apiVersion: extensions/v1beta1 metadata: name: dubbo-monitor namespace: infra spec: rules: - host: dubbo-monitor.od.com http: paths: - path: / backend: serviceName: dubbo-monitor servicePort: 8080 [root@hdss7-200 dubbo-monitor]# cat svc.yaml kind: Service apiVersion: v1 metadata: name: dubbo-monitor namespace: infra spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: dubbo-monitor
添加A 记录 dubbo-monitor.od.com IN A 10.4.7.10 测试访问 http://dubbo-monitor.od.com
7.交付 dubbo服务消费者到K8S 集群
jenkins 上新构建一个pipeline 项目,添加的参数和dubbo-serve 一样,,可以直接把dubbo-server 复制过来,然后填入参数 开始构建,参数如下图:
看下 console output
Step 1/2 : FROM harbor.od.com/base/jre8:8u112 ---> 621a01f7f85a Step 2/2 : ADD ./dubbo-client/target/project_dir /opt/project_dir ---> 7d5577992ecd Successfully built 7d5577992ecd Successfully tagged harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 + docker push harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 The push refers to repository [harbor.od.com/app/dubbo-demo-consumer] 会看到harbor 上多了 app/dubbo-demo-consumer:apollo_20200517_1650 的镜像
8. 准备consumer 的yaml 文件
# dubbo-consumer 资源配置清单 dp.yaml kind: Deployment apiVersion: extensions/v1beta1 metadata: name: dubbo-demo-consumer namespace: app labels: name: dubbo-demo-consumer spec: replicas: 1 selector: matchLabels: name: dubbo-demo-consumer template: metadata: labels: app: dubbo-demo-consumer name: dubbo-demo-consumer spec: containers: - name: dubbo-demo-consumer image: harbor.od.com/app/dubbo-demo-consumer:apollo_20200517_1650 #就是刚才新构建的consumer 镜像 ports: - containerPort: 8080 protocol: TCP - containerPort: 20880 protocol: TCP env: - name: JAR_BALL value: dubbo-client.jar imagePullPolicy: IfNotPresent imagePullSecrets: - name: harbor restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 cat svc.yaml kind: Service apiVersion: v1 metadata: name: dubbo-demo-consumer namespace: app spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: dubbo-demo-consumer cat ingress.yaml kind: Ingress apiVersion: extensions/v1beta1 metadata: name: dubbo-demo-consumer namespace: app spec: rules: - host: demo.od.com http: paths: - path: / backend: serviceName: dubbo-demo-consumer servicePort: 8080
kubectl apply -f 这几个文件,然后 访问 http://demo.od.com/hello?name=hello 看是否有返回结果,有返回就是正常的。
登录monitor 查看: 可以看到 server 和 consumer 都有了。