Helm部署应用实践
一、构建私有helm repo仓库
此处使用harbor仓库作为helm repo，使用内部统一的harbor仓库：https://harbor.xxx.com/
1、harbor仓库启用helmchart 服务
harbor之前没用启用helmchart服务，我们可以把服务停掉，然后将helm chart相关插件集成安装，最后启动即可。
[root@v-harbor harbor]# docker-compose down [+] Running 10/10 ⠿ Container nginx Removed 4.9s ⠿ Container registryctl Removed 10.9s ⠿ Container harbor-jobservice Removed 3.1s ⠿ Container harbor-portal Removed 2.7s ⠿ Container harbor-core Removed 1.5s ⠿ Container redis Removed 3.2s ⠿ Container harbor-db Removed 3.1s ⠿ Container registry Removed 2.9s ⠿ Container harbor-log Removed 10.6s ⠿ Network harbor_harbor Removed 0.3s [root@v-harbor harbor]# ./prepare --with-chartmuseum prepare base dir is set to /opt/harbor Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Clearing the configuration file: /config/nginx/nginx.conf Clearing the configuration file: /config/core/env Clearing the configuration file: /config/core/app.conf Clearing the configuration file: /config/registry/passwd Clearing the configuration file: /config/registry/config.yml Clearing the configuration file: /config/registry/root.crt Clearing the configuration file: /config/registryctl/env Clearing the configuration file: /config/registryctl/config.yml Clearing the configuration file: /config/db/env Clearing the configuration file: /config/jobservice/env Clearing the configuration file: /config/jobservice/config.yml Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml loaded secret from file: /data/secret/keys/secretkey Generated configuration file: /config/chartserver/env Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir [root@v-harbor harbor]# docker-compose up -d [+] Running 7/7 ⠿ chartmuseum Pulled 74.3s ⠿ d28e100d9e5c Already exists 0.0s ⠿ cbace8f09cd9 Pull complete 7.2s ⠿ 0a84ed3159d0 Pull complete 32.2s ⠿ d2bd3a22fe01 Pull complete 32.4s ⠿ 06dcd18a422c Pull complete 32.5s ⠿ b0386ab7aac1 Pull complete 35.2s [+] Running 12/12 ⠿ Network harbor_harbor Created 0.2s ⠿ Network harbor_harbor-chartmuseum Created 0.1s ⠿ Container harbor-log Started 2.8s ⠿ Container harbor-portal Started 5.5s ⠿ Container registry Started 5.1s ⠿ Container redis Started 5.5s ⠿ Container chartmuseum Started 3.6s ⠿ Container harbor-db Started 3.0s ⠿ Container registryctl Started 3.7s ⠿ Container harbor-core Started 6.0s ⠿ Container nginx Started 10.4s ⠿ Container harbor-jobservice Started 10.5s |
2、创建chartrepo
登录harbor页面，创建cusc-chartrepo。
二、添加helm-plugin插件
[root@k8s-node4 ~]# helm plugin install https://github.com/chartmuseum/helm-push [root@k8s-node4 ~]# helm plugin list NAME VERSION DESCRIPTION cm-push 0.10.3 Push chart package to ChartMuseum [root@k8s-node4 ~]# helm repo add --username admin --password xxxxxxxxxxx cusc-chartrepo https://harbor.xxx.com/chartrepo/cusc-chartrepo [root@k8s-node4 ~]# helm repo list NAME URL cusc-chartrepo https://harbor.xxx.com/chartrepo/cusc-chartrepo [root@k8s-node4 ~]# |
三、构建镜像推送chart repo
此处以http://172.23.xxx.xxx/devops/k8s-pod-restart-info-collector-master.git 项目工具为例。
1、构建k8s-pod-restart-info-collector工具镜像
[root@k8s-node4 k8s-pod-restart-info-collector-master]# docker build -t harbor.xxx.com/cusc-chartrepo/k8s-pod-restart-info-collector:v1.1.0 . Sending build context to Docker daemon 104MB Step 1/8 : FROM golang:1.17.5-alpine3.15 AS builder ---> d8bf44a3f6b4 Step 2/8 : COPY go.* / ---> Using cache ---> 4e343b6e56fa Step 3/8 : RUN go env -w GO111MODULE=on && go env -w GOPROXY=https://goproxy.cn,direct && go mod download ---> Running in b82a31e3a99f Removing intermediate container b82a31e3a99f ---> d44f99ea374c Step 4/8 : COPY *.go / ---> 9c8f3583187d Step 5/8 : RUN CGO_ENABLED=0 go build -o /k8s-pod-restart-info-collector / ---> Running in 9846447d7363 Removing intermediate container 9846447d7363 ---> e93f1fda7bf8 Step 6/8 : FROM alpine:3.15 3.15: Pulling from library/alpine 9621f1afde84: Downloading 3.15: Pulling from library/alpine 59bf1c3509f3: Already exists 3.15: Pulling from library/alpine 9621f1afde84: Pull complete Digest: sha256:cf34c62ee8eb3fe8aa24c1fab45d7e9d12768d945c3f5a6fd6a63d901e898479 Status: Downloaded newer image for alpine:3.15 ---> c4fc93816858 Step 7/8 : COPY --from=builder /k8s-pod-restart-info-collector /k8s-pod-restart-info-collector ---> d5dc28c81380 Step 8/8 : CMD ["/k8s-pod-restart-info-collector"] ---> Running in 747e37f48399 Removing intermediate container 747e37f48399 ---> 9e37776bbf4c Successfully built 9e37776bbf4c Successfully tagged harbor.xxx.com/cusc-chartrepo/k8s-pod-restart-info-collector:v1.1.0 [root@k8s-node4 k8s-pod-restart-info-collector-master]# docker images|grep kubecollect harbor.xxx.com/cusc-chartrepo/k8s-pod-restart-info-collector v1.1.0 9e37776bbf4c 3 minutes ago 57.6MB [root@k8s-node4 k8s-pod-restart-info-collector-master]# docker push harbor.xxx.com/cusc-chartrepo/k8s-pod-restart-info-collector:v1.1.0 The push refers to repository [harbor.xxx.com/cusc-chartrepo/k8s-pod-restart-info-collector] 5338088df6f5: Layer already exists 34d5ebaa5410: Layer already exists v1.1.0: digest: sha256:d766d555031df324885d9ce0b01f1913d191fe80d2b5a6502293d49c2e17cc23 size: 740 |
2、修改项目Helm chart 内容
[root@k8s-node4 k8s-pod-restart-info-collector-master]# cat helm/values.yaml ..... image: #repository: devopsairwallex/k8s-pod-restart-info-collector repository: cusc-chartrepo/k8s-pod-restart-info-collector tag: "v1.1.0" ... |
3、helm package 打包
[root@k8s-node4 k8s-pod-restart-info-collector-master]# helm package helm [root@k8s-node4 k8s-pod-restart-info-collector-master]# helm cm-push k8s-pod-restart-info-collector-0.1.0.tgz cusc-chartrepo Error: fork/exec /root/.local/share/helm/plugins/helm-push/bin/helm-cm-push: no such file or directory |
4、helm cm-push 无法运行报错
[root@k8s-node4 k8s-pod-restart-info-collector-master]# cd /root/.local/share/helm/plugins/helm-push [root@k8s-node4 k8s-pod-restart-info-collector-master]# make build ....... k8s.io/api/scheduling/v1alpha1 k8s.io/api/scheduling/v1beta1 k8s.io/api/storage/v1 k8s.io/api/storage/v1alpha1 k8s.io/api/storage/v1beta1 k8s.io/client-go/kubernetes/scheme k8s.io/client-go/discovery helm.sh/helm/v3/pkg/chartutil k8s.io/client-go/discovery/cached/disk k8s.io/client-go/restmapper k8s.io/cli-runtime/pkg/resource k8s.io/cli-runtime/pkg/genericclioptions helm.sh/helm/v3/pkg/cli helm.sh/helm/v3/pkg/plugin helm.sh/helm/v3/pkg/getter helm.sh/helm/v3/pkg/repo github.com/chartmuseum/helm-push/pkg/helm helm.sh/helm/v3/internal/resolver helm.sh/helm/v3/pkg/downloader command-line-arguments [root@k8s-node4 helm-push]# ll bin/ total 37716 drwxr-xr-x 3 root root 4096 Feb 3 13:02 darwin -rwxr-xr-x 1 root root 38607856 Feb 3 13:02 helm-cm-push drwxr-xr-x 3 root root 4096 Feb 3 13:01 linux drwxr-xr-x 3 root root 4096 Feb 3 13:03 windows [root@k8s-node4 k8s-pod-restart-info-collector-master]# helm cm-push k8s-pod-restart-info-collector-0.1.0.tgz cusc-chartrepo Error: fork/exec /root/.local/share/helm/plugins/helm-push/bin/helm-cm-push: exec format error #使用编译之后linux子文件夹的命令覆盖bin目录下的 [root@k8s-node4 k8s-pod-restart-info-collector-master]# cp /root/.local/share/helm/plugins/helm-push/bin/linux/amd64/helm-cm-push /root/.local/share/helm/plugins/helm-push/bin |
5、helm push chart包
[root@k8s-node4 k8s-pod-restart-info-collector-master]# helm cm-push k8s-pod-restart-info-collector-0.1.0.tgz cusc-chartrepo Pushing k8s-pod-restart-info-collector-0.1.0.tgz to cusc-chartrepo... Done. |
6、harbor portal查看推送结果
7、helm install 部署应用
[root@k8s-node4 k8s-pod-restart-info-collector-master]# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "cusc-chartrepo" chart repository [root@k8s-node4 k8s-pod-restart-info-collector-master]# helm install k8s-pod-restart-info-collector cusc-chartrepo/k8s-pod-restart-info-collector NAME: k8s-pod-restart-info-collector LAST DEPLOYED: Fri Feb 3 13:23:26 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None [root@k8s-node4 k8s-pod-restart-info-collector-master]# helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION deis-workflow-1 default 1 2023-01-27 16:26:04.861323863 +0800 CST deployed deis-workflow-0.1.0 1.16.0 k8s-pod-restart-info-collector default 1 2023-02-03 13:23:26.859389325 +0800 CST deployed k8s-pod-restart-info-collector-0.1.0 1.16.0 redis-demo default 1 2023-01-28 09:34:53.244617337 +0800 CST deployed redis-10.5.7 5.0.7 [root@k8s-node4 k8s-pod-restart-info-collector-master]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES k8s-pod-restart-info-collector-559989cf9c-46ppj 1/1 Running 0 20s <none> k8s-node11 <none> <none> redis-demo-master-0 1/1 Running 0 6d4h 10.42.2.105 k8s-node1 <none> <none> |
