Kubernetes K8S之存储Secret详解
K8S之存储Secret概述与类型说明，并详解常用Secret示例
主机配置规划
| 服务器名称(hostname) | 系统版本 | 配置 | 内网IP | 外网IP(模拟) |
|---|---|---|---|---|
| k8s-master | CentOS7.7 | 2C/4G/20G | 172.16.1.110 | 10.0.0.110 |
| k8s-node01 | CentOS7.7 | 2C/4G/20G | 172.16.1.111 | 10.0.0.111 |
| k8s-node02 | CentOS7.7 | 2C/4G/20G | 172.16.1.112 | 10.0.0.112 |
Secret概述
Secret解决了密码、token、秘钥等敏感数据的配置问题，而不需要把这些敏感数据暴露到镜像或者Pod Spec中。Secret可以以Volume或者环境变量的方式使用。
用户可以创建 secret，同时系统也创建了一些 secret。
要使用 secret，pod 需要引用 secret。Pod 可以用两种方式使用 secret：作为 volume 中的文件被挂载到 pod 中的一个或者多个容器里，或者当 kubelet 为 pod 拉取镜像时使用。
Secret类型
- Service Account：用来访问Kubernetes API，由Kubernetes自动创建，并且会自动挂载到Pod的 /run/secrets/kubernetes.io/serviceaccount 目录中。
- Opaque：base64编码格式的Secret，用来存储密码、秘钥等。
- kubernetes.io/dockerconfigjson：用来存储私有docker registry的认证信息。
Service Account
通过kube-proxy查看
1 [root@k8s-master ~]# kubectl get pod -A | grep 'kube-proxy' 2 kube-system kube-proxy-6bfh7 1/1 Running 12 7d3h 3 kube-system kube-proxy-6vfkf 1/1 Running 11 7d3h 4 kube-system kube-proxy-bvl9n 1/1 Running 11 7d3h 5 [root@k8s-master ~]# 6 [root@k8s-master ~]# kubectl exec -it -n kube-system kube-proxy-6bfh7 -- /bin/sh 7 # ls -l /run/secrets/kubernetes.io/serviceaccount 8 total 0 9 lrwxrwxrwx 1 root root 13 Jun 8 13:39 ca.crt -> ..data/ca.crt 10 lrwxrwxrwx 1 root root 16 Jun 8 13:39 namespace -> ..data/namespace 11 lrwxrwxrwx 1 root root 12 Jun 8 13:39 token -> ..data/token
Opaque Secret
创建secret
手动加密，基于base64加密
1 [root@k8s-master ~]# echo -n 'admin' | base64 2 YWRtaW4= 3 [root@k8s-master ~]# echo -n '1f2d1e2e67df' | base64 4 MWYyZDFlMmU2N2Rm
yaml文件
1 [root@k8s-master secret]# pwd 2 /root/k8s_practice/secret 3 [root@k8s-master secret]# cat secret.yaml 4 apiVersion: v1 5 kind: Secret 6 metadata: 7 name: mysecret 8 type: Opaque 9 data: 10 username: YWRtaW4= 11 password: MWYyZDFlMmU2N2Rm
或者通过如下命令行创建【secret名称故意设置不一样，以方便查看对比】，生成secret后会自动加密，而非明文存储。
kubectl create secret generic db-user-pass --from-literal=username=admin --from-literal=password=1f2d1e2e67df
生成secret，并查看状态
1 [root@k8s-master secret]# kubectl apply -f secret.yaml 2 secret/mysecret created 3 [root@k8s-master secret]# 4 [root@k8s-master secret]# kubectl get secret ### 查看默认名称空间的secret简要信息 5 NAME TYPE DATA AGE 6 basic-auth Opaque 1 2d12h 7 default-token-v48g4 kubernetes.io/service-account-token 3 27d 8 mysecret Opaque 2 23s ### 可见已创建 9 tls-secret kubernetes.io/tls 2 3d2h 10 [root@k8s-master secret]# 11 [root@k8s-master secret]# kubectl get secret mysecret -o yaml ### 查看mysecret详细信息 12 apiVersion: v1 13 data: 14 password: MWYyZDFlMmU2N2Rm 15 username: YWRtaW4= 16 kind: Secret 17 metadata: 18 annotations: 19 kubectl.kubernetes.io/last-applied-configuration: | 20 {"apiVersion":"v1","data":{"password":"MWYyZDFlMmU2N2Rm","username":"YWRtaW4="},"kind":"Secret","metadata":{"annotations":{},"name":"mysecret","namespace":"default"},"type":"Opaque"} 21 creationTimestamp: "2020-06-08T14:08:59Z" 22 name: mysecret 23 namespace: default 24 resourceVersion: "987419" 25 selfLink: /api/v1/namespaces/default/secrets/mysecret 26 uid: 27b58929-71c4-495b-99a5-0d411910a529 27 type: Opaque 28 [root@k8s-master secret]# 29 [root@k8s-master secret]# kubectl describe secret mysecret ### 查看描述信息 30 Name: mysecret 31 Namespace: default 32 Labels: <none> 33 Annotations: 34 Type: Opaque 35 36 Data 37 ==== 38 password: 12 bytes 39 username: 5 bytes
将Secret挂载到Volume中
yaml文件
1 [root@k8s-master secret]# pwd 2 /root/k8s_practice/secret 3 [root@k8s-master secret]# cat pod_secret_volume.yaml 4 apiVersion: v1 5 kind: Pod 6 metadata: 7 name: pod-secret-volume 8 spec: 9 containers: 10 - name: myapp 11 image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1 12 volumeMounts: 13 - name: secret-volume 14 mountPath: /etc/secret 15 readOnly: true 16 volumes: 17 - name: secret-volume 18 secret: 19 secretName: mysecret
启动pod并查看状态
1 [root@k8s-master secret]# kubectl apply -f pod_secret_volume.yaml 2 pod/pod-secret-volume created 3 [root@k8s-master secret]# 4 [root@k8s-master secret]# kubectl get pod -o wide 5 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 6 pod-secret-volume 1/1 Running 0 16s 10.244.2.159 k8s-node02 <none> <none>
查看secret信息
1 [root@k8s-master secret]# kubectl exec -it pod-secret-volume -- /bin/sh 2 / # ls /etc/secret 3 password username 4 / # 5 / # cat /etc/secret/username 6 admin/ # 7 / # 8 / # cat /etc/secret/password 9 1f2d1e2e67df/ #
由上可见，在pod中的secret信息实际已经被解密。
将Secret导入到环境变量中
yaml文件
1 [root@k8s-master secret]# pwd 2 /root/k8s_practice/secret 3 [root@k8s-master secret]# cat pod_secret_env.yaml 4 apiVersion: v1 5 kind: Pod 6 metadata: 7 name: pod-secret-env 8 spec: 9 containers: 10 - name: myapp 11 image: registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1 12 env: 13 - name: SECRET_USERNAME 14 valueFrom: 15 secretKeyRef: 16 name: mysecret 17 key: username 18 - name: SECRET_PASSWORD 19 valueFrom: 20 secretKeyRef: 21 name: mysecret 22 key: password 23 restartPolicy: Never
启动pod并查看状态
1 [root@k8s-master secret]# kubectl apply -f pod_secret_env.yaml 2 pod/pod-secret-env created 3 [root@k8s-master secret]# 4 [root@k8s-master secret]# kubectl get pod -o wide 5 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 6 pod-secret-env 1/1 Running 0 6s 10.244.2.160 k8s-node02 <none> <none>
查看secret信息
1 [root@k8s-master secret]# kubectl exec -it pod-secret-env -- /bin/sh 2 / # env 3 ……………… 4 HOME=/root 5 SECRET_PASSWORD=1f2d1e2e67df ### secret信息 6 MYAPP_SVC_PORT_80_TCP=tcp://10.98.57.156:80 7 TERM=xterm 8 NGINX_VERSION=1.12.2 9 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 10 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 11 KUBERNETES_PORT_443_TCP_PORT=443 12 KUBERNETES_PORT_443_TCP_PROTO=tcp 13 MYAPP_SVC_SERVICE_HOST=10.98.57.156 14 SECRET_USERNAME=admin ### secret信息 15 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 16 ………………
由上可见，在pod中的secret信息实际已经被解密。
docker-registry Secret
harbor镜像仓库
首先使用harbor搭建镜像仓库，搭建部署过程参考：「Harbor企业级私有Docker镜像仓库部署」
harbor部分配置文件信息
1 [root@k8s-master harbor]# pwd 2 /root/App/harbor 3 [root@k8s-master harbor]# vim harbor.yml 4 # Configuration file of Harbor 5 hostname: 172.16.1.110 6 7 # http related config 8 http: 9 # port for http, default is 80. If https enabled, this port will redirect to https port 10 port: 5000 11 12 # https related config 13 https: 14 # https port for harbor, default is 443 15 port: 443 16 # The path of cert and key files for nginx 17 certificate: /etc/harbor/cert/httpd.crt 18 private_key: /etc/harbor/cert/httpd.key 19 harbor_admin_password: Harbor12345
启动harbor后客户端http设置
集群所有机器都要操作
1 [root@k8s-master ~]# vim /etc/docker/daemon.json 2 { 3 "exec-opts": ["native.cgroupdriver=systemd"], 4 "log-driver": "json-file", 5 "log-opts": { 6 "max-size": "100m" 7 }, 8 "insecure-registries": ["172.16.1.110:5000"] 9 } 10 [root@k8s-master ~]# 11 [root@k8s-master ~]# systemctl restart docker # 重启docker服务
添加了 “insecure-registries”: [“172.16.1.110:5000”] 这行，其中172.16.1.110为内网IP地址。该文件必须符合 json 规范，否则 Docker 将不能启动。
如果在Harbor所在的机器重启了docker服务，记得要重新启动Harbor。
创建「私有」仓库
镜像上传
1 docker pull registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1 2 docker tag registry.cn-beijing.aliyuncs.com/google_registry/myapp:v1 172.16.1.110:5000/k8s-secret/myapp:v1 3 # 登录 4 docker login 172.16.1.110:5000 -u admin -p Harbor12345 5 # 上传 6 docker push 172.16.1.110:5000/k8s-secret/myapp:v1
退出登录
之后在操作机上退出harbor登录，便于后面演示
1 ### 退出harbor登录 2 [root@k8s-node02 ~]# docker logout 172.16.1.110:5000 3 Removing login credentials for 172.16.1.110:5000 4 ### 拉取失败，需要先登录。表明完成准备工作 5 [root@k8s-master secret]# docker pull 172.16.1.110:5000/k8s-secret/myapp:v1 6 Error response from daemon: pull access denied for 172.16.1.110:5000/k8s-secret/myapp, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
pod直接下载镜像
在yaml文件中指定image后，直接启动pod
1 [root@k8s-master secret]# pwd 2 /root/k8s_practice/secret 3 [root@k8s-master secret]# cat pod_secret_registry.yaml 4 apiVersion: v1 5 kind: Pod 6 metadata: 7 name: pod-secret-registry 8 spec: 9 containers: 10 - name: myapp 11 image: 172.16.1.110:5000/k8s-secret/myapp:v1
启动pod并查看状态
1 [root@k8s-master secret]# kubectl apply -f pod_secret_registry.yaml 2 pod/pod-secret-registry created 3 [root@k8s-master secret]# 4 [root@k8s-master secret]# kubectl get pod -o wide ### 可见镜像下载失败 5 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 6 pod-secret-registry 0/1 ImagePullBackOff 0 7s 10.244.2.161 k8s-node02 <none> <none> 7 [root@k8s-master secret]# 8 [root@k8s-master secret]# kubectl describe pod pod-secret-registry ### 查看pod详情 9 Name: pod-secret-registry 10 Namespace: default 11 Priority: 0 12 Node: k8s-node02/172.16.1.112 13 Start Time: Mon, 08 Jun 2020 23:59:07 +0800 14 Labels: <none> 15 Annotations: kubectl.kubernetes.io/last-applied-configuration: 16 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"pod-secret-registry","namespace":"default"},"spec":{"containers":[{"i... 17 Status: Pending 18 IP: 10.244.2.161 19 IPs: 20 IP: 10.244.2.161 21 Containers: 22 myapp: 23 Container ID: 24 Image: 172.16.1.110:5000/k8s-secret/myapp:v1 25 Image ID: 26 ……………… 27 Events: 28 Type Reason Age From Message 29 ---- ------ ---- ---- ------- 30 Normal Scheduled 23s default-scheduler Successfully assigned default/pod-secret-registry to k8s-node02 31 Normal BackOff 19s (x2 over 20s) kubelet, k8s-node02 Back-off pulling image "172.16.1.110:5000/k8s-secret/myapp:v1" 32 Warning Failed 19s (x2 over 20s) kubelet, k8s-node02 Error: ImagePullBackOff 33 Normal Pulling 9s (x2 over 21s) kubelet, k8s-node02 Pulling image "172.16.1.110:5000/k8s-secret/myapp:v1" 34 Warning Failed 9s (x2 over 21s) kubelet, k8s-node02 Failed to pull image "172.16.1.110:5000/k8s-secret/myapp:v1": rpc error: code = Unknown desc = Error response from daemon: pull access denied for 172.16.1.110:5000/k8s-secret/myapp, repository does not exist or may require 'docker login': denied: requested access to the resource is denied 35 Warning Failed 9s (x2 over 21s) kubelet, k8s-node02 Error: ErrImagePull 36 [root@k8s-master secret]# 37 [root@k8s-master secret]# kubectl delete -f pod_secret_registry.yaml
可见拉取私有镜像失败。
pod通过Secret下载镜像
通过命令行创建Secret，并查看其描述信息
1 [root@k8s-master secret]# kubectl create secret docker-registry myregistrysecret --docker-server='172.16.1.110:5000' --docker-username='admin' --docker-password='Harbor12345' 2 secret/myregistrysecret created 3 [root@k8s-master secret]# 4 [root@k8s-master secret]# kubectl get secret 5 NAME TYPE DATA AGE 6 basic-auth Opaque 1 2d14h 7 default-token-v48g4 kubernetes.io/service-account-token 3 27d 8 myregistrysecret kubernetes.io/dockerconfigjson 1 8s # 刚刚创建的 9 mysecret Opaque 2 118m 10 tls-secret kubernetes.io/tls 2 3d4h 11 [root@k8s-master secret]# 12 [root@k8s-master secret]# kubectl get secret myregistrysecret -o yaml ### 查看详细信息 13 apiVersion: v1 14 data: 15 .dockerconfigjson: eyJhdXRocyI6eyIxMC4wLjAuMTEwOjUwMDAiOnsidXNlcm5hbWUiOiJhZG1pbiIsInBhc3N3b3JkIjoiSGFyYm9yMTIzNDUiLCJhdXRoIjoiWVdSdGFXNDZTR0Z5WW05eU1USXpORFU9In19fQ== 16 kind: Secret 17 metadata: 18 creationTimestamp: "2020-06-08T16:07:32Z" 19 name: myregistrysecret 20 namespace: default 21 resourceVersion: "1004582" 22 selfLink: /api/v1/namespaces/default/secrets/myregistrysecret 23 uid: b95f4386-64bc-4ba3-b43a-08afb1c1eb9d 24 type: kubernetes.io/dockerconfigjson 25 [root@k8s-master secret]# 26 [root@k8s-master secret]# kubectl describe secret myregistrysecret ### 查看描述信息 27 Name: myregistrysecret 28 Namespace: default 29 Labels: <none> 30 Annotations: <none> 31 32 Type: kubernetes.io/dockerconfigjson 33 34 Data 35 ==== 36 .dockerconfigjson: 109 bytes
修改之前的yaml文件
1 [root@k8s-master secret]# cat pod_secret_registry.yaml 2 apiVersion: v1 3 kind: Pod 4 metadata: 5 name: pod-secret-registry 6 spec: 7 containers: 8 - name: myapp 9 image: 172.16.1.110:5000/k8s-secret/myapp:v1 10 imagePullSecrets: 11 - name: myregistrysecret
启动pod并查看状态
1 [root@k8s-master secret]# kubectl apply -f pod_secret_registry.yaml 2 pod/pod-secret-registry created 3 [root@k8s-master secret]# 4 [root@k8s-master secret]# kubectl get pod -o wide 5 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 6 pod-secret-registry 1/1 Running 0 8s 10.244.2.162 k8s-node02 <none> <none> 7 [root@k8s-master secret]# 8 [root@k8s-master secret]# kubectl describe pod pod-secret-registry 9 Name: pod-secret-registry 10 Namespace: default 11 Priority: 0 12 Node: k8s-node02/172.16.1.112 13 Start Time: Tue, 09 Jun 2020 00:22:40 +0800 14 Labels: <none> 15 Annotations: kubectl.kubernetes.io/last-applied-configuration: 16 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"pod-secret-registry","namespace":"default"},"spec":{"containers":[{"i... 17 Status: Running 18 IP: 10.244.2.162 19 IPs: 20 IP: 10.244.2.162 21 Containers: 22 myapp: 23 Container ID: docker://ef4d42f1f1616a44c2a6c0a5a71333b27f46dfe76eb392962813a28d69150c00 24 Image: 172.16.1.110:5000/k8s-secret/myapp:v1 25 Image ID: docker-pullable://172.16.1.110:5000/k8s-secret/myapp@sha256:9eeca44ba2d410e54fccc54cbe9c021802aa8b9836a0bcf3d3229354e4c8870e 26 Port: <none> 27 Host Port: <none> 28 State: Running 29 Started: Tue, 09 Jun 2020 00:22:41 +0800 30 Ready: True 31 Restart Count: 0 32 Environment: <none> 33 Mounts: 34 /var/run/secrets/kubernetes.io/serviceaccount from default-token-v48g4 (ro) 35 Conditions: 36 Type Status 37 Initialized True 38 Ready True 39 ContainersReady True 40 PodScheduled True 41 Volumes: 42 default-token-v48g4: 43 Type: Secret (a volume populated by a Secret) 44 SecretName: default-token-v48g4 45 Optional: false 46 QoS Class: BestEffort 47 Node-Selectors: <none> 48 Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s 49 node.kubernetes.io/unreachable:NoExecute for 300s 50 Events: 51 Type Reason Age From Message 52 ---- ------ ---- ---- ------- 53 Normal Scheduled 22s default-scheduler Successfully assigned default/pod-secret-registry to k8s-node02 54 Normal Pulling 22s kubelet, k8s-node02 Pulling image "172.16.1.110:5000/k8s-secret/myapp:v1" 55 Normal Pulled 22s kubelet, k8s-node02 Successfully pulled image "172.16.1.110:5000/k8s-secret/myapp:v1" 56 Normal Created 22s kubelet, k8s-node02 Created container myapp 57 Normal Started 21s kubelet, k8s-node02 Started container myapp
由上可见，通过secret认证后pod拉取私有镜像是可以的。
相关阅读
我的博客即将同步至腾讯云+社区，邀请大家一同入驻：https://cloud.tencent.com/developer/support-plan?invite_code=2e3ko8jzdg2s8
———END———
如果觉得不错就关注下呗 (-^O^-) ！
