deepin 安装 microk8s 1.17 踩坑记录
microk8s 1.17
环境: Debian 系 Deepin 15.11 桌面系统, ubuntu 理论上可参照
安装
sudo apt update
sudo apt install snapd snap
export $PATH=PATH:/snap/bin >> ~/.zshrc && source ~/.zshrc
sudo snap install microk8s --classic
sudo microk8s.status --wait-ready
## status 输出
microk8s is running
addons:
cilium: disabled
dashboard: enabled
dns: enabled
fluentd: disabled
gpu: disabled
helm: disabled
ingress: disabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled
监控 pods 状态
watch microk8s.kubectl get all --all-namespaces
这是问题解决后的状态, STATUS 都是 Running
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-9b8997588-hlqxz 1/1 Running 54 4h38m
kube-system pod/dashboard-metrics-scraper-687667bb6c-7f79n 0/1 Pending 0 6m50s
kube-system pod/dashboard-metrics-scraper-687667bb6c-r8tgq 0/1 Evicted 0 37m
kube-system pod/heapster-v1.5.2-5c58f64f8b-lj2nf 4/4 Running 0 37m
kube-system pod/kubernetes-dashboard-5c848cc544-47fqk 0/1 Evicted 0 6m53s
kube-system pod/kubernetes-dashboard-5c848cc544-4zdgs 0/1 Evicted 0 6m52s
kube-system pod/kubernetes-dashboard-5c848cc544-7mhmj 0/1 Evicted 0 6m52s
kube-system pod/kubernetes-dashboard-5c848cc544-7xwfw 0/1 Pending 0 6m50s
kube-system pod/kubernetes-dashboard-5c848cc544-c7t4v 0/1 Evicted 0 6m51s
kube-system pod/kubernetes-dashboard-5c848cc544-kfnds 0/1 Evicted 0 6m53s
kube-system pod/kubernetes-dashboard-5c848cc544-l8r6s 0/1 Evicted 0 6m54s
kube-system pod/kubernetes-dashboard-5c848cc544-ms8gg 0/1 Evicted 0 6m54s
kube-system pod/kubernetes-dashboard-5c848cc544-ngvlc 0/1 Evicted 0 6m54s
kube-system pod/kubernetes-dashboard-5c848cc544-p7xqc 0/1 Evicted 0 6m54s
kube-system pod/kubernetes-dashboard-5c848cc544-wlw5m 0/1 Evicted 0 37m
kube-system pod/monitoring-influxdb-grafana-v4-6d599df6bf-nvr62 2/2 Running 0 37m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 5h34m
kube-system service/dashboard-metrics-scraper ClusterIP 10.152.183.61 <none> 8000/TCP 37m
kube-system service/heapster ClusterIP 10.152.183.168 <none> 80/TCP 37m
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 4h38m
kube-system service/kubernetes-dashboard ClusterIP 10.152.183.29 <none> 443/TCP 37m
kube-system service/monitoring-grafana ClusterIP 10.152.183.195 <none> 80/TCP 37m
kube-system service/monitoring-influxdb ClusterIP 10.152.183.212 <none> 8083/TCP,8086/TCP 37m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 1/1 1 1 4h38m
kube-system deployment.apps/dashboard-metrics-scraper 0/1 1 0 37m
kube-system deployment.apps/heapster-v1.5.2 1/1 1 1 37m
kube-system deployment.apps/kubernetes-dashboard 0/1 1 0 37m
kube-system deployment.apps/monitoring-influxdb-grafana-v4 1/1 1 1 37m
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-9b8997588 1 1 1 4h38m
kube-system replicaset.apps/dashboard-metrics-scraper-687667bb6c 1 1 0 37m
kube-system replicaset.apps/heapster-v1.5.2-5c58f64f8b 1 1 1 37m
kube-system replicaset.apps/kubernetes-dashboard-5c848cc544 1 1 0 37m
kube-system replicaset.apps/monitoring-influxdb-grafana-v4-6d599df6bf 1 1 1 37m
修改 .zshrc
.bashrc 教程一大把
如果本地没有安装 kubectl 可以使用 alias; 否则请不要全部复制粘贴;
如果已经安装了 kubectl，可以用下面的命令覆盖配置文件：
microk8s.kubectl config view --raw > $HOME/.kube/config-- 来自开篇的安装教程 http://www.imooc.com/article/291860
以下仅限于 zsh 用户; bash 用户百度下即可
vim ~/.zshrc
export PATH=$PATH:/usr/local/go/bin:/snap/bin
alias kubectl='microk8s.kubectl'
# 命令补全
if [ $commands[microk8s.kubectl] ]; then
source <(microk8s.kubectl completion zsh |
sed "s/complete -o default -F __start_kubectl kubectl/complete -o default -F __start_kubectl microk8s.kubectl/g" |
sed "s/complete -o default -o nospace -F __start_kubectl kubectl/complete -o default -o nospace -F __start_kubectl microk8s.kubectl/g");
fi
添加ctr proxy
microk8s.docker 命令在 1.17 版本被移除; 由 containerd 代替;
之前版本的, 需要修改 dockerd-env 加代理
sudo vim /var/snap/microk8s/current/args/containerd-env
HTTPS_PROXY=http://127.0.0.1:1082
重启 containerd 服务
sudo systemctl restart snap.microk8s.daemon-containerd.service
没有proxy 的也可以参照开篇链接教程, 条条大路通罗马, 不是非要proxy才能完成这个事情
修改内存/硬盘空间限制
sudo vim /var/snap/microk8s/current/args/kubelet
# 酌情复制
--eviction-hard="memory.available<1024Mi,nodefs.available<1Gi,imagefs.available<1Gi"
## 意思是: 当本node宿主机的 内存小于 1024Mi / 硬盘存储 小于 1Gi 时, 会将 pod 强制驱逐
这里之前有一些理解上的错误, 原本以为是 允许多大 内存/硬盘 使用, 后来发现并不是这样子的; 鉴于这篇文章现在有 7 个阅读, 我对这 7 位读者表示抱歉...
后面针对这样不明确的地方一定查证后再上传
防火墙 ufw
关于 CrashLoopBackOff 问题
sudo iptables -P FORWARD ACCEPT
# 1.17版本是 cni0; 之前版本是 cnr0, 参照官网 TroubleShooting
sudo ufw allow in on cni0 && sudo ufw allow out on cni0
sudo ufw default allow routed
重启 microk8s
microk8s.stop && microk8s.start
开启 add-on
sudo microk8s.enable dns dashboard
# 安装输出备忘部分
# enable dashbord后的输出部分, RBAC 未开启状态下需要依赖 token开头的两个命令 获取 token
If RBAC is not enabled access the dashboard using the default token retrieved with:
token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1);microk8s.kubectl -n kube-system describe secret $token
In an RBAC enabled setup (microk8s.enable RBAC) you need to create a user with restricted
permissions as shown in:
https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md
开启dashboard
开启proxy
kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'
新开命令行, 使用 kubectl get service -n kube-system 查看dashboard的 ip
kubectl get service -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.152.183.169 <none> 8000/TCP 17h
heapster ClusterIP 10.152.183.76 <none> 80/TCP 17h
kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 22h
kubernetes-dashboard ClusterIP 10.152.183.237 <none> 443/TCP 17h
monitoring-grafana ClusterIP 10.152.183.197 <none> 80/TCP 17h
monitoring-influxdb ClusterIP 10.152.183.82 <none> 8083/TCP,8086/TCP 17h
浏览器访问 https://10.152.183.237 可以到达 k8s-dashboard 界面 *注意 https 一定要加上, 不能去访问 443 端口(10.152.183.237:443 是行不通的)
dashboard 用户
csdn 大佬路子 https://blog.csdn.net/wucong60/article/details/81911859
#### ### dashboard addon 启动之后, microk8s 会自动启动一个 token 认证服务 default-token-b96pr 在 default namespace 里面 ### 可以通过命令: ### $ token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1);microk8s.kubectl -n kube-system describe secret $token ### 直接获取到 token ####
# 下面是手动创建secret # 创建服务 kubectl create serviceaccount cluster-admin-dashboard-sa # 启动 kubectl create clusterrolebinding cluster-admin-dashboard-sa --clusterrole=cluster-admin --serviceaccount=default:cluster-admin-dashboard-sa # 获取 pod 名称 kubectl get secret | grep cluster-admin-dashboard-sa # 获取 token kubectl describe secrets/cluster-admin-dashboard-sa-token-82dwx # 查看 token 服务 kubectl get serviceaccount # 删除手动创建的服务 kubectl delete serviceaccount cluster-admin-dashboard-sa
https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md
操作流程:
开启 RBAC 认证后的 官方推荐 操作流程
Create Service Account
mkdir ~/microk8s && cd ~/microk8s
vim dashboard-adminuser.yaml
# 写入文件
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
# EOF
# 应用更改
kubectl apply -f ./dashboard-adminuser.yaml
# 输出 serviceaccount/admin-user created
Create ClusterRoleBinding
touch role-bind.yaml
echo 'apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard' > role-bind.yaml
kubectl apply -f ./role-bind.yaml
# 输出 clusterrolebinding.rbac.authorization.k8s.io/admin-user created
获取token
kubectl -n kube-system describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
coredns 不启动 - deepin
snap 的安装 触发了 apparmor 的启动;
pod错误: CrashLoopBackOff
coredns日志:
kubectl logs -f coredns-xxxxxxx-xxxxx -n kube-system
:: socket permission denied; listen tcp port failed
暂行解决办法:
option#1. 关闭 apparmor https://wiki.debian.org/AppArmor/HowToUse#Disable_AppArmor
$ sudo mkdir -p /etc/default/grub.d
$ echo 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT apparmor=0"' \
| sudo tee /etc/default/grub.d/apparmor.cfg
$ sudo update-grub
$ sudo reboot
option#2. 参考链接 https://blog.csdn.net/u014062332/article/details/100099196
删除 Evicted pod
microk8s 运行一天后, 发现有很多处于 Evicted(被k8s放弃) 的 pod, 原因不明,删了去求,原因是 kubectl 配置文件有问题->k8s检测到系统资源达到了阀值, 放弃了pod以释放资源
kubectl get pods -n kube-system | grep Evicted | awk '{print $1}' | xargs microk8s.kubectl delete pod -n kube-system
参考链接 https://serverfault.com/questions/972120/microk8s-keeps-evicting-pods
参考链接 https://blog.csdn.net/zzq900503/article/details/83788152
