rook 排错记录 + Orphaned pod found kube-controller-manager的日志输出

1.查看rook-agent(重要)和mysql-wordpress 的日志，如下：

MountVolume.SetUp failed for volume "pvc-f002e1fe-469c-11e8-9dca-90b8d0599f2f" : mount command failed, status: Failure, reason: Rook: Error getting RPC client: error connecting to socket /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: dial unix /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: connect: no such file or directory

解决：

rook-operator.yml

env:
- name: FLEXVOLUME_DIR_PATH
  value: "/var/lib/kubelet/volumeplugins"

kubelet 启动加上

--volume-plugin-dir=/var/lib/kubelet/volumeplugins

2.  kubectl describe pod   wordpress-69895668dd-zfkvq

Unable to mount volumes for pod "wordpress-mysql-5bbddff49c-vbv4q_default(f2949440-e838-11e8-999c-06c9540001ea)": timeout expired waiting for volumes to attach or mount for pod "default"/"wordpress-mysql-5bbddff49c-vbv4q". list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-sp8rj]
Warning FailedMount 58s (x2 over 1m) kubelet, node3 MountVolume.SetUp failed for volume "pvc-b044b1d8-e819-11e8-84c3-060f100001e9" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume replicapool/pvc-b044b1d8-e819-11e8-84c3-060f100001e9: failed to map image replicapool/pvc-b044b1d8-e819-11e8-84c3-060f100001e9 cluster rook-ceph. failed to map image replicapool/pvc-b044b1d8-e819-11e8-84c3-060f100001e9: Failed to complete 'rbd': signal: interrupt. . output:

方法：

进入容器手动执行

重启kubelet systemctl restart kubelet  -- 不起作用

直接重启所有host机器，删除wordpress 和 mysql pod ,机器重启后删除有错误osd pod ,等10分钟后

default wordpress-69895668dd-zfkvq 1/1 Running 0 14h 10.233.71.43 node3
default wordpress-mysql-5bbddff49c-vbv4q 1/1 Running 0 14h 10.233.71.42 node3

###
1. 先把镜像都下载下来
2. 把pod个数设置为0
3. 把pv , pvc都删除
4.重新apply就可以

 

 

1.MountVolume.SetUp failed for volume "pvc-f002e1fe-469c-11e8-9dca-90b8d0599f2f" : mount command failed, status: Failure, reason: Rook: Error getting RPC client: error connecting to socket /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: dial unix /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: connect: no such file or directory

解决：

rook-operator.yml

env:
- name: FLEXVOLUME_DIR_PATH
  value: "/var/lib/kubelet/volumeplugins"

kubelet 启动加上

--volume-plugin-dir=/var/lib/kubelet/volumeplugins