删除命名空间失败
今天在作测试的时候,清理集群。就把没用的都清理掉包括命名空间。但是发现失败了,一直卡在终止状态。
导致删除失败的原因一般有两种:
1、命名空间下还有资源在用,如果有删除,命名空间自动消失(原因:API服务器强制要求,只有在名称空间为空的情况下,并且只有在名称空间的情况下才能从存储中删除名称空间。)
2、就是名称空间下没有资源
开始分析原因
正常可以有三种方式
1、查看控制器日志 kube-controller-manager
2、查看api-resources会有报错信息在下面
获取所有注册ns的资源,提取能delete的部分(kubectl api-resources --namespaced=true --verbs=delete)
接下来查看注册服务kubectl get apiservice
3、失败查看当前yaml文件,通过get获取。其中有status字段中的message会有相关信息。
kubectl get ns t1 -o yaml
删除了一些留了主要的部分
[root@ECS1 ~]# kubectl get ns t1 -o yaml
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: "2021-06-12T08:43:19Z"
deletionTimestamp: "2021-06-12T08:43:24Z"
name: t1
resourceVersion: "546044"
uid: 62f4616c-ea45-441f-8ccf-4079d35c2796
spec:
finalizers:
- kubernetes
status:
conditions:
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: 'Discovery failed for some groups, 2 failing: unable to retrieve the
complete list of server APIs: discovery.k8s.io/v1: the server could not find
the requested resource, policy/v1: the server could not find the requested resource'
reason: DiscoveryFailed
status: "True"
type: NamespaceDeletionDiscoveryFailure
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All legacy kube types successfully parsed
reason: ParsedGroupVersions
status: "False"
type: NamespaceDeletionGroupVersionParsingFailure
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All content successfully deleted, may be waiting on finalization
reason: ContentDeleted
status: "False"
type: NamespaceDeletionContentFailure
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All content successfully removed
reason: ContentRemoved
status: "False"
type: NamespaceContentRemaining
- lastTransitionTime: "2021-06-12T08:43:29Z"
message: All content-preserving finalizers finished
reason: ContentHasNoFinalizers
status: "False"
type: NamespaceFinalizersRemaining
phase: Terminating
问题直接定位到,这个资源找不到。并且在查看资源与查看注册资源时,输出最下面会有error。不处理上文故障的处理办法如下:
正文:
处理方法
[root@ECS1 ~]# kubectl get ns
NAME STATUS AGE
app-team1 Terminating 3d7h
default Active 3d19h
internal Active 2d23h
kube-node-lease Active 3d19h
kube-public Active 3d19h
kube-system Active 3d19h
[root@ECS1 ~]# kubectl delete ns/app-team1
namespace "app-team1" deleted
^C
[root@ECS1 ~]#
没办法只能手动停止,不然卡到你天荒地老。
找到一个神奇的地方找到这么一段话
There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, finalize
. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:
大体意思就是如果删除了命名空间,在已经清除空间下所有对象后。空间还在,那么需要通过更新名称空间子资源来强制删除。这种方式通知名称空间控制器,我要从命名空间中删除终结器并且执行清理所有操作。
这东西用的是restful请求方式,但是我这不安全端口都封掉了,开个代理吧(可以选择用证书)
[root@ECS1 ~]# kubectl proxy --port=8081
Starting to serve on 127.0.0.1:8081
cat <<EOF | curl -X PUT \
localhost:8081/api/v1/namespaces/app-team1/finalize \
-H "Content-Type: application/json" \
--data-binary @-
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "app-team1"
},
"spec": {
"finalizers": null
}
}
EOF
查看结果
[root@ECS1 ~]# cat <<EOF | curl -X PUT \
> localhost:8081/api/v1/namespaces/app-team1/finalize \
> -H "Content-Type: application/json" \
> --data-binary @-
> {
> "kind": "Namespace",
> "apiVersion": "v1",
> "metadata": {
> "name": "app-team1"
> },
> "spec": {
> "finalizers": null
> }
> }
> EOF
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "app-team1",
"uid": "108e6665-9b70-422c-8f94-783347101836",
"resourceVersion": "533794",
"creationTimestamp": "2021-06-08T23:46:24Z",
"deletionTimestamp": "2021-06-12T06:27:33Z",
"managedFields": [
{
"manager": "curl",
"operation": "Update",
"apiVersion": "v1",
"time": "2021-06-12T06:58:32Z",
"fieldsType": "FieldsV1",
"fieldsV1": {"f:status":{"f:phase":{}}}
}
]
},
"spec": {
},
"status": {
"phase": "Terminating",
"conditions": [
{
"type": "NamespaceDeletionDiscoveryFailure",
"status": "True",
"lastTransitionTime": "2021-06-12T06:27:38Z",
"reason": "DiscoveryFailed",
"message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource"
},
{
"type": "NamespaceDeletionGroupVersionParsingFailure",
"status": "False",
"lastTransitionTime": "2021-06-12T06:27:38Z",
"reason": "ParsedGroupVersions",
"message": "All legacy kube types successfully parsed"
},
{
"type": "NamespaceDeletionContentFailure",
"status": "False",
"lastTransitionTime":