K&
随笔 - 71,  文章 - 0,  评论 - 7,  阅读 - 53084

删除命名空间失败

 

今天在作测试的时候,清理集群。就把没用的都清理掉包括命名空间。但是发现失败了,一直卡在终止状态。

 

导致删除失败的原因一般有两种:

 

1、命名空间下还有资源在用,如果有删除,命名空间自动消失(原因:API服务器强制要求,只有在名称空间为空的情况下,并且只有在名称空间的情况下才能从存储中删除名称空间。)

 

2、就是名称空间下没有资源

 

 

开始分析原因

 

正常可以有三种方式

 

1、查看控制器日志  kube-controller-manager

2、查看api-resources会有报错信息在下面

  • 获取所有注册ns的资源,提取能delete的部分(kubectl api-resources --namespaced=true --verbs=delete)
  • 接下来查看注册服务kubectl get apiservice

3、失败查看当前yaml文件,通过get获取。其中有status字段中的message会有相关信息。

 

 

kubectl get ns t1 -o yaml

 

 

删除了一些留了主要的部分

 

 

复制代码
[root@ECS1 ~]# kubectl get ns t1 -o yaml
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-06-12T08:43:19Z"
  deletionTimestamp: "2021-06-12T08:43:24Z"
  name: t1
  resourceVersion: "546044"
  uid: 62f4616c-ea45-441f-8ccf-4079d35c2796
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2021-06-12T08:43:29Z"
    message: 'Discovery failed for some groups, 2 failing: unable to retrieve the
      complete list of server APIs: discovery.k8s.io/v1: the server could not find
      the requested resource, policy/v1: the server could not find the requested resource'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2021-06-12T08:43:29Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2021-06-12T08:43:29Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2021-06-12T08:43:29Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2021-06-12T08:43:29Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating
复制代码

 

问题直接定位到,这个资源找不到。并且在查看资源与查看注册资源时,输出最下面会有error。不处理上文故障的处理办法如下:

 

正文:

 

处理方法

 

复制代码
[root@ECS1 ~]# kubectl get ns
NAME              STATUS        AGE
app-team1         Terminating   3d7h
default           Active        3d19h
internal          Active        2d23h
kube-node-lease   Active        3d19h
kube-public       Active        3d19h
kube-system       Active        3d19h


[root@ECS1 ~]# kubectl delete ns/app-team1
namespace "app-team1" deleted
^C
[root@ECS1 ~]# 
复制代码

 

没办法只能手动停止,不然卡到你天荒地老。

 

找到一个神奇的地方找到这么一段话

 

There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, finalize. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:

 

大体意思就是如果删除了命名空间,在已经清除空间下所有对象后。空间还在,那么需要通过更新名称空间子资源来强制删除。这种方式通知名称空间控制器,我要从命名空间中删除终结器并且执行清理所有操作。

 

这东西用的是restful请求方式,但是我这不安全端口都封掉了,开个代理吧(可以选择用证书)

 

[root@ECS1 ~]# kubectl proxy --port=8081
Starting to serve on 127.0.0.1:8081

 

开始删除

 

复制代码
cat <<EOF | curl -X PUT \
  localhost:8081/api/v1/namespaces/app-team1/finalize \
  -H "Content-Type: application/json" \
  --data-binary @-
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "app-team1"
  },
  "spec": {
    "finalizers": null
  }
}
EOF
复制代码

 

查看结果

 

复制代码
[root@ECS1 ~]# cat <<EOF | curl -X PUT \
>   localhost:8081/api/v1/namespaces/app-team1/finalize \
>   -H "Content-Type: application/json" \
>   --data-binary @-
> {
>   "kind": "Namespace",
>   "apiVersion": "v1",
>   "metadata": {
>     "name": "app-team1"
>   },
>   "spec": {
>     "finalizers": null
>   }
> }
> EOF
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "app-team1",
    "uid": "108e6665-9b70-422c-8f94-783347101836",
    "resourceVersion": "533794",
    "creationTimestamp": "2021-06-08T23:46:24Z",
    "deletionTimestamp": "2021-06-12T06:27:33Z",
    "managedFields": [
      {
        "manager": "curl",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2021-06-12T06:58:32Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:phase":{}}}
      }
    ]
  },
  "spec": {
    
  },
  "status": {
    "phase": "Terminating",
    "conditions": [
      {
        "type": "NamespaceDeletionDiscoveryFailure",
        "status": "True",
        "lastTransitionTime": "2021-06-12T06:27:38Z",
        "reason": "DiscoveryFailed",
        "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource"
      },
      {
        "type": "NamespaceDeletionGroupVersionParsingFailure",
        "status": "False",
        "lastTransitionTime": "2021-06-12T06:27:38Z",
        "reason": "ParsedGroupVersions",
        "message": "All legacy kube types successfully parsed"
      },
      {
        "type": "NamespaceDeletionContentFailure",
        "status": "False",
        "lastTransitionTime": "2021-06-12T06:27:38Z",
        "reason": "ContentDeleted",
        "message": "All content successfully deleted, may be waiting on finalization"
      },
      {
        "type": "NamespaceContentRemaining",
        "status": "False",
        "lastTransitionTime": "2021-06-12T06:27:38Z",
        "reason": "ContentRemoved",
        "message": "All content successfully removed"
      },
      {
        "type": "NamespaceFinalizersRemaining",
        "status": "False",
        "lastTransitionTime": "2021-06-12T06:27:38Z",
        "reason": "ContentHasNoFinalizers",
        "message": "All content-preserving finalizers finished"
      }
    ]
  }
}

[root@ECS1 ~]# kubectl get ns
NAME              STATUS   AGE
default           Active   3d19h
internal          Active   2d23h
kube-node-lease   Active   3d19h
kube-public       Active   3d19h
kube-system       Active   3d19h
[root@ECS1 ~]# 
复制代码

 

删除成功

 

 

 

理解不了上面开始删除那部分的代码可以参考下面这部分。

 

kubectl get namespace t1 -o json > ns.json

 

开始删除(先作错误示范)

 

 

复制代码
[root@ECS1 ~]# curl -X PUT localhost:8081/api/v1/namespaces/t1/finalize -H "Content-Type: application/json" --data-binary @ns.json
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "t1",
    "uid": "62f4616c-ea45-441f-8ccf-4079d35c2796",
    "resourceVersion": "546044",
    "creationTimestamp": "2021-06-12T08:43:19Z",
    "deletionTimestamp": "2021-06-12T08:43:24Z",
    "managedFields": [
      {
        "manager": "kubectl-create",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2021-06-12T08:43:19Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:phase":{}}}
      },
      {
        "manager": "kube-controller-manager",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2021-06-12T08:43:29Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}
      }
    ]
  },
 "spec": {
    "finalizers": [
      "kubernetes"
    ]
  },
  "status": {
    "phase": "Terminating",
    "conditions": [
      {
        "type": "NamespaceDeletionDiscoveryFailure",
        "status": "True",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "DiscoveryFailed",
        "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource"
      },
      {
        "type": "NamespaceDeletionGroupVersionParsingFailure",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ParsedGroupVersions",
        "message": "All legacy kube types successfully parsed"
      },
      {
        "type": "NamespaceDeletionContentFailure",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ContentDeleted",
        "message": "All content successfully deleted, may be waiting on finalization"
      },
      {
        "type": "NamespaceContentRemaining",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ContentRemoved",
        "message": "All content successfully removed"
      },
      {
        "type": "NamespaceFinalizersRemaining",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ContentHasNoFinalizers",
        "message": "All content-preserving finalizers finished"
      }
    ]
  }
}
复制代码

 

失败了

 

复制代码
[root@ECS1 ~]# kubectl get ns
NAME              STATUS        AGE
default           Active        3d22h
kube-node-lease   Active        3d22h
kube-public       Active        3d22h
kube-system       Active        3d22h
t1                Terminating   59m
复制代码

 

 

编辑ns.json文件删除spec部分保存文件,重新执行命令即可。

 

展示一下执行结果

 

 

复制代码
[root@ECS1 ~]# curl -X PUT localhost:8081/api/v1/namespaces/t1/finalize -H "Content-Type: application/json" --data-binary @ns.json
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "t1",
    "uid": "62f4616c-ea45-441f-8ccf-4079d35c2796",
    "resourceVersion": "546044",
    "creationTimestamp": "2021-06-12T08:43:19Z",
    "deletionTimestamp": "2021-06-12T08:43:24Z",
    "managedFields": [
      {
        "manager": "kubectl-create",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2021-06-12T08:43:19Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:phase":{}}}
      },
      {
        "manager": "kube-controller-manager",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2021-06-12T08:43:29Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}
      }
    ]
  },
  "spec": {
    
  },
  "status": {
    "phase": "Terminating",
    "conditions": [
      {
        "type": "NamespaceDeletionDiscoveryFailure",
        "status": "True",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "DiscoveryFailed",
        "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource"
      },
      {
        "type": "NamespaceDeletionGroupVersionParsingFailure",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ParsedGroupVersions",
        "message": "All legacy kube types successfully parsed"
      },
      {
        "type": "NamespaceDeletionContentFailure",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ContentDeleted",
        "message": "All content successfully deleted, may be waiting on finalization"
      },
      {
        "type": "NamespaceContentRemaining",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ContentRemoved",
        "message": "All content successfully removed"
      },
      {
        "type": "NamespaceFinalizersRemaining",
        "status": "False",
        "lastTransitionTime": "2021-06-12T08:43:29Z",
        "reason": "ContentHasNoFinalizers",
        "message": "All content-preserving finalizers finished"
      }
    ]
  }
}

[root@ECS1 ~]# kubectl get ns
NAME              STATUS        AGE
default           Active        3d22h
kube-node-lease   Active        3d22h
kube-public       Active        3d22h
kube-system       Active        3d22h
test              Terminating   61m
复制代码

 

 

 

这么作虽然能删除,但是有风险(尽量保证下面什么资源都没有后在删除空间)

 

This should be done with caution as it may delete the namespace only and leave orphan objects within the, now non-exiting, namespace - a confusing state for Kubernetes. If this happens, the namespace can be re-created manually and sometimes the orphaned objects will re-appear under the just-created namespace which will allow manual cleanup and recovery

 

译文:这样做时应该谨慎,因为它可能只删除名称空间,而将孤立对象留在现在不存在的名称空间中——这对Kubernetes来说是一种令人困惑的状态。 如果发生这种情况,可以手动重新创建名称空间,有时孤立的对象将重新出现在刚刚创建的名称空间下,这将需要手动清理和恢复

 

原因:

官方的一段话:

当执行删除后 Kubernetes 报告该对象已被删除,但是,它尚未在传统意义上被删除。相反,它处于删除过程中。当我们再次尝试该对象时,我们发现该对象已修改,以包括删除时间戳。

 

 

复制代码
[root@ECS1 ~]# curl -X PUT localhost:8081/api/v1/namespaces/internal/finalize -H "Content-Type: application/json" --data-binary @ns.json
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Operation cannot be fulfilled on namespaces \"internal\": the object has been modified; please apply your changes to the latest version and try again",
  "reason": "Conflict",
  "details": {
    "name": "internal",
    "kind": "namespaces"
  },
  "code": 409
}
复制代码

 

 

 

所发生的是对象已更新,而不是删除。这是因为 Kubernetes 看到对象包含终结器,并将其置于仅读取状态。删除时间戳表示对象只能读取,但删除终结者密钥更新除外。换句话说,删除将不完整,直到我们编辑对象并删除终结者。

 

 

 

 

 

作者K&

posted on   K&  阅读(1554)  评论(2编辑  收藏  举报
编辑推荐:
· .NET Core 中如何实现缓存的预热?
· 从 HTTP 原因短语缺失研究 HTTP/2 和 HTTP/3 的设计差异
· AI与.NET技术实操系列:向量存储与相似性搜索在 .NET 中的实现
· 基于Microsoft.Extensions.AI核心库实现RAG应用
· Linux系列:如何用heaptrack跟踪.NET程序的非托管内存泄露
阅读排行:
· TypeScript + Deepseek 打造卜卦网站:技术与玄学的结合
· 阿里巴巴 QwQ-32B真的超越了 DeepSeek R-1吗?
· 【译】Visual Studio 中新的强大生产力特性
· 2025年我用 Compose 写了一个 Todo App
· 张高兴的大模型开发实战:(一)使用 Selenium 进行网页爬虫

< 2025年3月 >
23 24 25 26 27 28 1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31 1 2 3 4 5
点击右上角即可分享
微信分享提示