Kubernetes删除一直处于terminating状态的namespace
k8s中namespace有两种常见的状态,即Active和Terminating状态,其中Terminating状态一般会比较少见,当对应的命名空间下还存在运行的资源,但该命名空间被删除时才会出现所谓的Terminating状态,这种情况下只要等待k8s本身将命名空间下的资源回收后,该命名空间将会被系统自动删除。
但是在某些情况下,即使命名空间下没有运行的资源,但依然无法删除Terminating状态的命名空间的情况,它会一直卡在Terminating状态下,即使用--force --grace-period=0 也删除不了。
接下来的步骤将描述如何手动删除卡住的命名空间:
-
Run the following command to view the namespaces that are stuck in the Terminating state:
kubectl get namespaces
-
Select a terminating namespace and view the contents of the namespace to find out the
finalizer
. Run the following command:kubectl get namespace <terminating-namespace> -o yaml
Your YAML contents might resemble the following output:
apiVersion: v1 kind: Namespace metadata: creationTimestamp: 2018-11-19T18:48:30Z deletionTimestamp: 2018-11-19T18:59:36Z name: <terminating-namespace> resourceVersion: "1385077" selfLink: /api/v1/namespaces/<terminating-namespace> uid: b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5 spec: finalizers: - kubernetes status: phase: Terminating
-
Run the following command to create a temporary JSON file:
kubectl get namespace <terminating-namespace> -o json >tmp.json
-
Edit your
tmp.json
file. Remove thekubernetes
value from thefinalizers
field and save the file.Your
tmp.json
file might resemble the following output:{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "creationTimestamp": "2018-11-19T18:48:30Z", "deletionTimestamp": "2018-11-19T18:59:36Z", "name": "<terminating-namespace>", "resourceVersion": "1385077", "selfLink": "/api/v1/namespaces/<terminating-namespace>", "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5" }, "spec": { "finalizers": }, "status": { "phase": "Terminating" } }
-
To set a temporary proxy IP and port, run the following command. Be sure to keep your terminal window open until you delete the stuck namespace:
kubectl proxy
Your proxy IP and port might resemble the following output:
Starting to serve on 127.0.0.1:8001
-
From a new terminal window, make an API call with your temporary proxy IP and port:
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize
Your output might resemble the following content:
{ "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "<terminating-namespace>", "selfLink": "/api/v1/namespaces/<terminating-namespace>/finalize", "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5", "resourceVersion": "1602981", "creationTimestamp": "2018-11-19T18:48:30Z", "deletionTimestamp": "2018-11-19T18:59:36Z" }, "spec": { }, "status": { "phase": "Terminating" } }
Note: The
finalizer
parameter is removed. -
Verify that the terminating namespace is removed, run the following command:
kubectl get namespaces
Continue to follow the steps for other namespaces that are stuck in the Terminating state.
参考:https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.1/troubleshoot/ns_terminating.html
参考:https://support.huaweicloud.com/cce_faq/cce_faq_00277.html