51CTO K8S常见故障速查指南常见的k8s pod crashloopbackoff排查
转载自博客：https://www.cnblogs.com/dalianpai/p/12258165.html
方案1：
转载自博客：https://www.cnblogs.com/dalianpai/p/12258165.html
现象如下：
[root@k8s1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE eureka-server-65695bbdc8-49b6v 0/1 CrashLoopBackOff 5 4m32s [root@k8s1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE eureka-server-65695bbdc8-49b6v 0/1 CrashLoopBackOff 5 5m16s
查找原因及解决如下：
[root@k8s1 ~]# kubectl describe pod eureka-server-65695bbdc8-49b6v
Name: eureka-server-65695bbdc8-49b6v
Namespace: default
Priority: 0
Node: k8s3/192.168.180.144
Start Time: Mon, 03 Feb 2020 21:03:24 +0800
Labels: app=eureka-server
pod-template-hash=65695bbdc8
Annotations: <none>
Status: Running
IP: 10.244.2.2
IPs: <none>
Controlled By: ReplicaSet/eureka-server-65695bbdc8
Containers:
eureka-server:
Container ID: docker://0a8de1c10a8bc42aea722d4994f8b53c06820030de30a2c014f6600d7a98d09d
Image: 112.124.104.171:1180/topcheer/eureka-server:1.0.1
Image ID: docker-pullable://112.124.104.171:1180/topcheer/eureka-server@sha256:b18dbacbc640c62b64d92f7240af0adcec2c4d9e870c60194d7ce9fa9d1e557c
Port: 9010/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 03 Feb 2020 21:07:15 +0800
Finished: Mon, 03 Feb 2020 21:07:22 +0800
Ready: False
Restart Count: 5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-tdbtd (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-tdbtd:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-tdbtd
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m30s default-scheduler Successfully assigned default/eureka-server-65695bbdc8-49b6v to k8s3
Normal Pulled 2m21s (x5 over 4m29s) kubelet, k8s3 Container image "112.124.104.171:1180/topcheer/eureka-server:1.0.1" already present on machine
Normal Created 2m21s (x5 over 4m28s) kubelet, k8s3 Created container eureka-server
Normal Started 2m21s (x5 over 4m28s) kubelet, k8s3 Started container eureka-server
Warning BackOff 105s (x10 over 4m6s) kubelet, k8s3 Back-off restarting failed container
[root@k8s1 ~]# kubectl logs eureka-server-65695bbdc8-49b6v
2020-02-03 13:07:18.750 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$2920e78d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.4.RELEASE)
2020-02-03 13:07:19.062 INFO 1 --- [ main] c.t.eureka.EurekaServerApplication : No active profile set, falling back to default profiles: default
2020-02-03 13:07:20.402 WARN 1 --- [ main] o.s.boot.actuate.endpoint.EndpointId : Endpoint ID 'service-registry' contains invalid characters, please migrate to a valid format.
2020-02-03 13:07:20.719 INFO 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=aab566da-2958-3e79-aa70-5d794b4e315b
2020-02-03 13:07:21.203 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$2920e78d] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-02-03 13:07:21.621 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9010 (http)
2020-02-03 13:07:21.638 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2020-02-03 13:07:21.639 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.30]
2020-02-03 13:07:21.727 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2020-02-03 13:07:21.728 INFO 1 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 2630 ms
2020-02-03 13:07:21.988 WARN 1 --- [ main] c.n.c.sources.URLConfigurationSource : No URLs will be polled as dynamic configuration sources.
2020-02-03 13:07:21.989 INFO 1 --- [ main] c.n.c.sources.URLConfigurationSource : To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
2020-02-03 13:07:22.018 INFO 1 --- [ main] c.netflix.config.DynamicPropertyFactory : DynamicPropertyFactory is initialized with configuration sources: com.netflix.config.ConcurrentCompositeConfiguration@3c73951
2020-02-03 13:07:22.358 ERROR 1 --- [ main] o.s.b.web.embedded.tomcat.TomcatStarter : Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'traceFilterRegistration' defined in class path resource [org/springframework/cloud/netflix/eureka/server/EurekaServerAutoConfiguration.class]: Unsatisfied dependency expressed through method 'traceFilterRegistration' parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'javax.servlet.Filter' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Qualifier(value=httpTraceFilter)}
2020-02-03 13:07:22.405 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-02-03 13:07:22.413 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
2020-02-03 13:07:22.442 INFO 1 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-02-03 13:07:22.778 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:156) ~[spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544) ~[spring-context-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at com.topcheer.eureka.EurekaServerApplication.main(EurekaServerApplication.java:13) [classes!/:0.0.1-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
才发现是boot的版本太高了，和boot有冲突。。。
案例2：
K8S 启动POD遇到CRASHLOOPBACKOFF的解决方法与思路
转载自博客：https://www.freesion.com/article/83191218638/
一直正常运的k8s，集群节点没问题，但启动pod出现异常
等待中: CrashLoopBackOff
1.登陆此节点主机使用kubctl获取pod状态
kubectl get pod
kubectl logs elkhost-944bcbcd4-8n9nj
[1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
错误原因在这里 经查elasticsearch运行要求:
vm.max_map_count内核参数必需大于262114，确认系统调过 $sysctl -w vm.max_map_count=262144
这里注意的问题是 上述属于临时性调整，主机重启后又恢复到默认状态，记久性修改 vi /etc/sysconfig 加入：
vm.max_map_count=262144
保存/etc/sysctl.conf,重新启动服务器以应用更改,或执行：sysctl -p以应用更改而不重新启动.他们将在重新启动时永久保持.
重启pod 进入控制台 查询状态 恢复
4.总结 出现故障可能很多种 但要学查看日志 排查具体原因及应用所在
posted on 2023-09-04 21:10 luzhouxiaoshuai 阅读(154) 评论(0) 编辑 收藏 举报
