RabbitMQ - Job for rabbitmq-server.service failed because the control process exited
昨天一早接到正在休假的领导的电话，我们部署在AWS上的核心业务进程挂了，原因是AWS的物理存储出问题了，正在紧急修复，已经从凌晨搞到了早上六七点，服务器是恢复了，但是我们的业务却一直起不起来，让赶紧看看什么情况。
登录到服务器，发现各个服务的异常都是因为无法连接到RabbitMQ导致的；
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rabbitMqInBoundAdapter': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.amqp.core.Queue com.nokia.app.datacollection.config.rabbitmq.RabbitMqInBoundAdapter.msmRequestQueue; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rabbitMqQueuesConfigurer': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:766)
at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:361)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1191)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1180)
at com.nokia.app.Application.main(Application.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.amqp.core.Queue com.nokia.app.datacollection.config.rabbitmq.RabbitMqInBoundAdapter.msmRequestQueue; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rabbitMqQueuesConfigurer': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:573)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
... 25 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rabbitMqQueuesConfigurer': Invocation of init method failed; nested exception is org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:368)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:545)
... 27 common frames omitted
Caused by: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:58)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:296)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:524)使用systemctl status rabbitmq-server.service查看rabbitmq的服务状态，发现服务启动是失败的；
Aug 12 07:27:38 ip-localhost rabbitmq-server[23580]: Crash dump was written to: erl_crash.dump
Aug 12 07:27:38 ip-localhost systemd[1]: rabbitmq-server.service: main process exited, c...URE
Aug 12 07:27:38 ip-localhost rabbitmqctl[23959]: Stopping and halting node 'rabbit@ip-172.....
Aug 12 07:27:38 ip-localhost rabbitmqctl[23959]: Error: unable to connect to node 'rabbit...wn
Aug 12 07:27:38 ip-localhost rabbitmqctl[23959]: DIAGNOSTICS
Aug 12 07:27:38 ip-localhost rabbitmqctl[23959]: ===========
Aug 12 07:27:38 ip-localhost rabbitmqctl[23959]: attempted to contact: ['rabbit@ip-172-31...']
Aug 12 07:27:38 ip-localhost systemd[1]: Failed to start RabbitMQ broker.
Aug 12 07:27:38 ip-localhost systemd[1]: Unit rabbitmq-server.service entered failed state.
Aug 12 07:27:38 ip-localhost systemd[1]: rabbitmq-server.service failed.检查了一遍rabbitmq的配置，并未发出现有差异的地方，依旧改了一些地方尝试一下，似乎并没有用。。。
vim /etc/rabbitmq/rabbitmq.config ##文件最后
{rabbitmq_auth_backend_http,
[{user_path, "http://localhost:9095/auth/user"},
{vhost_path, "http://localhost:9095/auth/vhost"},
{topic_path, "http://localhost:9095/auth/topic"},
{resource_path, "http://localhost:9095/auth/resource"}
]}
改为：
{rabbitmq_auth_backend_http,
[{user_path, "http://<内网ip>:9095/auth/user"},
{vhost_path, "http://<内网ip>:9095/auth/vhost"},
{topic_path, "http://<内网ip>:9095/auth/topic"},
{resource_path, "http://<内网ip>:9095/auth/resource"}
]}
参考这篇博客，问题得以解决https://blog.csdn.net/fvdfsdafdsafs/article/details/89332506
vim /etc/rabbitmq/rabbitmq-env.conf
在文件里面添加这一行：NODENAME=rabbit@localhost，保存; (注意：rabbitmq-env.conf这个文件没有，打开之后自动创建)
RabbitMQ服务是正常启动了，但是使用原来的用户来登录的时候，竟然说没有授权？刚开始还以为是密码错误了，后来查了rabbitmq的用户列表，才知道，原来的那个用户竟然‘消失了’。。。
=ERROR REPORT==== 12-Aug-2020::08:55:33 ===
Error on AMQP connection <0.469.0> (xxx.xx.xxx.xx:48402 -> xxx.xx.xxx.xx:5672, state: starting):
PLAIN login refused: user 'rabbitmqadmin' - invalid credentials
查询当前的用户列表；（原来创建的用户没有了，只剩下一个默认的guest）
# rabbitmqctl list_users
Listing users ...
guest [administrator]
创建用户，并赋予管理员权限；
# rabbitmqctl add_user <username> <password>
Creating user "username" ...
# rabbitmqctl list_users
Listing users ...
username []
guest [administrator]
##这个时候还是不能登录的，需要赋予管理员权限才行
# rabbitmqctl set_user_tags <username> administrator
Setting tags for user "username" to [administrator] ...
# rabbitmqctl list_users
Listing users ...
username [administrator]
guest [administrator]现在dashboard已经可以正常登陆了，重启我们的业务服务时，竟然还有报错，崩溃o(≧口≦)o。。。。
Caused by: java.io.IOException: null
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:378)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:647)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:283)
... 64 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method<connection.close>(reply-code=530, reply-text=NOT_ALLOWED - access to vhost '/' refused for user 'rabbitmqadmin', class-id=10, method-id=40)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:361)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:226)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 67 common frames omitted既然错误的日志说的是没有权限，我们就可以在web界面看看，我们的这个新添加的用户是不是要给它先赋权才对；
参考这篇文章解决了问题：https://blog.csdn.net/Dearbb1/article/details/84455591
