集群activemq重启报错java.lang.OutOfMemoryError: GC overhead limit exceeded

 

最近安全部门同事说我们环境有个弱密码漏洞，activemq后台的密码不够复杂，需要改为复杂密码。

我登录了他们扫出来的url，输入admin admin，果然进来了。不得不说上一位已离职同事的安全意识确实有待提高。

 

 

 

改吧那就。搜索引擎了好久，开始在测试环境动手。

主要更改了两个文件

/apache-activemq-5.15.10/conf/jetty-realm.properties ：

 

/apache-activemq-5.15.10/conf/activemq.xml ：

 

 

重启activemq，重新登录后台，发现可以正常登录。

 

但是后来发现有应用调用activemq是配置了默认无密码的，所以应用里面有个配置需要改：

 

 

于是我又重启一次。

这下出问题了。activemq一直起不来。报错：

2021-08-19T02:49:25.407552792Z INFO: Loading '/apache-activemq-5.15.8//bin/env'
2021-08-19T02:49:25.411478133Z INFO: Using java '/usr/lib/jvm/java-1.8-openjdk/jre/bin/java'
2021-08-19T02:49:25.411491881Z INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)
2021-08-19T02:49:25.411511505Z INFO: Creating pidfile /apache-activemq-5.15.8//data/activemq.pid
2021-08-19T02:49:25.751768482Z Java Runtime: Oracle Corporation 1.8.0_181 /usr/lib/jvm/java-1.8-openjdk/jre
2021-08-19T02:49:25.751793616Z   Heap sizes: current=62976k  free=58698k  max=932352k
2021-08-19T02:49:25.751799194Z     JVM args: -Xms64M -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/apache-activemq-5.15.8//conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/apache-activemq-5.15.8//tmp -Dactivemq.classpath=/apache-activemq-5.15.8//conf:/apache-activemq-5.15.8//../lib/: -Dactivemq.home=/apache-activemq-5.15.8/ -Dactivemq.base=/apache-activemq-5.15.8/ -Dactivemq.conf=/apache-activemq-5.15.8//conf -Dactivemq.data=/apache-activemq-5.15.8//data
2021-08-19T02:49:25.751805891Z Extensions classpath:
2021-08-19T02:49:25.75180961Z   [/apache-activemq-5.15.8/lib,/apache-activemq-5.15.8/lib/camel,/apache-activemq-5.15.8/lib/optional,/apache-activemq-5.15.8/lib/web,/apache-activemq-5.15.8/lib/extra]
2021-08-19T02:49:25.751813832Z ACTIVEMQ_HOME: /apache-activemq-5.15.8
2021-08-19T02:49:25.751817399Z ACTIVEMQ_BASE: /apache-activemq-5.15.8
2021-08-19T02:49:25.751821052Z ACTIVEMQ_CONF: /apache-activemq-5.15.8/conf
2021-08-19T02:49:25.751824782Z ACTIVEMQ_DATA: /apache-activemq-5.15.8/data
2021-08-19T02:49:25.940744695Z Loading message broker from: xbean:activemq.xml
2021-08-19T02:49:26.018554604Z  INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3fee9989: startup date [Thu Aug 19 10:49:26 HKT 2021]; root of context hierarchy
2021-08-19T02:49:27.108555122Z  INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/apache-activemq-5.15.8/data/kahadb]
2021-08-19T02:49:27.461050627Z  INFO | Page File: /apache-activemq-5.15.8/data/kahadb/db.data. Recovering pageFile free list due to prior unclean shutdown..
2021-08-19T02:49:27.466914828Z  INFO | KahaDB is version 6
2021-08-19T02:49:27.483650538Z  INFO | Recovering from the journal @176489:28
2021-08-19T02:49:27.54067786Z  INFO | Page File: /apache-activemq-5.15.8/data/kahadb/db.data. Recovered pageFile free list of size: 19423
2021-08-19T02:49:30.963686165Z  INFO | @176491:21197810, 100000 entries recovered ..
2021-08-19T02:49:32.87180072Z  INFO | @176495:3494384, 200000 entries recovered ..
2021-08-19T02:49:35.183203631Z  INFO | @176497:18130690, 300000 entries recovered ..
2021-08-19T02:49:37.270541382Z  INFO | @176499:29971068, 400000 entries recovered ..
2021-08-19T02:49:39.640094932Z  INFO | @176503:10449636, 500000 entries recovered ..
2021-08-19T02:49:41.872637011Z  INFO | @176505:23255288, 600000 entries recovered ..
2021-08-19T02:49:43.977470001Z  INFO | @176509:4025330, 700000 entries recovered ..
2021-08-19T02:49:46.250523453Z  INFO | @176511:19625713, 800000 entries recovered ..
2021-08-19T02:49:48.654289036Z  INFO | @176515:971190, 900000 entries recovered ..
2021-08-19T02:49:51.307804938Z  INFO | @176517:16905543, 1000000 entries recovered ..
2021-08-19T02:49:54.313955747Z  INFO | @176519:31789578, 1100000 entries recovered ..
2021-08-19T02:49:57.276056578Z  INFO | @176523:13477496, 1200000 entries recovered ..
2021-08-19T02:50:00.45483836Z  INFO | @176525:29133457, 1300000 entries recovered ..
2021-08-19T02:50:03.309667694Z  INFO | @224535:13329126, 1400000 entries recovered ..
2021-08-19T02:50:06.224617171Z  INFO | @224557:30182535, 1500000 entries recovered ..
2021-08-19T02:50:08.946220826Z  INFO | @241517:14076054, 1600000 entries recovered ..
2021-08-19T02:50:11.890695706Z  INFO | @241531:33039607, 1700000 entries recovered ..
2021-08-19T02:50:14.79864385Z  INFO | @278152:17618479, 1800000 entries recovered ..
2021-08-19T02:50:16.722956621Z  INFO | @366622:25067396, 1900000 entries recovered ..
2021-08-19T02:50:17.950009241Z  INFO | @366630:10175168, 2000000 entries recovered ..
2021-08-19T02:50:19.427774016Z  INFO | @366641:14624719, 2100000 entries recovered ..
2021-08-19T02:50:21.511216545Z  INFO | @366649:1925643, 2200000 entries recovered ..
2021-08-19T02:50:22.637640438Z  INFO | @366655:7512248, 2300000 entries recovered ..
2021-08-19T02:50:24.808706986Z  INFO | @366663:12956053, 2400000 entries recovered ..
2021-08-19T02:50:27.210699919Z  INFO | @366671:8714189, 2500000 entries recovered ..
2021-08-19T02:51:13.362637337Z  WARN | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-08-19T02:51:14.865452522Z ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-08-19T02:51:14.865492121Z org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-08-19T02:51:14.865500501Z     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1634)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865508023Z     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865514403Z     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865547458Z     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865555035Z     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865589395Z     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865596326Z     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865602646Z     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865610814Z     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.865617572Z     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
2021-08-19T02:51:14.86562315Z     at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-4.2.jar:4.2]
2021-08-19T02:51:14.865628908Z     at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-4.2.jar:4.2]
2021-08-19T02:51:14.865634419Z     at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865640543Z     at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865646674Z     at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865652615Z     at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865658748Z     at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865664786Z     at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865671111Z     at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865677188Z     at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865683306Z     at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865699953Z     at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.8.jar:5.15.8]
2021-08-19T02:51:14.865706911Z     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_181]
2021-08-19T02:51:14.865713097Z     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_181]
2021-08-19T02:51:14.86571918Z     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_181]
2021-08-19T02:51:14.865726669Z     at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_181]
2021-08-19T02:51:14.865732897Z     at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.8]
2021-08-19T02:51:14.865739022Z     at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.8]
2021-08-19T02:51:14.86634532Z ERROR: java.lang.RuntimeException: Failed to execute start task. Reason: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-08-19T02:51:14.866943961Z java.lang.RuntimeException: Failed to execute start task. Reason: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-08-19T02:51:16.459762881Z     at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:91)
2021-08-19T02:51:16.460137496Z     at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)
2021-08-19T02:51:16.46015272Z     at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)
2021-08-19T02:51:16.460157721Z     at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)

从日志看好像是activemq启动的时候在恢复数据jvm的内存不够。难道数据损坏了？我看了activemq挂载的数据盘文件大小，1.8G。而给activemq的内存是1G

 

 

加内存！

由于activemq是镜像启动的，只能重新构建一个镜像，将jvm内存从1G加到3G。再次启动，日志正常。

登录后台，输入admin 新密码，成功登录。