-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Critical
-
Resolution: Done
-
Affects Version/s: 1.5.0
-
Fix Version/s: 1.6.0
-
Component/s: None
-
Labels:
-
Environment:
Commit 3604bbcf498cafe06095416fb1ce264f41839e68
-
Story Points:5
-
Epic Link:
-
Sprint:Falcon Sprint #1 (1/5 - 1/22)
When restarting all nodes in a 7 node cluster, some nodes die with messages such as:
2016-01-11 13:04:11,909 | ERROR | entLoopGroup-4-1 | rejectedExecution | 46 - io.netty.common - 4.0.33.Final | Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:715)[46:io.netty.common:4.0.33.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:300)[46:io.netty.common:4.0.33.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:691)[46:io.netty.common:4.0.33.Final] at io.netty.util.concurrent.DefaultPromise.execute(DefaultPromise.java:671)[46:io.netty.common:4.0.33.Final] at io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:641)[46:io.netty.common:4.0.33.Final] at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138)[46:io.netty.common:4.0.33.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28)[48:io.netty.transport:4.0.33.Final] at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:252)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[48:io.netty.transport:4.0.33.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[46:io.netty.common:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[46:io.netty.common:4.0.33.Final] at java.lang.Thread.run(Thread.java:745)[:1.8.0_25] 2016-01-11 13:04:13,286 | ERROR | entLoopGroup-3-1 | MessageDecoder | 123 - org.onosproject.onlab-netty - 1.5.0.SNAPSHOT | Exception inside channel handling pipeline. io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Java heap space at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:431)[50:io.netty.codec:4.0.33.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)[50:io.netty.codec:4.0.33.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[48:io.netty.transport:4.0.33.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[48:io.netty.transport:4.0.33.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[46:io.netty.common:4.0.33.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[46:io.netty.common:4.0.33.Final] at java.lang.Thread.run(Thread.java:745)[:1.8.0_25] Caused by: java.lang.OutOfMemoryError: Java heap space
I have the logs from a test run, but they are too big to attach to Jira.
- relates to
-
ONOS-3679 HA test Stability issues
-
- Closed
-