Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-3740

ONOS nodes crash when all nodes are restarted via killing all processes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
    • Environment:

      Commit 3604bbcf498cafe06095416fb1ce264f41839e68

    • Story Points:
      5
    • Epic Link:
    • Sprint:
      Falcon Sprint #1 (1/5 - 1/22)

      Description

      When restarting all nodes in a 7 node cluster, some nodes die with messages such as:

      2016-01-11 13:04:11,909 | ERROR | entLoopGroup-4-1 | rejectedExecution                | 46 - io.netty.common - 4.0.33.Final | Failed to submit a listener notification task. Event loop shut down?
      java.util.concurrent.RejectedExecutionException: event executor terminated
              at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:715)[46:io.netty.common:4.0.33.Final]
              at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:300)[46:io.netty.common:4.0.33.Final]
              at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:691)[46:io.netty.common:4.0.33.Final]
              at io.netty.util.concurrent.DefaultPromise.execute(DefaultPromise.java:671)[46:io.netty.common:4.0.33.Final]
              at io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:641)[46:io.netty.common:4.0.33.Final]
              at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138)[46:io.netty.common:4.0.33.Final]
              at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28)[48:io.netty.transport:4.0.33.Final]
              at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:252)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[48:io.netty.transport:4.0.33.Final]
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[46:io.netty.common:4.0.33.Final]
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[46:io.netty.common:4.0.33.Final]
              at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
      2016-01-11 13:04:13,286 | ERROR | entLoopGroup-3-1 | MessageDecoder                   | 123 - org.onosproject.onlab-netty - 1.5.0.SNAPSHOT | Exception inside channel handling pipeline.
      io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Java heap space
              at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:431)[50:io.netty.codec:4.0.33.Final]
              at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)[50:io.netty.codec:4.0.33.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[48:io.netty.transport:4.0.33.Final]
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[48:io.netty.transport:4.0.33.Final]
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[46:io.netty.common:4.0.33.Final]
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[46:io.netty.common:4.0.33.Final]
              at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
      Caused by: java.lang.OutOfMemoryError: Java heap space
      

      I have the logs from a test run, but they are too big to attach to Jira.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              • Assignee:
                madan Madan Jampani
                Reporter:
                jhall Jon Hall
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: