Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-6347

Topology scaling test regression

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.10.0
    • Component/s: None
    • Labels:
    • Story Points:
      5

      Description

      Topology scaling test regressed from 40x40 to 30x30.
      To reproduce the problem:

      • Build a 3-node onos cluster
      • Build a 35x35 torus mininet
      • activate openflow app for onos cluster

      Then we see a lot of errors and exceptions in the log and onos is not able to recover from the failures. One suspicious commit to cause this regression is 13268. We occasionally saw these exceptions before 13268 but onos was able to recover.

      2017-04-18 11:36:02,345 | ERROR | w I/O worker #20 | OFChannelHandler                 | 166 - org.onosproject.onos-protocols-openflow-ctl - 1.10.0.SNAPSHOT | Error while processing message from switch org.onosproject.driver.handshaker.NiciraSwitchHandshaker [/10.254.1.200:55105 DPID[00:00:00:00:00:00:01:1a]]state WAIT_DESCRIPTION_STAT_REPLY
      org.onosproject.store.service.ConsistentMapException$Timeout: onos-network-configs
              at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:228)
              at org.onosproject.store.primitives.DefaultConsistentMap.get(DefaultConsistentMap.java:85)
              at org.onosproject.store.config.impl.DistributedNetworkConfigStore.getConfig(DistributedNetworkConfigStore.java:241)
              at org.onosproject.net.config.impl.NetworkConfigManager.getConfig(NetworkConfigManager.java:203)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at org.onosproject.net.device.impl.PortAnnotationOperator.lookupConfig(PortAnnotationOperator.java:56)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at org.onosproject.net.device.impl.PortAnnotationOperator.combine(PortAnnotationOperator.java:61)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at org.onosproject.net.device.impl.DeviceManager.applyAllPortOps(DeviceManager.java:1021)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at org.onosproject.net.device.impl.DeviceManager.applyAllPortOps(DeviceManager.java:1006)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at org.onosproject.net.device.impl.DeviceManager.access$900(DeviceManager.java:103)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at org.onosproject.net.device.impl.DeviceManager$InternalDeviceProviderService.lambda$updatePorts$3(DeviceManager.java:545)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)[:1.8.0_121]
              at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)[:1.8.0_121]
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)[:1.8.0_121]
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)[:1.8.0_121]
              at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)[:1.8.0_121]
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)[:1.8.0_121]
              at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)[:1.8.0_121]
              at org.onosproject.net.device.impl.DeviceManager$InternalDeviceProviderService.updatePorts(DeviceManager.java:546)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
              at org.onosproject.provider.of.device.impl.OpenFlowDeviceProvider$InternalDeviceProvider.switchAdded(OpenFlowDeviceProvider.java:405)[167:org.onosproject.onos-providers-openflow-device:1.10.0.SNAPSHOT]
              at org.onosproject.openflow.controller.impl.OpenFlowControllerImpl$OpenFlowSwitchAgent.addConnectedSwitch(OpenFlowControllerImpl.java:599)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
              at org.onosproject.openflow.controller.driver.AbstractOpenFlowSwitch.connectSwitch(AbstractOpenFlowSwitch.java:280)[161:org.onosproject.onos-protocols-openflow-api:1.10.0.SNAPSHOT]
              at org.onosproject.openflow.controller.impl.OFChannelHandler$ChannelState$6.processOFStatisticsReply(OFChannelHandler.java:475)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
              at org.onosproject.openflow.controller.impl.OFChannelHandler$ChannelState.processOFMessage(OFChannelHandler.java:994)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
              at org.onosproject.openflow.controller.impl.OFChannelHandler.messageReceived(OFChannelHandler.java:1274)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)[95:org.jboss.netty:3.10.5.Final]
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)[95:org.jboss.netty:3.10.5.Final]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]
              at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
      

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            You You Wang
            Reporter:
            You You Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: