Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-6347

Topology scaling test regression

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.10.0
    • Component/s: None
    • Labels:
    • Story Points:
      5

      Description

      Topology scaling test regressed from 40x40 to 30x30.
      To reproduce the problem:

      • Build a 3-node onos cluster
      • Build a 35x35 torus mininet
      • activate openflow app for onos cluster

      Then we see a lot of errors and exceptions in the log and onos is not able to recover from the failures. One suspicious commit to cause this regression is 13268. We occasionally saw these exceptions before 13268 but onos was able to recover.

      2017-04-18 11:36:02,345 | ERROR | w I/O worker #20 | OFChannelHandler                 | 166 - org.onosproject.onos-protocols-openflow-ctl - 1.10.0.SNAPSHOT | Error while processing message from switch org.onosproject.driver.handshaker.NiciraSwitchHandshaker [/10.254.1.200:55105 DPID[00:00:00:00:00:00:01:1a]]state WAIT_DESCRIPTION_STAT_REPLY
org.onosproject.store.service.ConsistentMapException$Timeout: onos-network-configs
        at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:228)
        at org.onosproject.store.primitives.DefaultConsistentMap.get(DefaultConsistentMap.java:85)
        at org.onosproject.store.config.impl.DistributedNetworkConfigStore.getConfig(DistributedNetworkConfigStore.java:241)
        at org.onosproject.net.config.impl.NetworkConfigManager.getConfig(NetworkConfigManager.java:203)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at org.onosproject.net.device.impl.PortAnnotationOperator.lookupConfig(PortAnnotationOperator.java:56)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at org.onosproject.net.device.impl.PortAnnotationOperator.combine(PortAnnotationOperator.java:61)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at org.onosproject.net.device.impl.DeviceManager.applyAllPortOps(DeviceManager.java:1021)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at org.onosproject.net.device.impl.DeviceManager.applyAllPortOps(DeviceManager.java:1006)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at org.onosproject.net.device.impl.DeviceManager.access$900(DeviceManager.java:103)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at org.onosproject.net.device.impl.DeviceManager$InternalDeviceProviderService.lambda$updatePorts$3(DeviceManager.java:545)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)[:1.8.0_121]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)[:1.8.0_121]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)[:1.8.0_121]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)[:1.8.0_121]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)[:1.8.0_121]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)[:1.8.0_121]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)[:1.8.0_121]
        at org.onosproject.net.device.impl.DeviceManager$InternalDeviceProviderService.updatePorts(DeviceManager.java:546)[127:org.onosproject.onos-core-net:1.10.0.SNAPSHOT]
        at org.onosproject.provider.of.device.impl.OpenFlowDeviceProvider$InternalDeviceProvider.switchAdded(OpenFlowDeviceProvider.java:405)[167:org.onosproject.onos-providers-openflow-device:1.10.0.SNAPSHOT]
        at org.onosproject.openflow.controller.impl.OpenFlowControllerImpl$OpenFlowSwitchAgent.addConnectedSwitch(OpenFlowControllerImpl.java:599)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
        at org.onosproject.openflow.controller.driver.AbstractOpenFlowSwitch.connectSwitch(AbstractOpenFlowSwitch.java:280)[161:org.onosproject.onos-protocols-openflow-api:1.10.0.SNAPSHOT]
        at org.onosproject.openflow.controller.impl.OFChannelHandler$ChannelState$6.processOFStatisticsReply(OFChannelHandler.java:475)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
        at org.onosproject.openflow.controller.impl.OFChannelHandler$ChannelState.processOFMessage(OFChannelHandler.java:994)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
        at org.onosproject.openflow.controller.impl.OFChannelHandler.messageReceived(OFChannelHandler.java:1274)[166:org.onosproject.onos-protocols-openflow-ctl:1.10.0.SNAPSHOT]
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)[95:org.jboss.netty:3.10.5.Final]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)[95:org.jboss.netty:3.10.5.Final]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            You You Wang (Inactive)
            Reporter:
            You You Wang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: