Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-3000

Stopping an onos node of a multi-node cluster causes default flow disappear in BM

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 1.3.0, 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: None
    • Labels:
    • Story Points:
      5
    • Epic Link:
    • Sprint:
      Emu Sprint 1 (9/21-10/9), Emu Sprint 2 (10/12-10/30), Emu Sprint 3 (11/2 - 11/20), Emu Sprint 4 (11/23-12/18), Dead Sprint (Emu Wrap-Up)

      Description

      this bug seems to happen in some clusters (BM, container) but not on others (VMs).

      Set up 3 nodes (with drivers, openflow, fwd),
      1) sudo mn --topo torus,3,3 --controller remote,ip=10.254.1.201 --controller remote,ip=10.254.1.202 --controller remote,ip=10.254.1.203
      2) onos see the correct topo and default flows:
      node=10.254.1.201, version=1.4.0.admin~2015/09/25@08:42
      nodes=3, devices=9, links=36, hosts=0, SCC(s)=1, flows=45, intents=0
      3) onos-service $OC3 stop:
      nodes=2, devices=9, links=0, hosts=0, SCC(s)=9, flows=0, intents=0
      node=10.254.1.201, version=1.4.0.admin~2015/09/25@08:42

      $OC1 has exception:
      2015-09-25 08:43:58,948 | ERROR | message-handlers | Tools | 73 - org.onosproject.onlab-misc - 1.4.0.SNAPSHOT | Uncaught exception on onos-store-flow-message-handlers
      java.lang.NullPointerException
      at org.onosproject.provider.of.flow.impl.OpenFlowRuleProvider.executeBatch(OpenFlowRuleProvider.java:325)[173:org.onosproject.onos-of-provider-flow:1.4.0.SNAPSHOT]
      at org.onosproject.net.flow.impl.FlowRuleManager$InternalStoreDelegate.notify(FlowRuleManager.java:493)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
      at org.onosproject.net.flow.impl.FlowRuleManager$InternalStoreDelegate.notify(FlowRuleManager.java:454)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
      at org.onosproject.store.AbstractStore.notifyDelegate(AbstractStore.java:58)[75:org.onosproject.onos-api:1.4.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.NewDistributedFlowRuleStore.storeBatchInternal(NewDistributedFlowRuleStore.java:439)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.NewDistributedFlowRuleStore.access$500(NewDistributedFlowRuleStore.java:99)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.NewDistributedFlowRuleStore$OnStoreBatch.handle(NewDistributedFlowRuleStore.java:591)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalClusterMessageHandler.apply(ClusterCommunicationManager.java:223)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalClusterMessageHandler.apply(ClusterCommunicationManager.java:213)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onlab.netty.NettyMessaging.lambda$null$4(NettyMessaging.java:214)[147:org.onosproject.onlab-netty:1.4.0.SNAPSHOT]
      at org.onlab.netty.NettyMessaging$$Lambda$438/1001788291.run(Unknown Source)[147:org.onosproject.onlab-netty:1.4.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_31]

      $OC2 some WARNINGs and exceptions:
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=arp}]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000102: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}

      ]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000101: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=arp}]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000103: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=bddp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000301: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000103: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}

      ]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000301: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000303: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}

      ]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000301: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=arp}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000102: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}

      ]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000303: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000302: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000302: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000101: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}

      ]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000101: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=arp}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000103: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000201: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]
      }, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000301: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000303: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000102: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}

      ]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000101: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=bddp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000303: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000201: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]
      }, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000302: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=bddp}

      ]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000302: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=bddp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000101: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,897 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000303: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000102: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=bddp}

      ]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000103: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=lldp}

      ]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000103: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=arp}]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000302: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}

      ]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000301: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:22,897 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=bddp}

      ]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000102: FLOWINSTALLATIONFAILED
      2015-09-25 08:44:30,523 | WARN | event-dispatch-0 | ListenerRegistry | 75 - org.onosproject.onos-api - 1.4.0.SNAPSHOT | Exception encountered while processing event ClusterEvent{time=2015-09-25T08:44:25.506, type=INSTANCE_DEACTIVATED, subject=DefaultControllerNode{id=10.254.1.203, ip=10.254.1.203, tcpPort=9876}}
      org.onosproject.store.service.ConsistentMapException$Timeout
      at org.onosproject.store.consistent.impl.DefaultConsistentMap.complete(DefaultConsistentMap.java:175)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.consistent.impl.DefaultConsistentMap.entrySet(DefaultConsistentMap.java:140)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.consistent.impl.ConsistentMapBackedJavaMap.entrySet(ConsistentMapBackedJavaMap.java:145)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.consistent.impl.ConsistentMapBackedJavaMap.forEach(ConsistentMapBackedJavaMap.java:153)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.consistent.impl.MutexExecutionManager$InternalClusterEventListener.event(MutexExecutionManager.java:192)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.store.consistent.impl.MutexExecutionManager$InternalClusterEventListener.event(MutexExecutionManager.java:184)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
      at org.onosproject.event.ListenerRegistry.process(ListenerRegistry.java:66)[75:org.onosproject.onos-api:1.4.0.SNAPSHOT]
      at org.onosproject.event.impl.CoreEventDispatcher$DispatchLoop.process(CoreEventDispatcher.java:141)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
      at org.onosproject.event.impl.CoreEventDispatcher$DispatchLoop.run(CoreEventDispatcher.java:124)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_31]

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              Assignee:
              madan Madan Jampani
              Reporter:
              suibin suibin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: