-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Critical
-
Resolution: Done
-
Affects Version/s: 1.3.0, 1.4.0
-
Fix Version/s: 1.4.0
-
Component/s: None
-
Story Points:5
-
Epic Link:
-
Sprint:Emu Sprint 1 (9/21-10/9), Emu Sprint 2 (10/12-10/30), Emu Sprint 3 (11/2 - 11/20), Emu Sprint 4 (11/23-12/18), Dead Sprint (Emu Wrap-Up)
this bug seems to happen in some clusters (BM, container) but not on others (VMs).
Set up 3 nodes (with drivers, openflow, fwd),
1) sudo mn --topo torus,3,3 --controller remote,ip=10.254.1.201 --controller remote,ip=10.254.1.202 --controller remote,ip=10.254.1.203
2) onos see the correct topo and default flows:
node=10.254.1.201, version=1.4.0.admin~2015/09/25@08:42
nodes=3, devices=9, links=36, hosts=0, SCC(s)=1, flows=45, intents=0
3) onos-service $OC3 stop:
nodes=2, devices=9, links=0, hosts=0, SCC(s)=9, flows=0, intents=0
node=10.254.1.201, version=1.4.0.admin~2015/09/25@08:42
$OC1 has exception:
2015-09-25 08:43:58,948 | ERROR | message-handlers | Tools | 73 - org.onosproject.onlab-misc - 1.4.0.SNAPSHOT | Uncaught exception on onos-store-flow-message-handlers
java.lang.NullPointerException
at org.onosproject.provider.of.flow.impl.OpenFlowRuleProvider.executeBatch(OpenFlowRuleProvider.java:325)[173:org.onosproject.onos-of-provider-flow:1.4.0.SNAPSHOT]
at org.onosproject.net.flow.impl.FlowRuleManager$InternalStoreDelegate.notify(FlowRuleManager.java:493)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
at org.onosproject.net.flow.impl.FlowRuleManager$InternalStoreDelegate.notify(FlowRuleManager.java:454)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
at org.onosproject.store.AbstractStore.notifyDelegate(AbstractStore.java:58)[75:org.onosproject.onos-api:1.4.0.SNAPSHOT]
at org.onosproject.store.flow.impl.NewDistributedFlowRuleStore.storeBatchInternal(NewDistributedFlowRuleStore.java:439)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.flow.impl.NewDistributedFlowRuleStore.access$500(NewDistributedFlowRuleStore.java:99)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.flow.impl.NewDistributedFlowRuleStore$OnStoreBatch.handle(NewDistributedFlowRuleStore.java:591)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalClusterMessageHandler.apply(ClusterCommunicationManager.java:223)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalClusterMessageHandler.apply(ClusterCommunicationManager.java:213)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onlab.netty.NettyMessaging.lambda$null$4(NettyMessaging.java:214)[147:org.onosproject.onlab-netty:1.4.0.SNAPSHOT]
at org.onlab.netty.NettyMessaging$$Lambda$438/1001788291.run(Unknown Source)[147:org.onosproject.onlab-netty:1.4.0.SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_31]
$OC2 some WARNINGs and exceptions:
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}
]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000101: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=bddp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000301: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000103: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}
]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000301: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}
]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000301: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,895 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}
]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000303: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000302: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000101: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}
]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000101: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000201: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000301: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000303: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000102: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}
]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000101: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000201: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=lldp}]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000302: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=bddp}
]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000302: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,897 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000303: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=ipv4}]}, priority=5, appId=DefaultApplicationId{id=31, name=org.onosproject.fwd}} to of:0000000000000102: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=bddp}
]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000103: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000103: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
2015-09-25 08:44:22,896 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE{ethType=arp}
]}, priority=40000, appId=DefaultApplicationId{id=44, name=org.onosproject.provider.host}} to of:0000000000000301: FLOWINSTALLATIONFAILED
2015-09-25 08:44:22,897 | WARN | message-handlers | PacketManager | 143 - org.onosproject.onos-core-net - 1.4.0.SNAPSHOT | Failed to install packet request DefaultPacketRequest{selector=DefaultTrafficSelector{criteria=[ETH_TYPE
]}, priority=40000, appId=DefaultApplicationId{id=43, name=org.onosproject.provider.lldp}} to of:0000000000000102: FLOWINSTALLATIONFAILED
2015-09-25 08:44:30,523 | WARN | event-dispatch-0 | ListenerRegistry | 75 - org.onosproject.onos-api - 1.4.0.SNAPSHOT | Exception encountered while processing event ClusterEvent{time=2015-09-25T08:44:25.506, type=INSTANCE_DEACTIVATED, subject=DefaultControllerNode{id=10.254.1.203, ip=10.254.1.203, tcpPort=9876}}
org.onosproject.store.service.ConsistentMapException$Timeout
at org.onosproject.store.consistent.impl.DefaultConsistentMap.complete(DefaultConsistentMap.java:175)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.consistent.impl.DefaultConsistentMap.entrySet(DefaultConsistentMap.java:140)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.consistent.impl.ConsistentMapBackedJavaMap.entrySet(ConsistentMapBackedJavaMap.java:145)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.consistent.impl.ConsistentMapBackedJavaMap.forEach(ConsistentMapBackedJavaMap.java:153)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.consistent.impl.MutexExecutionManager$InternalClusterEventListener.event(MutexExecutionManager.java:192)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.store.consistent.impl.MutexExecutionManager$InternalClusterEventListener.event(MutexExecutionManager.java:184)[145:org.onosproject.onos-core-dist:1.4.0.SNAPSHOT]
at org.onosproject.event.ListenerRegistry.process(ListenerRegistry.java:66)[75:org.onosproject.onos-api:1.4.0.SNAPSHOT]
at org.onosproject.event.impl.CoreEventDispatcher$DispatchLoop.process(CoreEventDispatcher.java:141)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
at org.onosproject.event.impl.CoreEventDispatcher$DispatchLoop.run(CoreEventDispatcher.java:124)[143:org.onosproject.onos-core-net:1.4.0.SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_31]
- relates to
-
ONOS-3622 Stopping ONOS nodes can cause links to disappear
-
- Closed
-