-
Type: Bug
-
Status: Closed (View Workflow)
-
Priority: Major
-
Resolution: Fixed
-
Affects Version/s: 1.13.0
-
Fix Version/s: 1.13.0
-
Component/s: Platform
-
Labels:
-
Environment:
3 ONOS cluster deployment + 1 Open flow device (OVS)
Version : ONOS 1.12
-
Story Points:2
GossipDeviceStore cannot do 'mark off line' for a removed device in a controller recovered from failure.
Reproduction steps are as follows.
- Starting multi node ONOS cluster deployment (in my case 3 node cluster)
- Adding a open flow device on the cluster
- Make a ONOS node shutdown (Remaining 2 ONOS nodes are working)
- Remove the open flow device by using 'DeviceAdminService.removeDevice(of device id)'
- Making the shut-downed ONOS alive (All 3 ONOS nodes are working)
- The ONOS recovered from failure prints the below Exception message continuously, and it cannot do 'mark off line' for the device which was deleted while the ONOS node was in failure.
2018-01-13 05:45:41,003 | WARN | onos-device-fg-1 | GossipDeviceStore | 147 | Exception thrown handling Device advertisements. java.lang.IllegalArgumentException: Must be MastershipBasedTimestamp [null] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:203)[94:com.google.guava:22.0.0] at org.onosproject.store.impl.MastershipBasedTimestamp.compareTo(MastershipBasedTimestamp.java:61)[147:org.onosproject.onos-core-dist:1.12.0] at org.onosproject.store.impl.MastershipBasedTimestamp.compareTo(MastershipBasedTimestamp.java:35)[147:org.onosproject.onos-core-dist:1.12.0] at org.onosproject.store.device.impl.GossipDeviceStore.handleAdvertisement(GossipDeviceStore.java:1463)[147:org.onosproject.onos-core-dist:1.12.0] at org.onosproject.store.device.impl.GossipDeviceStore.handleDeviceAdvertisement(GossipDeviceStore.java:1650)[147:org.onosproject.onos-core-dist:1.12.0] at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalMessageConsumer.accept(ClusterCommunicationManager.java:336)[147:org.onosproject.onos-core-dist:1.12.0] at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalMessageConsumer.accept(ClusterCommunicationManager.java:325)[147:org.onosproject.onos-core-dist:1.12.0] at org.onosproject.store.cluster.messaging.impl.NettyMessagingManager.lambda$null$15(NettyMessagingManager.java:439) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_102] at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
I checked the 'offline' of GossipDeviceStore with Debugger
The open flow device device id is of:c0a80ab30000002.
1. The offline of the ONOS (which became shutdown) is as below. It does not have of:c0a80ab30000002 in 'offline'.
The offline of the remaining ONOS are as below. They have of:c0a80ab30000002 in 'offline'.