Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-7372

GossipDeviceStore cannot do 'mark off line' for a removed device in a controller recovered from failure

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.13.0
    • Fix Version/s: 1.13.0
    • Component/s: Platform
    • Labels:
    • Environment:

      3 ONOS cluster deployment + 1 Open flow device (OVS)

      Version : ONOS 1.12

    • Story Points:
      2

      Description

      GossipDeviceStore cannot do 'mark off line' for a removed device in a controller recovered from  failure.

      Reproduction steps are as follows.

       

      1. Starting multi node ONOS cluster deployment (in my case 3 node cluster)
      2. Adding a open flow device on the cluster
      3. Make a ONOS node shutdown (Remaining 2 ONOS nodes are working)
      4. Remove the open flow device by using 'DeviceAdminService.removeDevice(of device id)'
      5. Making the shut-downed ONOS alive (All 3 ONOS nodes are working)
      6. The ONOS recovered from failure prints the below Exception message continuously, and it cannot do 'mark off line' for the device which was deleted while the ONOS node was in failure.

       

      2018-01-13 05:45:41,003 | WARN  | onos-device-fg-1 | GossipDeviceStore                | 147 | Exception thrown handling Device advertisements.
      java.lang.IllegalArgumentException: Must be MastershipBasedTimestamp [null]
              at com.google.common.base.Preconditions.checkArgument(Preconditions.java:203)[94:com.google.guava:22.0.0]
              at org.onosproject.store.impl.MastershipBasedTimestamp.compareTo(MastershipBasedTimestamp.java:61)[147:org.onosproject.onos-core-dist:1.12.0]
              at org.onosproject.store.impl.MastershipBasedTimestamp.compareTo(MastershipBasedTimestamp.java:35)[147:org.onosproject.onos-core-dist:1.12.0]
              at org.onosproject.store.device.impl.GossipDeviceStore.handleAdvertisement(GossipDeviceStore.java:1463)[147:org.onosproject.onos-core-dist:1.12.0]
              at org.onosproject.store.device.impl.GossipDeviceStore.handleDeviceAdvertisement(GossipDeviceStore.java:1650)[147:org.onosproject.onos-core-dist:1.12.0]
              at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalMessageConsumer.accept(ClusterCommunicationManager.java:336)[147:org.onosproject.onos-core-dist:1.12.0]
              at org.onosproject.store.cluster.messaging.impl.ClusterCommunicationManager$InternalMessageConsumer.accept(ClusterCommunicationManager.java:325)[147:org.onosproject.onos-core-dist:1.12.0]
              at org.onosproject.store.cluster.messaging.impl.NettyMessagingManager.lambda$null$15(NettyMessagingManager.java:439)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_102]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_102]
              at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]

       

      I checked the 'offline' of GossipDeviceStore with Debugger

      The open flow device device id is of:c0a80ab30000002.

       

      1. The offline of the ONOS (which became shutdown) is as below. It does not have of:c0a80ab30000002 in 'offline'.

       

       The offline of the remaining ONOS are as below. They have of:c0a80ab30000002 in 'offline'.

       

       

       

       

       

       

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            jaegonkim Jaegon Kim
            Reporter:
            jaegonkim Jaegon Kim
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: