Consistency issue in distributed store (Cluster environment)

From: https://groups.google.com/a/onosproject.org/forum/#!topic/onos-dev/QD_sAwnaK6I
I recently experienced that there was inconsistency of data in different ONOS controllers about a particular device where master device had the correct information while the other noded didn't.

Reproducible:
1) Make a cluster of three controllers:
onos> nodes
id=192.168.10.138, address=192.168.10.138:9876, state=READY, updated=1h35m ago *
id=192.168.10.139, address=192.168.10.139:9876, state=READY, updated=1h35m ago
id=192.168.10.140, address=192.168.10.140:9876, state=READY, updated=1h34m ago
2) Add the controller configuration on a openflow switch on a Rest Server (like Cisco switch or Arista switch)
3) Add a few physical ports to the switch
4) Execute masters cli:
onos> masters
192.168.10.138: 1 devices
of:000100c88b56a3af
192.168.10.139: 1 devices
rest:192.168.10.235:80
192.168.10.140: 0 devices
5) Execute ports cli on all the three nodes and I found the same configuration
onos> ports of:000100c88b56a3af
id=of:000100c88b56a3af, available=true, local-status=connected 2m46s ago, role=MASTER, type=SWITCH, mfr=Cisco Systems, Inc., hw=N9K-C9372PX-E V01, sw=7.0(3)I2(2)| of_agent 2.1.5, serial=SAL2003X3R5, driver=default, channelId=192.168.10.235:59862, managementAddress=192.168.10.235, protocol=OF_13
port=3, state=disabled, type=copper, speed=1410065 , portName=Eth1/3, portMac=00:c8:8b:56:a3:b2
port=49, state=enabled, type=copper, speed=1345294 , portName=Eth1/49, portMac=00:c8:8b:56:a3:e0
port=50, state=disabled, type=copper, speed=1345294 , portName=Eth1/50, portMac=00:c8:8b:56:a3:e1
port=51, state=disabled, type=copper, speed=1345294 , portName=Eth1/51, portMac=00:c8:8b:56:a3:e2
5) Delete ports from the switch
The switch shows: No interfaces configured for the switch
Controller 1:192.168.10.138
---------------------
onos> ports of:000100c88b56a3af
id=of:000100c88b56a3af, available=true, local-status=connected 7m46s ago, role=MASTER, type=SWITCH, mfr=Cisco Systems, Inc., hw=N9K-C9372PX-E V01, sw=7.0(3)I2(2)| of_agent 2.1.5, serial=SAL2003X3R5, driver=default, channelId=192.168.10.235:59862, managementAddress=192.168.10.235, protocol=OF_13
onos>
---------------------

Controller 2: 192.168.10.139
-----------------------
onos> ports of:000100c88b56a3af
id=of:000100c88b56a3af, available=true, local-status=connected 8m22s ago, role=STANDBY, type=SWITCH, mfr=Cisco Systems, Inc., hw=N9K-C9372PX-E V01, sw=7.0(3)I2(2)| of_agent 2.1.5, serial=SAL2003X3R5, driver=ciscoof, channelId=192.168.10.235:59862, managementAddress=192.168.10.235, protocol=OF_13
port=3, state=disabled, type=copper, speed=1410065 , portName=Eth1/3, portMac=00:c8:8b:56:a3:b2
port=49, state=disabled, type=copper, speed=1345294 , portName=Eth1/49, portMac=00:c8:8b:56:a3:e0
port=50, state=disabled, type=copper, speed=1345294 , portName=Eth1/50, portMac=00:c8:8b:56:a3:e1
port=51, state=disabled, type=copper, speed=1345294 , portName=Eth1/51, portMac=00:c8:8b:56:a3:e2
onos>
------------------------

Controller 3: 192.168.10.140
--------------------------
onos> ports of:000100c88b56a3af
id=of:000100c88b56a3af, available=true, local-status=connected 9m12s ago, role=STANDBY, type=SWITCH, mfr=Cisco Systems, Inc., hw=N9K-C9372PX-E V01, sw=7.0(3)I2(2)| of_agent 2.1.5, serial=SAL2003X3R5, driver=default, channelId=192.168.10.235:59862, managementAddress=192.168.10.235, protocol=OF_13
port=3, state=disabled, type=copper, speed=1410065 , portName=Eth1/3, portMac=00:c8:8b:56:a3:b2
port=49, state=disabled, type=copper, speed=1345294 , portName=Eth1/49, portMac=00:c8:8b:56:a3:e0
port=50, state=disabled, type=copper, speed=1345294 , portName=Eth1/50, portMac=00:c8:8b:56:a3:e1
port=51, state=disabled, type=copper, speed=1345294 , portName=Eth1/51, portMac=00:c8:8b:56:a3:e2
onos>
--------------------------

6) For the master controller ports were removed, but the other controllers, it wasn't.