Steps in my test before failure:
Start 7 node ONOS cluster
start MN network
assign each switch to all 7 controllers
reassign mastership through ONOS cli to have deterministic assignments
pingall(to discover hosts)
wait 10 secs for flow rules to timeout
add 10 host intents through ONOS4 cli
ping across intents
restart ONOS 1,2,3
Check the state of the system
At this point there are only 9 of the 10 intents. The intent view is consistent across all nodes and all 9 are in the installed state.
There are some random flow rules missing from the switches. The test pings across the intents and 3 of the 10 paths had all the needed flows for the pings to go through.
Later the flow entries are restored for the 9 remaining intents.
I have a pcap file but it is too large for Jira