We start SDN-IP and install some intents. Then we shutdown the FSFW which removes the entire topology, all intents go to FAILED. Then we start up the FSFW which causes the same topology to reconnect to ONOS. Some intents get reinstalled correctly, however some are left in the FAILED state.
We believe this is due to cascading topology changes of the multiple successive topology events which trying to recalculate the intents.
Solved by added FAILED intents to the IntentCleanup thread, meaning that intents that have been FAILED for a while get retried.