After BM test-station/Mininet server was upgraded to Ubuntu 16.04 recently we started to see an increased latency in switch discovery test between last TCP handshake message and OVS sending `OFPT_HELLO` message to ONOS
Most of the time it's taking over 50ms before OVS to send `OFPT_HELLO`
OVS logs also confirm the delay
It’s happening on all ONOS branches which seems to confirm that it’s coming from the infrastructure. Looks like the Ubuntu upgrade further slowed down OVS a little. But even before the upgrade, it’s much slower than OVS on VMs.
The current test flow is `ovs-vsctl set-controller` for switch-up and then `ovs-vsctl del-controller` for switch-down and repeat that. The large delay of sending `OFPT_HELLO` starts from the 2nd iteration. And if I change the flow to `switch s3 start` -> `ovs-vsctl set-controller` -> `switch s3 stop`. The delay goes away and every time it sends HELLO in `1ms`.
However it's unclear why using `ovs-vsctl del-controller` without restarting the switch slows down OVS next time it connects to ONOS as well as why it’s not the case on VMs.
So since this delay is really not controlled by ONOS, possible solutions are
- Ignore the regression as it’s not really an ONOS issue, or
- Change how we test the latency by 1) changing `tcp_to_feature_reply` to `hello_to_feature_reply` which starts from OVS sending the `HELLO` and removes TCP handshake (which basically happens within 1ms) and the wait time for OVS to send `HELLO` or 2) add `switch start` and `switch stop` into the test flow which will also get rid of the weird OVS waiting time.