-
Type: Bug
-
Status: Closed (View Workflow)
-
Priority: Minor
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: 1.3.0
-
Component/s: None
-
Labels:None
-
Epic Link:
-
Sprint:Drake Sprint 4 (8/31-9/4) 2
ONOS sometimes fails to detect all links in a mininet network regularly but not consistently. Detected in HA and FUNC tests. Below is debug information from FUNCintent afflicted with the problem.
Note: Problem encountered both before and after a topology change. The number and type of links found in each is variable. Additionally, re-runs occasionally yield different results, including a match vs a mismatch.
015-09-08 13:46:52,949 - FUNCintent - INFO - Command 'devices -j' sent to ONOScli1.
2015-09-08 13:46:52,950 - FUNCintent - ERROR - [{"id":"of:0000000000000001","type":"SWITCH","available":true,"role":"MASTER","mfr":"Nicira, Inc.","hw":"Open vSwitch","sw":"2.3.0","serial":"None","chassisId":"1","annotations":{"protocol":"OF_13","channelId":"10.128.50.10:42296"}},{"id":"of:0000000000000002","type":"SWITCH","available":true,"role":"MASTER","mfr":"Nicira, Inc.","hw":"Open vSwitch","sw":"2.3.0","serial":"None","chassisId":"2","annotations":{"protocol":"OF_13","channelId":"10.128.50.10:42298"}},{"id":"of:0000000000000003","type":"SWITCH","available":true,"role":"STANDBY","mfr":"Nicira, Inc.","hw":"Open vSwitch","sw":"2.3.0","serial":"None","chassisId":"3","annotations":{"protocol":"OF_13","channelId":"10.128.50.10:42162"}},{"id":"of:0000000000000004","type":"SWITCH","available":true,"role":"STANDBY","mfr":"Nicira, Inc.","hw":"Open vSwitch","sw":"2.3.0","serial":"None","chassisId":"4","annotations":{"protocol":"OF_13","channelId":"10.128.50.10:35235"}},{"id":"of:0000000000000005","type":"SWITCH","available":true,"role":"STANDBY","mfr":"Nicira, Inc.","hw":"Open vSwitch","sw":"2.3.0","serial":"None","chassisId":"5","annotations":{"protocol":"OF_13","channelId":"10.128.50.10:42308"}},{"id":"of:0000000000000006","type":"SWITCH","available":true,"role":"STANDBY","mfr":"Nicira, Inc.","hw":"Open vSwitch","sw":"2.3.0","serial":"None","chassisId":"6","annotations":{"protocol":"OF_13","channelId":"10.128.50.10:42171"}},{"id":"of:0000000000000007","type":"SWITCH","available":true,"role":"MASTER","mfr":"Nicira, Inc.","hw":"Open vSwitch","sw":"2.3.0","serial":"None","chassisId":"7","annotations":{"protocol":"OF_13","channelId":"10.128.50.10:42315"}}]
2015-09-08 13:46:52,951 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,951 - FUNCintent - INFO - ONOS1 Switches view is correct
2015-09-08 13:46:52,952 - FUNCintent - INFO - Assertion Passed
2015-09-08 13:46:52,952 - FUNCintent - DEBUG - ONOS links
2015-09-08 13:46:52,952 - FUNCintent - DEBUG - [{u'src':
, u'dst':
{u'device': u'of:0000000000000003', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000007', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000004', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'2'}, u'dst':
{u'device': u'of:0000000000000003', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000005', u'port': u'10'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'3'}, u'dst':
{u'device': u'of:0000000000000004', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000004', u'port': u'1'}, u'dst':
{u'device': u'of:0000000000000001', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'4'}, u'dst':
{u'device': u'of:0000000000000005', u'port': u'9'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000005', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000001', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000006', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'1'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}]
2015-09-08 13:46:52,952 - FUNCintent - ERROR - Mininet has 10 bidirectional links and ONOS has 10 unidirectional links
2015-09-08 13:46:52,952 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/1 -> 0000000000000001/1
2015-09-08 13:46:52,952 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/1 -> 0000000000000001/2
2015-09-08 13:46:52,953 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/2 -> 0000000000000003/2
2015-09-08 13:46:52,953 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/2 -> 0000000000000002/2
2015-09-08 13:46:52,953 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/3 -> 0000000000000005/10
2015-09-08 13:46:52,953 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/4 -> 0000000000000006/9
2015-09-08 13:46:52,953 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/3 -> 0000000000000004/2
2015-09-08 13:46:52,954 - FUNCintent - ERROR - ONOS does not have the link 0000000000000004/2 -> 0000000000000003/3
2015-09-08 13:46:52,954 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/4 -> 0000000000000006/10
2015-09-08 13:46:52,954 - FUNCintent - ERROR - ONOS does not have the link 0000000000000004/3 -> 0000000000000007/9
2015-09-08 13:46:52,954 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,954 - FUNCintent - ERROR - ONOS1 links view is incorrect
[REPORT] [08 Sep 2015 13:46:52] ONOS1 links view is incorrect
2015-09-08 13:46:52,955 - FUNCintent - WARNING - Assertion Failed
2015-09-08 13:46:52,955 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,955 - FUNCintent - INFO - ONOS1 hosts exist in Mininet
2015-09-08 13:46:52,955 - FUNCintent - INFO - Assertion Passed
2015-09-08 13:46:52,956 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,956 - FUNCintent - INFO - ONOS2 Switches view is correct
2015-09-08 13:46:52,956 - FUNCintent - INFO - Assertion Passed
2015-09-08 13:46:52,957 - FUNCintent - DEBUG - ONOS links
2015-09-08 13:46:52,957 - FUNCintent - DEBUG - [{u'src':
, u'dst':
{u'device': u'of:0000000000000003', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000007', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000004', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'2'}, u'dst':
{u'device': u'of:0000000000000003', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000005', u'port': u'10'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'3'}, u'dst':
{u'device': u'of:0000000000000004', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000004', u'port': u'1'}, u'dst':
{u'device': u'of:0000000000000001', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'4'}, u'dst':
{u'device': u'of:0000000000000005', u'port': u'9'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000006', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000005', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000001', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'1'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}]
2015-09-08 13:46:52,957 - FUNCintent - ERROR - Mininet has 10 bidirectional links and ONOS has 10 unidirectional links
2015-09-08 13:46:52,957 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/1 -> 0000000000000001/1
2015-09-08 13:46:52,957 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/1 -> 0000000000000001/2
2015-09-08 13:46:52,957 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/2 -> 0000000000000003/2
2015-09-08 13:46:52,958 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/2 -> 0000000000000002/2
2015-09-08 13:46:52,958 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/3 -> 0000000000000005/10
2015-09-08 13:46:52,958 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/4 -> 0000000000000006/9
2015-09-08 13:46:52,958 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/3 -> 0000000000000004/2
2015-09-08 13:46:52,958 - FUNCintent - ERROR - ONOS does not have the link 0000000000000004/2 -> 0000000000000003/3
2015-09-08 13:46:52,958 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/4 -> 0000000000000006/10
2015-09-08 13:46:52,959 - FUNCintent - ERROR - ONOS does not have the link 0000000000000004/3 -> 0000000000000007/9
2015-09-08 13:46:52,959 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,959 - FUNCintent - ERROR - ONOS2 links view is incorrect
[REPORT] [08 Sep 2015 13:46:52] ONOS2 links view is incorrect
2015-09-08 13:46:52,959 - FUNCintent - WARNING - Assertion Failed
2015-09-08 13:46:52,959 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,960 - FUNCintent - INFO - ONOS2 hosts exist in Mininet
2015-09-08 13:46:52,960 - FUNCintent - INFO - Assertion Passed
2015-09-08 13:46:52,961 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,961 - FUNCintent - INFO - ONOS3 Switches view is correct
2015-09-08 13:46:52,961 - FUNCintent - INFO - Assertion Passed
2015-09-08 13:46:52,961 - FUNCintent - DEBUG - ONOS links
2015-09-08 13:46:52,961 - FUNCintent - DEBUG - [{u'src':
, u'dst':
{u'device': u'of:0000000000000003', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000007', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000004', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'2'}, u'dst':
{u'device': u'of:0000000000000003', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000005', u'port': u'10'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'3'}, u'dst':
{u'device': u'of:0000000000000004', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000004', u'port': u'1'}, u'dst':
{u'device': u'of:0000000000000001', u'port': u'3'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'4'}, u'dst':
{u'device': u'of:0000000000000005', u'port': u'9'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000006', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000005', u'port': u'9'}, u'dst':
{u'device': u'of:0000000000000001', u'port': u'4'}, u'type': u'DIRECT', u'state': u'ACTIVE'}, {u'src':
{u'device': u'of:0000000000000001', u'port': u'1'}, u'dst':
{u'device': u'of:0000000000000002', u'port': u'1'}, u'type': u'DIRECT', u'state': u'ACTIVE'}]
2015-09-08 13:46:52,962 - FUNCintent - ERROR - Mininet has 10 bidirectional links and ONOS has 10 unidirectional links
2015-09-08 13:46:52,962 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/1 -> 0000000000000001/1
2015-09-08 13:46:52,962 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/1 -> 0000000000000001/2
2015-09-08 13:46:52,962 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/2 -> 0000000000000003/2
2015-09-08 13:46:52,963 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/2 -> 0000000000000002/2
2015-09-08 13:46:52,963 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/3 -> 0000000000000005/10
2015-09-08 13:46:52,963 - FUNCintent - ERROR - ONOS does not have the link 0000000000000002/4 -> 0000000000000006/9
2015-09-08 13:46:52,963 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/3 -> 0000000000000004/2
2015-09-08 13:46:52,963 - FUNCintent - ERROR - ONOS does not have the link 0000000000000004/2 -> 0000000000000003/3
2015-09-08 13:46:52,964 - FUNCintent - ERROR - ONOS does not have the link 0000000000000003/4 -> 0000000000000006/10
2015-09-08 13:46:52,964 - FUNCintent - ERROR - ONOS does not have the link 0000000000000004/3 -> 0000000000000007/9
2015-09-08 13:46:52,964 - FUNCintent - INFO - Verifying the Expected is equal to the actual or not using assert_equal
2015-09-08 13:46:52,964 - FUNCintent - ERROR - ONOS3 links view is incorrect
[REPORT] [08 Sep 2015 13:46:52] ONOS3 links view is incorrect
2015-09-08 13:46:52,965 - FUNCintent - WARNING - Assertion Failed