Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-4515

Cluster Device Role States out of Sync

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
    • Environment:

      3 node cluster

    • Story Points:
      3
    • Epic Link:
    • Sprint:
      Goldeneye Black Sprint #4

      Description

      the device roles in ONOS are out of sync between nodes in the cluster. one node is seeing five devices with the role none, and the other two nodes see the five devices with standby as the role.

      This bug was produced by starting the mininet topology first, then starting ONOS and connecting the switches in mininet using the sh ovs-vsctl set-controller command in the mininet prompt.

      There are seven switches in the topology that was started, but each onos node only saw the same five switches and reported that there were to SCC(s). All ONOS nodes are in a READY state and all the tcp connections exist on the mininet machine.

      When this problem occurs, the karaf.log very rapidly fills with the warnings seen and quickly grows to large sizes.

      The ONOS karaf.log from the node seeing the problem:

      2016-05-10 15:22:21,345 | INFO  | ew I/O worker #1 | OFChannelHandler
             | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | New switch connec
      tion from /10.128.50.10:51746
      2016-05-10 15:22:21,362 | INFO  | ew I/O worker #1 | OFChannelHandler
             | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | Sending OF_13 Hel
      lo to /10.128.50.10:51746
      2016-05-10 15:22:21,388 | INFO  | ew I/O worker #1 | OFChannelHandler
             | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | Received port des
      c reply for switch at [/10.128.50.10:51746 DPID[00:00:00:00:00:00:00:01]]
      2016-05-10 15:22:21,407 | INFO  | ew I/O worker #1 | OFChannelHandler
             | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | Received switch d
      escription reply OFDescStatsReplyVer13(xid=4294967289, flags=[], mfrDesc=Nicira
      , Inc., hwDesc=Open vSwitch, swDesc=2.3.1, serialNum=None, dpDesc=None) from sw
      itch at /10.128.50.10:51746
      2016-05-10 15:22:21,414 | INFO  | ew I/O worker #1 | Controller
             | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | OpenFlow handshak
      er found for device 1: org.onosproject.driver.handshaker.NiciraSwitchHandshaker
       [? DPID[00:00:00:00:00:00:00:01]]
      2016-05-10 15:22:21,414 | INFO  | ew I/O worker #1 | ntrollerImpl$OpenFlowSwitc
      hAgent | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | Added switch 00:0
      0:00:00:00:00:00:01
      2016-05-10 15:22:21,532 | INFO  | event-dispatch-0 | FlowObjectiveManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Driver ovs boun
      d to device of:0000000000000001 ... initializing driver
      2016-05-10 15:22:21,549 | INFO  | nos-topo-build-1 | TopologyManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Topology Defaul
      tTopology{time=710344648978180, creationTime=1462918941546, computeCost=535068,
       clusters=1, devices=1, links=0} changed
      2016-05-10 15:22:21,569 | INFO  | ew I/O worker #1 | DeviceManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Local role is S
      TANDBY for of:0000000000000001
      2016-05-10 15:22:21,585 | INFO  | ew I/O worker #1 | DeviceManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Device of:00000
      00000000001 connected
      2016-05-10 15:22:21,587 | INFO  | ew I/O worker #1 | PortStatsCollector
             | 180 - org.onosproject.onos-of-provider-device - 1.6.0.SNAPSHOT | Start
      ing Port Stats collection thread for 00:00:00:00:00:00:00:01
      2016-05-10 15:22:21,592 | INFO  | ew I/O worker #1 | GroupStatsCollector
             | 183 - org.onosproject.onos-of-provider-group - 1.6.0.SNAPSHOT | Starti
      ng Group Stats collection thread for 00:00:00:00:00:00:00:01
      2016-05-10 15:22:21,593 | INFO  | ew I/O worker #1 | MeterStatsCollector
             | 184 - org.onosproject.onos-of-provider-meter - 1.6.0.SNAPSHOT | Starti
      ng Meter Stats collection thread for 00:00:00:00:00:00:00:01
      2016-05-10 15:22:21,594 | INFO  | ew I/O worker #1 | OFChannelHandler
             | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | Processing 0 pend
      ing port status messages for 00:00:00:00:00:00:00:01
      2016-05-10 15:22:21,597 | WARN  | ew I/O worker #1 | DeviceManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Role mismatch o
      n of:0000000000000001. set to STANDBY, but store demands NONE
      2016-05-10 15:22:21,597 | INFO  | ew I/O worker #1 | ntrollerImpl$OpenFlowSwitc
      hAgent | 179 - org.onosproject.onos-of-ctl - 1.6.0.SNAPSHOT | Transitioned swit
      ch 00:00:00:00:00:00:00:01 to EQUAL
      2016-05-10 15:22:21,794 | WARN  | ew I/O worker #1 | DeviceManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Role mismatch o
      n of:0000000000000001. set to STANDBY, but store demands NONE
      2016-05-10 15:22:21,868 | WARN  | ew I/O worker #1 | DeviceManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Role mismatch o
      n of:0000000000000001. set to STANDBY, but store demands NONE
      2016-05-10 15:22:22,043 | WARN  | ew I/O worker #1 | DeviceManager
             | 129 - org.onosproject.onos-core-net - 1.6.0.SNAPSHOT | Role mismatch o
      n of:0000000000000001. set to STANDBY, but store demands NONE
      

      The mininet log:

      *** Creating network
      *** Adding hosts:
      h1 h2 h3 h4 h5 h6 h7 h8 h9 h10 h11 h12 h13 h14 h15 h16 h17 h18 h19 h20 h21 h22 h23 h24
      *** Adding switches:
      s1 s2 s3 s4 s5 s6 s7
      *** Adding links:
      (s1, s2) (s1, s3) (s1, s4) (s1, s5) (s2, s3) (s2, s5) (s2, s6) (s3, s4) (s3, s6) (s4, s7) (s5, h1) (s5, h2) (s5, h3) (s5, h4) (s5, h5) (s5, h6) (s5, h7) (s5, h8) (s6, h9) (s6, h10) (s6, h11) (s6, h12) (s6, h13) (s6, h14) (s6, h15) (s6, h16) (s7, h17) (s7, h18) (s7, h19) (s7, h20) (s7, h21) (s7, h22) (s7, h23) (s7, h24)
      *** Configuring hosts
      h1 h2 h3 h4 h5 h6 h7 h8 h9 h10 h11 h12 h13 h14 h15 h16 h17 h18 h19 h20 h21 h22 h23 h24
      *** Starting controller
      
      *** Starting 7 switches
      s1 s2 s3 s4 s5 s6 s7 ...
      *** Starting CLI:
      mininet> sh ovs-vsctl set-controller s1 tcp:10.128.50.11:6653 tcp:10.128.50.12:6653 tcp:10.128.50.13:6653
      mininet> sh ovs-vsctl set-controller s2 tcp:10.128.50.11:6653 tcp:10.128.50.12:6653 tcp:10.128.50.13:6653
      mininet> sh ovs-vsctl set-controller s3 tcp:10.128.50.11:6653 tcp:10.128.50.12:6653 tcp:10.128.50.13:6653
      mininet> sh ovs-vsctl set-controller s4 tcp:10.128.50.11:6653 tcp:10.128.50.12:6653 tcp:10.128.50.13:6653
      mininet> sh ovs-vsctl set-controller s5 tcp:10.128.50.11:6653 tcp:10.128.50.12:6653 tcp:10.128.50.13:6653
      mininet> sh ovs-vsctl set-controller s6 tcp:10.128.50.11:6653 tcp:10.128.50.12:6653 tcp:10.128.50.13:6653
      mininet> sh ovs-vsctl set-controller s7 tcp:10.128.50.11:6653 tcp:10.128.50.12:6653 tcp:10.128.50.13:6653
      

      The TCP connections:

      tcp        0      0 10.128.50.10:51766      10.128.50.11:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:33196      10.128.50.12:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:33188      10.128.50.12:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:55077      10.128.50.13:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:51753      10.128.50.11:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:33198      10.128.50.12:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:33184      10.128.50.12:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:55081      10.128.50.13:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:51759      10.128.50.11:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:55071      10.128.50.13:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:55085      10.128.50.13:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:51757      10.128.50.11:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:55072      10.128.50.13:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:33180      10.128.50.12:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:55087      10.128.50.13:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:55079      10.128.50.13:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:51746      10.128.50.11:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:33185      10.128.50.12:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:33193      10.128.50.12:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:51750      10.128.50.11:6653       ESTABLISHED
      tcp        0      0 10.128.50.10:51761      10.128.50.11:6653       ESTABLISHED
      

        Attachments

        1. newFuncTopo.py
          6 kB
        2. karaf.log
          7.76 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            ximara Jeremy Songster
            Reporter:
            ximara Jeremy Songster
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: