Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-1895

Always the first node in 5-node cluster fails to come up after system:shutdown

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
    • Epic Link:
    • Sprint:
      Cardinal Sprint 4 (5/4-5/15)

      Description

      It is observed that every time after system:shutdown is issued at once on all nodes in a 5-nodes cluster the very first ONOS node fails to come up and karaf console cannot be accessed for this instance.
      Below are the session logs and a detailed onos-log from ONOS-1 is attached as a text file to this issue.
      Logs show ERRORs at
      #####################################
      thirdparty - 1.2.0.SNAPSHOT | tcp://10.128.40.41:7238 - Received accepted poll from tcp://10.128.40.41:7238
      2015-05-10 21:26:42,912 | ERROR | edFlowRuleStore) | configadmin | 6 - org.apache.felix.configadmin - 1.8.0 | Unexpected problem delivering configuration event to [org.osgi.service.cm.ConfigurationListener, id=63, bundle=39/mvn:org.apache.felix/org.apache.felix.scr/1.8.2]
      java.lang.IllegalStateException: Could not obtain lock
      at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainLock(AbstractComponentManager.java:198)[39:org.apache.felix.scr:1.8.2]
      at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainActivationWriteLock(AbstractComponentManager.java:233)[39:org.apache.felix.scr:1.8.2]
      at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:614)[39:org.apache.felix.scr:1.8.2]
      at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.configurationUpdated(ConfigurableComponentHolder.java:328)[39:org.apache.felix.scr:1.8.2]
      at org.apache.felix.scr.impl.config.ConfigurationSupport.configurationEvent(ConfigurationSupport.java:290)[39:org.apache.felix.scr:1.8.2]
      at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.sendEvent(ConfigurationManager.java:2032)[6:org.apache.felix.configadmin:1.8.0]
      at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.run(ConfigurationManager.java:2002)[6:org.apache.felix.configadmin:1.8.0]
      at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)[6:org.apache.felix.configadmin:1.8.0]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]

      ##################################################
      From ONOS build machine
      ########################
      admin@chONOSbench:~/ONOS$ git log -1
      commit 16f857baa359445bf8dbb8e6cee6924ebd23c03c
      Author: Marc De Leenheer <marc@onlab.us>
      Date: Tue May 5 20:50:24 2015 -0700

      First round of cleanups in optical path provisioner. No more user input for packet/optical mininet script.

      Change-Id: Ibbfa6a17a97432da8dee63e9cd15fa6b1c2c1e46
      admin@chONOSbench:~/ONOS$ cell
      ONOS_CELL=choTest5
      OCI=10.128.40.41
      OC1=10.128.40.41
      OC2=10.128.40.42
      OC3=10.128.40.43
      OC4=10.128.40.44
      OC5=10.128.40.45
      ONOS_APPS=openflow,drivers,proxyarp,mobility,metrics
      ONOS_NIC=10.128.40.*
      admin@chONOSbench:~/ONOS$

      ############################
      ONOS-1 Session
      #########################
      admin@chONOS1:~$ /opt/onos/bin/onos
      Logging in as karaf
      458 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, e0:ed:b6:7a:24:69:d1:35:87:4e:95:37:4f:af:5e:96] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
      onos> add-host-intent 00:00:00:00:00:01/-1 00:00:00:00:00:03/-1
      Host to Host intent submitted:
      HostToHostIntent{id=0x200000, key=0x200000, appId=DefaultApplicationId

      {id=3, name=org.onosproject.cli}

      , priority=100, resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1], selector=DefaultTrafficSelector{criteria=[ETH_TYPE

      {ethType=800}]}, treatment=DefaultTrafficTreatment{immediate=[], deferred=[], transition=None, cleared=false}, constraints=[LinkTypeConstraint{inclusive=false, types=[OPTICAL]}], one=00:00:00:00:00:01/-1, two=00:00:00:00:00:03/-1}
      onos> summary
      node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=1
      onos> intents
      id=0x200000, state=FAILED, key=0x200000, type=HostToHostIntent, appId=org.onosproject.cli
      resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1]
      selector=[ETH_TYPE{ethType=800}

      ]
      constraints=[LinkTypeConstraint

      {inclusive=false, types=[OPTICAL]}]
      host1=00:00:00:00:00:01/-1, host2=00:00:00:00:00:03/-1
      onos> summary
      node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=2, links=2, hosts=0, SCC(s)=1, flows=10, intents=1
      onos> intents
      id=0x200000, state=INSTALLED, key=0x200000, type=HostToHostIntent, appId=org.onosproject.cli
      resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1]
      selector=[ETH_TYPE{ethType=800}]
      constraints=[LinkTypeConstraint{inclusive=false, types=[OPTICAL]}

      ]
      host1=00:00:00:00:00:01/-1, host2=00:00:00:00:00:03/-1
      onos> intents
      id=0x200000, state=INSTALLED, key=0x200000, type=HostToHostIntent, appId=org.onosproject.cli
      resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1]
      selector=[ETH_TYPE

      {ethType=800}

      ]
      constraints=[LinkTypeConstraint

      {inclusive=false, types=[OPTICAL]}

      ]
      host1=00:00:00:00:00:01/-1, host2=00:00:00:00:00:03/-1
      onos> summary
      node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=13, intents=1
      onos>
      onos>
      onos>
      onos>
      onos> nodes
      id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago *
      id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
      id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago
      onos>
      onos>
      onos> system:shutdown
      Confirm: halt instance root (yes/no): yes
      onos> admin@chONOS1:~$ /opt/onos/bin/onos
      Logging in as karaf
      Failed to get the session.
      admin@chONOS1:~$ /opt/onos/bin/onos
      Logging in as karaf
      Failed to get the session.
      admin@chONOS1:~$

      ###########################
      ONOS-2 Session
      ###########################
      admin@chONOS2:~$ /opt/onos/bin/onos
      Logging in as karaf
      459 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 6c:f4:17:b6:5c:c3:44:e6:d6:0c:d5:f8:f9:4a:f8:3f] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.42, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
      onos> summary
      node=10.128.40.42, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=13, intents=1
      onos>
      onos>
      onos> nodes
      id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago *
      id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
      id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago
      onos>
      onos>
      onos> system:shutdown
      Confirm: halt instance root (yes/no): yes
      onos> admin@chONOS2:~$ /opt/onos/bin/onos
      Logging in as karaf
      550 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 6c:f4:17:b6:5c:c3:44:e6:d6:0c:d5:f8:f9:4a:f8:3f] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.42, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=12, intents=1
      onos> log:exception-display
      onos>

      ############################
      ONOS-3 Session
      ############################
      admin@chONOS3:~$ /opt/onos/bin/onos
      Logging in as karaf
      474 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 90:a2:de:82:50:92:f6:5c:e1:00:c7:92:db:fd:b4:23] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.43, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
      onos>
      onos>
      onos> nodes
      id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=20m ago
      id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=20m ago
      id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=21m ago *
      id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=20m ago
      id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=20m ago
      onos>
      onos>
      onos> system:shutdown
      Confirm: halt instance root (yes/no): yes
      onos> admin@chONOS3:~$ /opt/onos/bin/onos
      Logging in as karaf
      522 [sshd-SshClient[55a1c291]-nio2-thread-3] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 90:a2:de:82:50:92:f6:5c:e1:00:c7:92:db:fd:b4:23] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.43, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=12, intents=1
      onos> log:exception-display
      onos>

      ####################################
      ONOS-4 session
      ##################################
      admin@chONOS4:~$ /opt/onos/bin/onos
      Logging in as karaf
      480 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 5a:28:d3:f1:c0:e5:0b:98:0b:c0:e4:26:a2:d6:66:61] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.44, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
      onos> summary
      node=10.128.40.44, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=13, intents=1
      onos>
      onos>
      onos> nodes
      id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
      id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago *
      id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago
      onos>
      onos>
      onos> system:shutdown
      Confirm: halt instance root (yes/no): yes
      onos> admin@chONOS4:~$ /opt/onos/bin/onos
      Logging in as karaf
      531 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 5a:28:d3:f1:c0:e5:0b:98:0b:c0:e4:26:a2:d6:66:61] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> nodes
      id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=1m ago
      id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=1m ago
      id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=1m ago
      id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=1m ago *
      id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=1m ago
      onos> log:exception-display
      onos>

      #################################
      ONOS-5 Session
      ########################
      admin@chONOS5:~$ /opt/onos/bin/onos
      Logging in as karaf
      446 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 13:2b:0b:ce:31:d3:28:69:ac:db:58:c5:43:8e:2c:c1] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.45, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
      onos>
      onos>
      onos> nodes
      id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
      id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago
      id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago *
      onos>
      onos>
      onos> system:shutdown
      Confirm: halt instance root (yes/no): yes
      onos> admin@chONOS5:~$ /opt/onos/bin/onos
      Logging in as karaf
      512 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 13:2b:0b:ce:31:d3:28:69:ac:db:58:c5:43:8e:2c:c1] presented unverified {} key: {}
      Welcome to Open Network Operating System (ONOS)!
      ____ _ ______ ____
      / __ \/ |/ / __ \/ __/
      / // / / // /\ \
      ___//|/__/__/

      Hit '<tab>' for a list of available commands
      and '[cmd] --help' for help on a specific command.
      Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

      onos> summary
      node=10.128.40.45, version=1.2.0.admin~2015/05/10@21:03
      nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=12, intents=1
      onos> log:exception-display
      onos>

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            hari hari
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: