-
Type: Bug
-
Status: Closed (View Workflow)
-
Priority: Critical
-
Resolution: Done
-
Affects Version/s: 1.2.0
-
Fix Version/s: 1.2.0
-
Component/s: None
-
Labels:
-
Epic Link:
-
Sprint:Cardinal Sprint 4 (5/4-5/15)
It is observed that every time after system:shutdown is issued at once on all nodes in a 5-nodes cluster the very first ONOS node fails to come up and karaf console cannot be accessed for this instance.
Below are the session logs and a detailed onos-log from ONOS-1 is attached as a text file to this issue.
Logs show ERRORs at
#####################################
thirdparty - 1.2.0.SNAPSHOT | tcp://10.128.40.41:7238 - Received accepted poll from tcp://10.128.40.41:7238
2015-05-10 21:26:42,912 | ERROR | edFlowRuleStore) | configadmin | 6 - org.apache.felix.configadmin - 1.8.0 | Unexpected problem delivering configuration event to [org.osgi.service.cm.ConfigurationListener, id=63, bundle=39/mvn:org.apache.felix/org.apache.felix.scr/1.8.2]
java.lang.IllegalStateException: Could not obtain lock
at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainLock(AbstractComponentManager.java:198)[39:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainActivationWriteLock(AbstractComponentManager.java:233)[39:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:614)[39:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.configurationUpdated(ConfigurableComponentHolder.java:328)[39:org.apache.felix.scr:1.8.2]
at org.apache.felix.scr.impl.config.ConfigurationSupport.configurationEvent(ConfigurationSupport.java:290)[39:org.apache.felix.scr:1.8.2]
at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.sendEvent(ConfigurationManager.java:2032)[6:org.apache.felix.configadmin:1.8.0]
at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.run(ConfigurationManager.java:2002)[6:org.apache.felix.configadmin:1.8.0]
at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)[6:org.apache.felix.configadmin:1.8.0]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
##################################################
From ONOS build machine
########################
admin@chONOSbench:~/ONOS$ git log -1
commit 16f857baa359445bf8dbb8e6cee6924ebd23c03c
Author: Marc De Leenheer <marc@onlab.us>
Date: Tue May 5 20:50:24 2015 -0700
First round of cleanups in optical path provisioner. No more user input for packet/optical mininet script.
Change-Id: Ibbfa6a17a97432da8dee63e9cd15fa6b1c2c1e46
admin@chONOSbench:~/ONOS$ cell
ONOS_CELL=choTest5
OCI=10.128.40.41
OC1=10.128.40.41
OC2=10.128.40.42
OC3=10.128.40.43
OC4=10.128.40.44
OC5=10.128.40.45
ONOS_APPS=openflow,drivers,proxyarp,mobility,metrics
ONOS_NIC=10.128.40.*
admin@chONOSbench:~/ONOS$
############################
ONOS-1 Session
#########################
admin@chONOS1:~$ /opt/onos/bin/onos
Logging in as karaf
458 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, e0:ed:b6:7a:24:69:d1:35:87:4e:95:37:4f:af:5e:96] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
onos> add-host-intent 00:00:00:00:00:01/-1 00:00:00:00:00:03/-1
Host to Host intent submitted:
HostToHostIntent{id=0x200000, key=0x200000, appId=DefaultApplicationId
, priority=100, resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1], selector=DefaultTrafficSelector{criteria=[ETH_TYPE
{ethType=800}]}, treatment=DefaultTrafficTreatment{immediate=[], deferred=[], transition=None, cleared=false}, constraints=[LinkTypeConstraint{inclusive=false, types=[OPTICAL]}], one=00:00:00:00:00:01/-1, two=00:00:00:00:00:03/-1}onos> summary
node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=1
onos> intents
id=0x200000, state=FAILED, key=0x200000, type=HostToHostIntent, appId=org.onosproject.cli
resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1]
selector=[ETH_TYPE{ethType=800}
]
constraints=[LinkTypeConstraint
host1=00:00:00:00:00:01/-1, host2=00:00:00:00:00:03/-1
onos> summary
node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=2, links=2, hosts=0, SCC(s)=1, flows=10, intents=1
onos> intents
id=0x200000, state=INSTALLED, key=0x200000, type=HostToHostIntent, appId=org.onosproject.cli
resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1]
selector=[ETH_TYPE{ethType=800}]
constraints=[LinkTypeConstraint{inclusive=false, types=[OPTICAL]}
]
host1=00:00:00:00:00:01/-1, host2=00:00:00:00:00:03/-1
onos> intents
id=0x200000, state=INSTALLED, key=0x200000, type=HostToHostIntent, appId=org.onosproject.cli
resources=[00:00:00:00:00:01/-1, 00:00:00:00:00:03/-1]
selector=[ETH_TYPE
]
constraints=[LinkTypeConstraint
]
host1=00:00:00:00:00:01/-1, host2=00:00:00:00:00:03/-1
onos> summary
node=10.128.40.41, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=13, intents=1
onos>
onos>
onos>
onos>
onos> nodes
id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago *
id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago
id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago
id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago
onos>
onos>
onos> system:shutdown
Confirm: halt instance root (yes/no): yes
onos> admin@chONOS1:~$ /opt/onos/bin/onos
Logging in as karaf
Failed to get the session.
admin@chONOS1:~$ /opt/onos/bin/onos
Logging in as karaf
Failed to get the session.
admin@chONOS1:~$
###########################
ONOS-2 Session
###########################
admin@chONOS2:~$ /opt/onos/bin/onos
Logging in as karaf
459 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 6c:f4:17:b6:5c:c3:44:e6:d6:0c:d5:f8:f9:4a:f8:3f] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.42, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
onos> summary
node=10.128.40.42, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=13, intents=1
onos>
onos>
onos> nodes
id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago
id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago *
id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago
id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago
onos>
onos>
onos> system:shutdown
Confirm: halt instance root (yes/no): yes
onos> admin@chONOS2:~$ /opt/onos/bin/onos
Logging in as karaf
550 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 6c:f4:17:b6:5c:c3:44:e6:d6:0c:d5:f8:f9:4a:f8:3f] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.42, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=12, intents=1
onos> log:exception-display
onos>
############################
ONOS-3 Session
############################
admin@chONOS3:~$ /opt/onos/bin/onos
Logging in as karaf
474 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 90:a2:de:82:50:92:f6:5c:e1:00:c7:92:db:fd:b4:23] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.43, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
onos>
onos>
onos> nodes
id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=20m ago
id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=20m ago
id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=21m ago *
id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=20m ago
id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=20m ago
onos>
onos>
onos> system:shutdown
Confirm: halt instance root (yes/no): yes
onos> admin@chONOS3:~$ /opt/onos/bin/onos
Logging in as karaf
522 [sshd-SshClient[55a1c291]-nio2-thread-3] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 90:a2:de:82:50:92:f6:5c:e1:00:c7:92:db:fd:b4:23] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.43, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=12, intents=1
onos> log:exception-display
onos>
####################################
ONOS-4 session
##################################
admin@chONOS4:~$ /opt/onos/bin/onos
Logging in as karaf
480 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 5a:28:d3:f1:c0:e5:0b:98:0b:c0:e4:26:a2:d6:66:61] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.44, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
onos> summary
node=10.128.40.44, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=13, intents=1
onos>
onos>
onos> nodes
id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago
id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago
id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago *
id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago
onos>
onos>
onos> system:shutdown
Confirm: halt instance root (yes/no): yes
onos> admin@chONOS4:~$ /opt/onos/bin/onos
Logging in as karaf
531 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 5a:28:d3:f1:c0:e5:0b:98:0b:c0:e4:26:a2:d6:66:61] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> nodes
id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=1m ago
id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=1m ago
id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=1m ago
id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=1m ago *
id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=1m ago
onos> log:exception-display
onos>
#################################
ONOS-5 Session
########################
admin@chONOS5:~$ /opt/onos/bin/onos
Logging in as karaf
446 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 13:2b:0b:ce:31:d3:28:69:ac:db:58:c5:43:8e:2c:c1] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.45, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=0, links=0, hosts=0, SCC(s)=0, flows=0, intents=0
onos>
onos>
onos> nodes
id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=21m ago
id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=21m ago
id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=20m ago
id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=21m ago
id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=21m ago *
onos>
onos>
onos> system:shutdown
Confirm: halt instance root (yes/no): yes
onos> admin@chONOS5:~$ /opt/onos/bin/onos
Logging in as karaf
512 [sshd-SshClient[55a1c291]-nio2-thread-2] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 13:2b:0b:ce:31:d3:28:69:ac:db:58:c5:43:8e:2c:c1] presented unverified {} key: {}
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ // / / // /\ \
___//|/__/__/
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.
onos> summary
node=10.128.40.45, version=1.2.0.admin~2015/05/10@21:03
nodes=5, devices=2, links=2, hosts=2, SCC(s)=1, flows=12, intents=1
onos> log:exception-display
onos>