-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Done
-
Affects Version/s: 1.0.0
-
Fix Version/s: 1.1.0
-
Component/s: None
-
Labels:None
-
Environment:
commit eb8e3ea61af693118bcee7d8984756a0d8bffa4f
Author: Brian O'Connor <bocon@onlab.us>
Date: Wed Dec 3 17:53:33 2014 -0800
-
Epic Link:
Under stressed push-test-intent (1000 x10 intents) repeatedly ~10 times simultaneously on three nodes, ONOS stops working with following exceptions:
2014-12-03 23:29:44,251 | ERROR | onos-flowtracker | Tools | 71 - org.onosproject.onlab-misc - 1.0.0.SNAPSHOT | Uncaught exception on onos-flowtracker
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93)[54:com.hazelcast:3.3.2]
at com.hazelcast.map.proxy.MapProxySupport.toData(MapProxySupport.java:1043)[54:com.hazelcast:3.3.2]
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:83)[54:com.hazelcast:3.3.2]
at org.onosproject.store.hz.SMap.get(SMap.java:131)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
at org.onosproject.store.intent.impl.HazelcastIntentStore.getIntent(HazelcastIntentStore.java:248)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.IntentManager.buildAndSubmitBatches(IntentManager.java:505)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.IntentManager.access$300(IntentManager.java:80)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.IntentManager$InternalTopoChangeDelegate.triggerCompile(IntentManager.java:548)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.ObjectiveTracker$TopologyChangeHandler.run(ObjectiveTracker.java:210)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_25]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
2014-12-03 23:29:44,268 | ERROR | onos-intent-26 | Tools | 71 - org.onosproject.onlab-misc - 1.0.0.SNAPSHOT | Uncaught exception on onos-intent-26
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:81)[54:com.hazelcast:3.3.2]
at com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:76)[54:com.hazelcast:3.3.2]
at com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:70)[54:com.hazelcast:3.3.2]
at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:86)[54:com.hazelcast:3.3.2]
at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:144)[54:com.hazelcast:3.3.2]
at com.hazelcast.queue.impl.proxy.QueueProxyImpl.remove(QueueProxyImpl.java:134)[54:com.hazelcast:3.3.2]
at org.onosproject.store.hz.SQueue.remove(SQueue.java:124)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
at org.onosproject.store.intent.impl.HazelcastIntentBatchQueue.removeIntentOperations(HazelcastIntentBatchQueue.java:179)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.abandonShip(IntentManager.java:854)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.run(IntentManager.java:945)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_25]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
2014-12-03 23:29:44,268 | ERROR | onos-intent-20 | Tools | 71 - org.onosproject.onlab-misc - 1.0.0.SNAPSHOT | Uncaught exception on onos-intent-20
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:81)[54:com.hazelcast:3.3.2]
at com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:76)[54:com.hazelcast:3.3.2]
at com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:70)[54:com.hazelcast:3.3.2]
at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:86)[54:com.hazelcast:3.3.2]
at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:144)[54:com.hazelcast:3.3.2]
at com.hazelcast.queue.impl.proxy.QueueProxyImpl.remove(QueueProxyImpl.java:134)[54:com.hazelcast:3.3.2]
at org.onosproject.store.hz.SQueue.remove(SQueue.java:124)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
at org.onosproject.store.intent.impl.HazelcastIntentBatchQueue.removeIntentOperations(HazelcastIntentBatchQueue.java:179)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.abandonShip(IntentManager.java:854)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.run(IntentManager.java:945)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_25]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
2014-12-03 23:29:44,268 | ERROR | onos-intent-27 | Tools