Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-389

ONOS stop working while stressing with push-test-intent

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      commit eb8e3ea61af693118bcee7d8984756a0d8bffa4f
      Author: Brian O'Connor <bocon@onlab.us>
      Date: Wed Dec 3 17:53:33 2014 -0800

      Description

      Under stressed push-test-intent (1000 x10 intents) repeatedly ~10 times simultaneously on three nodes, ONOS stops working with following exceptions:
      2014-12-03 23:29:44,251 | ERROR | onos-flowtracker | Tools | 71 - org.onosproject.onlab-misc - 1.0.0.SNAPSHOT | Uncaught exception on onos-flowtracker
      com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
      at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93)[54:com.hazelcast:3.3.2]
      at com.hazelcast.map.proxy.MapProxySupport.toData(MapProxySupport.java:1043)[54:com.hazelcast:3.3.2]
      at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:83)[54:com.hazelcast:3.3.2]
      at org.onosproject.store.hz.SMap.get(SMap.java:131)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
      at org.onosproject.store.intent.impl.HazelcastIntentStore.getIntent(HazelcastIntentStore.java:248)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.IntentManager.buildAndSubmitBatches(IntentManager.java:505)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.IntentManager.access$300(IntentManager.java:80)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.IntentManager$InternalTopoChangeDelegate.triggerCompile(IntentManager.java:548)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.ObjectiveTracker$TopologyChangeHandler.run(ObjectiveTracker.java:210)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_25]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_25]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
      2014-12-03 23:29:44,268 | ERROR | onos-intent-26 | Tools | 71 - org.onosproject.onlab-misc - 1.0.0.SNAPSHOT | Uncaught exception on onos-intent-26
      com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
      at com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:81)[54:com.hazelcast:3.3.2]
      at com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:76)[54:com.hazelcast:3.3.2]
      at com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:70)[54:com.hazelcast:3.3.2]
      at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:86)[54:com.hazelcast:3.3.2]
      at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:144)[54:com.hazelcast:3.3.2]
      at com.hazelcast.queue.impl.proxy.QueueProxyImpl.remove(QueueProxyImpl.java:134)[54:com.hazelcast:3.3.2]
      at org.onosproject.store.hz.SQueue.remove(SQueue.java:124)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
      at org.onosproject.store.intent.impl.HazelcastIntentBatchQueue.removeIntentOperations(HazelcastIntentBatchQueue.java:179)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.abandonShip(IntentManager.java:854)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.run(IntentManager.java:945)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_25]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_25]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
      2014-12-03 23:29:44,268 | ERROR | onos-intent-20 | Tools | 71 - org.onosproject.onlab-misc - 1.0.0.SNAPSHOT | Uncaught exception on onos-intent-20
      com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
      at com.hazelcast.spi.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:81)[54:com.hazelcast:3.3.2]
      at com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:76)[54:com.hazelcast:3.3.2]
      at com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:70)[54:com.hazelcast:3.3.2]
      at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:86)[54:com.hazelcast:3.3.2]
      at com.hazelcast.queue.impl.proxy.QueueProxyImpl.poll(QueueProxyImpl.java:144)[54:com.hazelcast:3.3.2]
      at com.hazelcast.queue.impl.proxy.QueueProxyImpl.remove(QueueProxyImpl.java:134)[54:com.hazelcast:3.3.2]
      at org.onosproject.store.hz.SQueue.remove(SQueue.java:124)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
      at org.onosproject.store.intent.impl.HazelcastIntentBatchQueue.removeIntentOperations(HazelcastIntentBatchQueue.java:179)[76:org.onosproject.onos-core-dist:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.abandonShip(IntentManager.java:854)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at org.onosproject.net.intent.impl.IntentManager$IntentInstallMonitor.run(IntentManager.java:945)[75:org.onosproject.onos-core-net:1.0.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_25]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_25]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_25]
      2014-12-03 23:29:44,268 | ERROR | onos-intent-27 | Tools

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            suibin suibin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: