Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-1601

Exception caught when issue "roles" cli command (which takes long time to return)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
    • Environment:
    • Epic Link:
    • Sprint:
      Cardinal Sprint 2 (4/6-4/17), Cardinal Sprint 3 (4/20-5/1), Cardinal Sprint 4 (5/4-5/15)

      Description

      running flow-tester.py with 35 devices each node, 7 nodes. When issue "roles" cmd, the command does not return. Check onos log and see the following exceptions:

      2015-04-08 14:23:24,020 | WARN | event-dispatch-0 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] While asking 'is-executing': BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=Operation

      {serviceName='hz:impl:mapService', callId=1585769, invocationTime=1428528079002, waitTimeout=-1, callTimeout=60000}

      , partitionId=-1, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      java.util.concurrent.TimeoutException: Call BasicInvocation{ serviceName='hz:impl:mapService', op=Operation

      {serviceName='hz:impl:mapService', callId=1587116, invocationTime=1428528199015, waitTimeout=-1, callTimeout=5000}

      , partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0} encountered a timeout
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponse(BasicInvocationFuture.java:352)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:320)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:194)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.isOperationExecuting(BasicInvocationFuture.java:421)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:244)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.getQueryResult(BasicMapContextQuerySupport.java:329)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.addResultsOfPredicate(BasicMapContextQuerySupport.java:316)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.query(BasicMapContextQuerySupport.java:227)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxySupport.query(MapProxySupport.java:1060)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:584)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:568)[55:com.hazelcast:3.4.0]
      at org.onosproject.store.hz.SMap.entrySet(SMap.java:421)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.mastership.impl.DistributedMastershipStore.relinquishAllRole(DistributedMastershipStore.java:370)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.cluster.impl.MastershipManager$InternalClusterEventListener.event(MastershipManager.java:308)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at org.onosproject.cluster.impl.MastershipManager$InternalClusterEventListener.event(MastershipManager.java:286)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at org.onosproject.event.AbstractListenerRegistry.process(AbstractListenerRegistry.java:63)[76:org.onosproject.onos-api:1.2.0.SNAPSHOT]
      at org.onosproject.event.impl.CoreEventDispatcher$DispatchLoop.run(CoreEventDispatcher.java:95)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_11]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_11]
      2015-04-08 14:23:24,023 | WARN | event-dispatch-0 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] 'is-executing': false -> BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=Operation

      {serviceName='hz:impl:mapService', callId=1585769, invocationTime=1428528079002, waitTimeout=-1, callTimeout=60000}

      , partitionId=-1, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:23:24,024 | WARN | event-dispatch-0 | BasicMapContextQuerySupport | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] Could not get results
      java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=Operation

      {serviceName='hz:impl:mapService', callId=1585769, invocationTime=1428528079002, waitTimeout=-1, callTimeout=60000}

      , partitionId=-1, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false} No response has been received! backups-expected:0 backups-completed: 0
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:343)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:194)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.getQueryResult(BasicMapContextQuerySupport.java:329)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.addResultsOfPredicate(BasicMapContextQuerySupport.java:316)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.query(BasicMapContextQuerySupport.java:227)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxySupport.query(MapProxySupport.java:1060)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:584)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:568)[55:com.hazelcast:3.4.0]
      at org.onosproject.store.hz.SMap.entrySet(SMap.java:421)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.mastership.impl.DistributedMastershipStore.relinquishAllRole(DistributedMastershipStore.java:370)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.cluster.impl.MastershipManager$InternalClusterEventListener.event(MastershipManager.java:308)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at org.onosproject.cluster.impl.MastershipManager$InternalClusterEventListener.event(MastershipManager.java:286)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at org.onosproject.event.AbstractListenerRegistry.process(AbstractListenerRegistry.java:63)[76:org.onosproject.onos-api:1.2.0.SNAPSHOT]
      at org.onosproject.event.impl.CoreEventDispatcher$DispatchLoop.run(CoreEventDispatcher.java:95)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_11]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_11]
      Caused by: com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=Operation

      {serviceName='hz:impl:mapService', callId=1585769, invocationTime=1428528079002, waitTimeout=-1, callTimeout=60000}

      , partitionId=-1, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false} No response has been received! backups-expected:0 backups-completed: 0
      at com.hazelcast.spi.impl.BasicInvocationFuture.newOperationTimeoutException(BasicInvocationFuture.java:309)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:246)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.getQueryResult(BasicMapContextQuerySupport.java:329)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.addResultsOfPredicate(BasicMapContextQuerySupport.java:316)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.BasicMapContextQuerySupport.query(BasicMapContextQuerySupport.java:227)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxySupport.query(MapProxySupport.java:1060)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:584)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:568)[55:com.hazelcast:3.4.0]
      at org.onosproject.store.hz.SMap.entrySet(SMap.java:421)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.mastership.impl.DistributedMastershipStore.relinquishAllRole(DistributedMastershipStore.java:370)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.cluster.impl.MastershipManager$InternalClusterEventListener.event(MastershipManager.java:308)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at org.onosproject.cluster.impl.MastershipManager$InternalClusterEventListener.event(MastershipManager.java:286)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at org.onosproject.event.AbstractListenerRegistry.process(AbstractListenerRegistry.java:63)[76:org.onosproject.onos-api:1.2.0.SNAPSHOT]
      at org.onosproject.event.impl.CoreEventDispatcher$DispatchLoop.run(CoreEventDispatcher.java:95)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_11]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_11]
      at ------ End remote and begin local stack-trace ------.(Unknown Source)
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponse(BasicInvocationFuture.java:387)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:320)[55:com.hazelcast:3.4.0]
      ... 17 more
      2015-04-08 14:24:17,105 | WARN | ice-installer-29 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] No response for 120000 ms. BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:17,109 | WARN | ice-installer-29 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] Asking if operation execution has been started: BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:18,700 | WARN | ice-installer-12 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] No response for 120000 ms. BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=PutIfAbsentOperation

      {flowtable_null:ffffffff97af0009}

      , partitionId=254, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:18,700 | WARN | ice-installer-12 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] Asking if operation execution has been started: BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=PutIfAbsentOperation

      {flowtable_null:ffffffff97af0009}

      , partitionId=254, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:20,303 | WARN | ice-installer-14 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] No response for 120000 ms. BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=PutIfAbsentOperation

      {flowtable_null:ffffffff97af0001}

      , partitionId=237, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:20,307 | WARN | ice-installer-14 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] Asking if operation execution has been started: BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=PutIfAbsentOperation

      {flowtable_null:ffffffff97af0001}

      , partitionId=237, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:20,312 | WARN | ice-installer-24 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] No response for 120000 ms. BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=233, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:20,315 | WARN | ice-installer-24 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] Asking if operation execution has been started: BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=233, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:22,110 | WARN | ice-installer-29 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] While asking 'is-executing': BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      java.util.concurrent.TimeoutException: Call BasicInvocation{ serviceName='hz:impl:mapService', op=Operation

      {serviceName='hz:impl:mapService', callId=1587389, invocationTime=1428528257108, waitTimeout=-1, callTimeout=5000}

      , partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0} encountered a timeout
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponse(BasicInvocationFuture.java:352)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:320)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:194)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.isOperationExecuting(BasicInvocationFuture.java:421)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:244)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:404)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:223)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:84)[55:com.hazelcast:3.4.0]
      at org.onosproject.store.hz.SMap.get(SMap.java:130)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore$UpdateBackup.lambda$run$13(DistributedFlowRuleStore.java:799)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore$UpdateBackup$$Lambda$219/338249682.accept(Unknown Source)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)[:1.8.0_11]
      at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)[:1.8.0_11]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore$UpdateBackup.run(DistributedFlowRuleStore.java:795)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_11]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_11]
      at org.onlab.util.BoundedThreadPool$CallerFeedbackPolicy.rejectedExecution(BoundedThreadPool.java:172)[74:org.onosproject.onlab-misc:1.2.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)[:1.8.0_11]
      at org.onlab.util.BoundedThreadPool.execute(BoundedThreadPool.java:96)[74:org.onosproject.onlab-misc:1.2.0.SNAPSHOT]
      at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)[:1.8.0_11]
      at org.onlab.util.BoundedThreadPool.submit(BoundedThreadPool.java:84)[74:org.onosproject.onlab-misc:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore.updateBackup(DistributedFlowRuleStore.java:531)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore.storeBatchInternal(DistributedFlowRuleStore.java:487)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore.storeBatch(DistributedFlowRuleStore.java:449)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.net.flow.impl.FlowRuleManager$FlowOperationsProcessor$1.run(FlowRuleManager.java:526)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_11]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_11]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_11]
      2015-04-08 14:24:22,114 | WARN | ice-installer-29 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] 'is-executing': false -> BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      2015-04-08 14:24:23,702 | WARN | ice-installer-12 | BasicInvocation | 55 - com.hazelcast - 3.4.0 | [10.254.1.207]:5701 [flowTP] [3.4] While asking 'is-executing': BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=PutIfAbsentOperation

      {flowtable_null:ffffffff97af0009}

      , partitionId=254, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false}
      java.util.concurrent.TimeoutException: Call BasicInvocation{ serviceName='hz:impl:mapService', op=Operation

      {serviceName='hz:impl:mapService', callId=1587390, invocationTime=1428528258699, waitTimeout=-1, callTimeout=5000}

      , partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[10.254.1.201]:5701, backupsExpected=0, backupsCompleted=0} encountered a timeout
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponse(BasicInvocationFuture.java:352)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:320)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:194)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.isOperationExecuting(BasicInvocationFuture.java:421)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:244)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)[55:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:404)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxySupport.putIfAbsentInternal(MapProxySupport.java:379)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.putIfAbsent(MapProxyImpl.java:134)[55:com.hazelcast:3.4.0]
      at com.hazelcast.map.impl.proxy.MapProxyImpl.putIfAbsent(MapProxyImpl.java:121)[55:com.hazelcast:3.4.0]
      at org.onosproject.store.hz.SMap.putIfAbsent(SMap.java:231)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore$UpdateBackup.lambda$run$13(DistributedFlowRuleStore.java:812)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore$UpdateBackup$$Lambda$219/338249682.accept(Unknown Source)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)[:1.8.0_11]
      at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)[:1.8.0_11]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore$UpdateBackup.run(DistributedFlowRuleStore.java:795)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_11]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_11]
      at org.onlab.util.BoundedThreadPool$CallerFeedbackPolicy.rejectedExecution(BoundedThreadPool.java:172)[74:org.onosproject.onlab-misc:1.2.0.SNAPSHOT]
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)[:1.8.0_11]
      at org.onlab.util.BoundedThreadPool.execute(BoundedThreadPool.java:96)[74:org.onosproject.onlab-misc:1.2.0.SNAPSHOT]
      at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)[:1.8.0_11]
      at org.onlab.util.BoundedThreadPool.submit(BoundedThreadPool.java:84)[74:org.onosproject.onlab-misc:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore.updateBackup(DistributedFlowRuleStore.java:531)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore.storeBatchInternal(DistributedFlowRuleStore.java:487)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.store.flow.impl.DistributedFlowRuleStore.storeBatch(DistributedFlowRuleStore.java:449)[79:org.onosproject.onos-core-dist:1.2.0.SNAPSHOT]
      at org.onosproject.net.flow.impl.FlowRuleManager$FlowOperationsProcessor$1.run(FlowRuleManager.java:526)[77:org.onosproject.onos-core-net:1.2.0.SNAPSHOT]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_11]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_11]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_11]

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            camfranke Cameron Franke
            Reporter:
            suibin suibin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: