Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-1163

Hazelcast Exceptions under load

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 1.0.0, 1.0.1
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
    • Environment:

      8eb4d3acdf0e42586f94108a55dac282d32b68e9

      Description

      log from $OC7

      2015-02-23 22:09:20,496 | ERROR | ration.thread-20 | WaitNotifyServiceImpl$WaitingOp | 53 - com.hazelcast - 3.4.0 | [192.168.200.207]:5701 [beast] [3.4] Op: Operation

      {serviceName='hz:impl:lockService', callId=1247, invocationTime=1424758084452, waitTimeout=-1080001, callTimeout=60000}

      , Error: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@2e08f444 to Address[192.168.200.205]:5701
      com.hazelcast.core.HazelcastException: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@2e08f444 to Address[192.168.200.205]:5701
      at com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:122)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.WaitNotifyServiceImpl$WaitingOp.run(WaitNotifyServiceImpl.java:325)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:733)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$500(BasicOperationService.java:709)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:560)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:466)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:458)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:432)[53:com.hazelcast:3.4.0]
      2015-02-23 22:09:20,496 | ERROR | ration.thread-23 | WaitNotifyServiceImpl$WaitingOp | 53 - com.hazelcast - 3.4.0 | [192.168.200.207]:5701 [beast] [3.4] Op: Operation

      {serviceName='hz:impl:lockService', callId=1245, invocationTime=1424758082439, waitTimeout=-1080001, callTimeout=60000}

      , Error: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@30d1162d to Address[192.168.200.205]:5701
      com.hazelcast.core.HazelcastException: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@30d1162d to Address[192.168.200.205]:5701
      at com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:122)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.WaitNotifyServiceImpl$WaitingOp.run(WaitNotifyServiceImpl.java:325)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:733)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$500(BasicOperationService.java:709)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:560)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:466)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:458)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:432)[53:com.hazelcast:3.4.0]
      2015-02-23 22:09:20,496 | ERROR | ration.thread-18 | WaitNotifyServiceImpl$WaitingOp | 53 - com.hazelcast - 3.4.0 | [192.168.200.207]:5701 [beast] [3.4] Op: Operation

      {serviceName='hz:impl:lockService', callId=1244, invocationTime=1424758082439, waitTimeout=-1080001, callTimeout=60000}

      , Error: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@5b52bfbd to Address[192.168.200.205]:5701
      com.hazelcast.core.HazelcastException: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@5b52bfbd to Address[192.168.200.205]:5701
      at com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:122)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.WaitNotifyServiceImpl$WaitingOp.run(WaitNotifyServiceImpl.java:325)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:733)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$500(BasicOperationService.java:709)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:560)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:466)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:458)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:432)[53:com.hazelcast:3.4.0]
      2015-02-23 22:09:20,528 | ERROR | ration.thread-18 | WaitNotifyServiceImpl$WaitingOp | 53 - com.hazelcast - 3.4.0 | [192.168.200.207]:5701 [beast] [3.4] Op: Operation

      {serviceName='hz:impl:lockService', callId=1129, invocationTime=1424758082435, waitTimeout=-1080001, callTimeout=60000}

      , Error: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@73ad81c to Address[192.168.200.206]:5701
      com.hazelcast.core.HazelcastException: Cannot send response: com.hazelcast.spi.impl.CallTimeoutResponse@73ad81c to Address[192.168.200.206]:5701
      at com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:122)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.WaitNotifyServiceImpl$WaitingOp.run(WaitNotifyServiceImpl.java:325)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:733)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$500(BasicOperationService.java:709)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:560)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:466)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:458)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:432)[53:com.hazelcast:3.4.0]

      ---------------

      log from $OC1

      2015-02-23 22:12:21,772 | WARN | ration.thread-20 | WaitNotifyServiceImpl$WaitingOp | 53 - com.hazelcast - 3.4.0 | [192.168.200.201]:5701 [beast] [3.4] Op: Operation

      {serviceName='hz:impl:lockService', callId=1377, invocationTime=1424758205278, waitTimeout=-1080001, callTimeout=60000}

      , com.hazelcast.spi.exception.WrongTargetException: WrongTarget! this:Address[192.168.200.201]:5701, target:Address[192.168.200.207]:5701, partitionId: 260, replicaIndex: 0, operation: com.hazelcast.spi.impl.WaitNotifyServiceImpl$WaitingOp, service: hz:impl:lockService
      2015-02-23 22:12:21,772 | WARN | ration.thread-23 | WaitNotifyServiceImpl$WaitingOp | 53 - com.hazelcast - 3.4.0 | [192.168.200.201]:5701 [beast] [3.4] Op: Operation

      {serviceName='hz:impl:lockService', callId=1547, invocationTime=1424758236896, waitTimeout=-1200001, callTimeout=60000}

      , com.hazelcast.spi.exception.WrongTargetException: WrongTarget! this:Address[192.168.200.201]:5701, target:Address[192.168.200.207]:5701, partitionId: 263, replicaIndex: 0, operation: com.hazelcast.spi.impl.WaitNotifyServiceImpl$WaitingOp, service: hz:impl:lockService

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            madan Madan Jampani
            Reporter:
            bocon Brian O'Connor
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: