Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-864

NPE in Hazelcast during backup

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Done
    • Affects Version/s: 1.0.0, 1.0.1, 1.1.0
    • Fix Version/s: 1.1.0, 1.0.2
    • Component/s: None
    • Labels:
      None

      Description

      Following NPE observed during QA test.
      Issue is fixed in Hazelcast 3.3.3 or later.

      ------
      2015-01-27 17:21:25,234 | INFO | nce_1_HA.event-1 | DistributedClusterStore | 157 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Member Member [10.128.30.17]:5701 added
      2015-01-27 17:21:25,237 | ERROR | eration.thread-5 | PutOperation | 52 - com.hazelcast - 3.3.2 | [10.128.30.11]:5701 [HA] [3.3.2] null
      java.lang.NullPointerException
      at com.hazelcast.map.record.Records.buildRecordInfo(Records.java:22)[52:com.hazelcast:3.3.2]
      at com.hazelcast.map.operation.BasePutOperation.getBackupOperation(BasePutOperation.java:84)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.initBackupOperation(BasicOperationService.java:932)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.newBackup(BasicOperationService.java:920)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.makeBackups(BasicOperationService.java:907)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.backup(BasicOperationService.java:889)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handleResponse(BasicOperationService.java:742)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:700)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$400(BasicOperationService.java:675)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:537)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:439)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:433)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:408)[52:com.hazelcast:3.3.2]
      2015-01-27 17:21:25,239 | ERROR | nce_1_HA.event-1 | EventServiceImpl | 52 - com.hazelcast - 3.3.2 | [10.128.30.11]:5701 [HA] [3.3.2] hz._hzInstance_1_HA.event-1 caught an exception while processing task:com.hazelcast.spi.impl.EventServiceImpl$LocalEventDispatcher@6a8b70f9
      java.lang.NullPointerException
      at com.hazelcast.map.record.Records.buildRecordInfo(Records.java:22)[52:com.hazelcast:3.3.2]
      at com.hazelcast.map.operation.BasePutOperation.getBackupOperation(BasePutOperation.java:84)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.initBackupOperation(BasicOperationService.java:932)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.newBackup(BasicOperationService.java:920)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.makeBackups(BasicOperationService.java:907)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationBackupHandler.backup(BasicOperationService.java:889)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handleResponse(BasicOperationService.java:742)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:700)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$400(BasicOperationService.java:675)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:537)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:439)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:433)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:408)[52:com.hazelcast:3.3.2]
      2015-01-27 17:21:31,235 | WARN | eration.thread-0 | ClusterService | 52 - com.hazelcast - 3.3.2 | [10.128.30.11]:5701 [HA] [3.3.2] While waiting finalize join calls...
      java.util.concurrent.TimeoutException: Call BasicInvocation{ serviceName='hz:core:clusterService', op=MembersUpdateCall {
      MemberInfo

      {address=Address[10.128.30.11]:5701}

      MemberInfo

      {address=Address[10.128.30.12]:5701}

      MemberInfo

      {address=Address[10.128.30.13]:5701}

      MemberInfo

      {address=Address[10.128.30.14]:5701}

      MemberInfo

      {address=Address[10.128.30.15]:5701}

      MemberInfo

      {address=Address[10.128.30.16]:5701}

      MemberInfo

      {address=Address[10.128.30.17]:5701}

      }, partitionId=-1, replicaIndex=0, tryCount=50, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.128.30.14]:5701} encountered a timeout
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponse(BasicInvocationFuture.java:321)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:289)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:181)[52:com.hazelcast:3.3.2]
      at com.hazelcast.util.FutureUtil.executeWithDeadline(FutureUtil.java:292)[52:com.hazelcast:3.3.2]
      at com.hazelcast.util.FutureUtil.waitWithDeadline(FutureUtil.java:276)[52:com.hazelcast:3.3.2]
      at com.hazelcast.util.FutureUtil.waitWithDeadline(FutureUtil.java:250)[52:com.hazelcast:3.3.2]
      at com.hazelcast.cluster.ClusterServiceImpl.startJoin(ClusterServiceImpl.java:899)[52:com.hazelcast:3.3.2]
      at com.hazelcast.cluster.ClusterServiceImpl.handleJoinRequest(ClusterServiceImpl.java:594)[52:com.hazelcast:3.3.2]
      at com.hazelcast.cluster.JoinRequestOperation.run(JoinRequestOperation.java:38)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:699)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$400(BasicOperationService.java:675)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:649)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:604)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.access$1400(BasicOperationService.java:591)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:543)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:439)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.processPriorityMessages(BasicOperationScheduler.java:452)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:432)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:408)[52:com.hazelcast:3.3.2]
      2015-01-27 17:21:31,236 | WARN | eration.thread-0 | ClusterService | 52 - com.hazelcast - 3.3.2 | [10.128.30.11]:5701 [HA] [3.3.2] While waiting finalize join calls...
      java.util.concurrent.TimeoutException
      at com.hazelcast.util.FutureUtil.executeWithDeadline(FutureUtil.java:289)[52:com.hazelcast:3.3.2]
      at com.hazelcast.util.FutureUtil.waitWithDeadline(FutureUtil.java:276)[52:com.hazelcast:3.3.2]
      at com.hazelcast.util.FutureUtil.waitWithDeadline(FutureUtil.java:250)[52:com.hazelcast:3.3.2]
      at com.hazelcast.cluster.ClusterServiceImpl.startJoin(ClusterServiceImpl.java:899)[52:com.hazelcast:3.3.2]
      at com.hazelcast.cluster.ClusterServiceImpl.handleJoinRequest(ClusterServiceImpl.java:594)[52:com.hazelcast:3.3.2]
      at com.hazelcast.cluster.JoinRequestOperation.run(JoinRequestOperation.java:38)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:699)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$400(BasicOperationService.java:675)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:649)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:604)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.access$1400(BasicOperationService.java:591)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:543)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:439)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.processPriorityMessages(BasicOperationScheduler.java:452)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:432)[52:com.hazelcast:3.3.2]
      at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:408)[52:com.hazelcast:3.3.2]
      2015-01-27 17:21:31,540 | INFO | e_1_HA.migration | InternalPartitionService | 52 - com.hazelcast - 3.3.2 | [10.128.30.11]:5701 [HA] [3.3.2] Re-partitioning cluster data... Migration queue size: 38
      ------

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            y-higuchi Yuta Higuchi
            Reporter:
            y-higuchi Yuta Higuchi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: