Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-1142

LeadershipEvent serialization exception

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      baremetal1 & baremetal2

      Description

      Every now and then, this error appears in the logs: (we also get "Couldn't parse partition ID 30(" and extra null partitions)

      2015-02-18 19:43:25,515 | WARN | e_1_bm-1.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.202]:5701 [bm-1] [3.4] Error while logging processing event
      com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1071493768, Size: 3
      Serialization trace:
      topic (org.onosproject.cluster.Leadership)
      subject (org.onosproject.cluster.LeadershipEvent)
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
      at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
      at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
      at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
      at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
      Caused by: java.lang.IndexOutOfBoundsException: Index: 1071493768, Size: 3
      at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
      at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
      at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
      ... 13 more
      2015-02-18 19:43:45,523 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 30`
      ========================================new log =========================================
      2015-02-19 16:13:48,650 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 23N
      2015-02-19 16:13:53,651 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 25b
      2015-02-19 16:14:03,652 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 30
      2015-02-19 16:14:13,654 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 30(
      2015-02-19 16:14:18,655 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 31:
      2015-02-19 16:14:23,656 | WARN | e_1_bm-2.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.201]:5701 [bm-2] [3.4] Error while logging processing event
      com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
      Serialization trace:
      topic (org.onosproject.cluster.Leadership)
      subject (org.onosproject.cluster.LeadershipEvent)
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
      at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
      at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
      at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
      at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
      Caused by: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
      at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
      at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
      at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
      ... 13 more
      2015-02-19 16:14:23,658 | WARN | e_1_bm-2.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.201]:5701 [bm-2] [3.4] Error while logging processing event
      com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
      Serialization trace:
      topic (org.onosproject.cluster.Leadership)
      subject (org.onosproject.cluster.LeadershipEvent)
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
      at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
      at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
      at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
      at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
      Caused by: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
      at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
      at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
      at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
      ... 13 more
      2015-02-19 16:14:28,656 | WARN | e_1_bm-2.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.201]:5701 [bm-2] [3.4] Error while logging processing event
      com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1219220060, Size: 3
      Serialization trace:
      topic (org.onosproject.cluster.Leadership)
      subject (org.onosproject.cluster.LeadershipEvent)
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
      at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
      at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
      at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
      at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
      at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
      at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
      Caused by: java.lang.IndexOutOfBoundsException: Index: 1219220060, Size: 3
      at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
      at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
      at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
      at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
      ... 13 more
      2015-02-19 16:15:03,662 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 28Z
      =====================end log===================

      onos> leaders
      Topic : Leader Epoch
      intent-partition-31 : 10.254.1.201 1
      intent-partition-30 : 10.254.1.201 1
      intent-partition-19 : 10.254.1.201 1
      intent-partition-17 : 10.254.1.201 1
      intent-partition-18 : 10.254.1.201 1
      intent-partition-16 : 10.254.1.201 1
      intent-partition-24 : 10.254.1.201 1
      intent-partition-25 : 10.254.1.201 1
      intent-partition-22 : 10.254.1.201 1
      intent-partition-23 : 10.254.1.201 1
      intent-partition-20 : 10.254.1.201 1
      intent-partition-21 : 10.254.1.201 1
      intent-partition-28 : 10.254.1.201 1
      intent-partition-29 : 10.254.1.201 1
      intent-partition-26 : 10.254.1.201 1
      intent-partition-27 : 10.254.1.201 1
      intent-partition-6 : 10.254.1.202 3
      intent-partition-7 : 10.254.1.202 3
      intent-partition-4 : 10.254.1.202 3
      intent-partition-5 : 10.254.1.202 3
      intent-partition-8 : 10.254.1.202 3
      intent-partition-9 : 10.254.1.202 3
      intent-partition-13 : 10.254.1.202 3
      intent-partition-14 : 10.254.1.202 3
      intent-partition-11 : 10.254.1.202 3
      intent-partition-12 : 10.254.1.202 3
      intent-partition-10 : 10.254.1.202 3
      intent-partition-15 : 10.254.1.202 3
      intent-partition-2 : 10.254.1.202 3
      intent-partition-3 : 10.254.1.202 3
      intent-partition-0 : 10.254.1.202 3
      intent-partition-1 : 10.254.1.202 3
      intent-partition-224: null 0
      intent-partition-17: null 0
      intent-partition-17t: null 0

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            madan Madan Jampani
            Reporter:
            tom Thomas Vachuska
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: