-
Type: Bug
-
Status: Closed (View Workflow)
-
Priority: Major
-
Resolution: Done
-
Affects Version/s: 1.1.0
-
Fix Version/s: 1.2.0
-
Component/s: None
-
Labels:None
-
Environment:
baremetal1 & baremetal2
Every now and then, this error appears in the logs: (we also get "Couldn't parse partition ID 30(" and extra null partitions)
2015-02-18 19:43:25,515 | WARN | e_1_bm-1.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.202]:5701 [bm-1] [3.4] Error while logging processing event
com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1071493768, Size: 3
Serialization trace:
topic (org.onosproject.cluster.Leadership)
subject (org.onosproject.cluster.LeadershipEvent)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
Caused by: java.lang.IndexOutOfBoundsException: Index: 1071493768, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
... 13 more
2015-02-18 19:43:45,523 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 30`
========================================new log =========================================
2015-02-19 16:13:48,650 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 23N
2015-02-19 16:13:53,651 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 25b
2015-02-19 16:14:03,652 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 30
2015-02-19 16:14:13,654 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 30(
2015-02-19 16:14:18,655 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 31:
2015-02-19 16:14:23,656 | WARN | e_1_bm-2.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.201]:5701 [bm-2] [3.4] Error while logging processing event
com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
Serialization trace:
topic (org.onosproject.cluster.Leadership)
subject (org.onosproject.cluster.LeadershipEvent)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
Caused by: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
... 13 more
2015-02-19 16:14:23,658 | WARN | e_1_bm-2.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.201]:5701 [bm-2] [3.4] Error while logging processing event
com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
Serialization trace:
topic (org.onosproject.cluster.Leadership)
subject (org.onosproject.cluster.LeadershipEvent)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
Caused by: java.lang.IndexOutOfBoundsException: Index: 1219210058, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
... 13 more
2015-02-19 16:14:28,656 | WARN | e_1_bm-2.event-3 | EventService | 53 - com.hazelcast - 3.4.0 | [10.254.1.201]:5701 [bm-2] [3.4] Error while logging processing event
com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 1219220060, Size: 3
Serialization trace:
topic (org.onosproject.cluster.Leadership)
subject (org.onosproject.cluster.LeadershipEvent)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:704)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)[57:com.esotericsoftware.kryo:3.0.0]
at org.onlab.util.KryoNamespace.deserialize(KryoNamespace.java:303)[72:org.onosproject.onlab-misc:1.1.0.SNAPSHOT]
at org.onosproject.store.serializers.KryoSerializer.decode(KryoSerializer.java:57)[156:org.onosproject.onos-core-serializers:1.1.0.SNAPSHOT]
at org.onosproject.store.cluster.impl.HazelcastLeadershipService.onMessage(HazelcastLeadershipService.java:200)[155:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT]
at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$EventPacketProcessor.process(EventServiceImpl.java:549)[53:com.hazelcast:3.4.0]
at com.hazelcast.spi.impl.EventServiceImpl$RemoteEventPacketProcessor.run(EventServiceImpl.java:630)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)[53:com.hazelcast:3.4.0]
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)[53:com.hazelcast:3.4.0]
Caused by: java.lang.IndexOutOfBoundsException: Index: 1219220060, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:653)[:1.8.0_31]
at java.util.ArrayList.get(ArrayList.java:429)[:1.8.0_31]
at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)[57:com.esotericsoftware.kryo:3.0.0]
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)[57:com.esotericsoftware.kryo:3.0.0]
... 13 more
2015-02-19 16:15:03,662 | WARN | event-dispatch-0 | PartitionManager | 155 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Couldn't parse partition ID 28Z
=====================end log===================
onos> leaders
Topic : Leader Epoch
intent-partition-31 : 10.254.1.201 1
intent-partition-30 : 10.254.1.201 1
intent-partition-19 : 10.254.1.201 1
intent-partition-17 : 10.254.1.201 1
intent-partition-18 : 10.254.1.201 1
intent-partition-16 : 10.254.1.201 1
intent-partition-24 : 10.254.1.201 1
intent-partition-25 : 10.254.1.201 1
intent-partition-22 : 10.254.1.201 1
intent-partition-23 : 10.254.1.201 1
intent-partition-20 : 10.254.1.201 1
intent-partition-21 : 10.254.1.201 1
intent-partition-28 : 10.254.1.201 1
intent-partition-29 : 10.254.1.201 1
intent-partition-26 : 10.254.1.201 1
intent-partition-27 : 10.254.1.201 1
intent-partition-6 : 10.254.1.202 3
intent-partition-7 : 10.254.1.202 3
intent-partition-4 : 10.254.1.202 3
intent-partition-5 : 10.254.1.202 3
intent-partition-8 : 10.254.1.202 3
intent-partition-9 : 10.254.1.202 3
intent-partition-13 : 10.254.1.202 3
intent-partition-14 : 10.254.1.202 3
intent-partition-11 : 10.254.1.202 3
intent-partition-12 : 10.254.1.202 3
intent-partition-10 : 10.254.1.202 3
intent-partition-15 : 10.254.1.202 3
intent-partition-2 : 10.254.1.202 3
intent-partition-3 : 10.254.1.202 3
intent-partition-0 : 10.254.1.202 3
intent-partition-1 : 10.254.1.202 3
intent-partition-224: null 0
intent-partition-17: null 0
intent-partition-17t: null 0