ONOS
  1. ONOS
  2. ONOS-5347

ONOS cluster not able to recover after killing one of cluster member

    Details

    • Type: Bug Bug
    • Status: Closed (View Workflow)
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.7.0, 1.8.0
    • Fix Version/s: 1.9.0, 1.8.2
    • Component/s: None
    • Labels:
    • Environment:

      Test Env: 3 node cluster
      Nodes running on LXC containers
      ONOS ver: ONOS 1.8 (Master)

      Also tested the scenario with ONOS 1.7 release

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      Ibis Sprint 2 - Platform, Ibis Sprint 3 - Platform, Junco Sprint #1 - Platform, Junco Sprint #2 - Platform, Junco Sprint #3 - Platform

      Description

      Issue during cluster capability testing of ONOS.
      Test Env: 3 node cluster (running in LXC containers)
      ONOS ver: ONOS 1.8 (Master)
      Steps:

      • All three nodes are up and running. ONOS is running lxc container.
      • Killed one of the instance.

      And this resulted in failure of entire cluster. i.e.\

      • ConsistentMapTimeout exceptions on other two nodes.
      • GUI for any node is not working.
      • ONOS cli on other nodes, output for other nodes as below:

      onos> masters
      Error executing command: org.onosproject.store.service.StorageException$Timeout

      Logs from all three nodes is attached.
      10.0.3.11 logs corresponds to killed instance (after restart)

      I see following two issues:

      1. Killing/crashing of one instance of ONOS resulting in entire cluster down.
      2. (on killed node) Unable to load the app from disk.

      Additional Info: Upon hitting "org.onosproject.store.service.StorageException$Timeout" exception in other scenarios, entire cluster seems to be not functional.

      Also shared detailed observations on testing with ONOS 1.7 & 1.8 in dev community group:
      https://groups.google.com/a/onosproject.org/forum/#!topic/onos-dev/e6EtzPrB1Pw

      1. 0001-debug.patch
        25 kB
        Thiago Sousa Santos
      2. 0001-debug-logs-cluster.patch
        14 kB
        Thiago Sousa Santos
      3. 10_0_3_10_karaf.log
        1.66 MB
        Srinivas Bandi
      4. 10_0_3_11_karaf.log
        653 kB
        Srinivas Bandi
      5. 10_0_3_12_karaf.log
        4.08 MB
        Srinivas Bandi
      6. onos1.log.bz2
        830 kB
        Thiago Sousa Santos

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          Mao Jianwei added a comment -

          It seem that we confront the same problem from 1.3.0 to now(1.10.0-SNAPSHOT)

          And I want to add that if we start every instances of one cluster not so simultaneously, it will easily cause the whole cluster can not start up and work.

          I think maybe there are some problem in the copycat or atomix?

          Show
          Mao Jianwei added a comment - It seem that we confront the same problem from 1.3.0 to now(1.10.0-SNAPSHOT) And I want to add that if we start every instances of one cluster not so simultaneously, it will easily cause the whole cluster can not start up and work. I think maybe there are some problem in the copycat or atomix?
          Hide
          Jon Hall added a comment -

          Mao Jianwei
          Can you give some more information on your setup and using the latest master give some debug logs?
          How many nodes are you using?
          How do you install ONOS( you mentioned a delay in starting the nodes, how long is the gap between the first and second, etc?*, What commands are you using?, ... )
          How do you kill the node?
          What are the exact symptoms you are seeing?

          Can you give us the karaf logs with debug turned on?
          You can enable atomix debug logging from startup by adding the following line to tools/test/bin/onos-config where there is a similar command:

          echo "log4j.logger.io.atomix= TRACE" \
                  >> $ONOS_INSTALL_DIR/$KARAF_DIST/etc/org.ops4j.pax.logging.cfg

          This will change the log configurations when installing onos remotely when using the installations scripts packaged with onos.

          This seems like two separate issues, so we might move the startup issue to it's won bug once we have more information.

          * This may be somewhat unavoidable, We need a quorum for the cluster to operate. We have tested installing 7 node clusters in sequence, and that does work, but I'm not sure what the maximum delay is from the first node starting to the last node in the quorum starting for ONOS to be able to successfully form a cluster.

          Show
          Jon Hall added a comment - Mao Jianwei Can you give some more information on your setup and using the latest master give some debug logs? How many nodes are you using? How do you install ONOS( you mentioned a delay in starting the nodes, how long is the gap between the first and second, etc?*, What commands are you using?, ... ) How do you kill the node? What are the exact symptoms you are seeing? Can you give us the karaf logs with debug turned on? You can enable atomix debug logging from startup by adding the following line to tools/test/bin/onos-config where there is a similar command: echo "log4j.logger.io.atomix= TRACE" \ >> $ONOS_INSTALL_DIR/$KARAF_DIST/etc/org.ops4j.pax.logging.cfg This will change the log configurations when installing onos remotely when using the installations scripts packaged with onos. This seems like two separate issues, so we might move the startup issue to it's won bug once we have more information. * This may be somewhat unavoidable, We need a quorum for the cluster to operate. We have tested installing 7 node clusters in sequence, and that does work, but I'm not sure what the maximum delay is from the first node starting to the last node in the quorum starting for ONOS to be able to successfully form a cluster.
          Hide
          Huseyin Ahmet AYDIN added a comment -

          Hello,
          We think, we met a similar issue at onos 1.10.0.

          This issue has become during cluster flow testing of ONOS.
          We have 6 node cluster, ONOS version = 1.10.0
          Controllers are running on docker and they have 10GB RAM limit.

          There is a JAVA env whüch we use in this case:
          export JAVA_HOME="/usr/lib/jvm/java-8-oracle"
          export JAVA_MIN_MEM="8192M"
          export JAVA_MAX_MEM="8192M"
          export EXTRA_JAVA_OPTS="-server -Xms8192M -Xmx8192M -Dfile.encoding=UTF-8 -Djavax.net.debug=all -Djavax.net.ssl.trustStore=/root/onos-1.10.0/Applications/apache-karaf-3.0.5/etc/cacerts -Djavax.net.ssl.trustStorePassword=argela"
          export KARAF_OPTS="-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:ConcGCThreads=4 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:MaxNewSize=256M -XX:NewSize=256M -Xloggc:/root/memory/gc/gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/memory/heap/heap_%p.hprof"

          In addition, there was an issue like this case : https://jira.onosproject.org/browse/ONOS-5347
          What should we do?

          Steps:
          All six nodes are up and running. ONOS is running docker container.
          We added 72 devices to controller. Each controller has 12.
          Case 1:
          We sended 32000 flows each controller. (Total=2.3 million flows)
          After tried 5-6 times this step; one of nodes going down(with org.onosproject.store.service.StorageException$Timeout) after the other nodes, also.

          Case 2:
          We sended 16000 flows each controller. (Total=1.15 million flows)
          After tried 11-12 times this step; one of nodes going down(with org.onosproject.store.service.StorageException$Timeout) after the other nodes, also.

          There might be a problem with setting leader.
          Any onos cli commands doesn't work and execute Exception on karaf.log.

          onos> summary
          Error executing command: io.atomix.copycat.session.ClosedSessionException: session closed
          onos> masters
          Error executing command: org.onosproject.store.service.StorageException$Timeout

          Logs (karaf.log and gc.log) from all 6 nodes is attached.

          Some outputs of exception;

                          -2017-06-21 08:04:35,877 | ERROR | s-change-handler | DistributedLeadershipStore       | 129 - org.onosproject.onos-core-dist - 1.10.0 | Uncaught exception on onos-store-dist-cluster-leadership-status-change-handler 
          org.onosproject.store.service.StorageException$Timeout 
                  at org.onosproject.store.primitives.DefaultLeaderElector.complete(DefaultLeaderElector.java:115) 
                  at org.onosproject.store.primitives.DefaultLeaderElector.run(DefaultLeaderElector.java:50) 
                  at org.onosproject.store.cluster.impl.DistributedLeadershipStore.lambda$handleStatusChange$3(DistributedLeadershipStore.java:102)[129:org.onosproject.onos-core-dist:1.10.0] 
                  at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)[:1.8.0_101] 
                  at org.onosproject.store.cluster.impl.DistributedLeadershipStore.handleStatusChange(DistributedLeadershipStore.java:102)[129:org.onosproject.onos-core-dist:1.10.0] 
                  at org.onosproject.store.cluster.impl.DistributedLeadershipStore.lambda$null$1(DistributedLeadershipStore.java:96)[129:org.onosproject.onos-core-dist:1.10.0] 
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101] 
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101] 
                  at java.lang.Thread.run(Thread.java:745)[:1.8.0_101] 
                   
                   
                          -2017-06-21 08:04:57,714 | ERROR | link-discovery-0 | LldpLinkProvider                 | 177 - org.onosproject.onos-providers-lldp - 1.10.0 | Exception thrown during synchronization process 
          org.onosproject.store.service.ConsistentMapException$Timeout: onos-network-configs 
                  at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:233) 
                  at org.onosproject.store.primitives.DefaultConsistentMap.get(DefaultConsistentMap.java:85) 
                  at org.onosproject.store.config.impl.DistributedNetworkConfigStore.getConfig(DistributedNetworkConfigStore.java:241) 
                  at org.onosproject.net.config.impl.NetworkConfigManager.getConfig(NetworkConfigManager.java:203) 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:401)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:409)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePort(LldpLinkProvider.java:473)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$updatePorts$4(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:397)[94:com.google.guava:22.0.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePorts(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$null$0(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at java.util.Optional.ifPresent(Optional.java:159)[:1.8.0_101] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$loadDevices$1(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at com.google.common.collect.Iterables$4.lambda$forEach$0(Iterables.java:584)[94:com.google.guava:22.0.0] 
                  at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707)[:1.8.0_101] 
                  at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_101] 
                  at com.google.common.collect.Iterables$4.forEach(Iterables.java:581)[94:com.google.guava:22.0.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.loadDevices(LldpLinkProvider.java:384)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.access$1500(LldpLinkProvider.java:96)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider$SyncDeviceInfoTask.run(LldpLinkProvider.java:683)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_101] 
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_101] 
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_101] 
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_101] 
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101] 
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101] 
                   
                          -2017-06-21 08:05:07,929 | ERROR | link-discovery-0 | LldpLinkProvider                 | 177 - org.onosproject.onos-providers-lldp - 1.10.0 | Exception thrown during synchronization process 
          org.onosproject.store.service.ConsistentMapException$Timeout: onos-network-configs 
                  at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:233) 
                  at org.onosproject.store.primitives.DefaultConsistentMap.get(DefaultConsistentMap.java:85) 
                  at org.onosproject.store.config.impl.DistributedNetworkConfigStore.getConfig(DistributedNetworkConfigStore.java:241) 
                  at org.onosproject.net.config.impl.NetworkConfigManager.getConfig(NetworkConfigManager.java:203) 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:401)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:409)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePort(LldpLinkProvider.java:473)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$updatePorts$4(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:397)[94:com.google.guava:22.0.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePorts(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$null$0(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at java.util.Optional.ifPresent(Optional.java:159)[:1.8.0_101] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$loadDevices$1(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at com.google.common.collect.Iterables$4.lambda$forEach$0(Iterables.java:584)[94:com.google.guava:22.0.0] 
                  at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707)[:1.8.0_101] 
                  at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_101] 
                  at com.google.common.collect.Iterables$4.forEach(Iterables.java:581)[94:com.google.guava:22.0.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.loadDevices(LldpLinkProvider.java:384)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider.access$1500(LldpLinkProvider.java:96)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at org.onosproject.provider.lldp.impl.LldpLinkProvider$SyncDeviceInfoTask.run(LldpLinkProvider.java:683)[177:org.onosproject.onos-providers-lldp:1.10.0] 
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_101] 
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_101] 
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_101] 
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_101] 
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101] 
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101] 
                  at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]
          
          
          Show
          Huseyin Ahmet AYDIN added a comment - Hello, We think, we met a similar issue at onos 1.10.0. This issue has become during cluster flow testing of ONOS. We have 6 node cluster, ONOS version = 1.10.0 Controllers are running on docker and they have 10GB RAM limit. There is a JAVA env whüch we use in this case: export JAVA_HOME="/usr/lib/jvm/java-8-oracle" export JAVA_MIN_MEM="8192M" export JAVA_MAX_MEM="8192M" export EXTRA_JAVA_OPTS="-server -Xms8192M -Xmx8192M -Dfile.encoding=UTF-8 -Djavax.net.debug=all -Djavax.net.ssl.trustStore=/root/onos-1.10.0/Applications/apache-karaf-3.0.5/etc/cacerts -Djavax.net.ssl.trustStorePassword=argela" export KARAF_OPTS="-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:ConcGCThreads=4 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:MaxNewSize=256M -XX:NewSize=256M -Xloggc:/root/memory/gc/gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/memory/heap/heap_%p.hprof" In addition, there was an issue like this case : https://jira.onosproject.org/browse/ONOS-5347 What should we do? Steps: All six nodes are up and running. ONOS is running docker container. We added 72 devices to controller. Each controller has 12. Case 1: We sended 32000 flows each controller. (Total=2.3 million flows) After tried 5-6 times this step; one of nodes going down(with org.onosproject.store.service.StorageException$Timeout) after the other nodes, also. Case 2: We sended 16000 flows each controller. (Total=1.15 million flows) After tried 11-12 times this step; one of nodes going down(with org.onosproject.store.service.StorageException$Timeout) after the other nodes, also. There might be a problem with setting leader. Any onos cli commands doesn't work and execute Exception on karaf.log. onos> summary Error executing command: io.atomix.copycat.session.ClosedSessionException: session closed onos> masters Error executing command: org.onosproject.store.service.StorageException$Timeout Logs (karaf.log and gc.log) from all 6 nodes is attached. Some outputs of exception; -2017-06-21 08:04:35,877 | ERROR | s-change-handler | DistributedLeadershipStore | 129 - org.onosproject.onos-core-dist - 1.10.0 | Uncaught exception on onos-store-dist-cluster-leadership-status-change-handler org.onosproject.store.service.StorageException$Timeout at org.onosproject.store.primitives.DefaultLeaderElector.complete(DefaultLeaderElector.java:115) at org.onosproject.store.primitives.DefaultLeaderElector.run(DefaultLeaderElector.java:50) at org.onosproject.store.cluster.impl.DistributedLeadershipStore.lambda$handleStatusChange$3(DistributedLeadershipStore.java:102)[129:org.onosproject.onos-core-dist:1.10.0] at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)[:1.8.0_101] at org.onosproject.store.cluster.impl.DistributedLeadershipStore.handleStatusChange(DistributedLeadershipStore.java:102)[129:org.onosproject.onos-core-dist:1.10.0] at org.onosproject.store.cluster.impl.DistributedLeadershipStore.lambda$ null $1(DistributedLeadershipStore.java:96)[129:org.onosproject.onos-core-dist:1.10.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101] at java.lang. Thread .run( Thread .java:745)[:1.8.0_101] -2017-06-21 08:04:57,714 | ERROR | link-discovery-0 | LldpLinkProvider | 177 - org.onosproject.onos-providers-lldp - 1.10.0 | Exception thrown during synchronization process org.onosproject.store.service.ConsistentMapException$Timeout: onos-network-configs at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:233) at org.onosproject.store.primitives.DefaultConsistentMap.get(DefaultConsistentMap.java:85) at org.onosproject.store.config.impl.DistributedNetworkConfigStore.getConfig(DistributedNetworkConfigStore.java:241) at org.onosproject.net.config.impl.NetworkConfigManager.getConfig(NetworkConfigManager.java:203) at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:401)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:409)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePort(LldpLinkProvider.java:473)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$updatePorts$4(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:397)[94:com.google.guava:22.0.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePorts(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$ null $0(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] at java.util.Optional.ifPresent(Optional.java:159)[:1.8.0_101] at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$loadDevices$1(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] at com.google.common.collect.Iterables$4.lambda$forEach$0(Iterables.java:584)[94:com.google.guava:22.0.0] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707)[:1.8.0_101] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_101] at com.google.common.collect.Iterables$4.forEach(Iterables.java:581)[94:com.google.guava:22.0.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.loadDevices(LldpLinkProvider.java:384)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.access$1500(LldpLinkProvider.java:96)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider$SyncDeviceInfoTask.run(LldpLinkProvider.java:683)[177:org.onosproject.onos-providers-lldp:1.10.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_101] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101] -2017-06-21 08:05:07,929 | ERROR | link-discovery-0 | LldpLinkProvider | 177 - org.onosproject.onos-providers-lldp - 1.10.0 | Exception thrown during synchronization process org.onosproject.store.service.ConsistentMapException$Timeout: onos-network-configs at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:233) at org.onosproject.store.primitives.DefaultConsistentMap.get(DefaultConsistentMap.java:85) at org.onosproject.store.config.impl.DistributedNetworkConfigStore.getConfig(DistributedNetworkConfigStore.java:241) at org.onosproject.net.config.impl.NetworkConfigManager.getConfig(NetworkConfigManager.java:203) at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:401)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.isBlacklisted(LldpLinkProvider.java:409)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePort(LldpLinkProvider.java:473)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$updatePorts$4(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:397)[94:com.google.guava:22.0.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.updatePorts(LldpLinkProvider.java:455)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$ null $0(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] at java.util.Optional.ifPresent(Optional.java:159)[:1.8.0_101] at org.onosproject.provider.lldp.impl.LldpLinkProvider.lambda$loadDevices$1(LldpLinkProvider.java:385)[177:org.onosproject.onos-providers-lldp:1.10.0] at com.google.common.collect.Iterables$4.lambda$forEach$0(Iterables.java:584)[94:com.google.guava:22.0.0] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707)[:1.8.0_101] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_101] at com.google.common.collect.Iterables$4.forEach(Iterables.java:581)[94:com.google.guava:22.0.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.loadDevices(LldpLinkProvider.java:384)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider.access$1500(LldpLinkProvider.java:96)[177:org.onosproject.onos-providers-lldp:1.10.0] at org.onosproject.provider.lldp.impl.LldpLinkProvider$SyncDeviceInfoTask.run(LldpLinkProvider.java:683)[177:org.onosproject.onos-providers-lldp:1.10.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_101] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101] at java.lang. Thread .run( Thread .java:745)[:1.8.0_101]
          Hide
          Huseyin Ahmet AYDIN added a comment -
          Show
          Huseyin Ahmet AYDIN added a comment - You could find attachments in here : https://groups.google.com/a/onosproject.org/forum/#!topic/onos-dev/VZdcUHrSQkQ
          Hide
          Jon Hall added a comment -

          I moved this to it's own ticket, ONOS-6682

          Show
          Jon Hall added a comment - I moved this to it's own ticket, ONOS-6682

            People

            • Assignee:
              Jon Hall
              Reporter:
              Srinivas Bandi
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile