Mar 10, 2015 5:16:36 PM org.apache.karaf.main.Main launch INFO: Installing and starting initial bundles Mar 10, 2015 5:16:36 PM org.apache.karaf.main.Main launch INFO: All initial bundles installed and set to start Mar 10, 2015 5:16:36 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Trying to lock /opt/onos/apache-karaf-3.0.2/lock Mar 10, 2015 5:16:36 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Lock acquired Mar 10, 2015 5:16:36 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired INFO: Lock acquired. Setting startlevel to 100 2015-03-10 17:16:37,055 | INFO | FelixStartLevel | fileinstall | 7 - org.apache.felix.fileinstall - 3.4.2 | Creating configuration from org.apache.karaf.jaas.cfg 2015-03-10 17:16:37,624 | INFO | -karaf-3.0.2/etc | fileinstall | 7 - org.apache.felix.fileinstall - 3.4.2 | Installing bundle samples / 0.0.0 2015-03-10 17:16:38,057 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-api 1.1.0-SNAPSHOT 2015-03-10 17:16:38,696 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr 3.0.2 2015-03-10 17:16:38,717 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-base 1.1.0-SNAPSHOT 2015-03-10 17:16:38,942 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature config 3.0.2 2015-03-10 17:16:38,948 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-gui 1.1.0-SNAPSHOT 2015-03-10 17:16:38,948 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-api 1.1.0-SNAPSHOT 2015-03-10 17:16:38,948 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr 3.0.2 2015-03-10 17:16:38,954 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-base 1.1.0-SNAPSHOT 2015-03-10 17:16:39,028 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-web 1.1.0-SNAPSHOT 2015-03-10 17:16:39,029 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature war 3.0.2 2015-03-10 17:16:39,029 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature http 3.0.2 2015-03-10 17:16:39,029 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature pax-http 3.1.2 2015-03-10 17:16:39,029 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature pax-jetty 8.1.15.v20140411 2015-03-10 17:16:39,107 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature pax-war 3.1.2 2015-03-10 17:16:39,107 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature pax-http-whiteboard 3.1.2 2015-03-10 17:16:39,197 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature region 3.0.2 2015-03-10 17:16:39,206 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-app-mobility 1.1.0-SNAPSHOT 2015-03-10 17:16:39,206 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-api 1.1.0-SNAPSHOT 2015-03-10 17:16:39,206 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr 3.0.2 2015-03-10 17:16:39,211 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-base 1.1.0-SNAPSHOT 2015-03-10 17:16:39,280 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-cli 1.1.0-SNAPSHOT 2015-03-10 17:16:39,280 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-api 1.1.0-SNAPSHOT 2015-03-10 17:16:39,280 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr 3.0.2 2015-03-10 17:16:39,285 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-base 1.1.0-SNAPSHOT 2015-03-10 17:16:39,365 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature management 3.0.2 2015-03-10 17:16:39,379 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr-condition-management_0_0_0 3.0.2 2015-03-10 17:16:39,382 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-openflow 1.1.0-SNAPSHOT 2015-03-10 17:16:39,382 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-api 1.1.0-SNAPSHOT 2015-03-10 17:16:39,382 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr 3.0.2 2015-03-10 17:16:39,386 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-base 1.1.0-SNAPSHOT 2015-03-10 17:16:39,514 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature ssh 3.0.2 2015-03-10 17:16:39,523 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature kar 3.0.2 2015-03-10 17:16:39,529 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature webconsole 3.0.2 2015-03-10 17:16:39,529 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature http 3.0.2 2015-03-10 17:16:39,530 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature pax-http 3.1.2 2015-03-10 17:16:39,530 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature pax-jetty 8.1.15.v20140411 2015-03-10 17:16:39,570 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr-condition-webconsole_0_0_0 3.0.2 2015-03-10 17:16:39,573 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature package 3.0.2 2015-03-10 17:16:39,577 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature standard 3.0.2 2015-03-10 17:16:39,601 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-core 1.1.0-SNAPSHOT 2015-03-10 17:16:39,601 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-api 1.1.0-SNAPSHOT 2015-03-10 17:16:39,601 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr 3.0.2 2015-03-10 17:16:39,606 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-base 1.1.0-SNAPSHOT 2015-03-10 17:16:39,698 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature webconsole-condition-scr_0_0_0 3.0.2 2015-03-10 17:16:39,700 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-app-proxyarp 1.1.0-SNAPSHOT 2015-03-10 17:16:39,700 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-api 1.1.0-SNAPSHOT 2015-03-10 17:16:39,701 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature scr 3.0.2 2015-03-10 17:16:39,709 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature onos-thirdparty-base 1.1.0-SNAPSHOT 2015-03-10 17:16:39,774 | INFO | FelixStartLevel | FeaturesServiceImpl | 20 - org.apache.karaf.features.core - 3.0.2 | Installing feature webconsole-condition-scr_0_0_0 3.0.2 2015-03-10 17:16:39,832 | INFO | FelixStartLevel | BlueprintContainerImpl | 15 - org.apache.aries.blueprint.core - 1.4.1 | Bundle org.apache.karaf.features.command is waiting for namespace handlers [http://karaf.apache.org/xmlns/shell/v1.1.0] 2015-03-10 17:16:39,934 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.feature into service ACL configs [org.apache.karaf.service.acl.command.feature.uninstall, org.apache.karaf.service.acl.command.feature.install] 2015-03-10 17:16:39,936 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.jaas into service ACL configs [org.apache.karaf.service.acl.command.jaas.update] 2015-03-10 17:16:39,937 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.config into service ACL configs [org.apache.karaf.service.acl.command.config.property-append, org.apache.karaf.service.acl.command.config.property-delete, org.apache.karaf.service.acl.command.config.property-set, org.apache.karaf.service.acl.command.config.edit, org.apache.karaf.service.acl.command.config.update, org.apache.karaf.service.acl.command.config.cancel, org.apache.karaf.service.acl.command.config.delete] 2015-03-10 17:16:39,940 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.shell into service ACL configs [org.apache.karaf.service.acl.command.shell.java, org.apache.karaf.service.acl.command.shell.new, org.apache.karaf.service.acl.command.shell.edit, org.apache.karaf.service.acl.command.shell.exec] 2015-03-10 17:16:39,942 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.system into service ACL configs [org.apache.karaf.service.acl.command.system.property, org.apache.karaf.service.acl.command.system.shutdown, org.apache.karaf.service.acl.command.system.start-level] 2015-03-10 17:16:39,944 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.scope_bundle into service ACL configs [org.apache.karaf.service.acl.command.scope_bundle.shell, org.apache.karaf.service.acl.command.scope_bundle.packages, org.apache.karaf.service.acl.command.scope_bundle.admin, org.apache.karaf.service.acl.command.scope_bundle.ssh, org.apache.karaf.service.acl.command.scope_bundle.osgi, org.apache.karaf.service.acl.command.scope_bundle.config, org.apache.karaf.service.acl.command.scope_bundle.jaas, org.apache.karaf.service.acl.command.scope_bundle.features, org.apache.karaf.service.acl.command.scope_bundle.log] 2015-03-10 17:16:39,948 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.kar into service ACL configs [org.apache.karaf.service.acl.command.kar.uninstall, org.apache.karaf.service.acl.command.kar.install] 2015-03-10 17:16:39,949 | INFO | FelixStartLevel | SecuredCommandConfigTransformer | 25 - org.apache.karaf.shell.console - 3.0.2 | Generating command ACL config org.apache.karaf.command.acl.bundle into service ACL configs [org.apache.karaf.service.acl.command.bundle.watch, org.apache.karaf.service.acl.command.bundle.update, org.apache.karaf.service.acl.command.bundle.install, org.apache.karaf.service.acl.command.bundle.uninstall, org.apache.karaf.service.acl.command.bundle.refresh, org.apache.karaf.service.acl.command.bundle.restart, org.apache.karaf.service.acl.command.bundle.stop, org.apache.karaf.service.acl.command.bundle.start] 2015-03-10 17:16:40,077 | INFO | FelixStartLevel | BlueprintContainerImpl | 15 - org.apache.aries.blueprint.core - 1.4.1 | Bundle org.apache.karaf.jaas.modules is waiting for namespace handlers [http://karaf.apache.org/xmlns/jaas/v1.0.0] 2015-03-10 17:16:40,243 | INFO | FelixStartLevel | DetailsCommandComponent | 40 - org.apache.karaf.scr.command - 3.0.2 | Activating the Apache Karaf SCR Details Command 2015-03-10 17:16:40,245 | INFO | FelixStartLevel | DeactivateCommandComponent | 40 - org.apache.karaf.scr.command - 3.0.2 | Activating the Apache Karaf SCR Deactivate Command 2015-03-10 17:16:40,253 | INFO | FelixStartLevel | ActivateCommandComponent | 40 - org.apache.karaf.scr.command - 3.0.2 | Activating the Apache Karaf SCR Activate Command 2015-03-10 17:16:40,255 | INFO | FelixStartLevel | ListCommandComponent | 40 - org.apache.karaf.scr.command - 3.0.2 | Activating the Apache Karaf SCR List Command 2015-03-10 17:16:40,424 | INFO | FelixStartLevel | Activator | 94 - org.ops4j.pax.web.pax-web-runtime - 3.1.2 | EventAdmin support is not available, no servlet events will be posted! 2015-03-10 17:16:40,429 | INFO | FelixStartLevel | Activator | 94 - org.ops4j.pax.web.pax-web-runtime - 3.1.2 | LogService support enabled, log events will be created. 2015-03-10 17:16:40,435 | INFO | FelixStartLevel | Activator | 94 - org.ops4j.pax.web.pax-web-runtime - 3.1.2 | Pax Web started 2015-03-10 17:16:40,602 | INFO | pool-4-thread-1 | Server | 86 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.15.v20140411 | jetty-8.1.15.v20140411 2015-03-10 17:16:40,666 | INFO | pool-4-thread-1 | AbstractConnector | 86 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.15.v20140411 | Started SelectChannelConnector@0.0.0.0:8181 2015-03-10 17:16:40,666 | INFO | pool-4-thread-1 | JettyServerImpl | 95 - org.ops4j.pax.web.pax-web-jetty - 3.1.2 | Pax Web available at [0.0.0.0]:[8181] 2015-03-10 17:16:40,823 | INFO | FelixStartLevel | RegionsPersistenceImpl | 119 - org.apache.karaf.region.persist - 3.0.2 | Loading region digraph persistence 2015-03-10 17:16:40,848 | INFO | FelixStartLevel | RegionsPersistenceImpl | 119 - org.apache.karaf.region.persist - 3.0.2 | initializing region digraph from etc/regions-config.xml 2015-03-10 17:16:40,995 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Starting JMX OSGi agent 2015-03-10 17:16:41,018 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Registering MBean with ObjectName [osgi.compendium:service=cm,version=1.3,framework=org.apache.felix.framework,uuid=3bf590e5-b7c0-4538-925c-f375d7641e13] for service with service.id [13] 2015-03-10 17:16:41,021 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Registering org.osgi.jmx.service.cm.ConfigurationAdminMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@245cff49 with name osgi.compendium:service=cm,version=1.3,framework=org.apache.felix.framework,uuid=3bf590e5-b7c0-4538-925c-f375d7641e13 2015-03-10 17:16:41,023 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Registering org.osgi.jmx.framework.BundleStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@245cff49 with name osgi.core:type=bundleState,version=1.7,framework=org.apache.felix.framework,uuid=3bf590e5-b7c0-4538-925c-f375d7641e13 2015-03-10 17:16:41,026 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Registering org.osgi.jmx.framework.PackageStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@245cff49 with name osgi.core:type=packageState,version=1.5,framework=org.apache.felix.framework,uuid=3bf590e5-b7c0-4538-925c-f375d7641e13 2015-03-10 17:16:41,026 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Registering org.osgi.jmx.framework.ServiceStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@245cff49 with name osgi.core:type=serviceState,version=1.7,framework=org.apache.felix.framework,uuid=3bf590e5-b7c0-4538-925c-f375d7641e13 2015-03-10 17:16:41,027 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Registering org.osgi.jmx.framework.FrameworkMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@245cff49 with name osgi.core:type=framework,version=1.7,framework=org.apache.felix.framework,uuid=3bf590e5-b7c0-4538-925c-f375d7641e13 2015-03-10 17:16:41,027 | INFO | FelixStartLevel | core | 125 - org.apache.aries.jmx.core - 1.1.2 | Registering org.osgi.jmx.framework.wiring.BundleWiringStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@245cff49 with name osgi.core:type=wiringState,version=1.1,framework=org.apache.felix.framework,uuid=3bf590e5-b7c0-4538-925c-f375d7641e13 2015-03-10 17:16:41,133 | INFO | FelixStartLevel | ScrServiceMBeanImpl | 129 - org.apache.karaf.scr.management - 3.0.2 | Activating the Apache Karaf SCR Service MBean 2015-03-10 17:16:41,619 | INFO | FelixStartLevel | HttpServiceFactoryImpl | 94 - org.ops4j.pax.web.pax-web-runtime - 3.1.2 | Binding bundle: [org.apache.karaf.webconsole.console [145]] to http service 2015-03-10 17:16:41,701 | INFO | rafOsgiManager]) | HttpServiceFactoryImpl | 94 - org.ops4j.pax.web.pax-web-runtime - 3.1.2 | Unbinding bundle: [org.apache.karaf.webconsole.console [145]] 2015-03-10 17:16:41,704 | INFO | rafOsgiManager]) | HttpServiceFactoryImpl | 94 - org.ops4j.pax.web.pax-web-runtime - 3.1.2 | Binding bundle: [org.apache.karaf.webconsole.console [145]] to http service 2015-03-10 17:16:41,742 | WARN | FelixStartLevel | ServletTracker | 99 - org.ops4j.pax.web.pax-web-extender-whiteboard - 3.1.2 | Registered servlet [org.apache.felix.webconsole.plugins.ds.internal.WebConsolePlugin@7f1edd2] did not contain a valid alias or url patterns property 2015-03-10 17:16:42,473 | WARN | FelixStartLevel | OSGiScriptEngineManager | 55 - com.hazelcast - 3.4.0 | Found ScriptEngineFactory candidate for com.sun.script.javascript.RhinoScriptEngineFactory, but cannot load class! -> java.lang.ClassNotFoundException: com.sun.script.javascript.RhinoScriptEngineFactory not found by com.hazelcast [55] 2015-03-10 17:16:42,786 | INFO | FelixStartLevel | HttpServiceFactoryImpl | 94 - org.ops4j.pax.web.pax-web-runtime - 3.1.2 | Binding bundle: [org.onosproject.onos-gui [117]] to http service 2015-03-10 17:16:42,821 | INFO | FelixStartLevel | UiExtensionManager | 117 - org.onosproject.onos-gui - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:43,084 | INFO | pool-5-thread-1 | WebApplicationImpl | 113 - com.sun.jersey.jersey-server - 1.18.1 | Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM' 2015-03-10 17:16:43,167 | INFO | FelixStartLevel | Controller | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Starting OpenFlow IO 2015-03-10 17:16:43,550 | INFO | FelixStartLevel | Controller | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Listening for switch connections on 0.0.0.0/0.0.0.0:6633 2015-03-10 17:16:43,671 | INFO | FelixStartLevel | CoreEventDispatcher | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:43,682 | INFO | event-dispatch-0 | CoreEventDispatcher | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Dispatch loop initiated 2015-03-10 17:16:43,779 | INFO | FelixStartLevel | DistributedTopologyStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:43,780 | INFO | FelixStartLevel | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:43,799 | INFO | FelixStartLevel | DistributedGroupStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:43,803 | INFO | FelixStartLevel | HostClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:43,840 | INFO | FelixStartLevel | FileSystemXmlConfig | 55 - com.hazelcast - 3.4.0 | Configuring Hazelcast from '/opt/onos/apache-karaf-3.0.2/etc/hazelcast.xml'. 2015-03-10 17:16:44,134 | INFO | FelixStartLevel | DefaultAddressPicker | 55 - com.hazelcast - 3.4.0 | [LOCAL] [DoubleHA] [3.4] Interfaces is enabled, trying to pick one address matching to one of: [10.128.30.*] 2015-03-10 17:16:44,134 | INFO | FelixStartLevel | DefaultAddressPicker | 55 - com.hazelcast - 3.4.0 | [LOCAL] [DoubleHA] [3.4] Prefer IPv4 stack is true. 2015-03-10 17:16:44,136 | INFO | FelixStartLevel | DefaultAddressPicker | 55 - com.hazelcast - 3.4.0 | [LOCAL] [DoubleHA] [3.4] Picked Address[10.128.30.11]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true 2015-03-10 17:16:44,356 | INFO | FelixStartLevel | OperationService | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Backpressure is disabled 2015-03-10 17:16:44,360 | INFO | FelixStartLevel | BasicOperationScheduler | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Starting with 2 generic operation threads and 4 partition operation threads. 2015-03-10 17:16:44,408 | INFO | FelixStartLevel | system | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Hazelcast 3.4 (20141224 - 3dc5214) starting at Address[10.128.30.11]:5701 2015-03-10 17:16:44,408 | INFO | FelixStartLevel | system | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Copyright (C) 2008-2014 Hazelcast.com 2015-03-10 17:16:44,422 | INFO | FelixStartLevel | Node | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Creating MulticastJoiner 2015-03-10 17:16:44,426 | INFO | FelixStartLevel | LifecycleService | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Address[10.128.30.11]:5701 is STARTING 2015-03-10 17:16:44,520 | INFO | pool-5-thread-1 | WebApplicationImpl | 113 - com.sun.jersey.jersey-server - 1.18.1 | Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM' 2015-03-10 17:16:44,792 | INFO | pool-5-thread-1 | WebApplicationImpl | 113 - com.sun.jersey.jersey-server - 1.18.1 | Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM' 2015-03-10 17:16:45,045 | INFO | pool-5-thread-1 | WebApplicationImpl | 113 - com.sun.jersey.jersey-server - 1.18.1 | Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM' 2015-03-10 17:16:45,205 | INFO | pool-5-thread-1 | WebApplicationImpl | 113 - com.sun.jersey.jersey-server - 1.18.1 | Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM' 2015-03-10 17:16:47,418 | INFO | FelixStartLevel | MulticastJoiner | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Trying to join to discovered node: Address[10.128.30.12]:5701 2015-03-10 17:16:47,425 | INFO | .cached.thread-3 | SocketConnector | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Connecting to /10.128.30.12:5701, timeout: 0, bind-any: true 2015-03-10 17:16:47,431 | INFO | .cached.thread-3 | TcpIpConnectionManager | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Established socket connection between /10.128.30.11:34063 and 10.128.30.12/10.128.30.12:5701 2015-03-10 17:16:54,434 | INFO | eration.thread-1 | ClusterService | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Members [2] { Member [10.128.30.12]:5701 Member [10.128.30.11]:5701 this } 2015-03-10 17:16:56,444 | INFO | FelixStartLevel | LifecycleService | 55 - com.hazelcast - 3.4.0 | [10.128.30.11]:5701 [DoubleHA] [3.4] Address[10.128.30.11]:5701 is STARTED 2015-03-10 17:16:56,444 | INFO | FelixStartLevel | StoreManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:56,581 | INFO | FelixStartLevel | DistributedApplicationIdStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:56,597 | INFO | FelixStartLevel | IntentClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:56,600 | INFO | FelixStartLevel | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:56,821 | INFO | FelixStartLevel | DistributedClusterStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Local Node: DefaultControllerNode{id=10.128.30.11, ip=10.128.30.11, tcpPort=9876} 2015-03-10 17:16:56,864 | WARN | FelixStartLevel | NettyMessagingService | 162 - org.onosproject.onlab-netty - 1.1.0.SNAPSHOT | Failed to initialize native (epoll) transport. Reason: no netty-transport-native-epoll in java.library.path. Proceeding with nio. 2015-03-10 17:16:56,917 | INFO | FelixStartLevel | DistributedClusterStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:56,918 | INFO | FelixStartLevel | ClusterManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:16:56,919 | INFO | FelixStartLevel | DatabaseManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Loading database definition: /opt/onos/apache-karaf-3.0.2/../config/tablets.json 2015-03-10 17:16:57,502 | INFO | ycat-coordinator | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to FOLLOWER 2015-03-10 17:17:01,422 | INFO | copycat-p1-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to FOLLOWER 2015-03-10 17:17:07,108 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to FOLLOWER 2015-03-10 17:17:10,312 | INFO | copycat-p2-1 | FollowerState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Heartbeat timed out in 3204 milliseconds 2015-03-10 17:17:10,313 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to CANDIDATE 2015-03-10 17:17:10,317 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Starting election 2015-03-10 17:17:10,318 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Polling members [tcp://10.128.30.11:7238, tcp://10.128.30.12:7238] 2015-03-10 17:17:10,335 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.11:7238 2015-03-10 17:17:10,348 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.12:7238 2015-03-10 17:17:10,348 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to LEADER 2015-03-10 17:17:10,356 | INFO | at-coordinator-1 | DatabaseManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Successfully opened database. 2015-03-10 17:17:10,357 | INFO | FelixStartLevel | DatabaseManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,384 | INFO | FelixStartLevel | DistributedMastershipStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,393 | INFO | FelixStartLevel | MastershipManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,394 | INFO | FelixStartLevel | ReplicaInfoManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,405 | WARN | FelixStartLevel | NettyMessagingService | 162 - org.onosproject.onlab-netty - 1.1.0.SNAPSHOT | Failed to initialize native (epoll) transport. Reason: Could not initialize class io.netty.channel.epoll.Native. Proceeding with nio. 2015-03-10 17:17:10,408 | INFO | FelixStartLevel | ClusterCommunicationManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started on 10.128.30.11:9876 2015-03-10 17:17:10,432 | INFO | FelixStartLevel | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,475 | INFO | FelixStartLevel | GossipApplicationStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,482 | INFO | FelixStartLevel | ApplicationManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,511 | INFO | FelixStartLevel | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started. 2015-03-10 17:17:10,544 | INFO | FelixStartLevel | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,546 | INFO | FelixStartLevel | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,547 | INFO | FelixStartLevel | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,554 | INFO | FelixStartLevel | GroupManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,555 | INFO | FelixStartLevel | OpenFlowGroupProvider | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,587 | INFO | FelixStartLevel | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,589 | INFO | FelixStartLevel | FlowRuleManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,591 | INFO | FelixStartLevel | OpenFlowRuleProvider | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,598 | INFO | FelixStartLevel | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,618 | INFO | FelixStartLevel | GossipComponentConfigStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,619 | INFO | FelixStartLevel | ComponentConfigManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,625 | INFO | FelixStartLevel | DefaultTopologyProvider | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Configured with maxEvents = 1000; maxBatchMs = 50; maxIdleMs = 10; accumulator=true 2015-03-10 17:17:10,625 | INFO | FelixStartLevel | DefaultTopologyProvider | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:10,632 | INFO | nos-topo-build-0 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061480288212719, creationTime=1426033030627, computeCost=3388231, clusters=0, devices=0, links=0} changed 2015-03-10 17:17:10,643 | INFO | FelixStartLevel | GossipHostStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,731 | INFO | FelixStartLevel | GossipIntentStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,742 | INFO | FelixStartLevel | HazelcastLinkResourceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,743 | INFO | FelixStartLevel | LinkResourceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,743 | INFO | FelixStartLevel | ObjectiveTracker | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,748 | INFO | FelixStartLevel | IntentManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,775 | INFO | FelixStartLevel | DistributedStatisticStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,776 | INFO | FelixStartLevel | StatisticManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,792 | INFO | FelixStartLevel | DistributedPacketStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,793 | INFO | FelixStartLevel | PacketManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,793 | INFO | FelixStartLevel | HostManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,798 | INFO | FelixStartLevel | PathManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,813 | INFO | FelixStartLevel | ProxyArpManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,821 | INFO | FelixStartLevel | HostMobility | 121 - org.onosproject.onos-app-mobility - 1.1.0.SNAPSHOT | Started with Application ID 2 2015-03-10 17:17:59,824 | INFO | FelixStartLevel | OpenFlowPacketProvider | 136 - org.onosproject.onos-of-provider-packet - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,831 | INFO | FelixStartLevel | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Reading suppression rules from ../config/lldp_suppression.json 2015-03-10 17:17:59,831 | INFO | FelixStartLevel | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Failed to load ../config/lldp_suppression.json, using built-in rules 2015-03-10 17:17:59,834 | INFO | FelixStartLevel | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Started 2015-03-10 17:17:59,839 | INFO | FelixStartLevel | HostLocationProvider | 134 - org.onosproject.onos-host-provider - 1.1.0.SNAPSHOT | Host removal on port/device down events is not configured, using current value of true 2015-03-10 17:17:59,839 | INFO | FelixStartLevel | HostLocationProvider | 134 - org.onosproject.onos-host-provider - 1.1.0.SNAPSHOT | Using IPv6 Neighbor Discovery is not configured, using current value of false 2015-03-10 17:17:59,839 | INFO | FelixStartLevel | HostLocationProvider | 134 - org.onosproject.onos-host-provider - 1.1.0.SNAPSHOT | Started with Application ID 4 2015-03-10 17:17:59,848 | INFO | FelixStartLevel | ProxyArp | 163 - org.onosproject.onos-app-proxyarp - 1.1.0.SNAPSHOT | IPv6 Neighbor Discovery is not configured, using current value of false 2015-03-10 17:17:59,848 | INFO | FelixStartLevel | ProxyArp | 163 - org.onosproject.onos-app-proxyarp - 1.1.0.SNAPSHOT | Started with Application ID 5 2015-03-10 17:17:59,925 | INFO | FelixStartLevel | BlueprintContainerImpl | 15 - org.apache.aries.blueprint.core - 1.4.1 | Bundle org.apache.karaf.bundle.command is waiting for dependencies [(objectClass=org.apache.karaf.bundle.core.BundleWatcher), (objectClass=org.apache.karaf.bundle.core.BundleService)] 2015-03-10 17:17:59,937 | INFO | rint Extender: 1 | BlueprintContainerImpl | 15 - org.apache.aries.blueprint.core - 1.4.1 | Bundle org.apache.karaf.bundle.command is waiting for dependencies [(objectClass=org.apache.karaf.bundle.core.BundleWatcher)] 2015-03-10 17:19:08,054 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52386 2015-03-10 17:19:08,068 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52386 2015-03-10 17:19:08,069 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52392 2015-03-10 17:19:08,070 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52392 2015-03-10 17:19:08,091 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52386 DPID[00:00:00:00:00:00:10:00]] 2015-03-10 17:19:08,091 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52392 DPID[00:00:00:00:00:00:20:00]] 2015-03-10 17:19:08,099 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52399 2015-03-10 17:19:08,102 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52399 2015-03-10 17:19:08,106 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52399 DPID[00:00:00:00:00:00:30:00]] 2015-03-10 17:19:08,114 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52386 2015-03-10 17:19:08,114 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52399 2015-03-10 17:19:08,114 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52392 2015-03-10 17:19:08,122 | INFO | ew I/O worker #3 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:00 2015-03-10 17:19:08,122 | INFO | ew I/O worker #1 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:10:00 2015-03-10 17:19:08,122 | INFO | ew I/O worker #2 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:20:00 2015-03-10 17:19:08,126 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000001000 connected 2015-03-10 17:19:08,126 | INFO | ew I/O worker #2 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000002000 connected 2015-03-10 17:19:08,126 | INFO | ew I/O worker #3 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003000 connected 2015-03-10 17:19:08,139 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52405 2015-03-10 17:19:08,139 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52405 2015-03-10 17:19:08,141 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52405 DPID[00:00:00:00:00:00:30:04]] 2015-03-10 17:19:08,142 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52405 2015-03-10 17:19:08,144 | INFO | ew I/O worker #4 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:04 2015-03-10 17:19:08,144 | INFO | ew I/O worker #4 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003004 connected 2015-03-10 17:19:08,150 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000002000 10.128.30.11 2015-03-10 17:19:08,151 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000002000 2015-03-10 17:19:08,151 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000002000 doesn't exist, or isn't there yet 2015-03-10 17:19:08,151 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000002000. Will load flow rules from backup 2015-03-10 17:19:08,158 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000001000 2015-03-10 17:19:08,158 | INFO | ew I/O worker #1 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000001000 10.128.30.11 2015-03-10 17:19:08,164 | INFO | ew I/O worker #1 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:10:00 2015-03-10 17:19:08,164 | INFO | ew I/O worker #1 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000001000 2015-03-10 17:19:08,165 | INFO | ew I/O worker #2 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000002000 2015-03-10 17:19:08,166 | INFO | ew I/O worker #2 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000002000 10.128.30.11 2015-03-10 17:19:08,166 | INFO | ew I/O worker #2 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:20:00 2015-03-10 17:19:08,166 | INFO | ew I/O worker #2 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000002000 2015-03-10 17:19:08,167 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52408 2015-03-10 17:19:08,168 | INFO | ew I/O worker #1 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000001000 2015-03-10 17:19:08,168 | INFO | ew I/O worker #4 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003004 2015-03-10 17:19:08,168 | INFO | ew I/O worker #4 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003004 10.128.30.11 2015-03-10 17:19:08,168 | INFO | ew I/O worker #4 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:04 2015-03-10 17:19:08,168 | INFO | ew I/O worker #4 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003004 2015-03-10 17:19:08,168 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52408 2015-03-10 17:19:08,169 | INFO | ew I/O worker #4 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003004 2015-03-10 17:19:08,173 | INFO | ew I/O worker #4 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003004 2015-03-10 17:19:08,174 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52408 DPID[00:00:00:00:00:00:50:00]] 2015-03-10 17:19:08,175 | INFO | ew I/O worker #2 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000002000 2015-03-10 17:19:08,177 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52408 2015-03-10 17:19:08,178 | INFO | ew I/O worker #2 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000002000 2015-03-10 17:19:08,180 | INFO | ew I/O worker #2 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:20:00 2015-03-10 17:19:08,180 | INFO | ew I/O worker #4 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:04 2015-03-10 17:19:08,181 | INFO | ew I/O worker #5 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:50:00 2015-03-10 17:19:08,181 | INFO | ew I/O worker #5 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000005000 connected 2015-03-10 17:19:08,183 | INFO | ew I/O worker #2 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:20:00 2015-03-10 17:19:08,182 | INFO | ew I/O worker #4 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:04 2015-03-10 17:19:08,182 | INFO | ew I/O worker #3 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003000 2015-03-10 17:19:08,188 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:04 2015-03-10 17:19:08,188 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003000 10.128.30.11 2015-03-10 17:19:08,188 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000003000 2015-03-10 17:19:08,188 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000003000 doesn't exist, or isn't there yet 2015-03-10 17:19:08,191 | INFO | ew I/O worker #4 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003004: asked for MASTER, and got MASTER 2015-03-10 17:19:08,187 | INFO | ew I/O worker #1 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000001000 2015-03-10 17:19:08,187 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:20:00 2015-03-10 17:19:08,188 | INFO | ew I/O worker #3 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003000 10.128.30.11 2015-03-10 17:19:08,192 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000001000 10.128.30.11 2015-03-10 17:19:08,194 | INFO | ew I/O worker #1 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:10:00 2015-03-10 17:19:08,194 | INFO | ew I/O worker #1 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:10:00 2015-03-10 17:19:08,194 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:10:00 2015-03-10 17:19:08,195 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000001000: asked for MASTER, and got MASTER 2015-03-10 17:19:08,196 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000001000 2015-03-10 17:19:08,199 | INFO | ew I/O worker #3 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:00 2015-03-10 17:19:08,199 | INFO | ew I/O worker #3 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003000 2015-03-10 17:19:08,199 | INFO | ew I/O worker #3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003000 2015-03-10 17:19:08,199 | INFO | ew I/O worker #2 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000002000: asked for MASTER, and got MASTER 2015-03-10 17:19:08,200 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:10:00 2015-03-10 17:19:08,200 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000001000 2015-03-10 17:19:08,200 | INFO | ew I/O worker #3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003000 2015-03-10 17:19:08,200 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000001000 2015-03-10 17:19:08,200 | INFO | ew I/O worker #3 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:00 2015-03-10 17:19:08,200 | INFO | ew I/O worker #3 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:00 2015-03-10 17:19:08,200 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:00 2015-03-10 17:19:08,201 | INFO | ew I/O worker #1 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:10:00 to MASTER 2015-03-10 17:19:08,201 | INFO | ew I/O worker #4 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:04 to MASTER 2015-03-10 17:19:08,205 | INFO | ew I/O worker #2 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:20:00 to MASTER 2015-03-10 17:19:08,208 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52415 2015-03-10 17:19:08,211 | INFO | ew I/O worker #3 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003000: asked for MASTER, and got MASTER 2015-03-10 17:19:08,211 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000001000: asked for MASTER, and got MASTER 2015-03-10 17:19:08,211 | INFO | ew I/O worker #3 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:00 to MASTER 2015-03-10 17:19:08,213 | INFO | ew I/O worker #5 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000005000 2015-03-10 17:19:08,214 | INFO | ew I/O worker #5 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:50:00 2015-03-10 17:19:08,214 | INFO | ew I/O worker #5 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000005000 2015-03-10 17:19:08,214 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52415 2015-03-10 17:19:08,221 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,224 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003000. Will load flow rules from backup 2015-03-10 17:19:08,221 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52415 DPID[00:00:00:00:00:00:60:00]] 2015-03-10 17:19:08,232 | INFO | ew I/O worker #5 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:50:00 2015-03-10 17:19:08,232 | INFO | ew I/O worker #5 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:50:00 2015-03-10 17:19:08,232 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:50:00 2015-03-10 17:19:08,232 | INFO | ew I/O worker #5 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000005000: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,238 | INFO | ew I/O worker #5 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:50:00 to EQUAL 2015-03-10 17:19:08,239 | INFO | onos-device-fg-1 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000001000 2015-03-10 17:19:08,239 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52415 2015-03-10 17:19:08,240 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003000 2015-03-10 17:19:08,240 | INFO | onos-device-fg-2 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000002000 2015-03-10 17:19:08,242 | INFO | ew I/O worker #6 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:00 2015-03-10 17:19:08,242 | INFO | ew I/O worker #6 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006000 connected 2015-03-10 17:19:08,252 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003004 10.128.30.11 2015-03-10 17:19:08,252 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003004 2015-03-10 17:19:08,252 | INFO | onos-device-fg-0 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003004 2015-03-10 17:19:08,256 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:04 2015-03-10 17:19:08,256 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003004 2015-03-10 17:19:08,256 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003004 2015-03-10 17:19:08,256 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,261 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52425 2015-03-10 17:19:08,261 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52425 2015-03-10 17:19:08,264 | INFO | ew I/O worker #4 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003004: asked for MASTER, and got MASTER 2015-03-10 17:19:08,264 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52425 DPID[00:00:00:00:00:00:60:07]] 2015-03-10 17:19:08,269 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52425 2015-03-10 17:19:08,279 | INFO | ew I/O worker #7 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:07 2015-03-10 17:19:08,279 | INFO | ew I/O worker #7 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006007 connected 2015-03-10 17:19:08,295 | INFO | ew I/O worker #6 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006000 2015-03-10 17:19:08,296 | INFO | ew I/O worker #6 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:00 2015-03-10 17:19:08,296 | INFO | ew I/O worker #6 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006000 2015-03-10 17:19:08,296 | INFO | ew I/O worker #6 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:00 2015-03-10 17:19:08,296 | INFO | ew I/O worker #6 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:00 2015-03-10 17:19:08,297 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:00 2015-03-10 17:19:08,297 | INFO | ew I/O worker #6 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006000: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,301 | INFO | ew I/O worker #6 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:00 to EQUAL 2015-03-10 17:19:08,304 | INFO | ew I/O worker #7 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006007 2015-03-10 17:19:08,304 | INFO | ew I/O worker #7 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:07 2015-03-10 17:19:08,304 | INFO | ew I/O worker #7 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006007 2015-03-10 17:19:08,305 | INFO | ew I/O worker #7 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:07 2015-03-10 17:19:08,305 | INFO | ew I/O worker #7 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:07 2015-03-10 17:19:08,305 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:07 2015-03-10 17:19:08,311 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52432 2015-03-10 17:19:08,317 | INFO | ew I/O worker #7 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006007: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,317 | INFO | ew I/O worker #7 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:07 to EQUAL 2015-03-10 17:19:08,318 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52432 2015-03-10 17:19:08,318 | INFO | nos-topo-build-1 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061597961275423, creationTime=1426033148300, computeCost=2337310, clusters=6, devices=6, links=0} changed 2015-03-10 17:19:08,320 | INFO | nos-topo-build-2 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061597979702744, creationTime=1426033148319, computeCost=144432, clusters=7, devices=7, links=0} changed 2015-03-10 17:19:08,322 | WARN | backup-publish-0 | BoundedThreadPool | 74 - org.onosproject.onlab-misc - 1.1.0.SNAPSHOT | queue size: 0 jobs, submitted: 0.01698629206230572 jobs/s, taken: 0.00849314603115286 jobs/s 2015-03-10 17:19:08,327 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52432 DPID[00:00:00:00:00:00:30:08]] 2015-03-10 17:19:08,334 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52432 2015-03-10 17:19:08,335 | INFO | ew I/O worker #8 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:08 2015-03-10 17:19:08,335 | INFO | ew I/O worker #8 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003008 connected 2015-03-10 17:19:08,354 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52438 2015-03-10 17:19:08,355 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52438 2015-03-10 17:19:08,363 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52438 DPID[00:00:00:00:00:00:30:09]] 2015-03-10 17:19:08,365 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52438 2015-03-10 17:19:08,366 | INFO | ew I/O worker #9 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:09 2015-03-10 17:19:08,367 | INFO | ew I/O worker #9 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003009 connected 2015-03-10 17:19:08,371 | INFO | ew I/O worker #8 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000003008 2015-03-10 17:19:08,371 | INFO | ew I/O worker #8 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:30:08 2015-03-10 17:19:08,371 | INFO | ew I/O worker #8 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003008 2015-03-10 17:19:08,372 | INFO | ew I/O worker #8 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:08 2015-03-10 17:19:08,372 | INFO | ew I/O worker #8 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:08 2015-03-10 17:19:08,372 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:08 2015-03-10 17:19:08,372 | INFO | ew I/O worker #8 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003008: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,373 | INFO | ew I/O worker #8 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:08 to EQUAL 2015-03-10 17:19:08,391 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52445 2015-03-10 17:19:08,394 | INFO | ew I/O worker #9 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003009 2015-03-10 17:19:08,394 | INFO | ew I/O worker #9 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003009 10.128.30.11 2015-03-10 17:19:08,394 | INFO | ew I/O worker #9 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:09 2015-03-10 17:19:08,394 | INFO | ew I/O worker #9 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003009 2015-03-10 17:19:08,395 | INFO | ew I/O worker #9 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003009 2015-03-10 17:19:08,396 | INFO | ew I/O worker #9 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003009 2015-03-10 17:19:08,396 | INFO | ew I/O worker #9 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:09 2015-03-10 17:19:08,396 | INFO | ew I/O worker #9 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:09 2015-03-10 17:19:08,396 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:09 2015-03-10 17:19:08,396 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52445 2015-03-10 17:19:08,397 | INFO | onos-device-fg-0 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003009 2015-03-10 17:19:08,398 | INFO | ew I/O worker #9 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003009: asked for MASTER, and got MASTER 2015-03-10 17:19:08,401 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52445 DPID[00:00:00:00:00:00:30:10]] 2015-03-10 17:19:08,404 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52445 2015-03-10 17:19:08,405 | INFO | w I/O worker #10 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:10 2015-03-10 17:19:08,405 | INFO | w I/O worker #10 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003010 connected 2015-03-10 17:19:08,409 | INFO | ew I/O worker #9 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:09 to MASTER 2015-03-10 17:19:08,426 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,429 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000001000. Will load flow rules from backup 2015-03-10 17:19:08,430 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52451 2015-03-10 17:19:08,435 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52451 2015-03-10 17:19:08,435 | INFO | nos-topo-build-3 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598094853308, creationTime=1426033148434, computeCost=615882, clusters=9, devices=9, links=0} changed 2015-03-10 17:19:08,445 | INFO | w I/O worker #10 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003010 2015-03-10 17:19:08,445 | INFO | w I/O worker #10 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003010 10.128.30.11 2015-03-10 17:19:08,446 | INFO | w I/O worker #10 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:10 2015-03-10 17:19:08,446 | INFO | w I/O worker #10 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003010 2015-03-10 17:19:08,446 | INFO | w I/O worker #10 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003010 2015-03-10 17:19:08,446 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52451 DPID[00:00:00:00:00:00:30:11]] 2015-03-10 17:19:08,447 | INFO | w I/O worker #10 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003010 2015-03-10 17:19:08,447 | INFO | w I/O worker #10 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:10 2015-03-10 17:19:08,447 | INFO | w I/O worker #10 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:10 2015-03-10 17:19:08,447 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:10 2015-03-10 17:19:08,447 | INFO | w I/O worker #10 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003010: asked for MASTER, and got MASTER 2015-03-10 17:19:08,448 | INFO | w I/O worker #10 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:10 to MASTER 2015-03-10 17:19:08,452 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52451 2015-03-10 17:19:08,455 | INFO | w I/O worker #11 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:11 2015-03-10 17:19:08,455 | INFO | ew I/O worker #2 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000001000, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000002000, portNumber=3} 2015-03-10 17:19:08,455 | INFO | ew I/O worker #1 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000002000, portNumber=3} and dst: ConnectPoint{elementId=of:0000000000001000, portNumber=2} 2015-03-10 17:19:08,457 | INFO | w I/O worker #11 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003011 connected 2015-03-10 17:19:08,457 | INFO | ew I/O worker #2 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000001000, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000002000, portNumber=3}, type=DIRECT} detected 2015-03-10 17:19:08,460 | INFO | ew I/O worker #1 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000002000, portNumber=3}, dst=ConnectPoint{elementId=of:0000000000001000, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:08,460 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000005000 2015-03-10 17:19:08,462 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:50:00 2015-03-10 17:19:08,462 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000005000 2015-03-10 17:19:08,462 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,463 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,463 | INFO | ew I/O worker #5 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000005000: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,465 | INFO | onos-device-fg-0 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003010 2015-03-10 17:19:08,469 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52462 2015-03-10 17:19:08,471 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52462 2015-03-10 17:19:08,472 | INFO | ew I/O worker #2 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003000, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000002000, portNumber=4} 2015-03-10 17:19:08,473 | INFO | ew I/O worker #2 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003000, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000002000, portNumber=4}, type=DIRECT} detected 2015-03-10 17:19:08,478 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003000, portNumber=4} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=2} 2015-03-10 17:19:08,479 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52462 DPID[00:00:00:00:00:00:30:12]] 2015-03-10 17:19:08,482 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52462 2015-03-10 17:19:08,484 | INFO | w I/O worker #12 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:12 2015-03-10 17:19:08,484 | INFO | w I/O worker #12 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003012 connected 2015-03-10 17:19:08,485 | INFO | nos-topo-build-4 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598140287665, creationTime=1426033148479, computeCost=3323022, clusters=9, devices=10, links=4} changed 2015-03-10 17:19:08,487 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003000, portNumber=4}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:08,487 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003004. Will load flow rules from backup 2015-03-10 17:19:08,494 | INFO | w I/O worker #11 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000003011 2015-03-10 17:19:08,494 | INFO | w I/O worker #11 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:30:11 2015-03-10 17:19:08,494 | INFO | w I/O worker #11 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003011 2015-03-10 17:19:08,496 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003008, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=3} 2015-03-10 17:19:08,499 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003008, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=3}, type=DIRECT} detected 2015-03-10 17:19:08,500 | INFO | w I/O worker #11 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:11 2015-03-10 17:19:08,500 | INFO | w I/O worker #11 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:11 2015-03-10 17:19:08,500 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:11 2015-03-10 17:19:08,500 | INFO | w I/O worker #11 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003011: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,506 | INFO | w I/O worker #11 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:11 to EQUAL 2015-03-10 17:19:08,506 | INFO | ew I/O worker #3 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000002000, portNumber=4} and dst: ConnectPoint{elementId=of:0000000000003000, portNumber=2} 2015-03-10 17:19:08,515 | INFO | ew I/O worker #3 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000002000, portNumber=4}, dst=ConnectPoint{elementId=of:0000000000003000, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:08,515 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52464 2015-03-10 17:19:08,516 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52464 2015-03-10 17:19:08,520 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52464 DPID[00:00:00:00:00:00:30:13]] 2015-03-10 17:19:08,522 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000006000 2015-03-10 17:19:08,523 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:00 2015-03-10 17:19:08,523 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52464 2015-03-10 17:19:08,523 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006000 2015-03-10 17:19:08,523 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,524 | INFO | ew I/O worker #6 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006000: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,526 | INFO | w I/O worker #12 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003012 2015-03-10 17:19:08,526 | INFO | w I/O worker #12 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003012 10.128.30.11 2015-03-10 17:19:08,526 | INFO | w I/O worker #12 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:12 2015-03-10 17:19:08,526 | INFO | w I/O worker #12 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003012 2015-03-10 17:19:08,526 | INFO | w I/O worker #13 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:13 2015-03-10 17:19:08,527 | INFO | w I/O worker #13 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003013 connected 2015-03-10 17:19:08,527 | INFO | w I/O worker #12 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003012 2015-03-10 17:19:08,529 | INFO | w I/O worker #12 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003012 2015-03-10 17:19:08,529 | INFO | w I/O worker #12 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:12 2015-03-10 17:19:08,529 | INFO | w I/O worker #12 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:12 2015-03-10 17:19:08,529 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:12 2015-03-10 17:19:08,529 | INFO | w I/O worker #12 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003012: asked for MASTER, and got MASTER 2015-03-10 17:19:08,530 | INFO | w I/O worker #12 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:12 to MASTER 2015-03-10 17:19:08,535 | INFO | nos-topo-build-5 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598194883706, creationTime=1426033148534, computeCost=376194, clusters=10, devices=12, links=6} changed 2015-03-10 17:19:08,540 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003012 2015-03-10 17:19:08,549 | INFO | w I/O worker #14 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52473 2015-03-10 17:19:08,552 | INFO | w I/O worker #14 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52473 2015-03-10 17:19:08,553 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000006007 2015-03-10 17:19:08,554 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:07 2015-03-10 17:19:08,554 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006007 2015-03-10 17:19:08,554 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,560 | INFO | ew I/O worker #7 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006007: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,562 | INFO | w I/O worker #14 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52473 DPID[00:00:00:00:00:00:30:14]] 2015-03-10 17:19:08,567 | INFO | w I/O worker #14 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52473 2015-03-10 17:19:08,571 | INFO | w I/O worker #14 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:14 2015-03-10 17:19:08,571 | INFO | w I/O worker #14 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003014 connected 2015-03-10 17:19:08,573 | INFO | w I/O worker #13 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003013 2015-03-10 17:19:08,573 | INFO | w I/O worker #13 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003013 10.128.30.11 2015-03-10 17:19:08,573 | INFO | w I/O worker #13 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:13 2015-03-10 17:19:08,574 | INFO | w I/O worker #13 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003013 2015-03-10 17:19:08,574 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003011, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=6} 2015-03-10 17:19:08,574 | INFO | w I/O worker #13 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003013 2015-03-10 17:19:08,575 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003011, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=6}, type=DIRECT} detected 2015-03-10 17:19:08,575 | INFO | w I/O worker #13 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003013 2015-03-10 17:19:08,575 | INFO | w I/O worker #13 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:13 2015-03-10 17:19:08,575 | INFO | w I/O worker #13 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:13 2015-03-10 17:19:08,576 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:13 2015-03-10 17:19:08,576 | INFO | w I/O worker #13 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003013: asked for MASTER, and got MASTER 2015-03-10 17:19:08,576 | INFO | w I/O worker #13 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:13 to MASTER 2015-03-10 17:19:08,577 | INFO | nos-topo-build-6 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598236039323, creationTime=1426033148575, computeCost=1259934, clusters=11, devices=13, links=7} changed 2015-03-10 17:19:08,579 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003013 2015-03-10 17:19:08,587 | INFO | w I/O worker #15 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52483 2015-03-10 17:19:08,592 | INFO | w I/O worker #15 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52483 2015-03-10 17:19:08,595 | INFO | w I/O worker #15 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52483 DPID[00:00:00:00:00:00:30:15]] 2015-03-10 17:19:08,596 | INFO | w I/O worker #15 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52483 2015-03-10 17:19:08,597 | INFO | w I/O worker #15 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:15 2015-03-10 17:19:08,597 | INFO | w I/O worker #15 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003015 connected 2015-03-10 17:19:08,626 | INFO | w I/O worker #16 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52489 2015-03-10 17:19:08,627 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000003008 2015-03-10 17:19:08,628 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:30:08 2015-03-10 17:19:08,628 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003008 2015-03-10 17:19:08,628 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,630 | INFO | w I/O worker #16 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52489 2015-03-10 17:19:08,631 | INFO | w I/O worker #14 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003014 2015-03-10 17:19:08,631 | INFO | w I/O worker #14 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003014 10.128.30.11 2015-03-10 17:19:08,631 | INFO | w I/O worker #14 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:14 2015-03-10 17:19:08,631 | INFO | w I/O worker #14 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003014 2015-03-10 17:19:08,632 | INFO | w I/O worker #14 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003014 2015-03-10 17:19:08,633 | INFO | ew I/O worker #8 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003008: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,633 | INFO | w I/O worker #14 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003014 2015-03-10 17:19:08,634 | INFO | w I/O worker #14 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:14 2015-03-10 17:19:08,634 | INFO | w I/O worker #14 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:14 2015-03-10 17:19:08,634 | INFO | w I/O worker #14 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:14 2015-03-10 17:19:08,634 | INFO | w I/O worker #14 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003014: asked for MASTER, and got MASTER 2015-03-10 17:19:08,635 | INFO | w I/O worker #14 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:14 to MASTER 2015-03-10 17:19:08,641 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003009 10.128.30.11 2015-03-10 17:19:08,641 | INFO | w I/O worker #16 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52489 DPID[00:00:00:00:00:00:30:16]] 2015-03-10 17:19:08,641 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003009 2015-03-10 17:19:08,643 | INFO | w I/O worker #16 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52489 2015-03-10 17:19:08,644 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003014 2015-03-10 17:19:08,644 | INFO | w I/O worker #16 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:16 2015-03-10 17:19:08,644 | INFO | w I/O worker #16 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003016 connected 2015-03-10 17:19:08,645 | INFO | w I/O worker #15 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003015 2015-03-10 17:19:08,645 | INFO | w I/O worker #15 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003015 10.128.30.11 2015-03-10 17:19:08,645 | INFO | w I/O worker #15 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:15 2015-03-10 17:19:08,645 | INFO | w I/O worker #15 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003015 2015-03-10 17:19:08,646 | INFO | w I/O worker #15 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003015 2015-03-10 17:19:08,649 | INFO | w I/O worker #15 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003015 2015-03-10 17:19:08,650 | INFO | w I/O worker #15 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:15 2015-03-10 17:19:08,650 | INFO | w I/O worker #15 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:15 2015-03-10 17:19:08,650 | INFO | w I/O worker #15 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:15 2015-03-10 17:19:08,650 | INFO | w I/O worker #15 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003015: asked for MASTER, and got MASTER 2015-03-10 17:19:08,651 | INFO | w I/O worker #15 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:15 to MASTER 2015-03-10 17:19:08,654 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:09 2015-03-10 17:19:08,654 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003009 2015-03-10 17:19:08,655 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003009 2015-03-10 17:19:08,655 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,656 | INFO | ew I/O worker #9 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003009: asked for MASTER, and got MASTER 2015-03-10 17:19:08,658 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003015 2015-03-10 17:19:08,659 | INFO | nos-topo-build-7 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598318865656, creationTime=1426033148658, computeCost=559279, clusters=13, devices=15, links=7} changed 2015-03-10 17:19:08,668 | INFO | nos-topo-build-0 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598327971552, creationTime=1426033148667, computeCost=395760, clusters=13, devices=15, links=7} changed 2015-03-10 17:19:08,673 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52498 2015-03-10 17:19:08,673 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52498 2015-03-10 17:19:08,673 | INFO | w I/O worker #16 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003016 2015-03-10 17:19:08,673 | INFO | w I/O worker #16 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003016 10.128.30.11 2015-03-10 17:19:08,674 | INFO | w I/O worker #16 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:16 2015-03-10 17:19:08,674 | INFO | w I/O worker #16 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003016 2015-03-10 17:19:08,675 | INFO | w I/O worker #16 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003016 2015-03-10 17:19:08,676 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003009, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=4} 2015-03-10 17:19:08,677 | INFO | w I/O worker #16 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003016 2015-03-10 17:19:08,677 | INFO | w I/O worker #16 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:16 2015-03-10 17:19:08,677 | INFO | w I/O worker #16 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:16 2015-03-10 17:19:08,677 | INFO | w I/O worker #16 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:16 2015-03-10 17:19:08,677 | INFO | w I/O worker #16 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003016: asked for MASTER, and got MASTER 2015-03-10 17:19:08,678 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003016 2015-03-10 17:19:08,678 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52498 DPID[00:00:00:00:00:00:30:17]] 2015-03-10 17:19:08,678 | INFO | w I/O worker #16 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:16 to MASTER 2015-03-10 17:19:08,681 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52498 2015-03-10 17:19:08,682 | INFO | ew I/O worker #1 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:30:17 2015-03-10 17:19:08,682 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000003017 connected 2015-03-10 17:19:08,686 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003010 10.128.30.11 2015-03-10 17:19:08,686 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003010 2015-03-10 17:19:08,687 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003009, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=4}, type=DIRECT} detected 2015-03-10 17:19:08,688 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:10 2015-03-10 17:19:08,688 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003010 2015-03-10 17:19:08,688 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003010 2015-03-10 17:19:08,688 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,689 | INFO | w I/O worker #10 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003010: asked for MASTER, and got MASTER 2015-03-10 17:19:08,694 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003010, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=5} 2015-03-10 17:19:08,695 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003010, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=5}, type=DIRECT} detected 2015-03-10 17:19:08,695 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000003017 2015-03-10 17:19:08,695 | INFO | ew I/O worker #1 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003017 10.128.30.11 2015-03-10 17:19:08,695 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000003011 2015-03-10 17:19:08,695 | INFO | ew I/O worker #1 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:17 2015-03-10 17:19:08,695 | INFO | ew I/O worker #1 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003017 2015-03-10 17:19:08,696 | INFO | ew I/O worker #1 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003017 2015-03-10 17:19:08,697 | INFO | ew I/O worker #1 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003017 2015-03-10 17:19:08,698 | INFO | ew I/O worker #1 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:30:17 2015-03-10 17:19:08,698 | INFO | ew I/O worker #1 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:30:17 2015-03-10 17:19:08,698 | INFO | ew I/O worker #1 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:30:17 2015-03-10 17:19:08,698 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003017: asked for MASTER, and got MASTER 2015-03-10 17:19:08,699 | INFO | ew I/O worker #1 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:30:17 to MASTER 2015-03-10 17:19:08,699 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000003017 2015-03-10 17:19:08,709 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:30:11 2015-03-10 17:19:08,709 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003011 2015-03-10 17:19:08,709 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,710 | INFO | w I/O worker #11 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003011: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,711 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52499 2015-03-10 17:19:08,711 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52499 2015-03-10 17:19:08,712 | INFO | nos-topo-build-1 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598372281069, creationTime=1426033148711, computeCost=495248, clusters=15, devices=17, links=9} changed 2015-03-10 17:19:08,714 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52499 DPID[00:00:00:00:00:00:60:18]] 2015-03-10 17:19:08,717 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003012 10.128.30.11 2015-03-10 17:19:08,717 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003012 2015-03-10 17:19:08,718 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:12 2015-03-10 17:19:08,718 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003012 2015-03-10 17:19:08,718 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003012 2015-03-10 17:19:08,718 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,720 | INFO | w I/O worker #12 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003012: asked for MASTER, and got MASTER 2015-03-10 17:19:08,723 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52499 2015-03-10 17:19:08,725 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003013 10.128.30.11 2015-03-10 17:19:08,725 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003013 2015-03-10 17:19:08,726 | INFO | ew I/O worker #2 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:18 2015-03-10 17:19:08,726 | INFO | ew I/O worker #2 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006018 connected 2015-03-10 17:19:08,726 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:13 2015-03-10 17:19:08,726 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003013 2015-03-10 17:19:08,726 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003013 2015-03-10 17:19:08,727 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,727 | INFO | w I/O worker #13 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003013: asked for MASTER, and got MASTER 2015-03-10 17:19:08,728 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003012, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=7} 2015-03-10 17:19:08,728 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003012, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=7}, type=DIRECT} detected 2015-03-10 17:19:08,732 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003014 10.128.30.11 2015-03-10 17:19:08,732 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003014 2015-03-10 17:19:08,732 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003013, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=8} 2015-03-10 17:19:08,733 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:14 2015-03-10 17:19:08,733 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003014 2015-03-10 17:19:08,733 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003014 2015-03-10 17:19:08,733 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003013, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=8}, type=DIRECT} detected 2015-03-10 17:19:08,733 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,734 | INFO | w I/O worker #14 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003014: asked for MASTER, and got MASTER 2015-03-10 17:19:08,735 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003009. Will load flow rules from backup 2015-03-10 17:19:08,736 | INFO | ew I/O worker #2 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006018 2015-03-10 17:19:08,738 | INFO | ew I/O worker #2 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:18 2015-03-10 17:19:08,738 | INFO | ew I/O worker #2 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006018 2015-03-10 17:19:08,738 | INFO | ew I/O worker #2 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:18 2015-03-10 17:19:08,738 | INFO | ew I/O worker #2 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:18 2015-03-10 17:19:08,739 | INFO | ew I/O worker #2 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:18 2015-03-10 17:19:08,740 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003015 10.128.30.11 2015-03-10 17:19:08,740 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003015 2015-03-10 17:19:08,741 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:15 2015-03-10 17:19:08,741 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003015 2015-03-10 17:19:08,741 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003015 2015-03-10 17:19:08,741 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,744 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52511 2015-03-10 17:19:08,747 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003016 10.128.30.11 2015-03-10 17:19:08,747 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003016 2015-03-10 17:19:08,747 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:16 2015-03-10 17:19:08,748 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003016 2015-03-10 17:19:08,748 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003016 2015-03-10 17:19:08,748 | INFO | w I/O worker #16 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003016: asked for MASTER, and got MASTER 2015-03-10 17:19:08,748 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,748 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003010. Will load flow rules from backup 2015-03-10 17:19:08,749 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52511 2015-03-10 17:19:08,750 | INFO | w I/O worker #15 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003015: asked for MASTER, and got MASTER 2015-03-10 17:19:08,751 | INFO | ew I/O worker #2 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006018: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,754 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52511 DPID[00:00:00:00:00:00:60:19]] 2015-03-10 17:19:08,755 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003014, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=9} 2015-03-10 17:19:08,755 | INFO | ew I/O worker #2 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:18 to EQUAL 2015-03-10 17:19:08,755 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003014, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=9}, type=DIRECT} detected 2015-03-10 17:19:08,757 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52511 2015-03-10 17:19:08,758 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003015, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=10} 2015-03-10 17:19:08,758 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003015, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=10}, type=DIRECT} detected 2015-03-10 17:19:08,759 | INFO | ew I/O worker #3 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:19 2015-03-10 17:19:08,759 | INFO | ew I/O worker #3 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006019 connected 2015-03-10 17:19:08,760 | INFO | nos-topo-build-2 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598419554655, creationTime=1426033148759, computeCost=791887, clusters=16, devices=18, links=13} changed 2015-03-10 17:19:08,763 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003016, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=11} 2015-03-10 17:19:08,763 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003016, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=11}, type=DIRECT} detected 2015-03-10 17:19:08,765 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000003017 10.128.30.11 2015-03-10 17:19:08,766 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000003017 2015-03-10 17:19:08,766 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:30:17 2015-03-10 17:19:08,767 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000003017 2015-03-10 17:19:08,767 | INFO | ew I/O worker #3 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000006019 2015-03-10 17:19:08,767 | INFO | ew I/O worker #3 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000006019 10.128.30.11 2015-03-10 17:19:08,767 | INFO | ew I/O worker #3 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:60:19 2015-03-10 17:19:08,767 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000003017: asked for MASTER, and got MASTER 2015-03-10 17:19:08,767 | INFO | ew I/O worker #3 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006019 2015-03-10 17:19:08,767 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000003017 2015-03-10 17:19:08,768 | INFO | ew I/O worker #3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006019 2015-03-10 17:19:08,768 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,768 | INFO | ew I/O worker #3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006019 2015-03-10 17:19:08,769 | INFO | ew I/O worker #3 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:19 2015-03-10 17:19:08,769 | INFO | ew I/O worker #3 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:19 2015-03-10 17:19:08,769 | INFO | ew I/O worker #3 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:19 2015-03-10 17:19:08,769 | INFO | ew I/O worker #3 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006019: asked for MASTER, and got MASTER 2015-03-10 17:19:08,770 | INFO | ew I/O worker #3 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:19 to MASTER 2015-03-10 17:19:08,770 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006019 2015-03-10 17:19:08,775 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003012. Will load flow rules from backup 2015-03-10 17:19:08,775 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003013. Will load flow rules from backup 2015-03-10 17:19:08,777 | INFO | ew I/O worker #4 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003017, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003004, portNumber=12} 2015-03-10 17:19:08,778 | INFO | ew I/O worker #4 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003017, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003004, portNumber=12}, type=DIRECT} detected 2015-03-10 17:19:08,781 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000006018 2015-03-10 17:19:08,782 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:18 2015-03-10 17:19:08,782 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006018 2015-03-10 17:19:08,782 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,783 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003014. Will load flow rules from backup 2015-03-10 17:19:08,783 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003015. Will load flow rules from backup 2015-03-10 17:19:08,785 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003016. Will load flow rules from backup 2015-03-10 17:19:08,787 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000006019 10.128.30.11 2015-03-10 17:19:08,787 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52516 2015-03-10 17:19:08,787 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role MASTER to of:0000000000006019 2015-03-10 17:19:08,788 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:60:19 2015-03-10 17:19:08,788 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006019 2015-03-10 17:19:08,788 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Triggering probe on device of:0000000000006019 2015-03-10 17:19:08,789 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,789 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000003017. Will load flow rules from backup 2015-03-10 17:19:08,789 | INFO | ew I/O worker #3 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006019: asked for MASTER, and got MASTER 2015-03-10 17:19:08,792 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52516 2015-03-10 17:19:08,792 | INFO | ew I/O worker #2 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006018: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,796 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000006019. Will load flow rules from backup 2015-03-10 17:19:08,807 | INFO | nos-topo-build-3 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598467144902, creationTime=1426033148806, computeCost=562124, clusters=17, devices=19, links=16} changed 2015-03-10 17:19:08,812 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52516 DPID[00:00:00:00:00:00:60:20]] 2015-03-10 17:19:08,820 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006020 doesn't exist, or isn't there yet 2015-03-10 17:19:08,822 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52516 2015-03-10 17:19:08,823 | INFO | ew I/O worker #4 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:20 2015-03-10 17:19:08,823 | INFO | ew I/O worker #4 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006020 connected 2015-03-10 17:19:08,825 | INFO | nos-topo-build-4 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598484670516, creationTime=1426033148824, computeCost=843247, clusters=17, devices=19, links=17} changed 2015-03-10 17:19:08,826 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,830 | INFO | ew I/O worker #4 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006020 2015-03-10 17:19:08,830 | INFO | ew I/O worker #4 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:20 2015-03-10 17:19:08,830 | INFO | ew I/O worker #4 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006020 2015-03-10 17:19:08,831 | INFO | ew I/O worker #4 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:20 2015-03-10 17:19:08,831 | INFO | ew I/O worker #4 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:20 2015-03-10 17:19:08,831 | INFO | ew I/O worker #4 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:20 2015-03-10 17:19:08,831 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52521 2015-03-10 17:19:08,836 | INFO | nos-topo-build-5 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598495554011, creationTime=1426033148835, computeCost=545255, clusters=18, devices=20, links=17} changed 2015-03-10 17:19:08,839 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52521 2015-03-10 17:19:08,841 | ERROR | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Disconnecting switch [/10.128.30.9:52521 DPID[?]] due to IO Error: Connection reset by peer 2015-03-10 17:19:08,841 | INFO | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Switch disconnected callback for sw:[/10.128.30.9:52521 DPID[?]]. Cleaning up ... 2015-03-10 17:19:08,841 | WARN | ew I/O worker #5 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | no dpid in channelHandler registered for disconnected switch [/10.128.30.9:52521 DPID[?]] 2015-03-10 17:19:08,854 | INFO | ew I/O worker #4 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006020: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,855 | INFO | ew I/O worker #4 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:20 to EQUAL 2015-03-10 17:19:08,871 | INFO | nos-topo-build-6 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598530774377, creationTime=1426033148870, computeCost=718559, clusters=18, devices=20, links=18} changed 2015-03-10 17:19:08,894 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52633 2015-03-10 17:19:08,896 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52633 2015-03-10 17:19:08,908 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52633 DPID[00:00:00:00:00:00:60:22]] 2015-03-10 17:19:08,910 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52633 2015-03-10 17:19:08,911 | INFO | ew I/O worker #6 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:22 2015-03-10 17:19:08,911 | INFO | ew I/O worker #6 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006022 connected 2015-03-10 17:19:08,915 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000006022 2015-03-10 17:19:08,915 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006022 doesn't exist, or isn't there yet 2015-03-10 17:19:08,922 | INFO | ew I/O worker #6 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006022 2015-03-10 17:19:08,922 | INFO | ew I/O worker #6 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:22 2015-03-10 17:19:08,922 | INFO | ew I/O worker #6 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006022 2015-03-10 17:19:08,923 | INFO | ew I/O worker #6 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:22 2015-03-10 17:19:08,923 | INFO | ew I/O worker #6 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:22 2015-03-10 17:19:08,923 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,923 | INFO | ew I/O worker #6 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:22 2015-03-10 17:19:08,924 | INFO | ew I/O worker #6 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006022: asked for STANDBY, and got STANDBY 2015-03-10 17:19:08,925 | INFO | ew I/O worker #6 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:22 to EQUAL 2015-03-10 17:19:08,928 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52636 2015-03-10 17:19:08,934 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52636 2015-03-10 17:19:08,934 | INFO | nos-topo-build-7 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598594097356, creationTime=1426033148933, computeCost=585650, clusters=19, devices=21, links=18} changed 2015-03-10 17:19:08,938 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52636 DPID[00:00:00:00:00:00:60:23]] 2015-03-10 17:19:08,940 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52636 2015-03-10 17:19:08,940 | INFO | ew I/O worker #7 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:23 2015-03-10 17:19:08,941 | INFO | ew I/O worker #7 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006023 connected 2015-03-10 17:19:08,946 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000006023 10.128.30.11 2015-03-10 17:19:08,946 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000006023 2015-03-10 17:19:08,946 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006023 doesn't exist, or isn't there yet 2015-03-10 17:19:08,946 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000006023. Will load flow rules from backup 2015-03-10 17:19:08,947 | INFO | ew I/O worker #7 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000006023 2015-03-10 17:19:08,947 | INFO | ew I/O worker #7 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000006023 10.128.30.11 2015-03-10 17:19:08,947 | INFO | ew I/O worker #7 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:60:23 2015-03-10 17:19:08,947 | INFO | ew I/O worker #7 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006023 2015-03-10 17:19:08,948 | INFO | ew I/O worker #7 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006023 2015-03-10 17:19:08,949 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,950 | INFO | ew I/O worker #7 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006023 2015-03-10 17:19:08,951 | INFO | ew I/O worker #7 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:23 2015-03-10 17:19:08,951 | INFO | ew I/O worker #7 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:23 2015-03-10 17:19:08,951 | INFO | ew I/O worker #7 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:23 2015-03-10 17:19:08,952 | INFO | ew I/O worker #7 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006023: asked for MASTER, and got MASTER 2015-03-10 17:19:08,952 | INFO | ew I/O worker #7 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:23 to MASTER 2015-03-10 17:19:08,953 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006023 2015-03-10 17:19:08,959 | INFO | nos-topo-build-0 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598618798282, creationTime=1426033148958, computeCost=527311, clusters=20, devices=22, links=19} changed 2015-03-10 17:19:08,974 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52647 2015-03-10 17:19:08,976 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52647 2015-03-10 17:19:08,979 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52647 DPID[00:00:00:00:00:00:60:24]] 2015-03-10 17:19:08,984 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52647 2015-03-10 17:19:08,985 | INFO | ew I/O worker #8 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:24 2015-03-10 17:19:08,985 | INFO | ew I/O worker #8 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006024 connected 2015-03-10 17:19:08,990 | INFO | event-dispatch-0 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000006024 10.128.30.11 2015-03-10 17:19:08,991 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000006024 2015-03-10 17:19:08,991 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006024 doesn't exist, or isn't there yet 2015-03-10 17:19:08,991 | INFO | event-dispatch-0 | DistributedFlowRuleStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | 10.128.30.11 is now the master for of:0000000000006024. Will load flow rules from backup 2015-03-10 17:19:08,992 | INFO | ew I/O worker #8 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is MASTER for of:0000000000006024 2015-03-10 17:19:08,992 | INFO | ew I/O worker #8 | DeviceClockManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | adding term info of:0000000000006024 10.128.30.11 2015-03-10 17:19:08,992 | INFO | ew I/O worker #8 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role MASTER to switch 00:00:00:00:00:00:60:24 2015-03-10 17:19:08,992 | INFO | ew I/O worker #8 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006024 2015-03-10 17:19:08,992 | INFO | ew I/O worker #8 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006024 2015-03-10 17:19:08,993 | INFO | ew I/O worker #8 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a ports update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006024 2015-03-10 17:19:08,993 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:08,993 | INFO | ew I/O worker #8 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:24 2015-03-10 17:19:08,993 | INFO | ew I/O worker #8 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:24 2015-03-10 17:19:08,993 | INFO | ew I/O worker #8 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:24 2015-03-10 17:19:08,994 | INFO | ew I/O worker #8 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006024: asked for MASTER, and got MASTER 2015-03-10 17:19:08,997 | INFO | ew I/O worker #8 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:24 to MASTER 2015-03-10 17:19:08,999 | INFO | onos-device-fg-3 | GossipDeviceStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a device update topology event for providerId: ProviderId{scheme=of, id=org.onosproject.provider.openflow, ancillary=false} and deviceId: of:0000000000006024 2015-03-10 17:19:09,004 | INFO | nos-topo-build-1 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598663629213, creationTime=1426033149003, computeCost=784373, clusters=21, devices=23, links=19} changed 2015-03-10 17:19:09,008 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52650 2015-03-10 17:19:09,012 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52650 2015-03-10 17:19:09,016 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52650 DPID[00:00:00:00:00:00:60:25]] 2015-03-10 17:19:09,016 | INFO | nos-topo-build-2 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598676522396, creationTime=1426033149016, computeCost=446182, clusters=21, devices=23, links=21} changed 2015-03-10 17:19:09,018 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52650 2015-03-10 17:19:09,021 | INFO | ew I/O worker #9 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:25 2015-03-10 17:19:09,021 | INFO | ew I/O worker #9 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006025 connected 2015-03-10 17:19:09,023 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000006025 2015-03-10 17:19:09,023 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006025 doesn't exist, or isn't there yet 2015-03-10 17:19:09,028 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:09,030 | INFO | ew I/O worker #9 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006025 2015-03-10 17:19:09,030 | INFO | ew I/O worker #9 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:25 2015-03-10 17:19:09,030 | INFO | ew I/O worker #9 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006025 2015-03-10 17:19:09,031 | INFO | ew I/O worker #9 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:25 2015-03-10 17:19:09,031 | INFO | ew I/O worker #9 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:25 2015-03-10 17:19:09,031 | INFO | ew I/O worker #9 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:25 2015-03-10 17:19:09,031 | INFO | ew I/O worker #9 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006025: asked for STANDBY, and got STANDBY 2015-03-10 17:19:09,031 | INFO | ew I/O worker #9 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:25 to EQUAL 2015-03-10 17:19:09,045 | INFO | nos-topo-build-3 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598704523690, creationTime=1426033149044, computeCost=601771, clusters=22, devices=24, links=22} changed 2015-03-10 17:19:09,045 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52661 2015-03-10 17:19:09,050 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52661 2015-03-10 17:19:09,053 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52661 DPID[00:00:00:00:00:00:60:26]] 2015-03-10 17:19:09,056 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52661 2015-03-10 17:19:09,057 | INFO | w I/O worker #10 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:26 2015-03-10 17:19:09,057 | INFO | w I/O worker #10 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006026 connected 2015-03-10 17:19:09,060 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000006026 2015-03-10 17:19:09,060 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006026 doesn't exist, or isn't there yet 2015-03-10 17:19:09,064 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:09,065 | INFO | w I/O worker #10 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006026 2015-03-10 17:19:09,066 | INFO | w I/O worker #10 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:26 2015-03-10 17:19:09,066 | INFO | w I/O worker #10 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006026 2015-03-10 17:19:09,067 | INFO | w I/O worker #10 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:26 2015-03-10 17:19:09,067 | INFO | w I/O worker #10 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:26 2015-03-10 17:19:09,067 | INFO | w I/O worker #10 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:26 2015-03-10 17:19:09,067 | INFO | w I/O worker #10 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006026: asked for STANDBY, and got STANDBY 2015-03-10 17:19:09,068 | INFO | w I/O worker #10 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:26 to EQUAL 2015-03-10 17:19:09,080 | INFO | nos-topo-build-4 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598740419536, creationTime=1426033149080, computeCost=533546, clusters=23, devices=25, links=23} changed 2015-03-10 17:19:09,082 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52666 2015-03-10 17:19:09,085 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52666 2015-03-10 17:19:09,089 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52666 DPID[00:00:00:00:00:00:60:27]] 2015-03-10 17:19:09,093 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52666 2015-03-10 17:19:09,094 | INFO | w I/O worker #11 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:27 2015-03-10 17:19:09,094 | INFO | w I/O worker #11 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006027 connected 2015-03-10 17:19:09,096 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000006027 2015-03-10 17:19:09,096 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006027 doesn't exist, or isn't there yet 2015-03-10 17:19:09,101 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:09,101 | INFO | w I/O worker #11 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006027 2015-03-10 17:19:09,102 | INFO | w I/O worker #11 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:27 2015-03-10 17:19:09,102 | INFO | w I/O worker #11 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006027 2015-03-10 17:19:09,103 | INFO | w I/O worker #11 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:27 2015-03-10 17:19:09,103 | INFO | w I/O worker #11 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:27 2015-03-10 17:19:09,103 | INFO | w I/O worker #11 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:27 2015-03-10 17:19:09,103 | INFO | w I/O worker #11 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006027: asked for STANDBY, and got STANDBY 2015-03-10 17:19:09,104 | INFO | w I/O worker #11 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:27 to EQUAL 2015-03-10 17:19:09,115 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52670 2015-03-10 17:19:09,117 | INFO | nos-topo-build-5 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598776757578, creationTime=1426033149116, computeCost=580284, clusters=24, devices=26, links=24} changed 2015-03-10 17:19:09,120 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52670 2015-03-10 17:19:09,124 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52670 DPID[00:00:00:00:00:00:28:00]] 2015-03-10 17:19:09,127 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52670 2015-03-10 17:19:09,131 | INFO | w I/O worker #12 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:28:00 2015-03-10 17:19:09,131 | INFO | w I/O worker #12 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000002800 connected 2015-03-10 17:19:09,135 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000002800 2015-03-10 17:19:09,135 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000002800 doesn't exist, or isn't there yet 2015-03-10 17:19:09,140 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:09,144 | INFO | w I/O worker #12 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000002800 2015-03-10 17:19:09,145 | INFO | w I/O worker #12 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:28:00 2015-03-10 17:19:09,145 | INFO | w I/O worker #12 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000002800 2015-03-10 17:19:09,146 | INFO | w I/O worker #12 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:28:00 2015-03-10 17:19:09,146 | INFO | w I/O worker #12 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:28:00 2015-03-10 17:19:09,146 | INFO | w I/O worker #12 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:28:00 2015-03-10 17:19:09,146 | INFO | w I/O worker #12 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000002800: asked for STANDBY, and got STANDBY 2015-03-10 17:19:09,147 | INFO | w I/O worker #12 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:28:00 to EQUAL 2015-03-10 17:19:09,150 | INFO | nos-topo-build-6 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598809785528, creationTime=1426033149149, computeCost=492982, clusters=25, devices=27, links=24} changed 2015-03-10 17:19:09,150 | INFO | ew I/O worker #3 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000002800, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003000, portNumber=3} 2015-03-10 17:19:09,152 | INFO | ew I/O worker #3 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000002800, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003000, portNumber=3}, type=DIRECT} detected 2015-03-10 17:19:09,163 | INFO | nos-topo-build-7 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061598823154084, creationTime=1426033149162, computeCost=465961, clusters=25, devices=27, links=26} changed 2015-03-10 17:19:09,840 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | New switch connection from /10.128.30.9:52687 2015-03-10 17:19:09,847 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Sending OF_13 Hello to /10.128.30.9:52687 2015-03-10 17:19:09,851 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received port desc reply for switch at [/10.128.30.9:52687 DPID[00:00:00:00:00:00:60:21]] 2015-03-10 17:19:09,854 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Received switch description reply from switch at /10.128.30.9:52687 2015-03-10 17:19:09,855 | INFO | w I/O worker #13 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Added switch 00:00:00:00:00:00:60:21 2015-03-10 17:19:09,856 | INFO | w I/O worker #13 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device of:0000000000006021 connected 2015-03-10 17:19:09,859 | WARN | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Device is not yet/no longer in the store: of:0000000000006021 2015-03-10 17:19:09,859 | WARN | event-dispatch-0 | LLDPLinkProvider | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Device of:0000000000006021 doesn't exist, or isn't there yet 2015-03-10 17:19:09,864 | INFO | event-dispatch-0 | LinkDiscovery | 133 - org.onosproject.onos-lldp-provider - 1.1.0.SNAPSHOT | Using BDDP to discover network 2015-03-10 17:19:09,866 | INFO | w I/O worker #13 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Role of this node is STANDBY for of:0000000000006021 2015-03-10 17:19:09,866 | INFO | w I/O worker #13 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:60:21 2015-03-10 17:19:09,866 | INFO | w I/O worker #13 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000006021 2015-03-10 17:19:09,867 | INFO | w I/O worker #13 | GroupStatsCollector | 138 - org.onosproject.onos-of-provider-group - 1.1.0.SNAPSHOT | Starting Group Stats collection thread for 00:00:00:00:00:00:60:21 2015-03-10 17:19:09,867 | INFO | w I/O worker #13 | FlowStatsCollector | 137 - org.onosproject.onos-of-provider-flow - 1.1.0.SNAPSHOT | Starting Stats collection thread for 00:00:00:00:00:00:60:21 2015-03-10 17:19:09,867 | INFO | w I/O worker #13 | OFChannelHandler | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Processing 0 pending port status messages for 00:00:00:00:00:00:60:21 2015-03-10 17:19:09,867 | INFO | w I/O worker #13 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000006021: asked for STANDBY, and got STANDBY 2015-03-10 17:19:09,868 | INFO | w I/O worker #13 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:60:21 to EQUAL 2015-03-10 17:19:09,875 | INFO | nos-topo-build-0 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061599535008561, creationTime=1426033149874, computeCost=545597, clusters=26, devices=28, links=26} changed 2015-03-10 17:19:09,945 | INFO | nos-topo-build-1 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061599605112250, creationTime=1426033149944, computeCost=447258, clusters=26, devices=28, links=27} changed 2015-03-10 17:19:11,403 | INFO | ew I/O worker #3 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000003000, portNumber=4} 2015-03-10 17:19:11,404 | INFO | ew I/O worker #3 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000003000, portNumber=4}, type=DIRECT} detected 2015-03-10 17:19:11,405 | INFO | ew I/O worker #9 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=4} and dst: ConnectPoint{elementId=of:0000000000003009, portNumber=2} 2015-03-10 17:19:11,406 | INFO | ew I/O worker #9 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=4}, dst=ConnectPoint{elementId=of:0000000000003009, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,406 | INFO | w I/O worker #10 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=5} and dst: ConnectPoint{elementId=of:0000000000003010, portNumber=2} 2015-03-10 17:19:11,406 | INFO | w I/O worker #10 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=5}, dst=ConnectPoint{elementId=of:0000000000003010, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,410 | INFO | w I/O worker #14 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=9} and dst: ConnectPoint{elementId=of:0000000000003014, portNumber=2} 2015-03-10 17:19:11,410 | INFO | w I/O worker #14 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=9}, dst=ConnectPoint{elementId=of:0000000000003014, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,411 | INFO | w I/O worker #13 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=8} and dst: ConnectPoint{elementId=of:0000000000003013, portNumber=2} 2015-03-10 17:19:11,411 | INFO | w I/O worker #13 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=8}, dst=ConnectPoint{elementId=of:0000000000003013, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,412 | INFO | w I/O worker #12 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=7} and dst: ConnectPoint{elementId=of:0000000000003012, portNumber=2} 2015-03-10 17:19:11,412 | INFO | w I/O worker #12 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=7}, dst=ConnectPoint{elementId=of:0000000000003012, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,414 | INFO | w I/O worker #15 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=10} and dst: ConnectPoint{elementId=of:0000000000003015, portNumber=2} 2015-03-10 17:19:11,414 | INFO | w I/O worker #15 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=10}, dst=ConnectPoint{elementId=of:0000000000003015, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,414 | INFO | ew I/O worker #1 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=12} and dst: ConnectPoint{elementId=of:0000000000003017, portNumber=2} 2015-03-10 17:19:11,415 | INFO | ew I/O worker #1 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=12}, dst=ConnectPoint{elementId=of:0000000000003017, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,415 | INFO | w I/O worker #16 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000003004, portNumber=11} and dst: ConnectPoint{elementId=of:0000000000003016, portNumber=2} 2015-03-10 17:19:11,416 | INFO | w I/O worker #16 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000003004, portNumber=11}, dst=ConnectPoint{elementId=of:0000000000003016, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,427 | INFO | nos-topo-build-2 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061601087092051, creationTime=1426033151426, computeCost=480827, clusters=15, devices=28, links=39} changed 2015-03-10 17:19:11,432 | INFO | ew I/O worker #1 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000005000, portNumber=3} and dst: ConnectPoint{elementId=of:0000000000001000, portNumber=3} 2015-03-10 17:19:11,432 | INFO | ew I/O worker #1 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000005000, portNumber=3}, dst=ConnectPoint{elementId=of:0000000000001000, portNumber=3}, type=DIRECT} detected 2015-03-10 17:19:11,433 | INFO | ew I/O worker #2 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000005000, portNumber=2} and dst: ConnectPoint{elementId=of:0000000000002000, portNumber=2} 2015-03-10 17:19:11,433 | INFO | ew I/O worker #2 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000005000, portNumber=2}, dst=ConnectPoint{elementId=of:0000000000002000, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,444 | INFO | nos-topo-build-3 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061601103793045, creationTime=1426033151443, computeCost=417663, clusters=14, devices=28, links=42} changed 2015-03-10 17:19:11,536 | INFO | ew I/O worker #3 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000006007, portNumber=4} and dst: ConnectPoint{elementId=of:0000000000006019, portNumber=2} 2015-03-10 17:19:11,536 | INFO | ew I/O worker #3 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000006007, portNumber=4}, dst=ConnectPoint{elementId=of:0000000000006019, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,538 | INFO | ew I/O worker #7 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000006007, portNumber=8} and dst: ConnectPoint{elementId=of:0000000000006023, portNumber=2} 2015-03-10 17:19:11,538 | INFO | ew I/O worker #7 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000006007, portNumber=8}, dst=ConnectPoint{elementId=of:0000000000006023, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,541 | INFO | ew I/O worker #8 | GossipLinkStore | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Notifying peers of a link update topology event from providerId: ProviderId{scheme=lldp, id=org.onosproject.provider.lldp, ancillary=false} between src: ConnectPoint{elementId=of:0000000000006007, portNumber=9} and dst: ConnectPoint{elementId=of:0000000000006024, portNumber=2} 2015-03-10 17:19:11,541 | INFO | ew I/O worker #8 | LinkManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Link DefaultLinkDescription{src=ConnectPoint{elementId=of:0000000000006007, portNumber=9}, dst=ConnectPoint{elementId=of:0000000000006024, portNumber=2}, type=DIRECT} detected 2015-03-10 17:19:11,559 | INFO | nos-topo-build-4 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061601218742316, creationTime=1426033151558, computeCost=658619, clusters=1, devices=28, links=56} changed 2015-03-10 17:19:11,615 | INFO | nos-topo-build-5 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061601274252882, creationTime=1426033151613, computeCost=781241, clusters=1, devices=28, links=58} changed 2015-03-10 17:19:19,737 | INFO | qtp620828969-717 | UiWebSocket | 117 - org.onosproject.onos-gui - 1.1.0.SNAPSHOT | GUI client connected 2015-03-10 17:21:02,575 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to FOLLOWER 2015-03-10 17:21:06,614 | INFO | lection-thread-0 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-0 2015-03-10 17:21:10,085 | INFO | lection-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-11 2015-03-10 17:21:10,086 | INFO | lection-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-10 2015-03-10 17:21:11,585 | INFO | lection-thread-3 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:21:13,791 | INFO | lection-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-13 2015-03-10 17:21:37,632 | INFO | copycat-p2-1 | FollowerState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Heartbeat timed out in 5826 milliseconds 2015-03-10 17:21:37,632 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to CANDIDATE 2015-03-10 17:21:37,633 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Starting election 2015-03-10 17:21:37,633 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Polling members [tcp://10.128.30.11:7238, tcp://10.128.30.12:7238] 2015-03-10 17:21:37,633 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.11:7238 2015-03-10 17:21:37,635 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.12:7238 2015-03-10 17:21:37,635 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to LEADER 2015-03-10 17:23:00,275 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000001000 2015-03-10 17:23:00,275 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:10:00 2015-03-10 17:23:00,275 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000001000 2015-03-10 17:23:00,276 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000001000: asked for STANDBY, and got STANDBY 2015-03-10 17:23:00,279 | INFO | ew I/O worker #1 | ntrollerImpl$OpenFlowSwitchAgent | 132 - org.onosproject.onos-of-ctl - 1.1.0.SNAPSHOT | Transitioned switch 00:00:00:00:00:00:10:00 to EQUAL 2015-03-10 17:23:00,287 | INFO | event-dispatch-0 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Applying role STANDBY to of:0000000000001000 2015-03-10 17:23:00,287 | INFO | event-dispatch-0 | OFSwitchImplOVS13 | 130 - org.onosproject.onos-of-api - 1.1.0.SNAPSHOT | Sending role EQUAL to switch 00:00:00:00:00:00:10:00 2015-03-10 17:23:00,287 | INFO | event-dispatch-0 | OpenFlowDeviceProvider | 135 - org.onosproject.onos-of-provider-device - 1.1.0.SNAPSHOT | Accepting mastership role change for device of:0000000000001000 2015-03-10 17:23:00,288 | INFO | ew I/O worker #1 | DeviceManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | got reply to a role request for of:0000000000001000: asked for STANDBY, and got STANDBY 2015-03-10 17:23:13,243 | INFO | nos-topo-build-6 | TopologyManager | 158 - org.onosproject.onos-core-net - 1.1.0.SNAPSHOT | Topology DefaultTopology{time=1061842902505055, creationTime=1426033393242, computeCost=585599, clusters=2, devices=28, links=56} changed 2015-03-10 17:27:01,732 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to FOLLOWER 2015-03-10 17:27:38,783 | INFO | copycat-p2-1 | FollowerState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Heartbeat timed out in 3541 milliseconds 2015-03-10 17:27:38,783 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to CANDIDATE 2015-03-10 17:27:38,784 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Starting election 2015-03-10 17:27:38,784 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Polling members [tcp://10.128.30.11:7238, tcp://10.128.30.12:7238] 2015-03-10 17:27:38,784 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.11:7238 2015-03-10 17:27:38,785 | INFO | copycat-p2-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.12:7238 2015-03-10 17:27:38,785 | INFO | copycat-p2-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to LEADER 2015-03-10 17:28:50,313 | INFO | copycat-p1-1 | FollowerState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Heartbeat timed out in 4024 milliseconds 2015-03-10 17:28:50,313 | INFO | copycat-p1-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to CANDIDATE 2015-03-10 17:28:50,313 | INFO | copycat-p1-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Starting election 2015-03-10 17:28:50,313 | INFO | copycat-p1-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Polling members [tcp://10.128.30.11:7238, tcp://10.128.30.12:7238] 2015-03-10 17:28:50,314 | INFO | copycat-p1-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.11:7238 2015-03-10 17:28:50,315 | INFO | copycat-p1-1 | CandidateState | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Received successful vote from tcp://10.128.30.12:7238 2015-03-10 17:28:50,315 | INFO | copycat-p1-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to LEADER 2015-03-10 17:28:51,291 | INFO | lection-thread-0 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-12 2015-03-10 17:28:51,292 | INFO | lection-thread-2 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-6 2015-03-10 17:28:53,290 | INFO | lection-thread-0 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:28:53,291 | INFO | lection-thread-2 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-9 2015-03-10 17:28:53,291 | INFO | lection-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-8 2015-03-10 17:28:53,291 | INFO | lection-thread-2 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:28:58,307 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-6 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:29:05,309 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-6 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:29:10,310 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:29:12,614 | INFO | copycat-p1-1 | CopycatStateContext | 72 - org.onosproject.onlab-thirdparty - 1.1.0.SNAPSHOT | tcp://10.128.30.11:7238 - Transitioning to FOLLOWER 2015-03-10 17:29:22,619 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:29:24,789 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:29:29,792 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:29:34,789 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:29:36,794 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:29:37,286 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:29:37,286 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:29:37,287 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:29:37,288 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:29:37,288 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:29:41,795 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:29:48,797 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:29:49,314 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:29:49,315 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:29:49,315 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:29:49,316 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:29:52,732 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:01,738 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:30:04,801 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:30:04,801 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:30:04,801 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:30:04,802 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:30:04,802 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:30:04,803 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:30:06,799 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:11,801 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:30:16,803 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:20,805 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:20,805 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:30:20,807 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:30:25,313 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:30,175 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:30:30,316 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:30:35,202 | INFO | qtp620828969-55 | UiWebSocket | 117 - org.onosproject.onos-gui - 1.1.0.SNAPSHOT | GUI client connected 2015-03-10 17:30:37,317 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:30:39,339 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:43,177 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:30:43,178 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:30:43,178 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:30:44,144 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:50,369 | INFO | Timer-2 | UiWebSocket | 117 - org.onosproject.onos-gui - 1.1.0.SNAPSHOT | GUI client disconnected 2015-03-10 17:30:51,164 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:30:55,296 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:30:55,321 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:30:55,322 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:30:55,323 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:30:58,286 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:31:03,288 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:31:07,303 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:31:07,352 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:31:12,305 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:31:14,762 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:31:16,769 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:31:16,770 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:31:19,193 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:31:19,198 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:31:24,195 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:31:29,363 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:31:29,364 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:31:29,365 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:31:34,364 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:31:41,366 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:31:43,319 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:31:43,319 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:31:43,320 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:31:44,786 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:31:51,789 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:31:54,206 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:31:56,836 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:31:56,837 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:31:56,837 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:31:59,213 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:32:06,379 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:32:08,840 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:32:08,840 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:32:08,840 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:32:08,842 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:32:15,843 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:32:18,290 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:32:21,387 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:32:21,388 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:32:24,294 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:32:29,296 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:32:32,786 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:32:33,389 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:32:38,391 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:32:40,847 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:32:45,344 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:32:45,345 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:32:45,345 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:32:47,852 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:32:54,854 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:32:57,348 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:32:58,312 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:32:58,312 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:32:58,338 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:33:04,351 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:33:07,248 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:33:13,305 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:33:13,359 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:33:13,360 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:33:15,399 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:33:20,401 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:33:23,257 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:33:25,312 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:33:25,312 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:33:27,403 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:33:34,404 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:33:35,367 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:33:38,228 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:33:38,788 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.lambda$doCommit$92(LeaderState.java:511)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica$$Lambda$125/1685348154.accept(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.CompletableFuture$AsyncWhenComplete.exec(CompletableFuture.java:648)[:1.8.0_31] at java.util.concurrent.CompletableFuture$Async.run(CompletableFuture.java:428)[:1.8.0_31] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:33:45,791 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:33:48,232 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:33:50,339 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:33:50,346 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:33:51,407 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:33:58,410 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:34:02,786 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:34:05,278 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:34:05,418 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:34:05,419 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:34:05,419 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:34:10,280 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:34:14,786 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:34:19,288 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:34:19,679 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:34:19,788 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:34:24,681 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$17(DefaultClusterCoordinator.java:271)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$67/171818210.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.commit(PassiveState.java:277)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$164/2119245163.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:34:29,684 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:34:31,286 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:34:34,252 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:34:34,286 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:34:34,286 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:34:41,257 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:34:44,891 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:34:45,299 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3107)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.remove(LocalCache.java:4173)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4765)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient$TcpProtocolClientHandler.channelRead(NettyTcpProtocolClient.java:220)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[48:io.netty.transport:4.0.23.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[50:io.netty.codec:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[48:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[48:io.netty.transport:4.0.23.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[46:io.netty.common:4.0.23.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[46:io.netty.common:4.0.23.Final] ... 1 more 2015-03-10 17:34:47,420 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:34:52,422 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:34:57,346 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:34:57,395 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:34:57,395 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:34:57,395 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:34:59,349 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:35:08,353 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:35:12,272 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:35:12,272 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:35:12,273 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:35:12,675 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:35:17,680 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:35:20,786 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:35:23,447 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:35:23,448 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:35:25,449 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:35:28,907 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:35:33,909 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:35:34,693 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:35:39,319 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:35:44,321 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:35:48,325 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:35:48,328 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:35:48,330 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:35:48,334 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:35:50,328 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:35:57,704 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:36:00,333 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:36:04,286 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:36:04,287 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:36:04,293 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:36:05,381 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:36:10,383 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:36:15,465 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:36:17,336 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:36:17,706 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:36:20,467 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:36:27,469 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:36:27,718 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:36:29,786 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:36:32,304 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$16(DefaultClusterCoordinator.java:270)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$66/1254894789.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.PassiveState.query(PassiveState.java:262)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:36:39,375 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:36:41,396 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:36:41,442 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:36:41,443 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:36:41,711 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:36:43,443 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:36:50,446 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:36:54,939 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:36:55,353 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:36:55,353 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:36:55,353 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:36:57,433 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:37:02,435 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:37:08,319 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:37:09,436 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:37:11,786 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:37:14,947 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:37:18,383 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:37:18,384 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:37:19,286 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-4 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.heartbeatMembers(LeaderState.java:96)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$$Lambda$130/915939043.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] ... 7 more 2015-03-10 17:37:21,365 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-7 2015-03-10 17:37:24,288 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:37:31,290 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] 2015-03-10 17:37:33,443 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-5 org.onosproject.store.service.ConsistentMapException: java.util.concurrent.TimeoutException at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:227)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31] Caused by: java.util.concurrent.TimeoutException at net.kuujo.copycat.netty.NettyTcpProtocolClient$1.onRemoval(NettyTcpProtocolClient.java:67)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1956)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3460)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3436)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$Segment.put(LocalCache.java:2891)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache.put(LocalCache.java:4149)[44:com.google.guava:18.0.0] at com.google.common.cache.LocalCache$LocalManualCache.put(LocalCache.java:4754)[44:com.google.guava:18.0.0] at net.kuujo.copycat.netty.NettyTcpProtocolClient.write(NettyTcpProtocolClient.java:94)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultRemoteMemberCoordinator.send(DefaultRemoteMemberCoordinator.java:62)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.CoordinatedMember.send(CoordinatedMember.java:84)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.handleOutboundRequest(DefaultClusterCoordinator.java:280)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter.lambda$createRoutes$15(DefaultClusterCoordinator.java:269)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.cluster.internal.coordinator.DefaultClusterCoordinator$ResourceRouter$$Lambda$65/1624060751.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:494)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.doCommit(LeaderState.java:474)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replica.commit(LeaderState.java:417)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState$Replicator.commit(LeaderState.java:301)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.LeaderState.query(LeaderState.java:152)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$148/2132387879.apply(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext.lambda$wrapCall$33(CopycatStateContext.java:548)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at net.kuujo.copycat.resource.internal.CopycatStateContext$$Lambda$88/987939935.run(Unknown Source)[72:org.onosproject.onlab-thirdparty:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_31] ... 3 more 2015-03-10 17:37:33,444 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-4 2015-03-10 17:37:33,446 | INFO | pool-19-thread-1 | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Already has leadership for intent-partition-5 2015-03-10 17:37:40,446 | WARN | ad-lock-detector | DistributedLeadershipManager | 160 - org.onosproject.onos-core-dist - 1.1.0.SNAPSHOT | Failed to purge stale lock held by 10.128.30.11 for intent-partition-7 org.onosproject.store.service.ConsistentMapException$Timeout at org.onosproject.store.consistent.impl.ConsistentMapImpl.complete(ConsistentMapImpl.java:225)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.ConsistentMapImpl.remove(ConsistentMapImpl.java:199)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.lambda$purgeStaleLocks$9(DistributedLeadershipManager.java:368)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$156/1241730088.accept(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.lang.Iterable.forEach(Iterable.java:75)[:1.8.0_31] at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)[:1.8.0_31] at org.onosproject.store.consistent.impl.DistributedLeadershipManager.purgeStaleLocks(DistributedLeadershipManager.java:349)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at org.onosproject.store.consistent.impl.DistributedLeadershipManager$$Lambda$142/441435249.run(Unknown Source)[160:org.onosproject.onos-core-dist:1.1.0.SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_31] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_31] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_31] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_31] at java.lang.Thread.run(Thread.java:745)[:1.8.0_31]