-
Type: Bug
-
Status: Open (View Workflow)
-
Priority: Major
-
Resolution: Unresolved
-
Affects Version/s: 1.14.0
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:3
-
Epic Link:
In topology scaling test, when starting 55x55 torus topology (3025 switches and 12100 links), we saw issues caused by "Too many open files" warnings.
2018-08-17 14:23:51,630 | WARN | onos-of-boss-1 | DefaultChannelPipeline | 106 | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)[:1.8.0_121] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)[:1.8.0_121] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)[:1.8.0_121] at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:110) at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:107) at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_121] at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:107)[106:io.netty.common:4.1.27.Final] at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:141)[108:io.netty.transport:4.1.27.Final] at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75)[108:io.netty.transport:4.1.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)[108:io.netty.transport:4.1.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)[108:io.netty.transport:4.1.27.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)[108:io.netty.transport:4.1.27.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)[108:io.netty.transport:4.1.27.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)[106:io.netty.common:4.1.27.Final] at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
The limit of open files on the server was already set to 999999. However during the test "lsof" gave over 7 million open files. ~97% of the files were opened by ONOS and ~2% by Atomix.