Uploaded image for project: 'ONOS'
  1. ONOS
  2. ONOS-7778

"Too many open files" when handling large Mininet topology

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Epic Link:

      Description

      In topology scaling test, when starting 55x55 torus topology (3025 switches and 12100 links), we saw issues caused by "Too many open files" warnings.

      2018-08-17 14:23:51,630 | WARN  | onos-of-boss-1   | DefaultChannelPipeline           | 106 | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
      java.io.IOException: Too many open files
              at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)[:1.8.0_121]
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)[:1.8.0_121]
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)[:1.8.0_121]
              at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:110)
              at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:107)
              at java.security.AccessController.doPrivileged(Native Method)[:1.8.0_121]
              at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:107)[106:io.netty.common:4.1.27.Final]
              at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:141)[108:io.netty.transport:4.1.27.Final]
              at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75)[108:io.netty.transport:4.1.27.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)[108:io.netty.transport:4.1.27.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)[108:io.netty.transport:4.1.27.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)[108:io.netty.transport:4.1.27.Final]
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)[108:io.netty.transport:4.1.27.Final]
              at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)[106:io.netty.common:4.1.27.Final]
              at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
      

      The limit of open files on the server was already set to 999999. However during the test "lsof" gave over 7 million open files. ~97% of the files were opened by ONOS and ~2% by Atomix.

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            tom Thomas Vachuska
            Reporter:
            You You Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: