TransportException: Connection has been closed

General
1

Hi,
I am doing a POT for YB. I created a 3 node cluster (RF = 3) on local machine (MacBook Pro) as explained in the documentation of YB.
I have written a java utility to create a key-space with 6 tables. It is a simple utility that does - Cluster → Session → Execute CQL.

The utility fails with the exception trace pasted below. I checked the logs and I see “Too many open files (error 24)”. Not sure if this is the root cause. Just to create a key-space with 6 tables, what’s the kind of settings are we looking on a local machine?

Local Machine Open Files Configuration:
kern.maxfiles: 49152
kern.maxfilesperproc: 24576

Exception Stack Trace:
com.datastax.driver.core.exceptions.TransportException: [/127.0.0.3:9042] Connection has been closed

  • at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1241)*
  • at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1226)*
  • at com.datastax.driver.core.Connection.defunct(Connection.java:464)*
  • at com.datastax.driver.core.Connection$ChannelCloseListener.operationComplete(Connection.java:1183)*
  • at com.datastax.driver.core.Connection$ChannelCloseListener.operationComplete(Connection.java:1173)*
  • at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)*
  • at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)*
  • at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)*
  • at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)*
  • at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)*
  • at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)*
  • at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1004)*
  • at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:633)*
  • at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:611)*
  • at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:554)*
  • at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)*
  • at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)*
  • at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)*
  • at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)*
  • at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)*
  • at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)*
  • at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)*
  • at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)*
  • at java.lang.Thread.run(Thread.java:748)*

Log Excerpt

E0510 17:54:20.340422 141635584 tablet.cc:508] T 2bbb83c56968467f8ab5b223b9dcdc2f P 0a6055a954b445e198ea2b225f8a7f8c: Failed to open a RocksDB database in directory /private/tmp/yugabyte-local-cluster/node-2/disk-2/yb-data/tserver/data/rocksdb/table-d626fab88fce4545a488b7687e21d7a4/tablet-2bbb83c56968467f8ab5b223b9dcdc2f: IO error (yb/rocksdb/util/env_posix.cc:337): /private/tmp/yugabyte-local-cluster/node-2/disk-2/yb-data/tserver/data/rocksdb/table-d626fab88fce4545a488b7687e21d7a4/tablet-2bbb83c56968467f8ab5b223b9dcdc2f: Too many open files

E0510 17:54:20.341192 143245312 ts_tablet_manager.cc:963] T c9bb00c885bf4bcb998d2453687efcf2 P 0a6055a954b445e198ea2b225f8a7f8c: Tablet failed to bootstrap: Illegal state (yb/tablet/tablet.cc:513): IO error (yb/rocksdb/util/env_posix.cc:225): /private/tmp/yugabyte-local-cluster/node-2/disk-1/yb-data/tserver/data/rocksdb/table-d626fab88fce4545a488b7687e21d7a4/tablet-c9bb00c885bf4bcb998d2453687efcf2/MANIFEST-000001: Too many open files

E0510 17:54:20.341200 141635584 ts_tablet_manager.cc:963] T 2bbb83c56968467f8ab5b223b9dcdc2f P 0a6055a954b445e198ea2b225f8a7f8c: Tablet failed to bootstrap: Illegal state (yb/tablet/tablet.cc:513): IO error (yb/rocksdb/util/env_posix.cc:337): /private/tmp/yugabyte-local-cluster/node-2/disk-2/yb-data/tserver/data/rocksdb/table-d626fab88fce4545a488b7687e21d7a4/tablet-2bbb83c56968467f8ab5b223b9dcdc2f: Too many open files

W0510 17:54:20.343843 144318464 db_impl.cc:5823] T 6e70f835dc6c4580aea19b8177f39113: Unable to persist options in DB::Open() – IO error (yb/rocksdb/util/env_posix.cc:225): /private/tmp/yugabyte-local-cluster/node-2/disk-1/yb-data/tserver/data/rocksdb/table-d626fab88fce4545a488b7687e21d7a4/tablet-6e70f835dc6c4580aea19b8177f39113/OPTIONS-000004.dbtmp: Too many open files

W0510 17:54:20.345109 144318464 log.cc:1086] Error closing log: Illegal state (yb/consensus/log.cc:938): Bad state for Close() 0

E0510 17:54:20.345659 144318464 ts_tablet_manager.cc:963] T 6e70f835dc6c4580aea19b8177f39113 P 0a6055a954b445e198ea2b225f8a7f8c: Tablet failed to bootstrap: IO error (yb/util/env_posix.cc:1093): Failed to open new log: Unable to read children from path: /private/tmp/yugabyte-local-cluster/node-2/disk-1/yb-data/tserver/wals/table-d626fab88fce4545a488b7687e21d7a4/tablet-6e70f835dc6c4580aea19b8177f39113: Too many open files (error 24)

E0510 17:54:20.347438 143781888 ts_tablet_manager.cc:985] T d2a6a67f0cc44d4bbaf86fa1f0e36ac4 P 0a6055a954b445e198ea2b225f8a7f8c: Tablet failed to init: IO error (yb/util/env_posix.cc:1013): /private/tmp/yugabyte-local-cluster/node-2/disk-1/yb-data/tserver/consensus-meta/d2a6a67f0cc44d4bbaf86fa1f0e36ac4: Too many open files (error 24)

E0510 17:54:20.347455 142708736 ts_tablet_manager.cc:985] T 400658370bbb4ae78f2a9e4b7d7165d1 P 0a6055a954b445e198ea2b225f8a7f8c: Tablet failed to init: IO error (yb/util/env_posix.cc:1013): /private/tmp/yugabyte-local-cluster/node-2/disk-1/yb-data/tserver/consensus-meta/400658370bbb4ae78f2a9e4b7d7165d1: Too many open files (error 24)

2

You should increase your kern.maxfiles to 1048576. Here is what my ulimits are:

Alans-MacBook-Pro-2:~ acaldera$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1418
virtual memory          (kbytes, -v) unlimited
3

Thanks Alan, it worked.

1 Like