Yugabyte produced 19GB log files in 6 hours

#1

My single node test environment (v2.0.8.0) went grazy and produced about 19GB log files in just 6 hours

Maybe yugabyte should somehow limit the number of log entries it write / second or minute especially because the following line occupies 98% of the space:

W0130 20:12:41.325667 12419 acceptor.cc:158] Acceptor: accept failed: Network error (yb/util/net/socket.cc:357): accept4(2) error: Too many open files (system error 24)

I read the manuals and noticed that the default log size is 1.6GB and that I can control it by setting e.g. --max_log_size=256 but where to put it if I start and stop the cluster by yb-ctl start / stop?

Any variation of
yb-ctl --max_log_size=256 start
will fail.

Thank you for your support!

#2

hi @jk-1:

Since you are running into the “Too many open files” error, can you please check the ulimit -a settings for max open files please?

Please see https://docs.yugabyte.com/latest/deploy/manual-deployment/system-config/#ulimits here.

#3

For the second part of your question, since you are using yb-ctl, the way to pass tserver gflags is:
./bin/yb-ctl start --tserver_flags "max_log_size=256"

#5

You have a point. I have several Yugabyte installations on different VMs and on this VM I forgot to run the prerequisities so the max number of open files was too low. Thank you for your support!

#6

I Try this immediately.

#7

I tried this but if failed.

jk@tiira:~/bin2/yugabyte-2.0.8.0/bin$ ./yb-ctl start  --tserver_flags “max_log_size=32”
Starting cluster with base directory /home/jk/yugabyte-data
Waiting for cluster to be ready.
Viewing file /home/jk/yugabyte-data/node-1/disk-1/tserver.err:
ERROR: unknown command line flag '“max_log_size'
Viewing file /tmp/tmpNIfAc5:
2020-01-31 17:39:40,845 INFO: Starting master-1 with:
/home/jk/bin2/yugabyte-2.0.8.0/bin/yb-master --fs_data_dirs "/home/jk/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 0 --version_file_json_path=/home/jk/bin2/yugabyte-2.0.8.0 --webserver_doc_root "/home/jk/bin2/yugabyte-2.0.8.0/www" --replication_factor=1 --yb_num_shards_per_tserver 2 --ysql_num_shards_per_tserver=2 --master_addresses 127.0.0.1:7100 --enable_ysql=true >"/home/jk/yugabyte-data/node-1/disk-1/master.out" 2>"/home/jk/yugabyte-data/node-1/disk-1/master.err" &
2020-01-31 17:39:40,882 INFO: Starting tserver-1 with:
/home/jk/bin2/yugabyte-2.0.8.0/bin/yb-tserver --fs_data_dirs "/home/jk/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 0 --version_file_json_path=/home/jk/bin2/yugabyte-2.0.8.0 --webserver_doc_root "/home/jk/bin2/yugabyte-2.0.8.0/www" --tserver_master_addrs=127.0.0.1:7100 --yb_num_shards_per_tserver=2 --redis_proxy_bind_address=127.0.0.1:6379 --cql_proxy_bind_address=127.0.0.1:9042 --local_ip_for_outbound_sockets=127.0.0.1 --use_cassandra_authentication=false --ysql_num_shards_per_tserver=2 --enable_ysql=true --pgsql_proxy_bind_address=127.0.0.1:5433 --“max_log_size=32” >"/home/jk/yugabyte-data/node-1/disk-1/tserver.out" 2>"/home/jk/yugabyte-data/node-1/disk-1/tserver.err" &
2020-01-31 17:39:40,887 INFO: Waiting for master and tserver processes to come up.
2020-01-31 17:39:40,952 INFO: Waiting for master leader election and tablet server registration.
2020-01-31 17:39:44,971 INFO: PIDs found: {'tserver': [None], 'master': [1614]}
2020-01-31 17:39:44,972 ERROR: At least one master or tserver process is down.
^^^ Encountered errors ^^^

Without the log paramter it starts fine:

jk@tiira:~/bin2/yugabyte-2.0.8.0/bin$ ./yb-ctl stop
Stopping cluster.
jk@tiira:~/bin2/yugabyte-2.0.8.0/bin$ ./yb-ctl start
Starting cluster with base directory /home/jk/yugabyte-data
Waiting for cluster to be ready.
----------------------------------------------------------------------------------------------------
| Node Count: 1 | Replication Factor: 1                                                            |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.1:5433/postgres                                  |
| YSQL Shell          : ysqlsh                                                                     |
| YCQL Shell          : cqlsh                                                                      |
| YEDIS Shell         : redis-cli                                                                  |
| Web UI              : http://127.0.0.1:7000/                                                     |
| Cluster Data        : /home/jk/yugabyte-data                                                     |
----------------------------------------------------------------------------------------------------

For more info, please use: yb-ctl status
#8

Please replace ” with ".
After replacing ^ your command works for me.

#9

I upgraded to 2.0.11.0 for other reasons and with that

./yb-ctl start --tserver_flags “max_log_size=32”

works fine.