jk-1
January 30, 2020, 9:55pm
1
My single node test environment (v2.0.8.0) went grazy and produced about 19GB log files in just 6 hours
Maybe yugabyte should somehow limit the number of log entries it write / second or minute especially because the following line occupies 98% of the space:
W0130 20:12:41.325667 12419 acceptor.cc:158] Acceptor: accept failed: Network error (yb/util/net/socket.cc:357): accept4(2) error: Too many open files (system error 24)
I read the manuals and noticed that the default log size is 1.6GB and that I can control it by setting e.g. --max_log_size=256 but where to put it if I start and stop the cluster by yb-ctl start / stop?
Any variation of
yb-ctl --max_log_size=256 start
will fail.
Thank you for your support!
kannan
January 30, 2020, 10:56pm
2
hi @jk-1 :
Since you are running into the “Too many open files” error, can you please check the ulimit -a
settings for max open files please?
Please see Deploy | YugabyteDB Docs here.
kannan
January 30, 2020, 11:18pm
3
For the second part of your question, since you are using yb-ctl, the way to pass tserver gflags is:
./bin/yb-ctl start --tserver_flags "max_log_size=256"
jk-1
January 31, 2020, 12:36pm
5
You have a point. I have several Yugabyte installations on different VMs and on this VM I forgot to run the prerequisities so the max number of open files was too low. Thank you for your support!
jk-1
January 31, 2020, 3:44pm
7
I tried this but if failed.
jk@tiira:~/bin2/yugabyte-2.0.8.0/bin$ ./yb-ctl start --tserver_flags “max_log_size=32”
Starting cluster with base directory /home/jk/yugabyte-data
Waiting for cluster to be ready.
Viewing file /home/jk/yugabyte-data/node-1/disk-1/tserver.err:
ERROR: unknown command line flag '“max_log_size'
Viewing file /tmp/tmpNIfAc5:
2020-01-31 17:39:40,845 INFO: Starting master-1 with:
/home/jk/bin2/yugabyte-2.0.8.0/bin/yb-master --fs_data_dirs "/home/jk/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 0 --version_file_json_path=/home/jk/bin2/yugabyte-2.0.8.0 --webserver_doc_root "/home/jk/bin2/yugabyte-2.0.8.0/www" --replication_factor=1 --yb_num_shards_per_tserver 2 --ysql_num_shards_per_tserver=2 --master_addresses 127.0.0.1:7100 --enable_ysql=true >"/home/jk/yugabyte-data/node-1/disk-1/master.out" 2>"/home/jk/yugabyte-data/node-1/disk-1/master.err" &
2020-01-31 17:39:40,882 INFO: Starting tserver-1 with:
/home/jk/bin2/yugabyte-2.0.8.0/bin/yb-tserver --fs_data_dirs "/home/jk/yugabyte-data/node-1/disk-1" --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --v 0 --version_file_json_path=/home/jk/bin2/yugabyte-2.0.8.0 --webserver_doc_root "/home/jk/bin2/yugabyte-2.0.8.0/www" --tserver_master_addrs=127.0.0.1:7100 --yb_num_shards_per_tserver=2 --redis_proxy_bind_address=127.0.0.1:6379 --cql_proxy_bind_address=127.0.0.1:9042 --local_ip_for_outbound_sockets=127.0.0.1 --use_cassandra_authentication=false --ysql_num_shards_per_tserver=2 --enable_ysql=true --pgsql_proxy_bind_address=127.0.0.1:5433 --“max_log_size=32” >"/home/jk/yugabyte-data/node-1/disk-1/tserver.out" 2>"/home/jk/yugabyte-data/node-1/disk-1/tserver.err" &
2020-01-31 17:39:40,887 INFO: Waiting for master and tserver processes to come up.
2020-01-31 17:39:40,952 INFO: Waiting for master leader election and tablet server registration.
2020-01-31 17:39:44,971 INFO: PIDs found: {'tserver': [None], 'master': [1614]}
2020-01-31 17:39:44,972 ERROR: At least one master or tserver process is down.
^^^ Encountered errors ^^^
Without the log paramter it starts fine:
jk@tiira:~/bin2/yugabyte-2.0.8.0/bin$ ./yb-ctl stop
Stopping cluster.
jk@tiira:~/bin2/yugabyte-2.0.8.0/bin$ ./yb-ctl start
Starting cluster with base directory /home/jk/yugabyte-data
Waiting for cluster to be ready.
----------------------------------------------------------------------------------------------------
| Node Count: 1 | Replication Factor: 1 |
----------------------------------------------------------------------------------------------------
| JDBC : jdbc:postgresql://127.0.0.1:5433/postgres |
| YSQL Shell : ysqlsh |
| YCQL Shell : cqlsh |
| YEDIS Shell : redis-cli |
| Web UI : http://127.0.0.1:7000/ |
| Cluster Data : /home/jk/yugabyte-data |
----------------------------------------------------------------------------------------------------
For more info, please use: yb-ctl status
jk-1:
“max_log_size=256”
Please replace ”
with "
.
After replacing ^ your command works for me.
jk-1
February 1, 2020, 2:44pm
9
I upgraded to 2.0.11.0 for other reasons and with that
./yb-ctl start --tserver_flags “max_log_size=32”
works fine.