Sysbench doesn't finished

Thank you for always providing useful software.I will report the issue because there was a problem while using it.

After reading this document, I decided to run the benchmark.

The YugabyteDB cluster worked normally, but the following error occurred and the benchmark could not be run. Is this the problem with this version of Sysbench?

YugabyteDB clusters run on AlmaLinux 8.10 and Python 3.9.20.
Thanks.

[vagrant@node4 ~]$ bash -x test.sh
+ sysbench oltp_read_only --pgsql-host=192.168.30.10,192.168.30.11,192.168.30.12 --tables=20 --table_size=5000000 --range_key_partitioning=false --serial_cache_size=1000 --create_secondary=true --pgsql-db=yugabyte --pgsql-user=yugabyte --db-driver=pgsql --pgsql-port=5433 prepare
sysbench 1.1.0-0b5aea2 (using bundled LuaJIT 2.1.0-beta3)

Server Endpoint: 192.168.30.10 
Server Endpoint: 192.168.30.11 
Server Endpoint: 192.168.30.12 
( 0:59:18) Creating table 'sbtest1'...
alter sequence with cache size: 1000
( 0:59:20) Creating a secondary index on 'sbtest1'...
( 0:59:23) Creating table 'sbtest2'...
alter sequence with cache size: 1000
( 0:59:24) Creating a secondary index on 'sbtest2'...
( 0:59:27) Creating table 'sbtest3'...
alter sequence with cache size: 1000
( 0:59:28) Creating a secondary index on 'sbtest3'...
( 0:59:31) Creating table 'sbtest4'...
alter sequence with cache size: 1000
( 0:59:32) Creating a secondary index on 'sbtest4'...
( 0:59:36) Creating table 'sbtest5'...
alter sequence with cache size: 1000
( 0:59:37) Creating a secondary index on 'sbtest5'...
( 0:59:40) Creating table 'sbtest6'...
alter sequence with cache size: 1000
( 0:59:41) Creating a secondary index on 'sbtest6'...
( 0:59:44) Creating table 'sbtest7'...
alter sequence with cache size: 1000
( 0:59:45) Creating a secondary index on 'sbtest7'...
( 0:59:49) Creating table 'sbtest8'...
alter sequence with cache size: 1000
( 0:59:51) Creating a secondary index on 'sbtest8'...
( 0:59:55) Creating table 'sbtest9'...
alter sequence with cache size: 1000
( 0:59:56) Creating a secondary index on 'sbtest9'...
( 1: 0: 3) Creating table 'sbtest10'...
alter sequence with cache size: 1000
( 1: 0: 4) Creating a secondary index on 'sbtest10'...
( 1: 0: 8) Creating table 'sbtest11'...
alter sequence with cache size: 1000
( 1: 0: 9) Creating a secondary index on 'sbtest11'...
( 1: 0:13) Creating table 'sbtest12'...
alter sequence with cache size: 1000
( 1: 0:14) Creating a secondary index on 'sbtest12'...
( 1: 0:17) Creating table 'sbtest13'...
alter sequence with cache size: 1000
( 1: 0:19) Creating a secondary index on 'sbtest13'...
( 1: 0:22) Creating table 'sbtest14'...
alter sequence with cache size: 1000
( 1: 0:23) Creating a secondary index on 'sbtest14'...
( 1: 0:28) Creating table 'sbtest15'...
alter sequence with cache size: 1000
( 1: 0:30) Creating a secondary index on 'sbtest15'...
( 1: 0:34) Creating table 'sbtest16'...
alter sequence with cache size: 1000
( 1: 0:35) Creating a secondary index on 'sbtest16'...
( 1: 0:41) Creating table 'sbtest17'...
alter sequence with cache size: 1000
( 1: 0:42) Creating a secondary index on 'sbtest17'...
( 1: 0:46) Creating table 'sbtest18'...
alter sequence with cache size: 1000
( 1: 0:47) Creating a secondary index on 'sbtest18'...
( 1: 0:50) Creating table 'sbtest19'...
alter sequence with cache size: 1000
( 1: 0:51) Creating a secondary index on 'sbtest19'...
( 1: 0:55) Creating table 'sbtest20'...
alter sequence with cache size: 1000
( 1: 0:57) Creating a secondary index on 'sbtest20'...
FATAL: PQexec() failed: 7 Invalid table definition: Error creating index yugabyte.k_20 on the master: The requested number of tablet replicas (9) would cause the total running tablet replica count (384) to exceed the safe system maximum (376)
FATAL: failed query was: CREATE INDEX k_20 ON sbtest20(k)
FATAL: `sysbench.cmdline.call_command' function failed: /usr/share/sysbench/oltp_common.lua:281: SQL error, errno = 0, state = 'XX000': Invalid table definition: Error creating index yugabyte.k_20 on the master: The requested number of tablet replicas (9) would cause the total running tablet replica count (384) to exceed the safe system maximum (376)
+ sysbench oltp_read_only --pgsql-host=192.168.30.10,192.168.30.11,192.168.30.12 --tables=20 --table_size=5000000 --range_key_partitioning=false --serial_cache_size=1000 --create_secondary=true --pgsql-db=yugabyte --pgsql-user=yugabyte --db-driver=pgsql --pgsql-port=5433 --time=1800 --warmup-time=300 --num_rows_in_insert=10 --point_selects=10 --index_updates=10 --non_index_updates=10 --range_selects=false --thread-init-timeout=90 --threads=60 run
sysbench 1.1.0-0b5aea2 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 60
Warmup time: 300s
Initializing random number generator from current time


Initializing worker threads...

Server Endpoint: 192.168.30.10 
Server Endpoint: 192.168.30.11 
Server Endpoint: 192.168.30.12 
Threads started!

Warming up for 300 seconds...    //Stop here

It was a storage problem. The problem has been solved.
Thanks!

Hi @ytooyama

The script failed because of this error:

( 1: 0:57) Creating a secondary index on 'sbtest20'...
FATAL: PQexec() failed: 7 Invalid table definition: Error creating index yugabyte.k_20 on the master: The requested number of tablet replicas (9) would cause the total running tablet replica count (384) to exceed the safe system maximum (376)

Looks like you’re hitting the “Max number of tablet replicas” limit yb-master configuration reference | YugabyteDB Docs.

You need to do one of:

  • increase the system resources for the yb-tserver
  • lower the default number of tablets used for a table/index
  • use a colocated database
1 Like

I just ran the command locally on a single node and it created 20 tables + 20 indexes (default to 40 tablets on a single node cluster). While you’re creating ~400 so there is room to lower it.

1 Like

Thank you, dorian.
I’ll take a look at the document.

I created a test environment with Vagrant, but I forgot to enlarge it on the OS side even though I set the storage size larger.
As a result, it seems that the storage was full and failed.

I adjusted the sysbench options and no longer caused the problem.

–tables=20, --table_size=1000000