Hi
I am receiving below error while creating a partition table in YSQLSH. Its 12 Tserver and 3 Master with replication factor 3.
ERROR: Invalid table definition: Error creating table db.test_202403 on the master: Not enough live tablet servers to create table with replication factor 3. Need at least 2 tablet servers whereas 0 are alive.
Please suggest.
Thanks
Subhankar
As the error suggeests, you don have tservers in all azs based on your cluster config. Can you share the cluster config and master and tserver list from web ui?
Hi, can you check that you have no 'tablespace" clause in the create table statement and that default_tablespace is not set?
show default_tablespace;
select * from pg_tablespace;
Hi image got cutoff. But can you post output for the command that @FranckPachot posted.
In your cluster, you are looking for 3 replicas, but you have only provided on replica config. That is the problem, I suppose. How are you creating this cluster? Using yugabyted command or yb-master yb-tserver commands directly?
Also share exact create table command… feel free to anonymize object and col names if its a concern.
I have created using yb-master, yb-tserver command.
I see too many underreplicated tables as below.
Also some of the tablets does not have 2 followers:
yugabyte@xxxxxxxxx:/pg_data_master/yugabyte/yugabyte-2.20.1.3$ bin/yb-admin --master_addresses xx.xx.xx.71:7100,xx.xx.xx.72:7100,xx.xx.xx.73:7100 list_tablets ysql.dldb test include_followers
Tablet-UUID Range Leader-IP Leader-UUID Followers
c5e946794e8c420d80158306f3da32ff partition_key_start: “” partition_key_end: “\025U” xx.xx.xx.65:9100 112babe077c44d4897c4337dcbe13541 xx.xx.xx.78:9100
2ea0698c89c24fa5900f9468578863b5 partition_key_start: “\025U” partition_key_end: “\252" xx.xx.xx.76:9100 12e16c9d501642b685a5d1dda1a8d5bf xx.xx.xx.77:9100,xx.xx.xx.79:9100
2c88283cc023496fa7785829fa80be7f partition_key_start: "\252” partition_key_end: “@\000” xx.xx.xx.65:9100 112babe077c44d4897c4337dcbe13541 xx.xx.xx.78:9100
81e3b07adcd44b55899153102be72a83 partition_key_start: “@\000” partition_key_end: “UU” xx.xx.xx.75:9100 e48bfa6399cc4aeeae3efb4f7ca3032f xx.xx.xx.81:9100,xx.xx.xx.74:9100
f66d054a69d84281b0ab2d8f60f533aa partition_key_start: “UU” partition_key_end: “j\252” xx.xx.xx.76:9100 12e16c9d501642b685a5d1dda1a8d5bf xx.xx.xx.77:9100,xx.xx.xx.79:9100
9a2892faa4514bfca3617147cafff701 partition_key_start: “j\252” partition_key_end: “\200\000” xx.xx.xx.76:9100 12e16c9d501642b685a5d1dda1a8d5bf xx.xx.xx.77:9100,xx.xx.xx.79:9100
be74af7fbdda46eb862d8121596dcc55 partition_key_start: “\200\000” partition_key_end: “\225U” xx.xx.xx.80:9100 5dd6d75260cd4796a7849a7226dcf6a6 xx.xx.xx.82:9100,xx.xx.xx.63:9100
2e3fc553c67446b7ad37f29eeba87d07 partition_key_start: “\225U” partition_key_end: “\252\252” xx.xx.xx.74:9100 16d3fdeb443340de995ff8720001788b xx.xx.xx.75:9100,xx.xx.xx.81:9100
697f7fe972f142a095ef7e5d28246ba4 partition_key_start: “\252\252” partition_key_end: “\300\000” xx.xx.xx.78:9100 1e8700e3305e431699ef7f1249751396 xx.xx.xx.65:9100
f33261b1a76a49eda70063b1dd7fceb1 partition_key_start: “\300\000” partition_key_end: “\325U” xx.xx.xx.75:9100 e48bfa6399cc4aeeae3efb4f7ca3032f xx.xx.xx.81:9100,xx.xx.xx.74:9100
table is a partitioned table:
create table test (
column1 character varying(255),
column2 character varying(255),
column3 timestamp with time zone,
column4 character varying(255),
column5 character varying(255),
primary key (column1, column2, column3))
Partition by range (column3);
create table test1 partition of test for values from (‘2024-03-01’) to (‘2024-04-01’);
We need to see the yb-tserver commands because what is displayed is strange. The UUID should a uuid - not the placement flags, and it seems you have nodes seen as read replicas but with tablet leaders 
Hi Frank
This is the yb–tserver conf file i used to start tserver.
–tserver_master_addrs=10.91.88.71:7100,10.91.88.72:7100,10.91.88.73:7100
–rpc_bind_addresses=10.91.88.74:9100
–enable_ysql
–pgsql_proxy_bind_address=10.91.88.74:5536
–ysql_max_connections=900
–fs_data_dirs=/data1/yugabyte,/data2/yugabyte
–fs_wal_dirs=/wal/yugabyte/waldir
–placement_cloud=xxxx-pvt-cloud
–placement_region=xxxx-chn
–placement_zone=xxxx-chn-1d
–webserver_interface=10.91.86.74
–webserver_port=9001
–ysql_timezone=Asia/Kolkata
–stream_compression_algo=3
–ysql_enable_packed_row=true
–enable_automatic_tablet_splitting=true
–ysql_num_shards_per_tserver=1
–cql_proxy_bind_address=10.91.88.74:9043
I used below command to start tserver:
nohup /pg_data_master/yugabyte/yugabyte-2.20.1.3/bin/yb-tserver --flagfile /pg_data_master/yugabyte/yugabyte-2.20.1.3/conf/tserver.conf > yb-tserver.out &
I also used below command later:
bin/yb-admin -master_addresses 10.91.88.71:7100,10.91.88.72:7100,10.91.88.73:7100 modify_placement_info replication_factor 3 xxxx-pvt-cloud.xxxx-chn.xxxx-chn-1d:1,xxxx-pvt-cloud.xxxx-chn.xxxx-chn-2d:1,xxxx-pvt-cloud.xxxx-chn.xxxx-chn-3d:1
Please help.
Thanks
Subhankar
This is how the rest of the Tservers looks like
Some of the Tservers does not have any system tablet Leaders/peers.
Please suggest how to make this right.
@FranckPachot, This currently in production. it was fine at benchmark time. some more nodes were added and one nodes needed to be blacklisted, before going into production. Please let us know what are the ways we can meke it correct.
Thanks & regards,
Dipanjan
@dipanjanghos , @subh14 :
In the screenshot shared for “Current Cluster Config”
the placement_cloud is shown as replication_factor that seems incorrect.
I believe the yb-admin’s modify_placement_info command is being used incorrectly.
See documentation:
Instead of:
bin/yb-admin -master_addresses ip1:7100,ip2:7100,ip3:7100 modify_placement_info replication_factor 3 c.r.z1:1,c.r.z2:1,c.r.z3:1
the command should be something like:
bin/yb-admin -master_addresses ip1:7100,ip2:7100,ip3:7100 modify_placement_info c.r.z1:1,c.r.z2:1,c.r.z3:1 3
If the above command is executed correctly, then the all the servers in the 2nd image in this link:
should show up in the PRIMARY section rather than in the READ REPLICA section (I believe).
Hi kannan
I have executed the query as you suggested, still those tservers are not coming up in the primary section.
We cannot create new tables also. Please help.
Thanks
Subhankar
Hi
I was able to solve this another way. Thanks for your help.
Thanks
Subhankar
Good to hear tht. can you share a bit about this “another way”?
I have just removed that additional placement info I was adding. then I saw it put all the nodes in primary cluster and I was able to create tables.
Thanks
Subhankar
