Hi,
we are currently running some tests with yugabyte 2.20.7.
The cluster consists of 3 masters and 3 tablet servers.
It is deployed with helm on kubernetes with the automatic tablet splitting feature enabled.
Each tablet server has two 74 GB disks.
We are using the YCQL API and created a test table like this:
CREATE TABLE IF NOT EXISTS ks.test(id text, secondary_id uuid, data1 text, data2 blob, ts timestamp, PRIMARY KEY (id, secondary_id));
We then loaded some test data (11,883,854 rows).
As expected more tablets were created over time.
At the end of the test, 10 tablets were created for this test table.
We noticed that the tablets were not evenly distributed across the available disks.
On each tserver more space is used on disk-0 than disk-1.
tserver-1:
- disk-0: 58.6 GB
- disk-1: 30.7 GB
tserver-2:
- disk-0: 59.2 GB
- disk-1: 42.2 GB
tserver-3:
- disk-0: 55.3 GB
- disk-1: 28.5 GB
We had a look at the tablets of the table:
Tablet-UUID Range Leader-IP Leader-UUID
17a7c4c8bc884ab78a7067d52fa96b87 partition_key_start: "" partition_key_end: "\031U" 10.240.10.50:9100 66789c933308415d96fbdffe094a6e68
6e56377eb86e4c2388788a6a4f482bf1 partition_key_start: "\031U" partition_key_end: "4\201" 10.240.2.166:9100 391a9a3cf1904782853499d4677258f9
1b5b6e2289eb4bdd86819bce3ec0a22e partition_key_start: "4\201" partition_key_end: "L^" 10.240.10.50:9100 66789c933308415d96fbdffe094a6e68
6ffcfc10fc1f4c2aadce41d58db9538d partition_key_start: "L^" partition_key_end: "UU" 10.240.13.226:9100 625e32c70d6149ff8ade08aded841e17
cf3e0593abdc46cd925c9da5217c5f93 partition_key_start: "UU" partition_key_end: "p\010" 10.240.2.166:9100 391a9a3cf1904782853499d4677258f9
465c68ff2ab34554934cfb97219bdfdf partition_key_start: "p\010" partition_key_end: "\212\274" 10.240.2.166:9100 391a9a3cf1904782853499d4677258f9
e9b064458236489096b449c163114d61 partition_key_start: "\212\274" partition_key_end: "\252\252" 10.240.13.226:9100 625e32c70d6149ff8ade08aded841e17
2abb97341c134bf8a2c95e38e2e71fae partition_key_start: "\252\252" partition_key_end: "\304[" 10.240.10.50:9100 66789c933308415d96fbdffe094a6e68
7d02316c432b4b11b8d82f0a02e06f29 partition_key_start: "\304[" partition_key_end: "\337\225" 10.240.2.166:9100 391a9a3cf1904782853499d4677258f9
6de2baf421664b48820576af307ab9fc partition_key_start: "\337\225" partition_key_end: "" 10.240.13.226:9100 625e32c70d6149ff8ade08aded841e17
And looked on tserver-1 how the tablets are distributed across the disks:
disk-0:
- tablet-17a7c4c8bc884ab78a7067d52fa96b87
- tablet-6e56377eb86e4c2388788a6a4f482bf1
- tablet-1b5b6e2289eb4bdd86819bce3ec0a22e
- tablet-6ffcfc10fc1f4c2aadce41d58db9538d
- tablet-cf3e0593abdc46cd925c9da5217c5f93
- tablet-465c68ff2ab34554934cfb97219bdfdf
- tablet-e9b064458236489096b449c163114d61
disk-1:
- tablet-2abb97341c134bf8a2c95e38e2e71fae
- tablet-7d02316c432b4b11b8d82f0a02e06f29
- tablet-6de2baf421664b48820576af307ab9fc
Is there are way to distribute the tablets evenly across the available tserver disks?
We also notice that the tservers had different number of leaders. We are not sure if this is related with the unbalanced disk usage, as the cluster status is shown as balanced.
The web ui shows this info for the tablets of the test table: