Hello team,
I have an issue with a table in my db, I have a cluster of 3 nodes, actually I see 2 tablets running and 2 stuck in “creating” stage. I cannot fetch the data from that table now!
How to proceed to unstuck the process please?
SplitDepth : 2 and state : Creating |
|---|
The table is now stuck in Altering mode!
When i try to do a SELECT on it I get GetTableLocationsRpc, 120s deadline, 486+ attempts
{"underreplicated_tablets":[{"table_uuid":"00004001000030008000000000004193","tablet_uuid":"1a092f17af2c4c98bef2e8aa35578273","underreplicated_placements":["78084298-4b65-48f0-9082-93f88ec4846f"]},{"table_uuid":"00004001000030008000000000004193","tablet_uuid":"d51000ef61e04a48b8614389526a2777","underreplicated_placements":["78084298-4b65-48f0-9082-93f88ec4846f"]}]}
{rosh-1.7.3}yugabytedb_prod@$hostname:/local/data/scratch/yugabyte/data/yb-data/master/logs$ grep -n ‘1a092f17af2c4c98bef2e8aa35578273|d51000ef61e04a48b8614389526a2777’ yb-master..INFO. yb-master..WARNING.
yb-master.$hostname.root.log.INFO.20260802-190013.22:2335:I0802 19:11:03.316890 1989 catalog_manager.cc:7965] Registered new tablet 1a092f17af2c4c98bef2e8aa35578273 (partition_key_start: , partition_key_end: 4DF0, split_depth: 2) to split the tablet 1a6e00b5ceb143d38a6b44b044e76238 (partition_key_start: , partition_key_end: 869F) for table Listing [id=00004001000030008000000000004193], new partition_list_version: 3
yb-master.$hostname.root.log.INFO.20260802-190013.22:2336:I0802 19:11:03.316942 1989 catalog_manager.cc:7965] Registered new tablet d51000ef61e04a48b8614389526a2777 (partition_key_start: 4DF0, partition_key_end: 869F, split_depth: 2) to split the tablet 1a6e00b5ceb143d38a6b44b044e76238 (partition_key_start: , partition_key_end: 869F) for table Listing [id=00004001000030008000000000004193], new partition_list_version: 3
yb-master.$hostname.root.log.INFO.20260802-190013.22:2337:I0802 19:11:03.316982 1989 tablet_split_manager.cc:1010] Processing split tablet result for table 00004001000030008000000000004193, split tablet ids: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} }
yb-master.$hostname.root.log.INFO.20260802-190013.22:2341:I0802 19:11:03.319175 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: dedd0eea6ebadba4f0436d3ea0700f04
yb-master.$hostname.root.log.INFO.20260802-190013.22:2342:I0802 19:11:03.319506 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: 2a53167c33be9e8ac64f3242617a8e1e
yb-master.$hostname.root.log.INFO.20260802-190013.22:2343:I0802 19:11:03.319825 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: e97ce36439392994134db0f215893728
yb-master.$hostname.root.log.INFO.20260802-190013.22:2344:I0802 19:11:03.320145 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: 99636c99d8b64a8b704cdd397df64540
yb-master.$hostname.root.log.INFO.20260802-190013.22:2345:I0802 19:11:03.320400 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: 226eeab6b7a4f78862470b7ca3d6a341
yb-master.$hostname.root.log.INFO.20260802-190013.22:2346:I0802 19:11:03.320667 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: f3a3d7cc12f7e5bed247cf8ee7fc9355
yb-master.$hostname.root.log.INFO.20260802-190013.22:2347:I0802 19:11:03.321017 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: bba6b28d54b5989507490d7aaa372dcb
yb-master.$hostname.root.log.INFO.20260802-190013.22:2348:I0802 19:11:03.321273 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: 71099581f57fe696c74bd870704af3d9
yb-master.$hostname.root.log.INFO.20260802-190013.22:2349:I0802 19:11:03.321525 1989 xrepl_catalog_manager.cc:3595] UpdateCDCProducerOnTabletSplit: Copy active time from parent to child tablets Tablets involved: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} } Consistent Snapshot StreamId: 9ebfe02ddab8549a3e4f9da5ee3a61de
yb-master.$hostname.root.log.INFO.20260802-190013.22:2351:W0802 19:11:03.323125 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “dedd0eea6ebadba4f0436d3ea0700f04” } } column_values { column_id: 2 expr { value { string_value: “25.184349” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775577709014135” } values { string_value: “7272764778033328128” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009656056 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2352:W0802 19:11:03.323379 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “2a53167c33be9e8ac64f3242617a8e1e” } } column_values { column_id: 2 expr { value { string_value: “27.185102” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775647184616936” } values { string_value: “7273039532722868224” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009607064 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2353:W0802 19:11:03.323412 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “e97ce36439392994134db0f215893728” } } column_values { column_id: 2 expr { value { string_value: “25.184124” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775570390536966” } values { string_value: “7272727287394136064” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009765592 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2354:W0802 19:11:03.323441 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “99636c99d8b64a8b704cdd397df64540” } } column_values { column_id: 2 expr { value { string_value: “11.160252” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1773944212052215” } values { string_value: “7266075491105927168” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009860216 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2355:W0802 19:11:03.323463 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “226eeab6b7a4f78862470b7ca3d6a341” } } column_values { column_id: 2 expr { value { string_value: “170.561753” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1780579026504134” } values { string_value: “7293242532020936704” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18040943736152 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2356:W0802 19:11:03.323485 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “f3a3d7cc12f7e5bed247cf8ee7fc9355” } } column_values { column_id: 2 expr { value { string_value: “11.160251” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1773944169870952” } values { string_value: “7266073791835602944” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18040943975480 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2357:W0802 19:11:03.323508 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “bba6b28d54b5989507490d7aaa372dcb” } } column_values { column_id: 2 expr { value { string_value: “514.903027” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1785697210302096” } values { string_value: “7314099619907739648” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041010020696 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2358:W0802 19:11:03.323527 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “71099581f57fe696c74bd870704af3d9” } } column_values { column_id: 2 expr { value { string_value: “132.541889” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1780254284662542” } values { string_value: “7289999457915379712” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18040854943544 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2359:W0802 19:11:03.323547 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “9ebfe02ddab8549a3e4f9da5ee3a61de” } } column_values { column_id: 2 expr { value { string_value: “39.259609” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775811561341865” } values { string_value: “7273724157304283136” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041010103096 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.INFO.20260802-190013.22:2360:W0802 19:11:03.323592 1989 catalog_manager.cc:3212] Failed to split tablet with GetSplitKey result for tablet: 1a6e00b5ceb143d38a6b44b044e76238: IO error (yb/client/batcher.cc:111): Encountered an error while updating the CDC producer metadata. Table id: 00004001000030008000000000004193, Split Tablets: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} }: Errors occurred while reaching out to the tablet servers
yb-master.$hostname.root.log.INFO.20260804-114251.22:1169:I0804 11:43:49.696677 810 catalog_loaders.cc:424] Loaded metadata for tablet 1a092f17af2c4c98bef2e8aa35578273 (first table Listing [id=00004001000030008000000000004193])
yb-master.$hostname.root.log.INFO.20260804-114251.22:1238:I0804 11:43:49.699491 810 catalog_loaders.cc:424] Loaded metadata for tablet d51000ef61e04a48b8614389526a2777 (first table Listing [id=00004001000030008000000000004193])
yb-master.$hostname.root.log.INFO.20260804-121300.22:1272:I0804 12:13:27.440539 198 catalog_loaders.cc:424] Loaded metadata for tablet 1a092f17af2c4c98bef2e8aa35578273 (first table Listing [id=00004001000030008000000000004193])
yb-master.$hostname.root.log.INFO.20260804-121300.22:1341:I0804 12:13:27.443123 198 catalog_loaders.cc:424] Loaded metadata for tablet d51000ef61e04a48b8614389526a2777 (first table Listing [id=00004001000030008000000000004193])
yb-master.$hostname.root.log.WARNING.20260802-190013.22:245:W0802 19:11:03.323125 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “dedd0eea6ebadba4f0436d3ea0700f04” } } column_values { column_id: 2 expr { value { string_value: “25.184349” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775577709014135” } values { string_value: “7272764778033328128” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009656056 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:246:W0802 19:11:03.323379 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “2a53167c33be9e8ac64f3242617a8e1e” } } column_values { column_id: 2 expr { value { string_value: “27.185102” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775647184616936” } values { string_value: “7273039532722868224” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009607064 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:247:W0802 19:11:03.323412 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “e97ce36439392994134db0f215893728” } } column_values { column_id: 2 expr { value { string_value: “25.184124” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775570390536966” } values { string_value: “7272727287394136064” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009765592 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:248:W0802 19:11:03.323441 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “99636c99d8b64a8b704cdd397df64540” } } column_values { column_id: 2 expr { value { string_value: “11.160252” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1773944212052215” } values { string_value: “7266075491105927168” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041009860216 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:249:W0802 19:11:03.323463 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “226eeab6b7a4f78862470b7ca3d6a341” } } column_values { column_id: 2 expr { value { string_value: “170.561753” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1780579026504134” } values { string_value: “7293242532020936704” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18040943736152 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:250:W0802 19:11:03.323485 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “f3a3d7cc12f7e5bed247cf8ee7fc9355” } } column_values { column_id: 2 expr { value { string_value: “11.160251” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1773944169870952” } values { string_value: “7266073791835602944” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18040943975480 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:251:W0802 19:11:03.323508 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “bba6b28d54b5989507490d7aaa372dcb” } } column_values { column_id: 2 expr { value { string_value: “514.903027” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1785697210302096” } values { string_value: “7314099619907739648” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041010020696 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:252:W0802 19:11:03.323527 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “71099581f57fe696c74bd870704af3d9” } } column_values { column_id: 2 expr { value { string_value: “132.541889” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1780254284662542” } values { string_value: “7289999457915379712” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18040854943544 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:253:W0802 19:11:03.323547 1602 cdc_state_table.cc:595] operator(): Flush of operation QL_WRITE type: QL_STMT_INSERT client: YQL_CLIENT_CQL request_id: 0 schema_version: 0 hash_code: 28963 hashed_column_values { value { string_value: “1a092f17af2c4c98bef2e8aa35578273” } } range_column_values { value { string_value: “9ebfe02ddab8549a3e4f9da5ee3a61de” } } column_values { column_id: 2 expr { value { string_value: “39.259609” } } } column_values { column_id: 3 expr { value { map_value { keys { string_value: “active_time” } keys { string_value: “cdc_sdk_safe_time” } values { string_value: “1775811561341865” } values { string_value: “7273724157304283136” } } } } } if_expr { condition { op: QL_OP_NOT_EXISTS } } query_id: 18041010103096 is_compatible_with_previous_version: false failed: IO error (yb/tserver/tablet_service.cc:2632): Write to tablet b04e91838f5a42d4be7dfe25e7d213fa rejected. Node 1dfa57107f7d433e93b7f68f3e39a86e has insufficient disk space
yb-master.$hostname.root.log.WARNING.20260802-190013.22:254:W0802 19:11:03.323592 1989 catalog_manager.cc:3212] Failed to split tablet with GetSplitKey result for tablet: 1a6e00b5ceb143d38a6b44b044e76238: IO error (yb/client/batcher.cc:111): Encountered an error while updating the CDC producer metadata. Table id: 00004001000030008000000000004193, Split Tablets: { source: 1a6e00b5ceb143d38a6b44b044e76238 children: {1a092f17af2c4c98bef2e8aa35578273, d51000ef61e04a48b8614389526a2777} }: Errors occurred while reaching out to the tablet servers
it failed due to insufficient disk space, but I solved it now , how to re trigger it ?