Timeout compacting table

General
1

Hi, I have a 3 server cluster. I imported some data but I noticed it uses more disk space than I expected. However, compacting table manually times out.

admin@yb-0:~$ yb-admin -master_addresses $MASTER_ADDRS compact_table ysql.yugabyte xyz
Error running compact_table: Timed out (yb/client/client-internal.cc:146): Unable to compact table yugabyte.xyz: Timed out waiting for FlushTables

Each server is doing about 800 IOPS and uses ~2 CPU cores at the moment, although the /tasks view doesn’t show any active task.

What is the DB doing? :slight_smile:

2

In the logs, I see lines like this:

W0105 14:39:06.360100 147224 async_rpc_tasks.cc:283] e731da422cc349e3b6f9cdeb7e019763 Flush Tablets RPC (task=0x000000000459d6d0, state=kRunning): TS e731da422cc349e3b6f9cdeb7e019763: Flush Tablets RPC failed for tablet : Timed out (yb/rpc/outbound_call.cc:503): FlushTablets RPC (request call id 2147) to 172.31.32.32:9100 timed out after 30.000s
W0105 14:39:06.360199 147224 async_rpc_tasks.cc:327] e731da422cc349e3b6f9cdeb7e019763 Flush Tablets RPC (task=0x000000000459d6d0, state=kRunning): Aborted (yb/master/async_rpc_tasks.cc:326): Reached maximum number of retries (0) for request e731da422cc349e3b6f9cdeb7e019763 Flush Tablets RPC, task=0x459d6d0 state=kRunning
3

For reference - I imported around 19 GB and 5 GB (uncompressed) into two tables. There are 7 indexes including the primary keys.

The CPU and IOPS load might be caused by the compaction command actually. It disappears after a few minutes and appears again if I try yb-admin compact_table again.

4

Hi @crabhi

It’s doing a major compaction, compacting the whole table into a single sstable.

Yes, because it’s a heavy operation.

Because it forces a major compaction even if it’s not necessary.

The task continues even if it timed out. You can increase the timeout with --timeout_in_seconds=20.

I don’t believe it’s supposed to show in /tasks.

5

In the /tasks list, there are the “flush” tasks for the three tablets. But they show as failed after the 30s timeout. So it means the operation succeeded but the success just wasn’t reported. Thank you for the explanation.