Tcmalloc issues

marknefedov · August 24, 2026, 9:25am

Hi, would you mind take a look? Having issues with 2024 series on new Ubuntu releases.

[DocDB] tserver/master heap corruption and crashes on Linux kernel >= 6.19: vendored tcmalloc incompatible with new rseq behavior

opened 09:56PM - 23 Aug 26 UTC

### Description yb-tserver and yb-master crash within 15-60 seconds under concu…rrent YSQL load on Linux kernels 6.19 and newer. The crashes look like random heap corruption: every run dies in a different place (RPC scheduler CHECK failure, protobuf serialization, boost multi_index rebalance), with different signals (SIGSEGV, SIGBUS, SIGTRAP, SIGABRT), and fault addresses that contain ASCII string data where pointers should be. The cause is the vendored tcmalloc. Kernels since 6.19 changed restartable sequences behavior (commit `39a167560a61`, "rseq: Optimize event setting"), which broke tcmalloc's per-CPU caches. This is google/tcmalloc#292, fixed upstream on 2026-05-13 by `95cd995b3e` ("Revert optimization to reference the __rseq_abi address relative to the tcmalloc sampler address"). The yugabyte/tcmalloc fork used by yugabyte-db-thirdparty is pinned at `3b79ccc-yb-2` (February 2025) and does not contain that fix. MongoDB hit the same bug (SERVER-121912) and resolved it by taking the tcmalloc fix. Tested and crashing: 2024.2.8.0-b85 and 2024.2.10.0-b62, official arm64 images. Since the thirdparty pin is shared, other release lines built after February 2025 should be affected as well. **Environment** - Kernels that crash: Ubuntu 26.04, 7.0.0-22-generic and 7.0.0-30-generic (aarch64) - Kernel that works: Ubuntu 24.04, 6.8.0-137-generic, same VM configuration, same image, same load - Hardware: Apple Silicon host, aarch64 Linux VM (Virtualization.framework), 6 vCPUs. Not tested on x86_64. **Reproduction** Single-node master + tserver from the official image, then generic YSQL load. No Kubernetes needed. ```bash docker network create ybtest docker run -d --name yb-master --net ybtest --entrypoint /home/yugabyte/bin/yb-master \ yugabytedb/yugabyte:2024.2.10.0-b62 \ --master_addresses=yb-master:7100 --rpc_bind_addresses=0.0.0.0:7100 \ --fs_data_dirs=/data --replication_factor=1 --enable_ysql=true --logtostderr docker run -d --name yb-tserver --net ybtest --entrypoint /home/yugabyte/bin/yb-tserver \ yugabytedb/yugabyte:2024.2.10.0-b62 \ --tserver_master_addrs=yb-master:7100 --rpc_bind_addresses=0.0.0.0 \ --server_broadcast_addresses=yb-tserver --pgsql_proxy_bind_address=0.0.0.0:5433 \ --fs_data_dirs=/data --use_memory_defaults_optimized_for_ysql=true --logtostderr ``` Load (crashes the tserver or master in under a minute): ```bash # in a third container on the same network, or via exec: ysqlsh -h yb-tserver -p 5433 -U yugabyte -c "create database loadtest" ysql_bench -h yb-tserver -p 5433 -U yugabyte -i -s 5 loadtest ysql_bench -h yb-tserver -p 5433 -U yugabyte -c 8 -j 4 -T 600 -n loadtest & # plus 4 parallel loops of: create table t (k bigint primary key, v text); # create index on t (v); insert 300 rows; select count(*); drop table t; # plus 4 parallel loops of short-lived connections running: select count(*) from pg_class; ``` **Sample crash signatures** (each from a separate run, same setup) ``` F0820 16:48:12 scheduler.cc:99] Check failed: pair.second @ yb::rpc::Scheduler::Impl::DoSchedule() [lambda] *** SIGSEGV (@0x5f796c70706156) received by PID 1 <- fault address is ASCII "Vapply_" ``` ``` F0821 07:18:54 serialization.cc:90] Check failed: dst - param_buf.udata() == param_buf.size() (1255 vs. 1212) *** SIGSEGV (@0x4974656c62615c) <- fault address is ASCII, fragment of "TabletI..." @ yb::tserver::PgPerformOptionsPB::~PgPerformOptionsPB() @ yb::client::internal::Batcher::FlushFinished() ``` ``` *** SIGSEGV (@0x8) received by PID 1 @ boost::multi_index::detail::ordered_index_node_impl<>::rebalance() ``` ``` core dump: SIGSEGV in google::protobuf::internal::WireFormatLite::VerifyUtf8String() <- yb::tablet::RaftGroupMetadata::SaveToDiskUnlocked() <- TSTabletManager::OpenTablet() ``` **Evidence that tcmalloc's rseq usage is the trigger** 1. `strace -e trace=rseq yb-tserver` shows tcmalloc registering rseq itself with the legacy 32-byte length: `rseq(addr, 0x20, 0, sig) = 0`. The container glibc (2.31) has no rseq support, so tcmalloc always self-registers and per-CPU caches stay active (`TCMalloc per cpu caches active: 1` in the startup log). 2. Blocking the rseq syscall with a seccomp profile (ENOSYS) makes tcmalloc fall back to thread caches (`per cpu caches active: 0`). All crashes stop, on both broken kernels. 3. The same image and load on kernel 6.8 with per-CPU caches active runs clean (23 minutes, zero fatals, zero restarts). 4. Ubuntu's kernel-side compatibility revert for legacy tcmalloc ("rseq: Revert to historical performance killing behaviour", in 7.0.0-30) does not help. The kernel's own rseq selftests pass on that kernel in the same registration mode tcmalloc uses, yet yb-tserver still corrupts. The remaining defect is in tcmalloc's addressing of the rseq area, which only the tcmalloc-side fix removes. **Suggested fix** Update the yugabyte/tcmalloc fork past upstream `95cd995b3e` (2026-05-13) and refresh the thirdparty pin. Backporting to 2024.2 would help LTS users on current distributions: Ubuntu 26.04 ships kernel 7.0 and any distribution tracking mainline past 6.19 is affected. **Workarounds until then** - Run on a kernel older than 6.19. - Block the rseq syscall for YB containers with a seccomp profile: ```json { "defaultAction": "SCMP_ACT_ALLOW", "syscalls": [ { "names": ["rseq"], "action": "SCMP_ACT_ERRNO", "errnoRet": 38 } ] } ``` Verify with `TCMalloc per cpu caches active: 0` in the startup log. Costs some allocator performance. `GLIBC_TUNABLES=glibc.pthread.rseq=0` (MongoDB's workaround) does not work here, because the image's glibc never registers rseq in the first place; tcmalloc does it directly. ### Issue Type kind/bug ### Warning: Please confirm that this issue does not contain any sensitive information - [x] I confirm this issue does not contain any sensitive information.

dorian_yugabyte · August 24, 2026, 9:55am

Hi @marknefedov

I’ve reported this issue internally.

Topic		Replies	Views
One node crushed suddenly General	24	8749	October 10, 2024
Yb-tserver unexpectedly stopped with code 1 and no msgs in logs General	7	3344	March 6, 2024
Tserver in dead status after bad narrow cast General	7	2960	December 3, 2024
A tserver node went down while processing a write heavy workload General	7	4447	March 15, 2019
Faster recovery after K8 outage General	11	9396	June 2, 2023

Tcmalloc issues

Related topics