Hi, would you mind take a look? Having issues with 2024 series on new Ubuntu releases.
opened 09:56PM - 23 Aug 26 UTC
### Description
yb-tserver and yb-master crash within 15-60 seconds under concu… rrent YSQL load on Linux kernels 6.19 and newer. The crashes look like random heap corruption: every run dies in a different place (RPC scheduler CHECK failure, protobuf serialization, boost multi_index rebalance), with different signals (SIGSEGV, SIGBUS, SIGTRAP, SIGABRT), and fault addresses that contain ASCII string data where pointers should be.
The cause is the vendored tcmalloc. Kernels since 6.19 changed restartable sequences behavior (commit `39a167560a61`, "rseq: Optimize event setting"), which broke tcmalloc's per-CPU caches. This is google/tcmalloc#292, fixed upstream on 2026-05-13 by `95cd995b3e` ("Revert optimization to reference the __rseq_abi address relative to the tcmalloc sampler address"). The yugabyte/tcmalloc fork used by yugabyte-db-thirdparty is pinned at `3b79ccc-yb-2` (February 2025) and does not contain that fix. MongoDB hit the same bug (SERVER-121912) and resolved it by taking the tcmalloc fix.
Tested and crashing: 2024.2.8.0-b85 and 2024.2.10.0-b62, official arm64 images. Since the thirdparty pin is shared, other release lines built after February 2025 should be affected as well.
**Environment**
- Kernels that crash: Ubuntu 26.04, 7.0.0-22-generic and 7.0.0-30-generic (aarch64)
- Kernel that works: Ubuntu 24.04, 6.8.0-137-generic, same VM configuration, same image, same load
- Hardware: Apple Silicon host, aarch64 Linux VM (Virtualization.framework), 6 vCPUs. Not tested on x86_64.
**Reproduction**
Single-node master + tserver from the official image, then generic YSQL load. No Kubernetes needed.
```bash
docker network create ybtest
docker run -d --name yb-master --net ybtest --entrypoint /home/yugabyte/bin/yb-master \
yugabytedb/yugabyte:2024.2.10.0-b62 \
--master_addresses=yb-master:7100 --rpc_bind_addresses=0.0.0.0:7100 \
--fs_data_dirs=/data --replication_factor=1 --enable_ysql=true --logtostderr
docker run -d --name yb-tserver --net ybtest --entrypoint /home/yugabyte/bin/yb-tserver \
yugabytedb/yugabyte:2024.2.10.0-b62 \
--tserver_master_addrs=yb-master:7100 --rpc_bind_addresses=0.0.0.0 \
--server_broadcast_addresses=yb-tserver --pgsql_proxy_bind_address=0.0.0.0:5433 \
--fs_data_dirs=/data --use_memory_defaults_optimized_for_ysql=true --logtostderr
```
Load (crashes the tserver or master in under a minute):
```bash
# in a third container on the same network, or via exec:
ysqlsh -h yb-tserver -p 5433 -U yugabyte -c "create database loadtest"
ysql_bench -h yb-tserver -p 5433 -U yugabyte -i -s 5 loadtest
ysql_bench -h yb-tserver -p 5433 -U yugabyte -c 8 -j 4 -T 600 -n loadtest &
# plus 4 parallel loops of: create table t (k bigint primary key, v text);
# create index on t (v); insert 300 rows; select count(*); drop table t;
# plus 4 parallel loops of short-lived connections running: select count(*) from pg_class;
```
**Sample crash signatures** (each from a separate run, same setup)
```
F0820 16:48:12 scheduler.cc:99] Check failed: pair.second
@ yb::rpc::Scheduler::Impl::DoSchedule() [lambda]
*** SIGSEGV (@0x5f796c70706156) received by PID 1 <- fault address is ASCII "Vapply_"
```
```
F0821 07:18:54 serialization.cc:90] Check failed: dst - param_buf.udata() == param_buf.size() (1255 vs. 1212)
*** SIGSEGV (@0x4974656c62615c) <- fault address is ASCII, fragment of "TabletI..."
@ yb::tserver::PgPerformOptionsPB::~PgPerformOptionsPB()
@ yb::client::internal::Batcher::FlushFinished()
```
```
*** SIGSEGV (@0x8) received by PID 1
@ boost::multi_index::detail::ordered_index_node_impl<>::rebalance()
```
```
core dump: SIGSEGV in google::protobuf::internal::WireFormatLite::VerifyUtf8String()
<- yb::tablet::RaftGroupMetadata::SaveToDiskUnlocked() <- TSTabletManager::OpenTablet()
```
**Evidence that tcmalloc's rseq usage is the trigger**
1. `strace -e trace=rseq yb-tserver` shows tcmalloc registering rseq itself with the legacy 32-byte length: `rseq(addr, 0x20, 0, sig) = 0`. The container glibc (2.31) has no rseq support, so tcmalloc always self-registers and per-CPU caches stay active (`TCMalloc per cpu caches active: 1` in the startup log).
2. Blocking the rseq syscall with a seccomp profile (ENOSYS) makes tcmalloc fall back to thread caches (`per cpu caches active: 0`). All crashes stop, on both broken kernels.
3. The same image and load on kernel 6.8 with per-CPU caches active runs clean (23 minutes, zero fatals, zero restarts).
4. Ubuntu's kernel-side compatibility revert for legacy tcmalloc ("rseq: Revert to historical performance killing behaviour", in 7.0.0-30) does not help. The kernel's own rseq selftests pass on that kernel in the same registration mode tcmalloc uses, yet yb-tserver still corrupts. The remaining defect is in tcmalloc's addressing of the rseq area, which only the tcmalloc-side fix removes.
**Suggested fix**
Update the yugabyte/tcmalloc fork past upstream `95cd995b3e` (2026-05-13) and refresh the thirdparty pin. Backporting to 2024.2 would help LTS users on current distributions: Ubuntu 26.04 ships kernel 7.0 and any distribution tracking mainline past 6.19 is affected.
**Workarounds until then**
- Run on a kernel older than 6.19.
- Block the rseq syscall for YB containers with a seccomp profile:
```json
{ "defaultAction": "SCMP_ACT_ALLOW",
"syscalls": [ { "names": ["rseq"], "action": "SCMP_ACT_ERRNO", "errnoRet": 38 } ] }
```
Verify with `TCMalloc per cpu caches active: 0` in the startup log. Costs some allocator performance.
`GLIBC_TUNABLES=glibc.pthread.rseq=0` (MongoDB's workaround) does not work here, because the image's glibc never registers rseq in the first place; tcmalloc does it directly.
### Issue Type
kind/bug
### Warning: Please confirm that this issue does not contain any sensitive information
- [x] I confirm this issue does not contain any sensitive information.
Hi @marknefedov
I’ve reported this issue internally.
Hi @marknefedov - Thank you for the detailed report. The crashes come from tcmalloc’s per-CPU cache, depending on rseq behavior that kernel 6.19 changed. Upstream treated the change as a kernel regression and reverted it, so the fix is on the kernel side. Two commits are relevant:
rseq: Revert to historical performance killing behaviour (Thomas Gleixner) — in mainline 7.0.10+, Ubuntu kernel 7.0.0-28+arm64/entry: Fix arm64-specific rseq brokenness (Mark Rutland) — in mainline 7.0.14+, Ubuntu kernel 7.0.0-31+
Since you’re on arm64, you need both fixes. The kernels you tested (7.0.0-22 and 7.0.0-30) predate the arm64 fix, which is why you still saw crashes on -30. The first Ubuntu kernel with both fixes is 7.0.0-31, which is currently in resolute-proposed and will likely reach the regular updates pocket soon (based on past experience). Until then, your options on arm64 are the seccomp workaround (blocking rseq, at an allocator-performance cost), installing the -31 kernel from proposed if you’re comfortable with that, or running on a supported OS.
On support status: Ubuntu 26.04 isn’t currently a supported OS for the 2024.x/2025.x or any current release. We don’t plan to add support for the affected kernel versions (6.19 through the pre-fix 7.0.x range). We are building a plan to support Ubuntu 26.04 in a future release. Potential backporting support is TBD.