Underreplicated Tablets

General
#1

Description:
Upgrading kubernets from 1.17 to 1.18 to 1.19. leads to underreplicated tablets in yugabytedb. I would expect that yugabytedb ensures that there is the expected number of replicas.

Any ideas how to fix the underreplicated tablets?

A short snippet from tablet-replication:

Underreplicated Tablets
Table Name	Table UUID	Tablet ID	Tablet Replication Count
cassandrakeyvalue	614f8afa1bd744b69079a6e0f1c64f55	fddb6ca3592a4bc6bb0013cd29714e44	2
cassandrakeyvalue	614f8afa1bd744b69079a6e0f1c64f55	a92c1a8a1e4746acb5c421be4a891bbf	2
cassandrakeyvalue	614f8afa1bd744b69079a6e0f1c64f55	b55b6ebb93f74617b121b665b81d5492	2
cassandrakeyvalue	614f8afa1bd744b69079a6e0f1c64f55	91ea6f835e4248438530b8972bfbd18c	2
...

Cluster Setup

  • Number Master Nodes: 3
  • Replication Factor 3
  • Num Nodes (TServers) 11
  • Num User Tables 1
  • Is Load Balanced? No
  • YugabyteDB Version 2.5.3.1
  • Build Type RELEASE
  • Installion with Helm Single DC Cluster on Kubernetes 1.19 and Istio 1.9
#2

@beatrausch are there any yb-master logs happening during this time ? That might show why it’s not re-replicating ?

The same with yb-tservers logs ?

#3

Hi, I try to get some logs, but I cannot promise, as it happens on dev stage.
Is there a way to instruct yugabyte to cteate the missing replicas e.g. via yb-admin tool?

#4

Unfortunately, there are no logs from the exact point in time. But from current leader I get the following logs:

I0325 13:59:21.342280    32 catalog_manager.cc:5529] d3d1bdabb110481e928163ca9608eb56 finished first full report: 45 tablets.
W0325 13:59:21.794284   128 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
W0325 13:59:22.391330   130 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
W0325 13:59:22.906467   130 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
W0325 13:59:23.351598   130 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
W0325 13:59:23.796777   130 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
…
W0325 14:00:18.731916   154 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
W0325 14:00:18.732136    49 consensus_peers.cc:471] T 00000000000000000000000000000000 P 2104441140464eba97f09e8e442509d0 -> Peer 58aa0a091a1242988bf9a13da9ab6077 ([host: "yb-master-1.yb-masters.yugabytedb.svc.cluster.local" port: 7100], [host: "yb-master-1.yb-masters.yugabytedb.svc.cluster.local" port: 7100]): Couldn't send request.  Status: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative). Retrying in the next heartbeat period. Already tried 118 times. State: 2
W0325 14:00:19.314105   154 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
W0325 14:00:19.831149   154 proxy.cc:269] Resolve yb-master-1.yb-masters.yugabytedb.svc.cluster.local failed: Network error (yb/util/net/dns_resolver.cc:64): Resolve failed yb-master-1.yb-masters.yugabytedb.svc.cluster.local: Host not found (authoritative)
I0325 14:00:19.836176    41 raft_consensus.cc:2707] T 00000000000000000000000000000000 P 2104441140464eba97f09e8e442509d0 [term 11 LEADER]:  Leader pre-election vote request: Denying vote to candidate 58aa0a091a1242988bf9a13da9ab6077 for term 12 because replica is either leader or believes a valid leader to be alive. Time left: 9222576832.789s
I0325 14:00:20.337493    41 consensus_queue.cc:1165] T 00000000000000000000000000000000 P 2104441140464eba97f09e8e442509d0 [LEADER]: Connected to new peer: { peer: 58aa0a091a1242988bf9a13da9ab6077 is_new: 0 last_received: 10.1736 next_index: 1737 last_known_committed_idx: 1736 is_last_exchange_successful: 0 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 0 last_applied: 10.1736 }
I0325 14:00:20.338557    41 log_cache.cc:323] T 00000000000000000000000000000000 P 2104441140464eba97f09e8e442509d0: Successfully read 1 ops from disk.
I0325 14:00:20.349576   154 replica_state.cc:1356] T 00000000000000000000000000000000 P 2104441140464eba97f09e8e442509d0 [term 11 LEADER]: Revoked old leader 58aa0a091a1242988bf9a13da9ab6077 ht lease: { physical: 1616680759229453 }
I0325 14:00:25.064741    21 reactor.cc:450] Master_R000: DEBUG: Closing idle connection: Connection (0x00000000040bca30) server 127.0.0.1:33612 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:00:29.164598    23 reactor.cc:450] Master_R002: DEBUG: Closing idle connection: Connection (0x00000000040bcc70) server 127.0.0.1:33898 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:00:29.164695    22 reactor.cc:450] Master_R001: DEBUG: Closing idle connection: Connection (0x00000000040bcb50) server 127.0.0.1:33896 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:00:47.365151    22 reactor.cc:450] Master_R001: DEBUG: Closing idle connection: Connection (0x00000000040bceb0) server 127.0.0.1:34650 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:00:47.365190    24 reactor.cc:450] Master_R003: DEBUG: Closing idle connection: Connection (0x00000000040bcd90) server 127.0.0.1:34648 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:00:58.965286    23 reactor.cc:450] Master_R002: DEBUG: Closing idle connection: Connection (0x0000000003710a30) server 127.0.0.1:35076 => 127.0.0.1:7100 - it has been idle for 65.0007s
I0325 14:00:59.065079    22 reactor.cc:450] Master_R001: DEBUG: Closing idle connection: Connection (0x00000000040bcfd0) server 127.0.0.1:35074 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:01:03.664491    21 reactor.cc:450] Master_R000: DEBUG: Closing idle connection: Connection (0x0000000003672eb0) server 127.0.0.1:35222 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:01:03.965220    24 reactor.cc:450] Master_R003: DEBUG: Closing idle connection: Connection (0x00000000036730f0) server 127.0.0.1:35256 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:01:06.865044    22 reactor.cc:450] Master_R001: DEBUG: Closing idle connection: Connection (0x0000000003673e70) server 127.0.0.1:35348 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:01:10.164521    24 reactor.cc:450] Master_R003: DEBUG: Closing idle connection: Connection (0x0000000003673570) server 127.0.0.1:35494 => 127.0.0.1:7100 - it has been idle for 65.0007s
I0325 14:01:13.964634    24 reactor.cc:450] Master_R003: DEBUG: Closing idle connection: Connection (0x0000000003793450) server 127.0.0.1:35620 => 127.0.0.1:7100 - it has been idle for 65.0997s
I0325 14:01:17.564548    23 reactor.cc:450] Master_R002: DEBUG: Closing idle connection: Connection (0x000000000386b690) server 127.0.0.1:35210 => 127.0.0.1:7100 - it has been idle for 65.0998s
W0325 14:01:20.451894    36 cluster_balance.cc:342] Skipping load balancing 614f8afa1bd744b69079a6e0f1c64f55: Leader not ready to serve requests. (../../src/yb/master/cluster_balance_util.h:341): Master leader has not yet received heartbeat from ts d443ab177bf3481cb0786cbfd3b254f6, either master just became leader or a network partition.
#5

Can you check if all the yb-tserver pods are running? kubectl get pods -n namespace should give us that list. Are there any crashing yb-tservers? Also it seems like leader is failing to resolve yb-master-1, can you check if that pod is running?

#6

Hi,

kubectl -n yugabyte get pods returns the

NAME           READY   STATUS    RESTARTS   AGE
yb-master-0    3/3     Running   0          24d
yb-master-1    3/3     Running   0          3d17h
yb-master-2    3/3     Running   1          25d
yb-tserver-0   3/3     Running   0          3d17h
yb-tserver-1   3/3     Running   1          24d
yb-tserver-2   3/3     Running   0          22d
yb-tserver-3   3/3     Running   1          25d
yb-tserver-4   3/3     Running   1          25d
yb-tserver-5   3/3     Running   0          23d
yb-tserver-6   3/3     Running   1          25d
yb-tserver-7   3/3     Running   0          3d17h

Right now it seems that all relevant pods are running. But there was an issue with master-1.