Illegal state: Node peer not initialized

I am trying to deploy a multi-region YugabyteDB cluster on Tencent Cloud. I followed this instruction: Deploy a multi-region cluster on Google Kubernetes Engine (GKE) using Helm Chart | YugabyteDB Docs

It went well until I tried to access the yb-master Admin UI. It shows

Illegal state (yb/master/catalog_manager.cc:9411): Unable to list masters during web request handling: Node 43b24a008446472e9f7c28ccb5d294d5 peer not initialized.

and on the log it shows:

I checked the pods and the services and they are all up and running normally:

This is one of the overrides files:

isMultiAz: True

AZ: "'330001'"

masterAddresses: "yb-master-0.yb-masters.yb-demo-150001.svc.cluster.local:7100,yb-master-0.yb-masters.yb-demo-900001.svc.cluster.local:7100,yb-master-0.yb-masters.yb-demo-330001.svc.cluster.local:7100"

storage:
  master:
    storageClass: "yb-storageclass"
  tserver:
    storageClass: "yb-storageclass"

replicas:
  master: 1
  tserver: 1
  totalMasters: 3

gflags:
  master:
    placement_cloud: "tke"
    placement_region: "ap-nanjing"
    placement_zone: "330001"
    leader_failure_max_missed_heartbeat_periods: 10
  tserver:
    placement_cloud: "tke"
    placement_region: "ap-nanjing"
    placement_zone: "330001"
    leader_failure_max_missed_heartbeat_periods: 10

I spent days to troubleshooting, but no luck. Can someone tell me which part I did wrong?

Can someone help me with this?

Hello @minze

Hope everything is well with you.

Can you please share the detailed steps as well as also share how many nodes for each region did you use to build your cluster.

Thanks

Hi @James007,

I follow this instruction and did the following steps:

  1. created 3 clusters and storageclass in 3 different regions.
  2. used the script provided in the doc and set up the global dns. The output is the same as posted in the doc.
  3. modified the overrides files as I posted and run helm to install the Yugabytedb.
  4. check the clusters status, and everything looks good except the UI shows illegal state.

I have 3 nodes created for each region, and each node has 4 cores and 16GB ram. Totally 9 nodes created for 3 regions.