Illegal state: Node peer not initialized

General
1

I am trying to deploy a multi-region YugabyteDB cluster on Tencent Cloud. I followed this instruction: Deploy | YugabyteDB Docs

It went well until I tried to access the yb-master Admin UI. It shows

Illegal state (yb/master/catalog_manager.cc:9411): Unable to list masters during web request handling: Node 43b24a008446472e9f7c28ccb5d294d5 peer not initialized.

and on the log it shows:

WeChatWorkScreenshot_feeb41e5-971a-467a-8019-3ca2bbec6ac6
WeChatWorkScreenshot_feeb41e5-971a-467a-8019-3ca2bbec6ac61920×889 246 KB

I checked the pods and the services and they are all up and running normally:

WeChatWorkScreenshot_b0b2209e-a7e7-4d16-9c9c-e0cfe8462611
WeChatWorkScreenshot_b0b2209e-a7e7-4d16-9c9c-e0cfe84626113236×1462 542 KB

This is one of the overrides files:

isMultiAz: True

AZ: "'330001'"

masterAddresses: "yb-master-0.yb-masters.yb-demo-150001.svc.cluster.local:7100,yb-master-0.yb-masters.yb-demo-900001.svc.cluster.local:7100,yb-master-0.yb-masters.yb-demo-330001.svc.cluster.local:7100"

storage:
  master:
    storageClass: "yb-storageclass"
  tserver:
    storageClass: "yb-storageclass"

replicas:
  master: 1
  tserver: 1
  totalMasters: 3

gflags:
  master:
    placement_cloud: "tke"
    placement_region: "ap-nanjing"
    placement_zone: "330001"
    leader_failure_max_missed_heartbeat_periods: 10
  tserver:
    placement_cloud: "tke"
    placement_region: "ap-nanjing"
    placement_zone: "330001"
    leader_failure_max_missed_heartbeat_periods: 10

I spent days to troubleshooting, but no luck. Can someone tell me which part I did wrong?

2

Can someone help me with this?

3

Hello @minze

Hope everything is well with you.

Can you please share the detailed steps as well as also share how many nodes for each region did you use to build your cluster.

Thanks

4

Hi @James007,

I follow this instruction and did the following steps:

  1. created 3 clusters and storageclass in 3 different regions.
  2. used the script provided in the doc and set up the global dns. The output is the same as posted in the doc.
  3. modified the overrides files as I posted and run helm to install the Yugabytedb.
  4. check the clusters status, and everything looks good except the UI shows illegal state.

I have 3 nodes created for each region, and each node has 4 cores and 16GB ram. Totally 9 nodes created for 3 regions.