Yugabyted writes logs frequently

General
1

[Update] I fixed it because it was wrong

I wanted to try out the tablespaces in YugabyteDB, so I created a cluster as follows.

All nodes are on the same network and are accessible.

yugabyted start \
  --base_dir=/home/yugabyte/az1-node1/yugabyte-data \
  --advertise_address=az1-node1.maas \
  --cloud_location=mycompany.regionone.az1

yugabyted start                           \
  --base_dir=/home/yugabyte/az1-node2/yugabyte-data \
  --advertise_address=az1-node2.maas                 \
  --join=az1-node1.maas                         \
  --cloud_location=mycompany.regionone.az1

yugabyted start                           \
  --base_dir=/home/yugabyte/az1-node3/yugabyte-data \
  --advertise_address=az1-node3.maas                 \
  --join=az1-node1.maas                         \
  --cloud_location=mycompany.regionone.az1

yugabyted start                           \
  --base_dir=/home/yugabyte/az3-node1/yugabyte-data \
  --advertise_address=az2-node1.maas             \
  --join=az1-node1.maas                         \
  --cloud_location=mycompany.regiontwo.az2

yugabyted start                           \
  --base_dir=/home/yugabyte/az3-node2/yugabyte-data \
  --advertise_address=az2-node2.maas              \
  --join=az1-node1.maas                         \
  --cloud_location=mycompany.regiontwo.az2

yugabyted start                           \
  --base_dir=/home/yugabyte/az3-node3/yugabyte-data \
  --advertise_address=az2-node3.maas                  \
  --join=az1-node1.maas                         \
  --cloud_location=mycompany.regiontwo.az2

Only nodes with --cloud_location=mycompany.regiontwo.az2 set will frequently log to yb-master.WARNING. What is happening?

W0414 06:00:44.965507 531343 catalog_manager.cc:1634] Failed to get current config: Illegal state (yb/master/catalog_manager.cc:11754): Node fbeb1e1a47b84fe38b411ae8ed81ff78 peer not initialized.
W0414 06:00:44.965581 530339 async_client_initializer.cc:94] Failed to initialize client: Illegal state (yb/client/client-internal.cc:2787): Could not locate the leader master: Unable to determine master addresses
W0414 06:00:44.965684 530325 async_client_initializer.cc:94] Failed to initialize client: Illegal state (yb/client/client-internal.cc:2787): Could not locate the leader master: Unable to determine master addresses

The problem will be reproduced in the next version.

  • yugabyte-2.25.0.0-b489-linux-x86_64 on Ubuntu 22.04.5 LTS, Python 3.10.12
  • yugabyte-2.25.1.0-b381-linux-x86_64 on Ubuntu 22.04.5 LTS, Python 3.10.12
2

Are you still having issues?

3

@ytooyama, Let us know if you are still experiencing the issue. If so, please run yugabyted collect_logs --base_dir= on the nodes where you see these messages. You can upload the tar.gz file to a Gdrive or something similar and send it to us for troubleshooting.

Thanks,
Nikhil

4

Yes, nothing has been solved.
The LTS version has not yet been tested, so we plan to work on it today.

Before that, let’s run yugabyted collect_logs.

5

I have prepared a log here.

az1-node1.maas is the main node.
az3-node1.maas,az3-node2.maas,az3-node3.maas output a lot of errors.

6

I changed the configuration and tried again.

The node is IP-reachable at the specified IP address.
The domain name “c7000-node1.maas” is set so that it can be named by /etc/hosts.
However, despite the fact that there is no problem with the az1 node, the nodes belonging to az2 and az3 continue to issue the following errors every second.

I0417 09:53:40.225172  2460 client-internal.cc:2820] New master addresses: []
W0417 09:53:40.225217  2203 async_client_initializer.cc:94] Failed to initialize client: Illegal state (yb/client/client-internal.cc:2823): Could not locate the leader master: Unable to determine master addresses
W0417 09:53:40.225260  2209 async_client_initializer.cc:94] Failed to initialize client: Illegal state (yb/client/client-internal.cc:2823): Could not locate the leader master: Unable to determine master addresses
W0417 09:53:41.225677  2461 catalog_manager.cc:1655] Failed to get current config: Illegal state (yb/master/catalog_manager.cc:11898): Node 2837d56ad7024f28a916cdc9ce6041a7 peer not initialized.
W0417 09:53:41.225751  2462 catalog_manager.cc:1655] Failed to get current config: Illegal state (yb/master/catalog_manager.cc:11898): Node 2837d56ad7024f28a916cdc9ce6041a7 peer not initialized.

Is this a problem of composition?
Just in case, the log has been uploaded to the same Google Drive.

config

master

yugabyted start \
  --base_dir=/home/yugabyte/c7000-node1.maas/yugabyte-data \
  --advertise_address=172.17.28.44 \
  --cloud_location=vtj.c7000.az1

worker

//az1
yugabyted start                           \
  --base_dir=/home/yugabyte/ml110g9-node1.maas/yugabyte-data \
  --advertise_address=172.17.28.47                 \
  --join=c7000-node1.maas                         \
  --cloud_location=vtj.ml110g9.az1

yugabyted start                           \
  --base_dir=/home/yugabyte/proxmox-node1.localdomain/yugabyte-data \
  --advertise_address=172.16.214.224                \
  --join=c7000-node1.maas                         \
  --cloud_location=vtj.proxmox.az1

//az2
yugabyted start                           \
  --base_dir=/home/yugabyte/c7000-node2.maas/yugabyte-data \
  --advertise_address=172.17.28.42             \
  --join=c7000-node1.maas                         \
  --cloud_location=vtj.c7000.az2

yugabyted start                           \
  --base_dir=/home/yugabyte/ml110g9-node2.maas/yugabyte-data \
  --advertise_address=172.17.28.48              \
  --join=c7000-node1.maas                         \
  --cloud_location=vtj.ml110g9.az2

yugabyted start                           \
  --base_dir=/home/yugabyte/proxmox-node2.localdomain/yugabyte-data \
  --advertise_address=172.16.214.225                 \
  --join=c7000-node1.maas                         \
  --cloud_location=vtj.proxmox.az2

//az3
yugabyted start                           \
  --base_dir=/home/yugabyte/c7000-node3.maas/yugabyte-data \
  --advertise_address=172.17.28.45             \
  --join=c7000-node1.maas                         \
  --cloud_location=vtj.c7000.az3

yugabyted start                           \
  --base_dir=/home/yugabyte/ml110g9-node3.maas/yugabyte-data \
  --advertise_address=172.17.28.46              \
  --join=c7000-node1.maas                         \
  --cloud_location=vtj.ml110g9.az3

yugabyted start                           \
  --base_dir=/home/yugabyte/proxmox-node3.localdomain/yugabyte-data \
  --advertise_address=172.16.214.226                  \
  --join=c7000-node1.maas                       \
  --cloud_location=vtj.proxmox.az3
7

Hi @ytooyama

I reviewed the logs of the nodes you sent in the Gdrive.

Issue Analysis

Consider the node az3-node3.maas. In this node, the master process is only a shell master, and it doesn’t actively participate in the master quorum. All the logs below are harmless and should be ignored.

0414 05:54:22.425798 529226 client-internal.cc:2784] New master addresses: []
W0414 05:54:22.425933 529223 async_client_initializer.cc:94] Failed to initialize client: Illegal state (yb/client/client-internal.cc:2787): Could not locate the leader master: Unable to determine master addresses
I0414 05:54:22.428207 529227 total_mem_watcher.cc:78] Root memtracker limit: 1247167488 (1189 MiB); this server will stop if memory usage exceeds 200% of that: 2494334976 bytes (2378 MiB).
W0414 05:54:23.396571 529324 catalog_manager.cc:1634] Failed to get current config: Illegal state (yb/master/catalog_manager.cc:11754): Node b61128a42ba2462bb4bfcd59fb290ad6 peer not initialized.

The cluster you created has started fine. You should be able to run DDLs and DMLs on it. Let us know if you have any issues running your workload.

Further Observations - 1

You have created the nodes in the following --cloud_location -

  1. mycompany.regionone.az1 - az1-node1.maas, az1-node2.maas, az1-node3.maas
  2. mycompany.regiontwo.az2 - az2-node1.maas, az2-node2.maas, az2-node3.maas

In your deployment, you have two regions, since you have started all the nodes in the mycompany.regionone.az1 first so all the masters have started in this zone. yugabyted automatically starts the masters in the first three nodes you bring up and eventually the masters gets reassigned based on the deployment topology depending of whether its a multi-zone or multi-region deployment

If you want to distribute masters to all the available regions, you will need to execute the command -

./bin/yugabyted configure data_placement --fault_tolerance=region --base_dir=--base_dir=/home/yugabyte/az1-node1/yugabyte-data

Further Observations - 2

Question for you. Are you trying to start a multi-region cluster?

If yes, we recommend having an odd number of regions so that the cluster can survive a region outage. To create a multi-region cluster, you must create an additional region.

Also, for multi-region and multi-zone deployment, you will need to set the flag --fault_tolerance=region See the docs here

yugabyted start \
  --base_dir=/home/yugabyte/az1-node1/yugabyte-data \
  --advertise_address=az1-node1.maas \
  --cloud_location=mycompany.regionone.az1 \
  --fault_tolerance=region
1 Like
8

Thank you very much, @nmalladi!
I will check what you have taught me and try again.
I will check about the fault_tolerance option.

9

Actually, I was interested in YugabyteDB’s Geo-placement, so I decided to try it.
Thank you for your kindness.