Unable to determine master addresses

General
5

Edit: solved. read the very bottom.

I’m facing a similar situation. I had a Docker-Compose file with just Yugabyte in it, and that worked fine, but then I added a pgAdmin service and that made the Yugabyte container start with a different IP and ysqlsh started freezing when attempting to connect, then timing out with Timed out waiting kResponseSent, state: kProcessingRequest.

I checked the config and saw that it had both the old (now pgAdmin’s) and new IPs in “current_masters”, but no matter what I do I can’t get it to work again… I emptied that config field and added --master_flags "master_addresses=127.0.0.1:7100" to the yugabyted start command and I’m still seeing the following errors in /root/var/logs/yugabyted.log:

OUT >>

<< ERR >>
Illegal state (yb/client/client-internal.cc:2787): Unable to establish connection to leader master at []. Please verify the addresses and check if server is up, or if you're missing --certs_dir_name.

: Could not locate the leader master: Unable to determine master addresses

<<
[yugabyted start] 2025-02-04 18:56:51,721 INFO:  | 2.3s | thread-uml: current masters ['']
[yugabyted start] 2025-02-04 18:56:51,721 INFO:  | 2.3s | thread-uml: Unable to query for all masters list, keeping masters list: ['']
[yugabyted start] 2025-02-04 18:57:51,755 INFO:  | 62.4s | thread-uml: current masters ['']
[yugabyted start] 2025-02-04 18:57:51,756 INFO:  | 62.4s | thread-uml: Unable to query for all masters list, keeping masters list: ['']
[yugabyted start] 2025-02-04 18:58:51,816 INFO:  | 122.4s | thread-uml: current masters ['']
[yugabyted start] 2025-02-04 18:58:51,817 INFO:  | 122.4s | thread-uml: Unable to query for all masters list, keeping masters list: ['']

And now the config has "current_masters": ",172.19.0.2:7100" (the container’s IP) which seems like it should work, yet it doesn’t.

At this point I could totally nuke it all and start from scratch since I have no important data in the database, but what if I did? I need to know how to solve this problem.

  • Edit 1: I temporarily fixed it by removing the pgAdmin service from the compose file, reverting the yugabyted start command to what it was before and starting it up, then re-adding the pgAdmin service. It works for now but I’m sure if the IP changes again the same problem will happen. Should I really not be persisting /root/var/conf/yugabyted.conf in a Docker volume??

  • Edit 2: yep… just as a test I downed both services and up’ped them again and it immediately broke. Here’s the logs now:

[yugabyted start] 2025-02-04 19:28:49,327 INFO:  | 2.2s | run_process: cmd: ['/home/yugabyte/bin/yb-admin', '--master_addresses', '', 'get_universe_config']
[yugabyted start] 2025-02-04 19:28:49,392 INFO:  | 2.2s | run_process returned 1: 
OUT >>

<< ERR >>
Illegal state (yb/client/client-internal.cc:2787): Unable to establish connection to leader master at []. Please verify the addresses and check if server is up, or if you're missing --certs_dir_name.

: Could not locate the leader master: Unable to determine master addresses

<<
[yugabyted start] 2025-02-04 19:28:49,479 INFO:  | 2.3s | thread-uml: current masters ['']
[yugabyted start] 2025-02-04 19:28:49,480 INFO:  | 2.3s | thread-uml: Unable to query for all masters list, keeping masters list: ['']
[yugabyted start] 2025-02-04 19:29:49,533 INFO:  | 62.4s | thread-uml: current masters ['']
[yugabyted start] 2025-02-04 19:29:49,534 DEBUG:  | 62.4s | Tserver 172.19.0.3 returned the following set of the current masters 172.19.0.2:7100,172.19.0.3:7100.
[yugabyted start] 2025-02-04 19:29:49,534 INFO:  | 62.4s | thread-uml: master list updated, old list: [''] new list: ['172.19.0.2:7100', '172.19.0.3:7100']
[yugabyted start] 2025-02-04 19:29:49,534 INFO:  | 62.4s | run_process: cmd: ['/home/yugabyte/bin/yb-ts-cli', '--server_address=172.19.0.3:9100', 'set_flag', 'tserver_master_addrs', '172.19.0.2:7100,172.19.0.3:7100', '--force']

172.19.0.2 is the old IP, now belonging to the pgAdmin service, yet it is still being used as a master address.

  • Edit 3: solved it by setting a static hostname for the container, using it in the config file (current_masters) and enabling DNS (dns_enabled), which is turned off by default for whatever reason.