Hello,
I have a setup with multiple servers that each will run a YugabyteDB node to form a multi-site cluster. Each node will run in a docker container, and will be managed by docker compose.
One node will be master and the other nodes will join that master. Let’s assume two nodes for simplicity:
- Node 1: This container will run on a host with IP: 192.168.120.243 and will be the master. The container will get the IP: 172.18.0.2.
- Node 2: This container will run on a host with IP: 192.168.120.244 and intends to join the master. This would create a 2 node cluster. The container will also get the IP: 172.18.0.2, but will, as mentioned, run on a different host.
Here is my docker compose file for node 1 to run on the host with IP 192.168.120.243:
name: my_project
services:
db:
image: yugabytedb/yugabyte:2025.1.0.1-b3
container_name: my_container_name_1
hostname: my_hostname_1
restart: always
command: [ "bin/yugabyted",
"start",
"--background=false",
"--advertise_address=my_container_name_1",
"--cloud_location=my_cloud.my_region_1.my_zone_1" ]
ports:
- 7000:7000
- 7100:7100
- 9000:9000
- 9100:9100
- 15433:15433
- 5433:5433
- 9042:9042
and here it is for node 2 to run on the host with IP 192.168.120.244:
name: my_project
services:
db:
image: yugabytedb/yugabyte:2025.1.0.1-b3
container_name: my_container_name_2
hostname: my_hostname_2
restart: always
command: [ "bin/yugabyted",
"start",
"--background=false",
"--advertise_address=my_container_name_2",
"--join=192.168.120.243",
"--cloud_location=my_cloud.my_region_2.my_zone_2" ]
ports:
- 7000:7000
- 7100:7100
- 9000:9000
- 9100:9100
- 15433:15433
- 5433:5433
- 9042:9042
The networks created for each docker project get the name “my_project_default” (same for both).
If I run “docker compose up -d” on the host with IP: 192.168.120.243, the node comes up fine and becomes master of a new cluster.
If I now go to the host with IP: 192.168.120.244 and run “docker compose up -d”, the container fails to start. In the docker logs one finds a message similar to:
ERROR: Master node present at my_container_name_1:7000 is not reachable.
The above setup works fine when the docker containers are started on the same host, but as soon as the containers run on different hosts, there seems to be a communication problem.
I have deactivated the firewalls on both servers, so there is nothing blocking the communication.
It does seem that the “join” communication reaches node 1 as the “my_container_name_1“ name is correctly identified. However, it fails to bind to the master.
My suspicion is that node 2 tries to find “my_container_name_1:7000“ on IP: 192.168.120.244 instead of IP: 192.168.120.243, but that will fail as IP: 192.168.120.244 has no such container.
Some question:
- How can I resolve this issue?
- Do I need to do some network configuration in docker compose to “expand“ the docker network?
- Can I configure YubabyteDB somehow to make it direct the incoming join request to the correct container?
I have searched almost all available documentation online for days, but both the official documentation and most other use cases only consider docker containers running on one-and-the-same host.
I am only interested in using the free version of YugabyteDB. The two servers in my example above (192.168.120.243 and 192.168.120.244) are run “on-prem”. We are not using and 3rd party cloud providers.
Thank you!