How to configure Docker YugabyteDB cluster on 3 servers?

General
1

While searching for a simple Postgres cluster solution using Docker, I came across YugabyteDB. I did find the documentation to create a single Docker instance, but I didn’t find any documentation or tutorial how to setup a 3 server cluster using the Docker image.

Is there a Docker compose example using simple env vars like those for self-hosting?

HOST    = db-1.internal   # per node
HOST_IP = 100.64.0.1      # per node
HOST_1  = db-1.internal
HOST_2  = db-2.internal
HOST_3  = db-3.internal
2

Hi @bluepuma77

Note that docker compose is for starting multiple containers in a single server. Do you need cluster in a single server or a cluster over multiple servers?

For multi-server setups, you can see docs in here: yugabyted reference | YugabyteDB Docs

The deployment docs are being changed to add examples with yugabyted in [doc] Deploy using yugabyted by ddhodge · Pull Request #27433 · yugabyte/yugabyte-db · GitHub. You can see a preview of that in Deploy YugabyteDB | YugabyteDB Docs

You’ll see that the docker deployment calls the same yugabyted command as the links above.

1 Like
3

Thanks, will look into it. We are thinking about testing YugabyteDB with our current infrastructure provider. Self-hosting for GDPR reasons. We got 3 VMs connected via a mesh VPN to enable private networking. The idea was to simply run the compose file on all 3 nodes.

Docker Swarm is still a thing, but Swarm issues can result in restart of instances, so I rather manage the 3 instances individually by hand. This also enables individual upgrading.

Trying to cross the chasm between a single local instance for development and a big professional build-out. Sitting in between with just 3 simple servers trying to run a tiny production system with HA for initial product MVP.

My personal preference would be to use the same compose file for all instances, so no dedicated --join on secondary nodes, but a list of hosts instead (like etcd --initial-cluster=IP1,IP2,IP3).

4

Trying to piece it together in plain Docker CLI:

export HOST=db-1.internal 
export HOST_IP=$(tailscale ip -4)
export HOST_1=db-1.internal 
export HOST_2=db-2.internal 
export HOST_3=db-3.internal

docker rm -fv yugabyte

docker run \
  --name yugabyte \
  -p ${HOST_IP}:7000:7000 \
  -p ${HOST_IP}:9000:9000 \
  -p ${HOST_IP}:15433:15433 \
  -p ${HOST_IP}:5433:5433 \
  -p ${HOST_IP}:9042:9042 \
  -v ~/yb_data:/home/yugabyte/yb_data \
  -d yugabytedb/yugabyte:2.25.2.0-b359 \
    bin/yugabyted start \
    --advertise_address=${HOST_IP} \
    --base_dir=/home/yugabyte/yb_data \
    --background=false \
    --cloud_location=hetzner.europe.central \
    --fault_tolerance=zone \
    --secure \
    --tserver_flags="enable_ysql_conn_mgr=true"

But the process just ends without an error message:

docker logs -f yugabyte
Starting yugabyted...
✅ System checks
\ Starting the YugabyteDB Processes...
5

This isn’t possible with yugabyted, it needs --join. cc @nmalladi

Can you run yugabyted collect_logs --base_dir=/home/yugabyte/yb_data inside the container and link them here?

6

I’m using the next manifesto.
If it is a simple 3 node, you can create it with the following compose file. It’s enough to evaluate.
I wish there was a manifest example like this in YugabyteDB’s Docs.

services:
  yugabyte1:
    image: software.yugabyte.com/yugabytedb/yugabyte:2.25.2.0-b359
    restart: on-failure
    command: sh -c "bin/yugabyted start --advertise_address=yugabyte1 --background=ture"
    ports:
      - "15433:15433"
      - "5433:5433"
    volumes:
      - yugabyte1-store:/root/var
  yugabyte2:
    image: software.yugabyte.com/yugabytedb/yugabyte:2.25.2.0-b359
    restart: on-failure
    command: sh -c "sleep 10 && bin/yugabyted start --advertise_address=yugabyte2 --background=ture --join=yugabyte1"
    ports:
      - "5434:5433"
    volumes:
      - yugabyte2-store:/root/var

  yugabyte3:
    image: software.yugabyte.com/yugabytedb/yugabyte:2.25.2.0-b359
    restart: on-failure
    command: sh -c "sleep 10 && bin/yugabyted start --advertise_address=yugabyte3 --background=ture --join=yugabyte1"
    ports:
      - "5435:5433"
    volumes:
      - yugabyte3-store:/root/var

volumes:
  yugabyte1-store:
  yugabyte2-store:
  yugabyte3-store:
1 Like
7

Update: background=ture option is not stable, so false seems better.

8

Hi @bluepuma77 ,

I am facing a very similar issue as you. I also want to run multiple nodes, deployed in docker containers, on different hosts. I see you have marked @ytooyama ‘s reply as solution, but I have not got that to work, except from when deploying all containers on a single host.

When I have deployed the master node on HOST_1 and attempt to join it from a follower node on HOST_2, using HOST_1’s IP for the join, it fails to join the cluster saying it is not reachable.

Your code above likely would work for deployment of the master if you would swap out

--advertise_address=${HOST_IP}

to instead use the container name or the host name. You can even delete it altogether (I think it then defaults to the hostname). It does not seem to like when you add the host IP as the advertise_address from within a container (most often it has an IP of 172.18.0.2 or similar).

How did you end up resolving your issue? Did you end up using docker swarm?

9

I think Docker support is only for development purposes.
If you want to run it with a multi-node Docker, I think it’s better to use Kubernetes.
There is native support for YugabyteDB.

10

Hi @ytooyama ,

I think Docker support is only for development purposes.

That is what I am afraid of. But at the same time, it would not make any sense. There are sources saying it should work. This link for example. It claims it should be possible to use docker to create a multi region universe. I have not seen any source saying it is impossible.

For me, Kubernetes is not an option right now.