How to configure Docker YugabyteDB cluster on 3 servers?

General
1

While searching for a simple Postgres cluster solution using Docker, I came across YugabyteDB. I did find the documentation to create a single Docker instance, but I didn’t find any documentation or tutorial how to setup a 3 server cluster using the Docker image.

Is there a Docker compose example using simple env vars like those for self-hosting?

HOST    = db-1.internal   # per node
HOST_IP = 100.64.0.1      # per node
HOST_1  = db-1.internal
HOST_2  = db-2.internal
HOST_3  = db-3.internal
2

Hi @bluepuma77

Note that docker compose is for starting multiple containers in a single server. Do you need cluster in a single server or a cluster over multiple servers?

For multi-server setups, you can see docs in here: yugabyted reference | YugabyteDB Docs

The deployment docs are being changed to add examples with yugabyted in [doc] Deploy using yugabyted by ddhodge · Pull Request #27433 · yugabyte/yugabyte-db · GitHub. You can see a preview of that in Deploy YugabyteDB | YugabyteDB Docs

You’ll see that the docker deployment calls the same yugabyted command as the links above.

1 Like
3

Thanks, will look into it. We are thinking about testing YugabyteDB with our current infrastructure provider. Self-hosting for GDPR reasons. We got 3 VMs connected via a mesh VPN to enable private networking. The idea was to simply run the compose file on all 3 nodes.

Docker Swarm is still a thing, but Swarm issues can result in restart of instances, so I rather manage the 3 instances individually by hand. This also enables individual upgrading.

Trying to cross the chasm between a single local instance for development and a big professional build-out. Sitting in between with just 3 simple servers trying to run a tiny production system with HA for initial product MVP.

My personal preference would be to use the same compose file for all instances, so no dedicated --join on secondary nodes, but a list of hosts instead (like etcd --initial-cluster=IP1,IP2,IP3).

4

Trying to piece it together in plain Docker CLI:

export HOST=db-1.internal 
export HOST_IP=$(tailscale ip -4)
export HOST_1=db-1.internal 
export HOST_2=db-2.internal 
export HOST_3=db-3.internal

docker rm -fv yugabyte

docker run \
  --name yugabyte \
  -p ${HOST_IP}:7000:7000 \
  -p ${HOST_IP}:9000:9000 \
  -p ${HOST_IP}:15433:15433 \
  -p ${HOST_IP}:5433:5433 \
  -p ${HOST_IP}:9042:9042 \
  -v ~/yb_data:/home/yugabyte/yb_data \
  -d yugabytedb/yugabyte:2.25.2.0-b359 \
    bin/yugabyted start \
    --advertise_address=${HOST_IP} \
    --base_dir=/home/yugabyte/yb_data \
    --background=false \
    --cloud_location=hetzner.europe.central \
    --fault_tolerance=zone \
    --secure \
    --tserver_flags="enable_ysql_conn_mgr=true"

But the process just ends without an error message:

docker logs -f yugabyte
Starting yugabyted...
✅ System checks
\ Starting the YugabyteDB Processes...
5

This isn’t possible with yugabyted, it needs --join. cc @nmalladi

Can you run yugabyted collect_logs --base_dir=/home/yugabyte/yb_data inside the container and link them here?

6

I’m using the next manifesto.
If it is a simple 3 node, you can create it with the following compose file. It’s enough to evaluate.
I wish there was a manifest example like this in YugabyteDB’s Docs.

services:
  yugabyte1:
    image: software.yugabyte.com/yugabytedb/yugabyte:2.25.2.0-b359
    restart: on-failure
    command: sh -c "bin/yugabyted start --advertise_address=yugabyte1 --background=ture"
    ports:
      - "15433:15433"
      - "5433:5433"
    volumes:
      - yugabyte1-store:/root/var
  yugabyte2:
    image: software.yugabyte.com/yugabytedb/yugabyte:2.25.2.0-b359
    restart: on-failure
    command: sh -c "sleep 10 && bin/yugabyted start --advertise_address=yugabyte2 --background=ture --join=yugabyte1"
    ports:
      - "5434:5433"
    volumes:
      - yugabyte2-store:/root/var

  yugabyte3:
    image: software.yugabyte.com/yugabytedb/yugabyte:2.25.2.0-b359
    restart: on-failure
    command: sh -c "sleep 10 && bin/yugabyted start --advertise_address=yugabyte3 --background=ture --join=yugabyte1"
    ports:
      - "5435:5433"
    volumes:
      - yugabyte3-store:/root/var

volumes:
  yugabyte1-store:
  yugabyte2-store:
  yugabyte3-store:
1 Like