Initdb with a docker stack

General
1

Hello everybody,

I come back on a subject already discuss with no real solution on a system approach.
I start a docker stack with a yb-master and a yb-tserver. This stack must be use by an other application but this one can’t start because it miss a table and a piece of sql code is given to create this table.
I thought “nice, yugabytedb is postgres compatible, I could the a initdb start to inject my sql”. After a while I read this subject who said in sumary that this feature is not implemented.
Well , searching again I found this one where @FranckPachot propose to use a third docker service to inject the code. Right , I’m not so bad because I thought to the same thing. But When I’m trying this way, it does’nt work. My yb-tserve refuse the connection with this message :

LOG: invalid length of startup packet

Is t here someone who have an idea to soluce this situation ?
PS > I’m an sysadmin, not a dev :face_with_thermometer:

2

Hi @Guillaume

Can you describe exactly the commands that you ran & at what order and the full error message?

3

Hi Dorian

My third docker service , the one who ‘replace/follow’ the initdb is like this :

  init-tserver:
    image: yugabytedb/yugabyte:2.16.6.0-b43
    environment:
      CQLSH_HOST: "yb-tserver"
      CQLSH_PORT: 5433
      CQLSH_USER: "postgres"
    command: 'bash -c bin/ycqlsh -f ./initdb.sql'
    volumes:
      - './pginit/initdb.sql:/home/yugabyte/initdb.sql:ro'
    depends_on:
      - yb-tserver

The log from it is :

yugabytedb-init-tserver-1 | Connection error: (‘Unable to connect to any servers’, {‘192.168.176.2’: ConnectionShutdown(‘Connection to 192.168.176.2 was closed’,)})
yugabytedb-init-tserver-1 exited with code 1

The log from yb-tserver is :

yugabytedb-yb-tserver-1 | 2023-08-03 11:05:29.255 UTC [110] LOG: invalid length of startup packet

I add in case of my tserver service :

  yb-tserver:
    image: yugabytedb/yugabyte:2.16.6.0-b43
    hostname: ybtserver2
    ports:
      - 9000:9000 # UI Tservev
      - 9100:9100 # port RPC YB-TServer
      - 5433:5433 # YSQL
    command: 'bin/yb-tserver --flagfile tserver.conf'
    volumes:
      - './tserver.conf:/home/yugabyte/tserver.conf:ro'
      - 'tserver_disk0:/mnt/disk0:rw'
      - 'tserver_disk1:/mnt/disk1:rw'

In addition, again, my initdb.sql file is just few lines to add a table

CREATE TABLE IF NOT EXISTS filemeta (
  dirhash     BIGINT,
  name        VARCHAR(65535),
  directory   VARCHAR(65535),
  meta        bytea,
  PRIMARY KEY (dirhash, name)
);
4

cc @FranckPachot for any insights

5

Hi @Guillaume, today I use yugabyted that starts yb-master and yb-tserver and has some automations like starting scripts from a directory, moving to replication factor 3 when there are 3 nodes, starting a new GUI…
If your goal is to have the same behavior as the PostgreSQL image, I build an image with similar behavior: GitHub - FranckPachot/yb-pglike: Builds a docker image with YugabyteDB that is compatible with the PostgreSQL image for a drop-in replacement

6

I think I got it. In your docker compose, I see bin/ycqlsh -f ./initdb.sql and CQLSH variables. Note that the PostgreSQL API is YSQL (and the command line is ysqlsh. YCQL is the cassandra like API. invalid length of startup packet may come from sending to the PostgreSQL endpoint (5433) from with the Cassandra protocol.

7

Hello @FranckPachot , thank your for yours answers.

I’m not sure to understand.
Do you mean I can launch my services yb-master and yb-tserver with a starting sql script who will initialise my database ?
If yes, how can I do that ? I try a lot of things without success.

( I note you’re Swiss, may be you speak french ? :slight_smile: )

8

Oui :slight_smile:
En fait yugabyted (d pour daemon) est un utilitaire qui lance les yb-master et yb-tserver et fait des choses en plus, comme lancer un script d’init: yugabyted reference | YugabyteDB Docs

Il y a un example sympa ici: How to run yugabytedb in docker and docker-compose
Il suffit de rajouter --initial_scripts_dir pour lancer tous les scripts .sql du directory

9

Merci beaucoup, je vais regarder tout cela, n’ayant pas réussi a lancé yugabyted pour créer un cluster je n’étais pas aller plus loin. Je me suis raté quelque part de toute évidence.
Merci pour ces liens je vais regarder de plus près.

10

Et navré @FranckPachot , de retour avec une question. J’ai tenté d’utiliser le flag --config dans le lancement du conteneur avec bin/yugabyted mais il me dit :

yugadb-yugabyte-1 | ERROR:root: | 0.0s | Failed to read config file /home/yugabyte/yugabyte.conf: Expecting value: line 1 column 1 (char 0)
yugadb-yugabyte-1 | Failed to read config file /home/yugabyte/yugabyte.conf: Expecting value: line 1 column 1 (char 0)

Ma commande docker est celle ci :

services:
  yugabyte:
    image: yugabytedb/yugabyte:2.16.6.0-b43
    restart: always
    command: bin/yugabyted start --base_dir=/home/yugabyte/yb_data --config /home/yugabyte/yugabyte.conf --daemon=false
    ports:
      - 7000:7000
      - 5433:5433
      - 9000:9000
      - 9042:9042
    volumes:
      - yugabyte_db:/var/lib/yugabyteql/data
      - ./yugabyte.conf:/home/yugabyte/yugabyte.conf:ro
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_DB=yugabyte

Avec un fichier de le configuration suivant étant donné qu’on est hors réseau :

--callhome false

J’ai également testé

--callhome=false

Est ce qu’il y a un formalisme particulier ?

11

Ah oui, ce fichier de config est en format JSON.
Mais je ne pense pas que ce soit possible de passer des paramètres. C’est yugabyted qui y met les paramètres pour le prochain restart

La doc devrait le préciser, merci pour le feedback