Database not visible after creation

General
1

Hi,

I installed YugabyteDB 2.1.5.0 and tried to test with some databases. Trying to do that I dumped users and a database from another server (PostgreSQL).
I managed to import users succesfully but am having trouble with database import.
Firstly I tried to create a database:

ysqlsh -U yugabyte -h 10.10.10.27 -p 5433

and then

yugabyte=# CREATE DATABASE test;

and got following error message:

ERROR: Service unavailable: Operation failed, tablet 00000000000000000000000000000000 operation memory consumption (0) has exceeded its limit (1073741824) or the limit of an ancestral tracker

if i try to list database it’s not listed:

yugabyte-# \l
                                   List of databases
      Name       |  Owner   | Encoding | Collate |    Ctype    |   Access privileges   
-----------------+----------+----------+---------+-------------+-----------------------
 postgres        | postgres | UTF8     | C       | en_US.UTF-8 | 
 system_platform | postgres | UTF8     | C       | en_US.UTF-8 | 
 template0       | postgres | UTF8     | C       | en_US.UTF-8 | =c/postgres          +
                 |          |          |         |             | postgres=CTc/postgres
 template1       | postgres | UTF8     | C       | en_US.UTF-8 | =c/postgres          +
                 |          |          |         |             | postgres=CTc/postgres
 yugabyte        | postgres | UTF8     | C       | en_US.UTF-8 | 
(5 rows)

if I try to create it again:

yugabyte=# CREATE DATABASE test;
ERROR:  Already present: Keyspace 'test' already exists

and if I try to change database to created database:

yugabyte=# \c test;
FATAL:  database "test" does not exist
Previous connection kept

Tserver ERROR log:

E0513 09:19:34.750120 27137 tablet_metadata.cc:402] Failed to destroy regular DB at: /var/yugabytedb/yb-data/tserver/data/rocksdb/table-3c22896d54a84fb486c02cfcb9a1cd42/tablet-37dab9c61e65494ab5e9d353a28edc3d: IO error (yb/rocksdb/util/env_posix.cc:326): /var/yugabytedb/yb-data/tserver/data/rocksdb/table-3c22896d54a84fb486c02cfcb9a1cd42/tablet-37dab9c61e65494ab5e9d353a28edc3d/LOCK: No such file or directory

Does anyone have any idea why is this happening?

2

Hi @puska

Can you list the specs of ther server that you’re using ?

3

6 VMs, 3 of them master and tserver, 3 of them only tservers. All VMs 2 CPU, 2GB RAM (as per minimal requirements), ulimit -a gives:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7742
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7742
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

/etc/security/limits.conf:

*                -       core            unlimited
*                -       data            unlimited
*                -       fsize           unlimited
*                -       sigpending      119934
*                -       memlock         64
*                -       rss             unlimited
*                -       nofile          1048576
*                -       msgqueue        819200
*                -       stack           8192
*                -       cpu             unlimited
*                -       nproc           12000
*                -       locks           unlimited
4

Hi @puska

Can you try this setting this option on yb-master:

-tcmalloc_max_free_bytes_percentage (Maximum percentage of the RSS that
      tcmalloc is allowed to use for reserved but unallocated memory.)
      type: int32 default: 10

Set it to 30. And see if that fixes it

5

Hi

It did not work, I deleted data directories and tried creating a database again but result was the same.

new master config:

--master_addresses=10.10.10.27:7100,10.20.10.27:7100,10.20.10.29:7100
--fs_data_dirs=/var/yugabytedb
--server_broadcast_addresses=10.10.10.27:7100
--tcmalloc_max_free_bytes_percentage=30
--placement_cloud=ZTM
--placement_region=HR
--placement_zone=Planinska

error message:

yugabyte=# CREATE DATABASE test;
ERROR:  Service unavailable: Operation failed, tablet 00000000000000000000000000000000 operation memory consumption (0) has exceeded its limit (1073741824) or the limit of an ancestral tracker

Message mentions unavailable service, services listing:

36256 ?        Sl     0:06 /opt/yugabyte-2.1.5.0/bin/yb-master --replication_factor=3 --flagfile /opt/yugabyte-2.1.5.0/etc/master.conf
 36297 ?        Sl     0:02 /opt/yugabyte-2.1.5.0/bin/yb-tserver --flagfile /opt/yugabyte-2.1.5.0/etc/tserver.conf
 36335 ?        S      0:00 /opt/yugabyte-2.1.5.0/postgres/bin/postgres -D /var/yugabytedb/pg_data -p 5433 -h 10.10.10.27 -k  -c logging_collector=on -c log_directory=/var/yugabytedb/yb-data/tserver/logs -c shared_preload_libraries=pg_stat_statements,yb_pg_metrics -c yb_pg_metrics.node_name=DEFAULT_NODE_NAME -c yb_pg_metrics.port=13000 -c config_file=/var/yugabytedb/pg_data/ysql_pg.conf -c hba_file=/var/yugabytedb/pg_data/ysql_hba.conf

tserver config file:

--tserver_master_addrs=10.10.10.27:7100,10.20.10.27:7100,10.20.10.29:7100
--fs_data_dirs=/var/yugabytedb
--rpc_bind_addresses=0.0.0.0:9101
--server_broadcast_addresses=10.10.10.27:9101
--use_private_ip=never
--cql_proxy_bind_address=10.10.10.27:9042
--redis_proxy_bind_address=10.10.10.27:6379
--webserver_port=9000
--pgsql_proxy_bind_address=10.10.10.27:5433
--placement_cloud=ZTM
--placement_region=HR
--placement_zone=Planinska
6

Hi @puska, please try to add --default_memory_limit_to_ram_ratio=0.35 in master conf file.

7

Yeepee yeah,

It works.