Read replica cluster setup in manual deployment gives error for creating new table

General
1

Issue: Data is not replicating from primary to read replica cluster. Creating a new table in primary cluster DB is failing.

I created primary and read replica clusters in GCP instances running in same availability zone using manual deployment process. To setup read replication between clusters, I followed steps described in Deploy | YugabyteDB Docs.

using commands:
.1. /bin/yb-admin modify_placement_info gcp.us-central.us-central1-b 3 <<primary_cluster_UUID>>
2. ./bin/yb-admin add_read_replica_placement_info gcp.us-central.us-central1-b_rr:3 3 <<READ_REPLICA_CLUSTER_UUID>>
3. Update primary tserver.conf as per documentation, and restart them
4. Update read replica tserver.conf as per documentation and restart them

I verified configuration from primary cluster config in web console using http://IP1:7000/cluster-config

version: 5
replication_info {
  live_replicas {
    num_replicas: 1
    placement_blocks {
      cloud_info {
        placement_cloud: "gcp"
        placement_region: "us-central"
        placement_zone: "us-central1-b"
      }
      min_num_replicas: 1
    }
    placement_uuid: "c04dbaa4-3abb-4fde-85d7-43bb9b239ca3”   <- primary yugabyte cluster UUID
  }
  read_replicas {
    num_replicas: 3
    placement_blocks {
      cloud_info {
        placement_cloud: "gcp"
        placement_region: "us-central"
        placement_zone: "us-central1-b_rr"
      }
      min_num_replicas: 3
    }
    placement_uuid: "5ba7fa51-f132-41f0-a447-45eaefe112c1”.      <- read replica yugabyte cluster UUID
  }
}
cluster_uuid: "c04dbaa4-3abb-4fde-85d7-43bb9b239ca3"

Primary cluster master.conf
——————————————

--master_addresses=IP1:7100,IP2:7100,IP3:7100
--rpc_bind_addresses=IP1:7100
--fs_data_dirs=/yb/data/disk1
--placement_cloud=gcp
--placement_uuid=c04dbaa4-3abb-4fde-85d7-43bb9b239ca3.  <- primary yugabyte cluster UUID
--placement_region=us-central
--placement_zone=us-central1-b
--default_memory_limit_to_ram_ratio=0.35
--durable_wal_write=true

Primary cluster tserver.conf
—————————————-

-tserver_master_addrs=IP1:7100,IP2:7100,IP3:7100
--rpc_bind_addresses=IP1:9100
--start_pgsql_proxy
--pgsql_proxy_bind_address=IP1:5433
--cql_proxy_bind_address=IP1:9042
--fs_data_dirs=/yb/data/disk1
--placement_cloud=gcp
--placement_region=us-central
--placement_zone=us-central1-b
--placement_uuid=c04dbaa4-3abb-4fde-85d7-43bb9b239ca3.   <- primary yugabyte cluster UUID

Read replica cluster tserver.conf
—————————————————

--tserver_master_addrs=IP4:7100,IP5:7100,IP6:7100
--rpc_bind_addresses=IP4:9100
--start_pgsql_proxy
--pgsql_proxy_bind_address=IP4:5433
--cql_proxy_bind_address=IP4:9042
--fs_data_dirs=/yb/data/disk1
--placement_region=us-central
--placement_zone=us-central1-b_rr
--placement_cloud=gcp
--placement_uuid=5ba7fa51-f132-41f0-a447-45eaefe112c1 <- read replica yugabyte cluster UUID

I verified from primary cluster web console

  1. Connect to primary cluster web console: http://<<primary_cluster_node_ip>:7000/
  2. Go to Home. All three master servers are running and alive
  3. Go to tablet-server tab. All three tablet servers up, running and alive

I also verified read replica cluster web console, master and tablet servers are up and running

Then, I created table as describe below:

  1. Connect to the primary cluster DB using /bin/ysqlsh -h <<primary_cluster_node_ip>>. Connection to DB successful with
  2. At yugabyte=# console, execute create table xys(id integer, name text);
    Result:
    ERROR: Invalid argument: Invalid table definition: Error creating table yugabyte.xys on the master: Not enough live tablet servers to create table with replication factor 3. 0 tablet servers are alive.

Any idea why creating a table failed?

Inserting or updating records in the existing tables did not replicated in the read replica cluster.

2

@ars1206 sorry for the late reply.

Can you paste a screenshot of http://yb-master-ip:7000/tablet-servers ?