Setting up read replica cluster

cyberboy · February 16, 2021, 7:55am

I am aware that it is possible to set up read replica cluster with Yugabyte Platform.
However, is it also possible to set it up with YugabyteDB only?

dorian_yugabyte · February 16, 2021, 8:51am

@cyberboy
Yes see here Deploy | YugabyteDB Docs.

Read replica is a part of 2DC deployment Deploy | YugabyteDB Docs.

cyberboy · February 17, 2021, 8:07am

@dorian_yugabyte
I deployed YugabyteDB on Amazon EKS by following the instructions below

So I got al three yb-master and three yb-tservers running on primary cluster

Then, I did step 2 and 3 from the link you provided below

Accessing http://:7000/cluster-config, I confirmed that read replica already defined

version: 2
replication_info {
  live_replicas {
    num_replicas: 3
    placement_blocks {
      cloud_info {
        placement_cloud: "aws"
        placement_region: "ap-northeast-1"
        placement_zone: "ap-northeast-1a"
      }
      min_num_replicas: 1
    }
    placement_blocks {
      cloud_info {
        placement_cloud: "aws"
        placement_region: "ap-northeast-1"
        placement_zone: "ap-northeast-1b"
      }
      min_num_replicas: 1
    }
    placement_blocks {
      cloud_info {
        placement_cloud: "aws"
        placement_region: "ap-northeast-1"
        placement_zone: "ap-northeast-1c"
      }
      min_num_replicas: 1
    }
  }
  read_replicas {
    num_replicas: 3
    placement_blocks {
      cloud_info {
        placement_cloud: "aws"
        placement_region: "ap-northeast-1"
        placement_zone: "ap-northeast-1a_rr"
      }
      min_num_replicas: 1
    }
    placement_blocks {
      cloud_info {
        placement_cloud: "aws"
        placement_region: "ap-northeast-1"
        placement_zone: "ap-northeast-1b_rr"
      }
      min_num_replicas: 1
    }
    placement_blocks {
      cloud_info {
        placement_cloud: "aws"
        placement_region: "ap-northeast-1"
        placement_zone: "ap-northeast-1c_rr"
      }
      min_num_replicas: 1
    }
    placement_uuid: "Fe1rwwa6shkJm7wX"
  }
}
cluster_uuid: "8d3cc0c7-73a9-411a-9e32-b960cc8c29be"

I created a new EKS cluster dedicated for the read replica with the same region and AZ as primary cluster.
However, now I don’t know how to start read replica yb-tserver services (step 5)
Could you please help?
Thank you in advance

dorian_yugabyte · February 17, 2021, 9:13am

Just like you started 3 yb-tservers in step 4, now you do the same again with another 3 yb-tservers but chaning the -placement_zone -placement_region -placement_cloud -placement_uuid for to those of the read replica (the ones ending with “_rr”).

cyberboy · February 17, 2021, 10:48am

@dorian_yugabyte
I actually skipped step 1 and 4 as all yb-masters and yb-tservers in primary cluster already running after I deployed YugabyteDB on Amazon EKS.
Could you give me an example of working command to start yb-tserver?

I checked the instruction below but I don’t know how to install YugabyteDB in read replica cluster, therefore I cannot use 「./bin/yb-tserver」

When I set up primary cluster, I used helm to install YugabyteDB.
For example : helm install yb-demo-us-east-1a yugabytedb/yugabyte
–namespace yb-demo-us-east-1a
-f overrides-us-east-1a.yaml --wait
Doing so will start yb-master and yb-tserver which is not what we want for read replica cluster.

cyberboy · February 18, 2021, 11:35am

@dorian_yugabyte
Is it actually possible to setup read replica cluster after running primary data cluster on EKS?

Since setting up read replica cluster with primary data cluster on EKS not yet working, I tried manual deployment by following the instruction below

I set up 3 nodes and got yb-masters and yb-tservers running (confirmed with step 8)

However, if I set up read replica by following the instruction below, when I used ysqlsh to try to create database, it did not work

Error message:
Timed out: Timed out waiting for Table Creation: . Errors from tablet servers: [Timed out (yb/client/client-internal.cc:144): Timed out waiting for Table Creation]

I also tried with ycqlsh and it also did not work when I tried to create a table.
Error message:
InvalidRequest: Error from server: code=2200 [Invalid query] message=“Invalid Table Definition. Error creating table app.user_actions on the master: Not enough live tablet servers to create table with replication factor 3. 0 tablet servers are alive.
CREATE TABLE user_actions (userid int, action_id int, payload text,
^^^^^^^^^^^^
PRIMARY KEY ((userid), action_id))
WITH CLUSTERING ORDER BY (action_id DESC);
(ql error -302)”

It seemed like yb-tservers were not running
However, when I checked master-ui page (port no 7000) on web browser, it showed that all yb-tservers were all alive
One thing that was weird was that the number of nodes for yb-tservers became 6
Were yb-tservers from read replica cluster supposed to be shown here?

arnav · February 18, 2021, 1:16pm

Hey @cyberboy , we have never actually tested creating read-replicas on k8s. So while in theory they should work, we don’t have proper documentation to help you out. Our helm charts are also lacking in this regard since as you mentioned, there is currently no way to bring up a deployment without creating the master services. Also, we do a lot of templating, and it is possible we will have to manually fix/update certain flags in our helm chart to support read-replicas.
We also haven’t ever tested multi-cluster deployments on EKS. But from what I understand, it seems like you have successfully established connection between the two clusters, since you can see the tservers of the read replica in the master UI. Did you successfully manage to run all the steps in Deploy | YugabyteDB Docs? Or did you get an error while updating the placement config?

cyberboy · February 18, 2021, 2:19pm

@arnav
Thank you for your follow-up
I actually stopped trying out the EKS for read replica cluster
I tried manual deployment as described in the link below

Then, I combined it with read replica cluster deployment

As I mentioned in the previous reply, I did see the read replica in the master UI.
So I believe all yb-masters and yb-tservers were all running
However, when I used ysqlsh to create database (eg. CREATE DATABASE yb_test;), it did not work.
Error message:
Timed out: Timed out waiting for Table Creation: . Errors from tablet servers: [Timed out (yb/client/client-internal.cc:144): Timed out waiting for Table Creation]

I also tried with ycqlsh and it also did not work when I tried to create a table.
Error message:
InvalidRequest: Error from server: code=2200 [Invalid query] message=“Invalid Table Definition. Error creating table app.user_actions on the master: Not enough live tablet servers to create table with replication factor 3. 0 tablet servers are alive.
CREATE TABLE user_actions (userid int, action_id int, payload text,
^^^^^^^^^^^^
PRIMARY KEY ((userid), action_id))
WITH CLUSTERING ORDER BY (action_id DESC);
(ql error -302)”

Do you have any idea why this happened?

dorian_yugabyte · February 19, 2021, 4:04pm

Paste all the config that you passed to the yb-tservers.

Also did you run ./bin/yb-admin modify_placement_info and ./bin/yb-admin add_read_replica_placement_info commands ?

cyberboy · February 22, 2021, 12:48am

@dorian_yugabyte
Yes, I did run the commands.

I did my setup based on documentations below
doc1

doc2

The followings are my step-by-step

I followed the instructions in doc1 all the way to step 4
Finishing step 4, I got all configurations for 3 yb-masters and 3 yb-tservers of primary data cluster and 3 yb-tservers of read replica cluster (I used replication factor of 3 with 3 EC2 instances for primary data and read replica cluster)
I started yb-masters using the instruction in doc1 step 5
I confirmed that I could display admin-ui on my web browser (http://any-master-ip:7000)
I defined the placement for primary data and read replica cluster using the instruction in doc2 step 2 and 3
I confirmed that cluster configuration already changed (http://any-master-ip:7000/cluster-config)
I started yb-tservers of primary data cluster using the instruction in doc1 step 6
I confirmed in admin-ui that the number of nodes for yb-tservers became 3
I started yb-tservers of read replica cluster using the instruction in doc1 step 6
I confirmed in admin-ui that the number of nodes for yb-tservers became 6
I tested YSQL API using the instruction in doc1 step 8
CREATE DATABASE yb_test; failed with the following error message
Timed out: Timed out waiting for Table Creation: . Errors from tablet servers: [Timed out (yb/client/client-internal.cc:144): Timed out waiting for Table Creation]
I tested YCQL API using the instruction in doc1 step 9
CREATE TABLE user_actions (userid int, action_id int, payload text,
PRIMARY KEY ((userid), action_id))
WITH CLUSTERING ORDER BY (action_id DESC);
failed with the following error message
InvalidRequest: Error from server: code=2200 [Invalid query] message=“Invalid Table Definition. Error creating table app.user_actions on the master: Not enough live tablet servers to create table with replication factor 3. 0 tablet servers are alive.
CREATE TABLE user_actions (userid int, action_id int, payload text,
^^^^^^^^^^^^
PRIMARY KEY ((userid), action_id))
WITH CLUSTERING ORDER BY (action_id DESC);
(ql error -302)”

The followings are the configuration for t-servers
=======PRIMARY-IP1=======
–tserver_master_addrs=PRIMARY-IP1:7100,PRIMARY-IP2:7100,PRIMARY-IP3:7100
–fs_data_dirs=/mnt/d0
–rpc_bind_addresses=PRIMARY-IP1:9100
–cql_proxy_bind_address=PRIMARY-IP1:9042
–redis_proxy_bind_address=PRIMARY-IP1:6379
–webserver_interface=PRIMARY-IP1
–placement_cloud=aws
–placement_region=ap-northeast-1
–placement_zone=ap-northeast-1a
–pgsql_proxy_bind_address=PRIMARY-IP1:5433
=======PRIMARY-IP2=======
–tserver_master_addrs=PRIMARY-IP1:7100,PRIMARY-IP2:7100,PRIMARY-IP3:7100
–fs_data_dirs=/mnt/d0
–rpc_bind_addresses=PRIMARY-IP2:9100
–cql_proxy_bind_address=PRIMARY-IP2:9042
–redis_proxy_bind_address=PRIMARY-IP2:6379
–webserver_interface=PRIMARY-IP2
–placement_cloud=aws
–placement_region=ap-northeast-1
–placement_zone=ap-northeast-1c
–pgsql_proxy_bind_address=PRIMARY-IP2:5433
=======PRIMARY-IP3=======
–tserver_master_addrs=PRIMARY-IP1:7100,PRIMARY-IP2:7100,PRIMARY-IP3:7100
–fs_data_dirs=/mnt/d0
–rpc_bind_addresses=PRIMARY-IP3:9100
–cql_proxy_bind_address=PRIMARY-IP3:9042
–redis_proxy_bind_address=PRIMARY-IP3:6379
–webserver_interface=PRIMARY-IP3
–placement_cloud=aws
–placement_region=ap-northeast-1
–placement_zone=ap-northeast-1d
–pgsql_proxy_bind_address=PRIMARY-IP3:5433
=======READREPLICA-IP1=======
–tserver_master_addrs=PRIMARY-IP1:7100,PRIMARY-IP2:7100,PRIMARY-IP3:7100
–fs_data_dirs=/mnt/d0
–rpc_bind_addresses=READREPLICA-IP1:9100
–cql_proxy_bind_address=READREPLICA-IP1:9042
–redis_proxy_bind_address=READREPLICA-IP1:6379
–webserver_interface=READREPLICA-IP1
–placement_cloud=aws
–placement_region=ap-northeast-1
–placement_zone=ap-northeast-1a_rr
–pgsql_proxy_bind_address=READREPLICA-IP1:5433
=======READREPLICA-IP2=======
–tserver_master_addrs=PRIMARY-IP1:7100,PRIMARY-IP2:7100,PRIMARY-IP3:7100
–fs_data_dirs=/mnt/d0
–rpc_bind_addresses=READREPLICA-IP2:9100
–cql_proxy_bind_address=READREPLICA-IP2:9042
–redis_proxy_bind_address=READREPLICA-IP2:6379
–webserver_interface=READREPLICA-IP2
–placement_cloud=aws
–placement_region=ap-northeast-1
–placement_zone=ap-northeast-1c_rr
–pgsql_proxy_bind_address=READREPLICA-IP2:5433
=======READREPLICA-IP3=======
–tserver_master_addrs=PRIMARY-IP1:7100,PRIMARY-IP2:7100,PRIMARY-IP3:7100
–fs_data_dirs=/mnt/d0
–rpc_bind_addresses=READREPLICA-IP3:9100
–cql_proxy_bind_address=READREPLICA-IP3:9042
–redis_proxy_bind_address=READREPLICA-IP3:6379
–webserver_interface=READREPLICA-IP3
–placement_cloud=aws
–placement_region=ap-northeast-1
–placement_zone=ap-northeast-1d_rr
–pgsql_proxy_bind_address=READREPLICA-IP3:5433

Note: I used the real IPs when setting up

dorian_yugabyte · February 22, 2021, 3:44pm

Can you paste your http://any-master-ip:7000/cluster-config ?
And a screenshot of http://any-master-ip:7000/tablet-servers ?

cyberboy · February 26, 2021, 1:44am

@dorian_yugabyte
Above are the requested screenshot

Topic		Replies	Views
Deploy on Amazon EKS using Fargate General	3	713	February 15, 2021
Yugabyte onto AWS EKS Cluster General	11	1089	December 17, 2020
Could not connect to yb-tserver-service with external-ip using python application General	5	658	February 10, 2021
Running yb-ctl status does not give the node information General	5	1429	October 4, 2019
How to create a multi-cluster deployment on Kubernetes? General	10	2587	June 5, 2019

Related Topics