# Cannot add read replica node

**URL:** <https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976>\
**Category:** General\
**Created:** [November 6, 2024, 11:45pm UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976 "2024-11-06T23:45:42Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tryyuga](https://avatars.discourse-cdn.com/v4/letter/t/d07c76/32.png) [@tryyuga](https://forum.yugabyte.com/u/tryyuga)\
**Post date:** [November 6, 2024, 11:45pm UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/1 "2024-11-06T23:45:42Z")

</div>

I’m trying to add 1 read replica on a west region to an existing yugabyte cluster in east coast region.  
I generated the certicate on the master node for the new ip replica, on master host is xx.xx.xx.96.

```auto
 /opt/yugabyte//bin/yugabyted cert generate_server_certs --hostnames=xx.xx.xx.46

```

I copy the certs to the replica certs dir and tried to started the replica the same way I started the other nodes but it errored out.

```auto
yugabyted start --secure --certs_dir=/db/yugabyte/certs --cloud_location=xx.xx.rr-1a --base_dir=/db/yugabyte --advertise_address=xx.xx.xx.46 --join=xx.xx.xx.96 --fault_tolerance=region

```

On the master, I ran the yb-admin command :

```auto
/opt/yugabyte/bin/yb-admin --certs_dir_name /db/yugabyte/certs --master_addresses xx.xx.xx.96:7100,xx.xx.xx.101:7100,xx.xx.xx.93:7100 add_read_replica_placement_info xx.xx.rr-1a:1 1 rr

Fetching configs from join IP...
Starting yugabyted...
/ Starting the YugabyteDB Processes...Failed to setup master. Exception: Traceback (most recent call last):
  File "/opt/yugabyte/bin/yugabyted", line 4565, in setup_master
    master_uuids = retry_op_with_argument(self.get_master_uuids, master_addrs)

```

The log has reference the list\_all\_masters command and looks like it could not run this command.

```auto
[yugabyted start] INFO: | 189.5s | run_process: ['/opt/yugabyte-2024.1.1.0/bin/yb-admin', '--certs_dir_name=/db/yugabyte/certs', 
'--master_addresses', 'xx.xx.xx.96:7100,xx.xx.xx.101:7100,xx.xx.xx.93:7100', 'list_all_masters'] timeout expired for command:
[yugabyted start] 2024-11-06 14:54:07,450 ERROR: | 189.5s | Failed to setup master. Exception: Traceback (most recent call last):

```

If I run on the replica the list\_all\_masters command:

```auto
Timed out (yb/rpc/rpc.cc:223): Unable to establish connection to leader master at [xx.xx.xx.96:7100,xx.xx.xx.101:7100,xx.xx.xx.93:7100]. Please verify the addresses and check if server is up,
 or if you're missing --certs_dir_name.
: Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.29.196.96:7100, 10.29.196.101:7100, 10.161.129.93:7100], num_attempts: 249) passed its deadline 15146.954s 
(passed: 60.031s): Network error (yb/rpc/secure_stream.cc:914): Handshake failed: Network error (yb/rpc/secure_stream.cc:1129): Unverified certificate: certificate signature failure, 
address: xx.xx.xx.101, hostname: xx.xx.xx.101

```

---

<div class="post-metadata">

**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\
**Post date:** [November 7, 2024, 10:27am UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/2 "2024-11-07T10:27:07Z")

</div>

Hi @tryyuga

What version are you using?

> [@tryyuga](#):
>
> I copy the certs to the replica certs dir and tried to started the replica the same way I started the other nodes but it errored out.

I think you used the incorrect command compared to docs: [yugabyted reference | YugabyteDB Docs](https://docs.yugabyte.com/preview/reference/configuration/yugabyted/#create-and-manage-read-replica-clusters)

> [@tryyuga](#):
>
> On the master, I ran the yb-admin command :

Did you get a more complete error here?

---

<div class="post-metadata">

**Author:** ![tryyuga](https://avatars.discourse-cdn.com/v4/letter/t/d07c76/32.png) [@tryyuga](https://forum.yugabyte.com/u/tryyuga)\
**Post date:** [November 7, 2024, 5:19pm UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/3 "2024-11-07T17:19:39Z")

</div>

Hi dorian\_yugabyte,  
Version is YugabyteDB Version 2024.1.1.0.

> > I think you used the incorrect command compared to docs: [yugabyted reference | YugabyteDB Docs](https://docs.yugabyte.com/preview/reference/configuration/yugabyted/#create-and-manage-read-replica-clusters) \<\<  
> > You are right, I restarted the command including missing param, but now I get below error:  
> > yugabyted start --secure --certs\_dir=/db/yugabyte/certs --cloud\_location=xx.xx.rr-1a --base\_dir=/db/yugabyte --advertise\_address=xx.xx.xx.46 --join=xx.xx.xx.96 --read\_replica

Got error list\_all\_tablet\_servers instead of list\_all\_masters this time.  
error:  
[yugabyted start] 2024-11-07 09:08:13,445 INFO: | 180.7s |  
run\_process: [‘/opt/yugabyte-2024.1.1.0/bin/yb-admin’, ‘–certs\_dir\_name=/db/yugabyte/certs’, ‘–master\_addresses’, ‘xx.xx.xx.93:7100,xx.xx.xx.101:7100,xx.xx.xx.96:7100’, ‘list\_all\_tablet\_servers’]  
timeout expired for command:  
[yugabyted start] 2024-11-07 09:08:13,455 INFO: | 180.7s | wait\_tserver: exception: Traceback (most recent call last):  
File “/opt/yugabyte/bin/yugabyted”, line 4658, in wait\_tserver  
if retry\_op\_with\_argument(self.is\_tserver\_up, cluster\_type, timeout):  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/opt/yugabyte/bin/yugabyted”, line 8569, in retry\_op\_with\_argument  
raise RuntimeError(“Failed after retrying operation for {} secs.”.format(  
RuntimeError: Failed after retrying operation for 180.06600880622864 secs.

I ran this list\_all\_tablet\_servers command itself on new replica and got the same error, ran on other existing nodes and it ran ok.  
Is this a network issue? admin told me they don’t see anything in firewall issues…  
I’m not sure how to debug.  
/opt/yugabyte/bin/yb-admin --certs\_dir\_name /db/yugabyte/certs --master\_addresses xx.xx.xx.96:7100,xx.xx.xx.196.99:7100,xx.xx.xx.93:7100 list\_all\_tablet\_servers

---

<div class="post-metadata">

**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\
**Post date:** [November 8, 2024, 8:12am UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/4 "2024-11-08T08:12:26Z")

</div>

Please paste both the full command & full response/error that you got.

---

<div class="post-metadata">

**Author:** ![tryyuga](https://avatars.discourse-cdn.com/v4/letter/t/d07c76/32.png) [@tryyuga](https://forum.yugabyte.com/u/tryyuga)\
**Post date:** [November 8, 2024, 6:09pm UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/5 "2024-11-08T18:09:35Z")

</div>

command:

```auto
  /opt/yugabyte/bin/yugabyted start --secure --certs_dir=/db/yugabyte/certs --cloud_location=xx.xx.west_rr-1a --base_dir=/db/yugabyte --advertise_address=xxx.46 --join=xxx.96 --read_replica

```

response:

```auto
Fetching configs from join IP...
Starting yugabyted...
❌ Database failed to start
Failed to start tserver yugabyted
For more information, check the logs in /db/yugabyte/logs

```

/db/yugabyte/logs/yugabyted.log:

```auto
[yugabyted start] 2024-11-08 10:03:36,454 INFO: | 180.7s | run_process: ['/opt/yugabyte-2024.1.1.0/bin/yb-admin', '--certs_dir_name=/db/yugabyte/certs', '--master_addresses', 'xxx.93:7100,xxs.101:7100,xxx.96:7100',
 'list_all_tablet_servers'] timeout expired for command:
[yugabyted start] 2024-11-08 10:03:36,463 INFO: | 180.7s | wait_tserver: exception: Traceback (most recent call last):
  File "/opt/yugabyte/bin/yugabyted", line 4658, in wait_tserver
    if retry_op_with_argument(self.is_tserver_up, cluster_type, timeout):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/yugabyte/bin/yugabyted", line 8569, in retry_op_with_argument
    raise RuntimeError("Failed after retrying operation for {} secs.".format(
RuntimeError: Failed after retrying operation for 180.0666904449463 secs.

[yugabyted start] 2024-11-08 10:03:36,463 INFO: | 180.7s | Failed to wait for tserver.
[yugabyted start] 2024-11-08 10:03:36,552 ERROR: | 180.8s | Failed to start tserver yugabyted
For more information, check the logs in /db/yugabyte/logs
[yugabyted start] 2024-11-08 10:03:36,552 INFO: | 180.8s | Shutting down...

```

---

<div class="post-metadata">

**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\
**Post date:** [November 11, 2024, 7:46am UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/6 "2024-11-11T07:46:34Z")

</div>

@tryyuga can you please do `yugabyted --base_dir=/db/yugabyte collect_logs` on your read replica node? And upload the zip in a cloud storage and share the link?

---

<div class="post-metadata">

**Author:** ![tryyuga](https://avatars.discourse-cdn.com/v4/letter/t/d07c76/32.png) [@tryyuga](https://forum.yugabyte.com/u/tryyuga)\
**Post date:** [November 12, 2024, 12:17am UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/7 "2024-11-12T00:17:10Z")

</div>

dorian\_yugabyte, Here is the link, let me know if you can get it.

> **[Google Drive: Sign-in](https://accounts.google.com/v3/signin/identifier?continue=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1fR5xJ6E1GCbEhENYa93ka7Sv8MSgHm47%2Fview%3Fusp%3Ddrive_link&followup=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1fR5xJ6E1GCbEhENYa93ka7Sv8MSgHm47%2Fview%3Fusp%3Ddrive_link&ifkv=AcMMx-co7PCq1nRJjJvv_d7IygjTcMA-i7_XezkVTuZknz2xsKXqZWh3HB3i0PjiFNsLTFLKCQXocg&osid=1&passive=1209600&service=wise&flowName=GlifWebSignIn&flowEntry=ServiceLogin&dsh=S-1340875237%3A1731370611970494&ddm=1)**
>
> Access Google Drive with a Google account (for personal use) or Google Workspace account (for business use).

---

<div class="post-metadata">

**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\
**Post date:** [November 12, 2024, 3:28pm UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/8 "2024-11-12T15:28:12Z")

</div>

> [@tryyuga](#):
>
> dorian\_yugabyte, Here is the link, let me know if you can get it.

The file is completely empty 127bytes. Maybe you uploaded it wrong?

---

<div class="post-metadata">

**Author:** ![tryyuga](https://avatars.discourse-cdn.com/v4/letter/t/d07c76/32.png) [@tryyuga](https://forum.yugabyte.com/u/tryyuga)\
**Post date:** [November 12, 2024, 6:19pm UTC](https://forum.yugabyte.com/t/cannot-add-read-replica-node/3976/9 "2024-11-12T18:19:10Z")

</div>

Ok sorry to waste your time, but I figure it out after redoing my test nodes completely.  
I had created the certs on a different “master” than what it was suppose to be, maybe that was why it coludn’t reach it, did give me certs error sporadically. created it on xxx.xxx.96 and copied it over to replica and ran same start replica command and it started.