How can we setup 3 node master master cluster in one data center?

General
1

Can Anyone help in this to resolve ?

Starting Master

./bin/yb-master
–master_addresses 10.151.16.189:7100,10.151.16.199:7100,10.151.16.119:7100
–rpc_bind_addresses 10.151.16.119
–fs_data_dirs “/u01/yugabytedata/disk1,/u01/yugabytedata/disk2”
–replication_factor=3
>& /u01/yb-master3.out &

=======================================================
Getting below errors From Yugabytedb - 1

======================================================

==> yb-master.ERROR <==
E0804 11:54:04.914026 7730 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.16.119:7100], num_attempts: 53) passed its deadline 384.874s (passed: 1.536s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111)

==> yb-master.INFO <==
I0804 11:54:04.409971 7728 tcp_stream.cc:308] { local: 10.151.16.189:39030 remote: 10.151.16.199:7100 }: Recv failed: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111)
E0804 11:54:04.914026 7730 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.16.119:7100], num_attempts: 53) passed its deadline 384.874s (passed: 1.536s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111)

2

Can you follow this guide here: Deploy | YugabyteDB Docs

3

Hi Already following yugabyte docs.

When I am starting yb-master we are getting below errors.

./bin/yb-master
–master_addresses 10.151.16.189:7100,10.151.16.199:7100,10.151.17.119:7100
–rpc_bind_addresses 10.151.16.189
–fs_data_dirs “/u01/yugabytedata/disk1,/u01/yugabytedata/disk2”
–replication_factor=3
>& /u01/yb-master1.out &

Error
Node 1
E0810 11:19:11.110110 14080 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 53) passed its deadline 516691.079s (passed: 1.526s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111)
E0810 11:19:13.654752 14080 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 53) passed its deadline 516693.606s (passed: 1.544s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111)
E0810 11:19:16.211920 14080 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 53) passed its deadline 516696.151s (passed: 1.556s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)

Node 2
LeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)
E0810 11:19:59.449920 13805 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 51) passed its deadline 516676.299s (passed: 1.521s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)
E0810 11:20:01.957839 13805 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 50) passed its deadline 516678.820s (passed: 1.507s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)
E0810 11:20:04.465477 13805 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 48) passed its deadline 516681.328s (passed: 1.507s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)

Node 3
E0810 11:19:55.909801 13862 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 52) passed its deadline 516675.307s (passed: 1.535s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)
E0810 11:19:58.457140 13862 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 52) passed its deadline 516677.844s (passed: 1.545s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)
E0810 11:20:00.995076 13862 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 41) passed its deadline 516680.390s (passed: 1.537s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)
E0810 11:20:03.517774 13862 async_initializer.cc:95] Failed to initialize client: Timed out (yb/rpc/rpc.cc:211): Could not locate the leader master: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 50) passed its deadline 516682.928s (passed: 1.522s): Not found (yb/master/master_rpc.cc:286): no leader found: GetLeaderMasterRpc(addrs: [10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100, 10.151.16.189:7100, 10.151.16.199:7100, 10.151.17.119:7100], num_attempts: 1)

4

Hi @BijalShah01

Did you try stopping the yb-masters, emptying the data dirs, and starting them again like I wrote you on slack ?

5

yes …i already did it.