Unable to add master node to a cluster

General
1

Step1: add 10.0.8.21 as a new master to my cluster by following command without error:
./bin/yb-admin -master_addresses 10.0.3.47:7100 change_master_config ADD_SERVER 10.0.8.21 7100

Step2: try add 10.0.1.22 as mster to the cluster:
./bin/yb-admin -master_addresses 10.0.3.47:7100 change_master_config ADD_SERVER 10.0.1.22 7100

however step2 always failed even after I tried many times and waited more than 15 days. It complains step1 not finished. The error output is:

Error running change_master_config: Illegal state (yb/consensus/raft_consensus.cc:2495): Unable to change master config: Leader is not ready for Config Change, there is already one server in transition. Try again once the transition completes or remove the server in transition. Type: ADD_SERVER. Has opid: 1. Committed config: opid_index: 446243 peers { permanent_uuid: “5e7473101cd64b17b36c73fb984c1862” member_type: VOTER last_known_private_addr { host: “10.0.3.47” port: 7100 } cloud_info { placement_cloud: “cloud_tx” placement_region: “dc_tx” placement_zone: “rack_fc” } } peers { permanent_uuid: “501378bf70584372bde9cf54612f4a6f” member_type: PRE_VOTER last_known_private_addr { host: “10.0.8.21” port: 7100 } }. Pending config: . Current term: 8. Committed op id: 8.476918. Num peers in transit: 1

Btw, my cluster is functional and web ui is Ok and shows two masters (10.0.3.47 and 10.0.1.22). But the new master(10.0.1.22) does not work. Can anyone help me? I am blocked by the problem for days.

The version is yugabyte-2024.1.1.0-b137-linux-x86_64.tar.gz

2

Hi @song

What was the initial state of the cluster? How many servers in total and what RF?

3

There are 2 master and 3 t-servers, and RF is 3. The 2nd master shows on web ui, but it cannot be connected as master. And other master cannot be added to the cluster.

I tried to remove the 2nd master by its IP and Port, but it always failed.

Today I read related code, and tried to remove the 2nd master by IP and Port and id, it succeeded !!!. Now everything is ok.

Thank you.