# Newly added Tserver not have any load **URL:** \ **Category:** General\ **Created:** [April 20, 2024, 5:32pm UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586 "2024-04-20T17:32:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![subh14](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@subh14](https://forum.yugabyte.com/u/subh14)\ **Post date:** [April 20, 2024, 5:32pm UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/1 "2024-04-20T17:32:51Z")
Hi I have a cluster of 3 masters and 12 Tservers, I have added one node to blacklist. ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/f9e70ce6c1a763c4c541c5b116688a2c4345525d.png) ``` 1. Is there a way to bring back the node to normal? Also the load is not getting split after blacklisting. ``` And when I added a new Tserver its not receiving any load. Also newly added Tserver and few other Tservers does not have any system tablet leaders/peers. ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/6c6347bb2ee77592f27fb78010823882d423c703.png) Please suggest, 1. how to split load with newly added tserver. 2. Do we have to have System tablet Leader/peers in every Tserver? If yes, how to make sure System tablets gets splitted in all Tservers. Thanks Subhankar ---
**Author:** ![kannan](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/kannan/32/18_2.png) [@kannan](https://forum.yugabyte.com/u/kannan)\ **Post date:** [April 21, 2024, 3:57am UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/2 "2024-04-21T03:57:26Z")
Hi Subhankar, 1. For many of your questions (like in this thread or previous threads), without knowing the [http://master-ip:7000/cluster-config](http://master-ip:7000/cluster-config) details, and COMPLETE output of [http://master-ip:7000/tablet-servers](http://master-ip:7000/tablet-servers) it is not going to be obvious what the issue is. Based on earlier threads also it is most likely a case of misconfiguration. In future, can you always share those details for this flavor of question? 2. Regarding the node that you added to bring back to normal: I am guessing you added the node to the blacklist using the ADD option of yb-admin utility’s change\_blacklist command, correct? > **[yb-admin - command line tool for advanced YugabyteDB administration](https://docs.yugabyte.com/preview/admin/yb-admin/#change-blacklist)** > > Use the yb-admin command line tool for advanced administration of YugabyteDB clusters. To remove it from the blacklist and bring it back as a normal node, have you tried using the REMOVE option of the same command? ---
**Author:** ![subh14](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@subh14](https://forum.yugabyte.com/u/subh14)\ **Post date:** [April 23, 2024, 3:51pm UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/3 "2024-04-23T15:51:20Z")
Hi Kannan Sorry for not adding those details initially. Please find below cluster config and tablet server details. ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/cca026a187c6298b912c1f7458f8d79e1a7bcd64.png) ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/35f32cdf6bdba0df404cb6344d8ca579ba41f8f2.png) ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/11da1d42ad568cc9a0b3ccf154e742ec6fedfb53.png) ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/d6bb6015a6b0ce9779a0d4cc4764a5beed721a1f.png) regarding blacklisted server, thanks for your suggestion removing the server then adding it worked. However, few nodes does not have any system tablet peers/leaders and also system transaction table is showing leader peer not found. Inside that transaction table I see it says that tablet reported with an active leader, however no leader/follower details are there. Also some of the transaction tablets only showing follower. I have attached screenshots of those. ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/a6cbe99b1797c636239da3253ae785e10f2304e1.png) ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/d49da18de578f781b0ec51b91b67b7ac91120dc8.png) ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/679ab2a405ac46d90c5cf979d75d62bda6864f57.png) Cluster load is not balanced for too long. ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/27fea7aa9d087ef237d5ed586009a3db59dad38d.png) Please suggest. Thanks Subhankar ---
**Author:** ![kannan](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/kannan/32/18_2.png) [@kannan](https://forum.yugabyte.com/u/kannan)\ **Post date:** [April 23, 2024, 5:42pm UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/4 "2024-04-23T17:42:50Z")
a) Just an observation… from the /cluster-config, looks like, for this cluster you have not run the yb-admin modify\_placement\_info step to configure a zone aware data placement policy. b) Separately, with respect to why you have leaderless tablets – it is possible (for example) that in a RF=3 configuration you had two concurrent node failures or some other issue that needs to be investigated. Note that an RF=3 configuration is designed to handle 1 concurrent failure. You’ll probably have to share the logs (yb-tserver & yb-master logs especially) for YB side folks to be able to help. Or, perhaps ping us on community slack channel, and some folks can help get on a call with your team to help investigate further. ---
**Author:** ![subh14](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@subh14](https://forum.yugabyte.com/u/subh14)\ **Post date:** [April 23, 2024, 6:09pm UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/5 "2024-04-23T18:09:59Z")
Yes, I did not run modify\_placement\_info command separately this time. Actually it was working fine without specifically mentioning that command. I did execute this command to check if this makes cluster load balanced. It ended up being in read replica. As you suggested, it did happen once that 2 master was down out of 3, also 3/4 Tservers were also down. I will share master log. Is there a way I can make system tablets split between servers? like if I separately go for compact table system.transactions. Will it help solving Leaderless tablets, system.transactions? or is this even a right approach, manual compaction of system.transactions table? Thanks Subhankar ---
**Author:** ![subh14](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@subh14](https://forum.yugabyte.com/u/subh14)\ **Post date:** [April 25, 2024, 9:26am UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/6 "2024-04-25T09:26:49Z")
Additionally, what I found that Insertion rate is decreasing gradually, and once I restart all Tservers, insertion rates increases and rate stays high for 2/3 hours, then it decreases again. Please suggest. Thanks Subhankar ---
**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\ **Post date:** [April 25, 2024, 10:28am UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/7 "2024-04-25T10:28:33Z")
> [@subh14](#): > > Is there a way I can make system tablets split between servers? Why? > [@subh14](#): > > like if I separately go for compact table system.transactions Compact just does rocksdb level compaction. > [@subh14](#): > > Will it help solving Leaderless tablets, system.transactions? But do you have any pending issues with the cluster still? > [@subh14](#): > > or is this even a right approach, manual compaction of system.transactions table? It’s not. > [@subh14](#): > > Additionally, what I found that Insertion rate is decreasing gradually, and once I restart all Tservers, insertion rates increases and rate stays high for 2/3 hours, then it decreases again. > Please suggest. There is no information here. It should be very detailed, with exact numbers, how the inserts are spread over the servers, etc. It could be many things that go wrong (like compactions, or bad client code, or inserts going into 1 tablet, that after it gets split and the load is spread). Or maybe the load isn’t spread. So would need a lot more details here. ---
**Author:** ![subh14](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@subh14](https://forum.yugabyte.com/u/subh14)\ **Post date:** [May 1, 2024, 2:14pm UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/8 "2024-05-01T14:14:11Z")
I was thinking that if system tablets are not split evenly, it might affect cluster performance. So it is okay to have system tablets like this. Inserts are spread over the servers more or less evenly, however insert rate drops suddenly and increases once all tservers are restarted. Could it be because of network issues? Cluster has some issues 1. Leaderless Tablets (table -transactions) 2. Tserver UI shows deleted tables. ![image](https://canada1.discourse-cdn.com/flex027/uploads/yugabyte/original/1X/d01472f6a344454d5058fea6d52b484c8f2b073d.png) 3. Insert rate drops suddenly. after tserver restart it increases. Thanks Subhankar ---
**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\ **Post date:** [May 3, 2024, 2:14pm UTC](https://forum.yugabyte.com/t/newly-added-tserver-not-have-any-load/2586/9 "2024-05-03T14:14:54Z")
> [@subh14](#): > > Inserts are spread over the servers more or less evenly, however insert rate drops suddenly and increases once all tservers are restarted. Could it be because of network issues? It could be many things. You have to be really careful when benchmarking and see where the bottleneck is (network, disk io, CPU usage, bad client code, not enough parallelism, too much concurrency on single rows/tablets/server/load-balancer, bad configuration somewhere in the stack, etc etc). > [@subh14](#): > > I will share master log. You never did. > [@kannan](#): > > b) Separately, with respect to why you have leaderless tablets – it is possible (for example) that in a RF=3 configuration you had two concurrent node failures or some other issue that needs to be investigated. Note that an RF=3 configuration is designed to handle 1 concurrent failure. You’ll probably have to share the logs (yb-tserver & yb-master logs especially) for YB side folks to be able to help. Or, perhaps ping us on community slack channel, and some folks can help get on a call with your team to help investigate further. Also see this response again.