YB Data Migration

[dorian_yugabyte]

Hi @mahir

@mahir

is there a reason why we have to use change_blacklist instead of change_leader_blacklist after adding the new nodes?

We want to fully remove those nodes. When we do change_blacklist, the tablets are fully removed from the nodes, and then re-replicated again on the remaining nodes, so each tablet should still be RF3 and we can safely remove them.

If we did change_leader_blacklist instead, the leaders would move much faster (mostly metadata move), but the reamining nodes would only have RF2 for the tablets that had 1 tablet on the server(s) that was being removed. After removing those servers, after ~15min, the tablets would go to RF3 again.

@mahir

Wouldn’t data still be copied to both sets either way?

While availability is the same, what if you lose another node (normal random failure) after removing the node with no leaders, but before the re-bootstrapping has finished which triggers after 15min? Some tablets might go from RF2 → RF1, so you’d lose majority of peers on those tablets, and be unable to read/write to them, until you did a manual recovery: Manual remote bootstrap of failed peer | YugabyteDB Docs

@mahir

Also if we go with the recommended process, will the old nodes still work as entrypoints for queries to the cluster until our applications are updated to use the new nodes, or should applications be switched over prior to blacklisting?

The nodes will accept connections & queries and route correctly, they just won’t have data themselves.

[mahir]

That makes sense, thank you!