I was skimming through the docs and was not able to find how to best replicate postgres functionality to minimize data loss in case of disaster. The closest I was able to find is: [docdb] PITR: Metadata restore from external backups · Issue #8847 · yugabyte/yugabyte-db · GitHub
There are currently no way to keep S3 backups up-to-date or make incrimental backups in yb / yba?
Hi @hispebarzu
“incremental backup” is actually the only feature that is not open source. But it is available in YBA YugabyteDB Anywhere Incremental Backups 101 | Yugabyte
But it doesn’t do log shipping, but only backups the changed sst files on disk instead.
We also have PITR but it doesn’t offload the logs to S3 and is only in-cluster: Point-in-time recovery | YugabyteDB Docs
Thanks, I’ve overlooked YBA incremental backups implementation. Is there any plans to implement log shipping? We somewhat recently had a disater that wiped out our DB clusters, and business now values data availability more and there is more push towards sharded postgres, I doubt YBA can handle 10-minute incremetal backups.
Hi @hispebarzu -
We aren’t planning to implement log shipping because our logs aren’t used for recovery in the way that other relational databases use them. That’s why we implemented Incremental Backups that actually capture changed files and can run on a shorter schedule. The time required for an incremental backup depends on the number of objects in your databases as we must extract the schema at every iteration. So if you have a relatively small number of objects, this could be a very short time period. All of our backups perform a ysql_dump using the schema-only option per database, so you can test for yourself to see how much time that phase takes. Once the schema extraction phase is complete, the changed SST files are uploaded to your cloud store. This could occur in the 10 minute window that you mention.
Restoring from backups is rare if the cluster is properly configured with 3 independent fault domains. Do you need assistance configuring the proper resilience/availability strategy?
Alan
Log shipping and rollforward PIT recovery is one of the critical requirement for any OLTP database systems. I would say with honest feedback is that biggest feature lagging. Its impossible to have snapshots every 5 to 10 min when we considering 10+TB database size.
-Sid
It works if you do incremental snapshot, and only backup the differing sstables instead of the whole db.