I understand that, we can perform rollback to before snapshot time ..
For: Lets say I have taken snapshot at 09:00 AM and I can able to restore the at 09:00 AM or before 09:00AM ..for example . 08:30
My Ask is:
I have taken snapshot at 09:00 AM , Now time is 10:00AM
I have all the logs ( docDB Raft logs )
Now, I want to restore the database at 09:30AM PIT rollforward
Question: Is it possible to restore PIT rollforwad to 09:30AM. ? If yes please provide document reference or Some high level overview or in case this feature not available in the current release any scope for this feature? I heard its available in MongoDB.
Yes because the db by default will keep all row versions for 24H. From the link above:
Flashback provides a way to rewind a database back to any microsecond in time over a short history retention period. Flashback is automatically and internally managed by YugabyteDB for performance purposes. The history retention period defaults to 24 hours and may be reduced to improve DB performance.
Thanks for the response …
I understand how flashback database works, Good feature .. when n issues with systesm.
By default snapshot will be taken to Local disk, Yes we can perform incremental snapshots. I am not sure how much time it takes for 50TB
OLTP database.
We always prefer to take snapshot and copy over to S3. Time taken for incremental snapshot and copying over difinately it takes lot of time.
Instead of this, Like PostgreSQl or oracle databases, We perform Weekly fullbackup, daily incremental and we archive the wal logs.
Increase of any system failure , we restore and recover to PITR using wal files. Looks like to me its missing here.Which is really required
for OLTP systems.
Incremental backups copy changed SST files created between backups. See Incremental Backups for a more complete explanation. We don’t need to copy the entire 50TB database; we only need to copy the new files created since the last full or incremental backup.
So if only a few gigabytes change between incremental snapshots/backups, only those new immutable SST files get copied to S3. The manifest tracks which files have already been uploaded, preventing repeated uploads.
Our write-ahead logs perform the following functions:
Primary target for all database writes (this includes heartbeats and other consensus operations in addition to data)
Rebuilds memstores after a node restart. Memstores flush to disk to become our SST data files.
Used in RBS (Remote bootstrap) operations to catch a peer up before it can participate in quorum.
Source of truth for XCluster and CDC operations. In XCluster (our asynchronous solution), the target reads WALs from the source and then applies them on the target. A similar operation occurs for CDC. CDC pulls data from the WAL to create a Virtual WAL for publications (pg_logical).
BAR (Backup and Restore) is not on that list, because we don’t use WALs for that purpose. WAL rollforward is a very expensive operation because it replays all of the consensus changes as well.
I hope this helps you understand that your perceived gap is not really there, our architecture doesn’t require the same things that other RDBMSs do.
–Alan
Thanks Alan for the details .. I completely got it , but being OLTP system we can’t offered to lose one minute of data as well. How we can achieve this. ?
We can create snapshot every 10 min and copy over to S3 or let say be in local disk itself. With Snapshot , we can only restore the database until snapshot completion time only, Even though we have wals of no use in case if we want restore after snapshot..
Also, I read the some documentation , regarding snapshot restore can’t restore the Transctional DDL during point in time recovery.
Example:
09:00 : Snapshot#1
09:15 : Some DDLs
09:30 : Snapshot#2
I want to restore to PIT 9:20, I am getting error relation doesn’t exist. If we have have wal support, it might have restored the DDLs as well. Wals greatly help in PITR and must for Critical OLTP systems.