Backup failed in the semi-automatic mode replication step

I decided to try YugabyteDB’s semi-automatic transactional xCluster replication by following this procedure.

The version is yugabyte-2.25.2.0-b359-linux, and this time it is not a container, but trying to run Linux natively with AlmaLinux 8.10.

Looking at this document, I added ybc’s PATH.

sudo dnf install -y rpcbind nfs-utils
sudo mkdir -p /share/nfs

sudo vi /etc/exports
/share/nfs 172.17.28.0/24(rw,no_root_squash,async)

sudo systemctl enable rpcbind nfs-server
sudo systemctl restart rpcbind nfs-server
sudo chmod 777 /share/nfs

sudo firewall-cmd --permanent --zone=public --add-service=nfs
sudo firewall-cmd --permanent --zone=public --add-service=rpc-bind
sudo firewall-cmd --reload
  • ALL nodes (NFS Client)
sudo dnf install -y nfs-utils
sudo firewall-cmd --permanent --zone=public --add-service=nfs
sudo firewall-cmd --reload

sudo mkdir -p /share/nfsc
sudo chmod 777 /share/nfsc

sudo mount -v -t nfs 172.17.28.26:/share/nfs /share/nfsc
  • Create YugabyteDB Cluster
//Source Nodes
sudo yugabyted start --base_dir=/home/yugabyte/data  --advertise_address=172.17.28.20 --backup_daemon=true
sudo yugabyted start --base_dir=/home/yugabyte/data --advertise_address=172.17.28.21 --join=yuganode1 --backup_daemon=true
sudo yugabyted start --base_dir=/home/yugabyte/data --advertise_address=172.17.28.22 --join=yuganode1 --backup_daemon=true

//Target Nodes
sudo yugabyted start --base_dir=/home/yugabyte/data  --advertise_address=172.17.28.23 --backup_daemon=true
sudo yugabyted start --base_dir=/home/yugabyte/data --advertise_address=172.17.28.24 --join=yuganode4 --backup_daemon=true
sudo yugabyted start --base_dir=/home/yugabyte/data --advertise_address=172.17.28.25 --join=yuganode4 --backup_daemon=true

If you perform a backup, it will fail.

sudo yugabyted backup --database=sysbench --cloud_storage_uri=/share/nfsc --base_dir=/home/yugabyte/data

+--------------------------------------------------+
|                    yugabyted                     |
+--------------------------------------------------+
| Status        : Backup failed. Check the logs.   |
+--------------------------------------------------+

When I executed the command, there was the following output.

[yugabyted backup] 2025-06-09 14:33:41,809 INFO:  | 0.0s | Running yugabyted command: '/opt/yugabyte/latest/bin/yugabyted bac
kup --database=sysbench --cloud_storage_uri=/share/nfsc --base_dir=/home/yugabyte/data'
[yugabyted backup] 2025-06-09 14:33:41,809 INFO:  | 0.0s | cmd = backup using config file: /home/yugabyte/data/conf/yugabyted.conf
[yugabyted backup] 2025-06-09 14:33:41,810 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/bin for file openssl_proxy.sh
[yugabyted backup] 2025-06-09 14:33:41,810 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/bin for file yb-admin
[yugabyted backup] 2025-06-09 14:33:41,810 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/bin for file yb-ts-cli
[yugabyted backup] 2025-06-09 14:33:41,810 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/postgres/bin for file pg_upgrade
[yugabyted backup] 2025-06-09 14:33:41,818 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/ybc/bin for file yb-controller-cli
[yugabyted backup] 2025-06-09 14:33:41,818 INFO:  | 0.0s | run_process: cmd: ['/opt/yugabyte/yugabyte-2.25.2.0/ybc/bin/yb-controller-cli', 'download_md', '--cloud_dir=nfsc', '--bucket=share', '--cloud_type=nfs', '--local_dir=/tmp/nfsc']
[yugabyted backup] 2025-06-09 14:33:41,901 INFO:  | 0.1s | run_process returned 0: 
OUT >>
download: Time Taken (ms): 0
download: Start Delay (ms): 0
Bytes Transferred: 0
Object Size: 0
Num Objects: 0
Status: INVALID

<< ERR >>
I0609 14:33:41.894712 163214 controller_object_flags.cc:130] Set --per_upload_num_objects = 2
I0609 14:33:41.894762 163214 controller_object_flags.cc:130] Set --per_download_num_objects = 2
I0609 14:33:41.894771 163214 async_op_resources.h:53] MaxConcurrentUploads = 2
I0609 14:33:41.894775 163214 async_op_resources.h:55] MaxConcurrentDownloads = 2
I0609 14:33:41.894779 163214 async_op_resources.h:57] PerUploadNumObjects = 2
I0609 14:33:41.894784 163214 async_op_resources.h:59] PerDownloadNumObjects = 2
I0609 14:33:41.894819 163214 op.cc:198] Starting Execution
E0609 14:33:41.895043 163220 dir_ops_executor.cc:182] Got error downloading dir manifest from nfsc/backup.md
E0609 14:33:41.895067 163220 dir_ops_executor.cc:485] Unable to get Manifest for nfsc/backup.md
I0609 14:33:41.895110 163214 op.cc:222] Done

<<
[yugabyted backup] 2025-06-09 14:33:41,902 INFO:  | 0.1s | run_process: cmd: ['/opt/yugabyte/yugabyte-2.25.2.0/ybc/bin/yb-controller-cli', 'backup', '--cloud_type=nfs', '--bucket=share', '--cloud_dir=nfsc', '--tserver_ip=172.17.28.20', '--ns=sysbench', '--ns_type=ysql']
[yugabyted backup] 2025-06-09 14:33:41,958 INFO:  | 0.2s | run_process returned 0: 
OUT >>
Bucket does not exist
Error creating creds for region: '' : COMMAND_FAILED
Usage: yb-controller-cli backup
        [--backup_config_data]: Arbitrary string that is uploaded to the backup and returned after restore.
        --bucket: Default bucket to use.
        [--cloud_dir]: Location relative to bucket where the backup should be stored.
        --cloud_type: Which cloud to put the backup on s3/google/azure/nfs.
        [--compression]: If specified, backups are compressed. (Default: false)
        --ns: Namespace.
        --ns_type: Namespace type ysql/ycql.
        [--passphrase]: Passphrase to encrypt the backup with.
        [--prev_cloud_dir]: Location of previous backup relative to bucket. Specify if differential backup is required
        [--tables]: Comma seperated list of tables to backup/restore.
        [--task_id]: Id of task. (Default: Random UUID)
        --tserver_ip: IP of YB master server on which the backup should be run.
        [--use_tablespaces]: Store YSQL TABLESPACE objects into the backup, restore if set to true. (Default: false)
        [--wait]: Wait till task is complete when monitoring task progress. (Default: false)
        [--certs_dir_name]: Certificates directory
        [--server_port]: Port the controller server is listening on (Default: 18018)

<< ERR >>

<<

Hi @ytooyama

The NFS backup is is not very intuitive right now. We’re planning to make some changes to this to reduce the friction and make it more obvious for taking backups on NFS. Can you try these steps we used with a recent user:

Actually, yugabyted supports setting custom NFS path. We will need to export a ENV variable YBC_NFS_DIR. Before running the backup command, you’ll need to the following

-if your NFS is mounted on this point - /filecoin/Backup_longterm/yugabyte1-ybctl

export YBC_NFS_DIR=/filecoin/Backup_longterm/yugabyte1-ybctl

and then --cloud_storage_uri needs to be used to storage bucket and storage dir

./yugabyted backup --keyspace=idx --cloud_storage_uri=/idk_bkup/04-17-2025 --base_dir=/media/PM1725b/yugabyte1

storage bucket will be set to idk_bkup and storage dir will be set to 04-17-2025. Your back will be stored in the following location /filecoin/Backup_longterm/yugabyte1-ybctl/idk_bkup/04-17-2025

cc @nmalladi

Hi @dorian_yugabyte.
I tried it. The backup worked, but the restore failed.

[root@c7000-node1]# export YBC_NFS_DIR=/share/nfsc
[root@c7000-node1]# mkdir -p /share/nfsc/sysbench_bkup
[root@c7000-node1 06-09-2025]# yugabyted backup --database=sysbnech --cloud_storage_uri=/sysbench_bkup/06-09-2025 --base_dir=/home/yugabyte/data --status

+---------------------------------------+
|               yugabyted               |
+---------------------------------------+
| Status        : Backup is complete.   |
+---------------------------------------+

[root@c7000-node4 ~]# yugabyted restore --database=sysbench --cloud_storage_uri=/sysbench_bkup/06-09-2025 --base_dir=/home/yugabyte/data

+-------------------------------------------------+
|                    yugabyted                    |
+-------------------------------------------------+
| Status        : Restore started successfully.   |
+-------------------------------------------------+

[root@c7000-node4 ~]# yugabyted restore --database=sysbench --cloud_storage_uri=/sysbench_bkup/06-09-2025 --base_dir=/home/yugabyte/data --status

+------------------------------------------------------------------------+
|                               yugabyted                                |
+------------------------------------------------------------------------+
| Status        : Restore has failed. Check the logs for more details.   |
+------------------------------------------------------------------------+

log:

[yugabyted restore] 2025-06-09 19:49:12,597 INFO:  | 0.0s | Running yugabyted command: '/opt/yugabyte/latest/bin/yugabyted restore --database=sysbench --cloud_storage_uri=/sysbench_bkup/06-09-2025 --base_dir=/home/yugabyte/data'
[yugabyted restore] 2025-06-09 19:49:12,597 INFO:  | 0.0s | cmd = restore using config file: /home/yugabyte/data/conf/yugabyted.conf
[yugabyted restore] 2025-06-09 19:49:12,597 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/bin for file openssl_proxy.sh
[yugabyted restore] 2025-06-09 19:49:12,597 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/bin for file yb-admin
[yugabyted restore] 2025-06-09 19:49:12,597 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/bin for file yb-ts-cli
[yugabyted restore] 2025-06-09 19:49:12,598 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/postgres/bin for file pg_upgrade
[yugabyted restore] 2025-06-09 19:49:12,603 INFO:  | 0.0s | Found directory /opt/yugabyte/yugabyte-2.25.2.0/ybc/bin for file yb-controller-cli
[yugabyted restore] 2025-06-09 19:49:12,603 INFO:  | 0.0s | run_process: cmd: ['/opt/yugabyte/yugabyte-2.25.2.0/ybc/bin/yb-controller-cli', 'download_md', '--cloud_dir=06-09-2025', '--bucket=sysbench_bkup', '--cloud_type=nfs', '--local_dir=/tmp/06-09-2025']
[yugabyted restore] 2025-06-09 19:49:12,898 INFO:  | 0.3s | run_process returned 0: 
OUT >>
download: Time Taken (ms): 214
download: Start Delay (ms): 0
Bytes Transferred: 64527
Object Size: 64512
Num Objects: 1
Status: OK

<< ERR >>
I0609 19:49:12.675433 165751 controller_object_flags.cc:130] Set --per_upload_num_objects = 2
I0609 19:49:12.675483 165751 controller_object_flags.cc:130] Set --per_download_num_objects = 2
I0609 19:49:12.675493 165751 async_op_resources.h:53] MaxConcurrentUploads = 2
I0609 19:49:12.675499 165751 async_op_resources.h:55] MaxConcurrentDownloads = 2
I0609 19:49:12.675504 165751 async_op_resources.h:57] PerUploadNumObjects = 2
I0609 19:49:12.675510 165751 async_op_resources.h:59] PerDownloadNumObjects = 2
I0609 19:49:12.675554 165751 op.cc:198] Starting Execution
I0609 19:49:12.890115 165751 op.cc:222] Done

@ytooyama : Can you please share the log archive from yugabyted collect_logs either here or in our community Slack linked from Community | Yugabyte? It should have some ybc logs that can help us understand what is happening?

@ytooyama, along with the logs that Sanketh mentioned to collect. Can you please collect the logs from the following directory as well <data-dir>/yb-data/ybc/logs.

Unfortunately, the YBC logs are not getting collected with yugabyted collect_logs. We are making changes to get this included.

Thanks all.

The log has been uploaded to the following directory.

There is the following error in the error log of ybc due to typos, but it is normal.

E0610 08:16:29.613396 21915 yb_snapshot_resources.cc:192] Error Output: ysql_dump: error: connection to server at "172.17.28.40", port 5433 failed: FATAL:  database "sysbnech" does not exist

This database has been created, and it is assumed that there is an environment where tables generated by the following methods exist in the DB.

sysbench oltp_read_write \
  --pgsql-host=primary-node1.maas,primary-node2.maas,primary-node3.maas \
  --tables=3 \
  --table_size=5000 \
  --range_key_partitioning=false \
  --serial_cache_size=1000 \
  --create_secondary=true \
  --pgsql-db=sysbench \
  --pgsql-user=yugabyte \
  --db-driver=pgsql \
  --pgsql-port=5433 \
  prepare

I don’t understand. On your command it’s not “sysbnech” but “sysbench”

It was caused by the lack of understanding of storage bucket and storage dir and typos when executing commands.
After reviewing the commands and options again and trying, backup from NFS and recovery from NFS worked fine.

I apologize. This problem has been solved. I will close it.

[root@primary-node1]# export YBC_NFS_DIR=/share/nfsc
[root@primary-node1]# mkdir -p /share/nfsc/sysbench_bkup
[root@primary-node1]# yugabyted backup --database=sysbench --cloud_storage_uri=/sysbench_bkup/06-13-2025 --base_dir=/home/yugabyte/data
+------------------------------------------------+
|                   yugabyted                    |
+------------------------------------------------+
| Status        : Backup started successfully.   |
+------------------------------------------------+
[root@primary-node1]# yugabyted backup --database=sysbench --cloud_storage_uri=/sysbench_bkup/06-13-2025 --base_dir=/home/yugabyte/data --status
+---------------------------------------------------------------------------+
|                                 yugabyted                                 |
+---------------------------------------------------------------------------+
| Status        : Backup is in Progress. Check the logs for more details.   |
+---------------------------------------------------------------------------+
...
[root@secondary-node1]# yugabyted restore --database=sysbench --cloud_storage_uri=/sysbench_bkup/06-13-2025 --base_dir=/home/yugabyte/data
+-------------------------------------------------+
|                    yugabyted                    |
+-------------------------------------------------+
| Status        : Restore started successfully.   |
+-------------------------------------------------+

[root@secondary-node1 ~]# ysqlsh -h secondary-node1.maas -U yugabyte -d yugabyte 
ysqlsh (15.12-YB-2.25.2.0-b0)
Type "help" for help.

yugabyte=# \c sysbench 
You are now connected to database "sysbench" as user "yugabyte".
sysbench=# \d
Did not find any relations.

[root@secondary-node1]# yugabyted restore --database=sysbench --cloud_storage_uri=/sysbench_bkup/06-13-2025 --base_dir=/home/yugabyte/data --status
+-------------------------------------------+
|                 yugabyted                 |
+-------------------------------------------+
| Status        : Restore is in progress.   |
+-------------------------------------------+

[root@secondary-node1]# ysqlsh -h secondary-node1.maas -U yugabyte -d sysbench -c "\d"
               List of relations
 Schema |      Name      |   Type   |  Owner   
--------+----------------+----------+----------
 public | sbtest1        | table    | yugabyte
 public | sbtest1_id_seq | sequence | yugabyte
 public | sbtest2        | table    | yugabyte
 public | sbtest2_id_seq | sequence | yugabyte
 public | sbtest3        | table    | yugabyte
 public | sbtest3_id_seq | sequence | yugabyte
(6 rows)
1 Like

@ytooyama Glad you were able to get this working. Your feedback on the difficulty in understanding the storage bucket and storage dir is also faced by other users.

We are modifying the usage of NFS backups to make it more straightforward for users to specify the NFS directory path. We are adding a new flag to the backup command, --nfs_storage_location.

The --nfs_storage_location flag will take in the NFS directory path in the following format -

nfs_mount_path:parent_directory/sub_directory

Example usage:

yugabyted backup --database=sysbench --nfs_storage_location=/share/nfsc/sysbench_bkup:/sysbench_bkup/06-13-2025
1 Like

I think it’s very easy to understand.