Backing up FoundationDB | Tigris Object Storage
Backing Up FoundationDB
We are running FoundationDB with the official kubernetes operator. FoundationDB supports logical backups (with backup_agent) and disaster recovery (with dr_agent) through copying the database/streaming changes logically. It also supports binary backups through disk snapshots.
In this blog post, we will describe how to make a backup of FoundationDB via backup_agent. The FoundationDB operator supports making logical backups via backup_agent, but it does not support running DR with dr_agent. We decided to run backup_agent as a separate deployment to allow a symmetric setup with dr_agent.
Prerequisites
In order to run the backup_agent pods, we will need a deployment. That deployment will need the following:
- Network level access to the FoundationDB cluster
- The cluster file to use. This is provided from the config volume that is exposed from the operator. In the setup described below, the cluster file will be available at
/mnt/fdb-config-volume/cluster-file. - If the backup destination is S3 or a similar object store, the secrets are needed to connect to it. In the example below, it is coming from an external secret.
- If the backup destination is not object store, supply a volume for the backup agents where the backup will be stored.
There is no limit to the number of backup agents that can connect to a cluster, each one will be responsible for a range of keys, which they will either store locally (or in the case of Kubernetes on the attached volumes), or stream to the object store. If the backup is stored locally and the database is for example 4 GiB, and there are 4 backup agents available, each of them will roughly store 1 GiB worth of data. If the backup destination is an object store, the number of backup agents will mostly determine the degree of parallelism both for reading the data and for writing the streamed data to the object store. This blog post will focus on the backups being stored on an object store, which I assume most of the Tigris users will do.
When using an object storage like S3, versioning should be disabled. (FoundationDB will do the versioning), and there should not be any automatic deletion/archival policy on the bucket. Only fdbbackup has a way to know what key range is still needed. Deleting old backups should be done with the expire subcommand of fdbbackup.
Example Kubernetes Deployment
Below is an example of backup_agent as a kubernetes deployment with 4 replicas.
apiVersion: apps/v1
kind: Deployment
metadata:
name: fdb-backup-agent
labels:
app.kubernetes.io/name: fdb-backup-agent
spec:
replicas: 4
selector:
matchLabels:
app.kubernetes.io/name: fdb-backup-agent
template:
metadata:
labels:
app.kubernetes.io/name: fdb-backup-agent
spec:
containers:
- name: fdb-backup-agent
resources:
limits:
cpu: "0.5"
memory: "4Gi"
requests:
cpu: "0.5"
memory: "2Gi"
image: foundationdb/foundationdb:7.1.20
imagePullPolicy: IfNotPresent
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: sample-fdb-backup-user-secrets
key: accessKey
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: sample-fdb-backup-user-secrets
key: secretKey
volumeMounts:
- mountPath: /tmp
name: tmp
- mountPath: /mnt/fdb-config-volume
name: sample-fdb-cluster-config-volume
command:
- "/usr/bin/backup_agent"
- "-C"
- "/mnt/fdb-config-volume/cluster-file"
- "--knob_http_request_aws_v4_header=true"
volumes:
- name: tmp
emptyDir: {}
- name: sample-fdb-cluster-config-volume
configMap:
name: sample-fdb-cluster-config
A continuous backup pushes multiple logs and inconsistent snapshots to the destination. At restore time, the changes will be used to "roll forward" the database.
Backup Command Example
fdbbackup start -t sample-test -z -s 7200 -C /mnt/fdb-config-volume/cluster-file -d 'blobstore://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@s3.us-east-2.amazonaws.com:443/path/to/backup/backup_name?bucket=sample-bucket&sc=1' --knob_http_request_aws_v4_header=true --log
The fdbbackup itself is a client utility (can be used from the backup agent pods) for managing backup jobs. Once the job level manipulation is done, the fdbbackup process itself will exit, the actual work is done by the backup_agent processes.
Those are quite a lot of parameters, let me explain them.
-t TAG_NAME
Specifies the tag that the backup will use. By default this is default. This will appear in the tags part of status json output.
-z
Specifies that the backup is continuous. Without this, the backup is just made as a one off. The fdbbackup
-s 7200
Specifies the snapshot interval, which determines how often a new snapshot should be made. Please note that 7200 is way too low for production settings, but for the purpose of this blog post, and showing the status output based on this, it's perfect.
[root@sample-fdb-backup-agent-776ff99c68-5xztw /]# fdbbackup status -C /mnt/fdb-config-volume/cluster-file -t conttest
The backup on tag conttest is restorable but continuing to blobstore://CREDS:HERE@s3.us-east-2.amazonaws.com:443/path/to/backup/backup_name?bucket=sample-bucket&sc=1.
BackupUID: 5579b197e04484d932ee267834086e47
BackupURL: blobstore://CREDS:HERE@s3.us-east-2.amazonaws.com:443/path/to/backup/backup_name?bucket=sample-bucket&sc=1
Snapshot interval is 7200 seconds. Current snapshot progress target is 43.74% (>100% means the snapshot is supposed to be done)
Details:
LogBytes written - 402464503164
RangeBytes written - 172474426856
Last complete log version and timestamp - 21435815219, 2023/02/28.17:32:15+0000
Last complete snapshot version and timestamp - 18299527573, 2023/02/28.16:39:57+0000
Current Snapshot start version and timestamp - 18304439087, 2023/02/28.16:40:02+0000
Expected snapshot end version and timestamp - 25504439087, 2023/02/28.18:40:04
Backup supposed to stop at next snapshot completion - No
```shell
[root@sample-fdb-backup-agent-776ff99c68-5xztw /]# fdbbackup expire -C /mnt/fdb-config-volume/cluster-file -d 'blobstore://CREDS:HERE@s3.us-east-2.amazonaws.com:443/path/to/backup/backup_name?bucket=sample-bucket&sc=1' --expire_before_version 25500608280 --knob_http_request_aws_v4_header=true --log
Final metadata update...
All data before version 25500608280 has been deleted.
Instead of specifying a version, a timestamp can be specified as well.
--expire-before-timestamp '2023/03/01.01:00:00+0000'
In this case, fdbbackup will find a version that is close to this, and will expire the data before that version.
After expiring, the backup size should reduced. This can be verified with an AWS client.
aws s3 ls s3://sample-bucket/path/to/backup/backup_name/ --recursive --human-readable --summarize
Restoring Backups
Restorations can be triggered with the fdbrestore utility. If no target time or version is specified, fdbrestore will restore the latest available version.
Example:
[root@sample-fdb-backup-agent-776ff99c68-dcpxr /]# fdbrestore start --dest-cluster-file /mnt/fdb-config-volume/cluster-file -r 'blobstore://CREDS:HERE@s3.us-east-2.amazonaws.com:443/path/to/backup/backup_name?bucket=sample-bucket&sc=1' -t conttest --knob_http_request_aws_v4_header=true --log
No restore target version given, will use maximum restorable version from backup description.
Using target restore version 86927163669
Backup Description
URL: blobstore://CREDS:HERE@s3.us-east-2.amazonaws.com:443/path/to/backup/backup_name?bucket=sample-bucket&sc=1
Restorable: true
Partitioned logs: false
Conclusion
Having strong backup and disaster recovery capabilities in a database is crucial to ensure that your data is safe and secure. FoundationDB provides a robust foundation for Tigris, with excellent backup and disaster recovery capabilities.