Our Scaleway cluster backup configuration (Velero)

1

We are experimenting with velero as a framework for cluster disaster recovery and data protection.

Up until now we have relied on our own sql-backup , sql-recovery and volume replicator containers deployed as CronJobs in our numerous Namespaces - all of which need maintenance, documentation, and often require time-consuming manual effort to ensure recovery is complete.

Velero simplifies our processes by providing a simple-to-use but powerful “all-in-one” backup and recovery framework for Kubernetes.

Sever installation

Velero relies on the installation of a “server” in the cluster and the definition of “schedules” that define what needs to be backed up along with its schedule. Installing and configuring velero relies on a CLI, whcih you install, then identify your backup destination before installing the server logic in the cluster.

You will need to set KUBECONFIG to the cluster you wish to manage.

Install the CLI. Here we’re on macos, and will be using Scaleway’s S3 object storage service in the region fr-par, the same region as the im-main cluster: -

brew install velero

Next, in the Scaleway console: -

  1. Create a private S3 bucket (ours is called im-main-velero-cluster-backups)
  2. Create an API key (to obtain an Access Key and Secret Key)
  3. Create a bucket policy (allowing everything for the chosen bucket and the user who created the API key)

With this done, create a credentials-velero ini file using the API key you created…

[default]
aws_access_key_id = SCW????????????????
aws_secret_access_key = ?????????????

…install velero (once) in the cluster. Version 1.142. was the latest at the time of writing: -

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.2 \
  --bucket im-main-velero-cluster-backups \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --uploader-type kopia \
  --backup-location-config \
      region=fr-par,s3ForcePathStyle=true,s3Url=https://s3.fr-par.scw.cloud

This will install a velero Pod and a node-agent for each Node in the cluster.

Once the installation is complete you should be able to check that access to the bucket is successfull by inspecting the velero Pod logs in the velero Namespace, where you will probably find a line containing the message “BackupStorageLocations is valid, marking as available”.

Creating schedules

Now we create schedules that define a backup frequency and (in our case) the Namespaces they apply to.

For each backup we typically create two schedules, a daily schedule (at 00:30) that is kept for 7 days, and a weekly schedule (running on Saturday morning, at 02:30) kept for the default period (30 days). The schedule name starts with the frequency, followed by the Namespaces (or application suite) it applies to. In our first example we create schedules that cover all of our production Squonk Namespaces: -

  • daily-prod-squonk
  • weekly-prod-squonk
velero schedule create daily-prod-squonk \
  --schedule="30 0 * * *" \
  --include-namespaces im-account-server,im-data-manager,im-data-manager-ui,im-data-manager-job-operator,im-data-manager-jupyter-operator,im-data-manager-viz-operator \
  --ttl 168h0m0s \
  --default-volumes-to-fs-backup \
  --snapshot-volumes=false

velero schedule create weekly-prod-squonk \
  --schedule="30 2 * * 6" \
  --include-namespaces im-account-server,im-data-manager,im-data-manager-ui,im-data-manager-job-operator,im-data-manager-jupyter-operator,im-data-manager-viz-operator \
  --ttl 720h0m0s \
  --default-volumes-to-fs-backup \
  --snapshot-volumes=false

As the Squonk deployment may contain objects created by operators outside of its Namespace we should include the operator Namespace in each backup, as we have done in the schedules above. Restoring Squonk without also restoring the operators it relies upon is likely to result in a mis-configured application state.

Using --default-volumes-to-fs-backup we ensure that all the volumes in the Namespace are backed-up (i.e. they do not need to be annotated). If we do not add this option to the schedule we have to identify the volumes that need to be backed-up using a root-level annotation. In this example we declare (in the Pod) that the media volume needs to be backed up: -

metadata:
  annotations:
    backup.velero.io/backup-volumes: media

Other schedules

  • daily/weekly-infra (covering im-infra, rabbitmq-system, and oos)
  • daily/weekly-website
  • daily/weekly-fragnet
  • daily-awx

Viewing schedules and backups

You can view existing schedules with: -

velero schedule get

You can run kubectl edit schedule <schedule-name> -n velero to change cron expressions, labels, or included namespaces in your default text editor.

You can get and display information about all existing backups with: -

velero backup get

velero backup describe <backup-name>

If you’re interested only in backups created by a specific schedule you can utilise selectors, based on the fact that backups are labelled with the schedule that created them. So, to see all the daily-prod-squonk backups you can run: -

velero backup get --selector velero.io/schedule-name=daily-prod-squonk

Editing a schedule

To edit a schedult you can use kubectl: -

kubectl edit schedule <schedule-name> -n velero

Running a schedule (manually)

If you want to create a backup now, rather than waiting for the scheduled time, you can run: -

velero backup create --from-schedule <schedule-name>