I am trying to setup a restic job to backup my docker stacks, and with half of everything owned by root it becomes problematic. I’ve been wanting to look at podman so everything isn’t owned by root, but for now I want to backup my work I built.
Also, how do you deal with some docker containers having databases. Do you have to create exports for all docker containers that have some form of database?
I’ve spent the last few days moving all my docker containers to a dedicated machine. I was using a mix of NFS and local storage before, but now I am doing everything on local NVME. My original plan was having everything on NFS so I would worry about backups there, and I might go back to that.
Don’t backup the container!!
Map volumes with your data to physical storage and then simply backup those folders with the rest of your data. Docker containers are already either backed up in your development directory (if you wrote them) or GitHub so like the operating system itself, no need tk backup anything. The whole idea of Docker is the containers are ephemeral. They are reset at every reboot.
This is the only correct answer.
Containers are ephemeral and stateless. If you’re not mounting a volume, think of what happens if the container dies during your process for exporting the data. A failure mode not possible if you mount.
I have my config and data volumes mounted to a share on truenas, that share replicates its snapshots to another truenas server. This is likely not ideal for everyone, but it works for me. My friend that also uses docker has it backed up with duplicati.
Can’t you run a restic container where you mount everything? If the restic container is insecure, everything is of course.
But yes, I also migrated to rootless Podman for this reason and a bunch of others.
I have a backup-script (running as root) that
- stops all containers
- runs the restic-backup
- starts all containers
Have a look at https://github.com/minituff/nautical-backup, it does a similar thing
A simpler method would be to stop/start the Docker daemon instead of containers, it works smoothly.
I had this before but this created struggles with some containers since they do start specific checks and scans during startup which resulted in high cpu and disk load.
Since unraid supports zfs, i am using this for the docker stuff and do snapshots to external disk as backup
no need to stop containers anymorw
If you work with databases it’s still safer to stop incoming data for the time of the backup. I don’t know why a higher CPU load would be a problem, those checks don’t run long or do so much your system would be under much stress. Do your backups at 3am if you still think the minute of highe load would cause any problems.
Don’t run docker as root, don’t run containers as root, pretty simple to be honest.
KISS method: Script that copy the data on the fly to the /tmp dir, compress it, encrypt it and move it to destination using rclone. Running every hour, 4 hours or 24 hours, depending the container.
Never fails. The backups nor the restore.
I mean, snapshotting and piping it to an rclone mount is arguably simpler than trying to do your own ad hoc file syncronization, also does not require 2x the storage space.
I do this at the file system level, not the file level, using zfs.
Unless the container has a database, I use zfs snapshots. If it has a database, my script dumps the database first and then does a ZFS snapshot. Then that snapshot is sent via sanoid to a zfs disk that is in a different backup pool.
This is a block level backup, so it only backs up the actual data blocks that changed.
ZFS snapshots combined with replication to another box. That and a cronjob on packaging up my compose/config files.
Try to not run containers as root?
All Docker containers have their persistent data in Docker volumes located on a BTRFS mount. A cronjob takes a snapshot of the BTRFS volume, then calls
btrfs send, pipes that throughtarandgpgand then directly to AWS S3.Great idea. I already do something similar(minus the btrfs part) for Vaultwarden. Mind sharing the script/commands?
I setup my host with btrfs but I have 0 knowledge of it so I didn’t take advantage of it until now. I already have my docker volumes mapped to /docker/stack so I’m gonna create a sub-volume and move that there.
I’m mostly interest in your btrfs snapshot and send commands but if you don’t mind sharing the whole thing that would be great.
This is one reason I prefer FreeBSD jails. They are each in a separate ZFS filesystem, with a separate filesystem for configuration files. So all I have to do is regular snapshots and send those to a backup pool.
i don’t, i created a docker VM (and a couple others) and then i backup the VMs (proxmox + PBS make this very easy) with all their data in /home/docker/config/*
i used to have them run off networked storage but i found it to be to slow/have other issues
this also means for the primary important services that VM runs in HA and moves to another node when needed
sudo crontab -e
I use a Synology Nas with Active backup for business and back up the VMs that host the containers.
only backup of the data i need to backup (mapped volumes).
Restore: create fresh container and map volumes again.