Historically I've backed it up using some home grown scripts around rsync to a server at my house. But for various reasons that server is not running at the moment, and so I was looking for some other way to do backups.
My criteria is:
- All data encrypted before leaving my server
- Efficient incremental backups
- Storage on an open platform, not locked away in some proprietary service
- Easy to setup
The fact it uses GnuPG is really nice as it is a public key encryption system as opposed to just a symmetric cipher, so I can put the public key on the server to encrypt the data, but I can keep the private key (to decrypt the data) offline.
Duplicity is in the OpenBSD packages collection as well, which makes it a simple to install:
traci# pkg_add duplicity quirks-3.124 signed on 2019-10-16T20:27:45Z duplicity-0.7.18.2:py-monotonic-1.3: ok duplicity-0.7.18.2:py-fasteners-0.14.1: ok duplicity-0.7.18.2:py-ptyprocess-0.5.1p0: ok duplicity-0.7.18.2:py-pexpect-4.2.1p0: ok duplicity-0.7.18.2:libb2-0.98.1v0: ok duplicity-0.7.18.2:popt-1.16p1: ok duplicity-0.7.18.2:librsync-2.0.2: ok duplicity-0.7.18.2:ncftp-3.2.6: ok duplicity-0.7.18.2:py-PyNaCl-1.3.0: ok duplicity-0.7.18.2:py-bcrypt-3.1.6: ok duplicity-0.7.18.2:py-paramiko-2.4.1: ok duplicity-0.7.18.2:py-yaml-3.13: ok duplicity-0.7.18.2:py-boto-2.49.0: ok duplicity-0.7.18.2:py-lockfile-0.12.2p2: ok duplicity-0.7.18.2: ok
My OpenBSD version is a bit old, hence the version of Duplicity is a bit old (current version as of writing is
0.8.15). But this is one impetus for me to get the backups sorted as I'm going to be upgrading this server soon.
As for storage, I'm using IBM Cloud Object Storage (COS). Which is an S3 compatible object storage and has a nice feature that it will move object to 'colder' (cheaper) storage if they are not accessed for a while. The free tier for COS allows up to 25GB/month storage and 2,000 PUT requests and 20,000 GET requests. So should be enough for my usage. If I do daily backups, it will likely only be putting 1 "volume" up per day. So even after a full backup (which will be several volumes) there should be enough capacity. If I go over I can always upgrade, the prices are pretty cheap.
So first, I needed to create a Cloud Object Storage instance in my IBM Cloud Account. You can find Object Storage in the catalog:
Select it and create your Object Storage instance:
Once created, go into the Object Storage instance and select 'Service Credentials' on the left menu, then 'New Credential' at the top right. Give the service credential a name. These credentials are what are going to allow Duplicity to upload to your COS instance. Make sure you enable "Include HMAC Credential" in order to generate S3 compatible credentials for
Duplicity to use.
Once created, click on the arrow on the left to expand the credential information and copy the
Make note of those keys as you will need them later.
Next you need to setup GnuPG to create a key to encrypt the data before it gets sent to COS.
$ gpg --gen-key gpg (GnuPG) 1.4.23; Copyright (C) 2015 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y You need a user ID to identify your key; the software constructs the user ID from the Real Name, Comment and Email Address in this form: "Heinrich Heine (Der Dichter) <email@example.com>" Real name: Duplicity Backups Email address: firstname.lastname@example.org Comment: You selected this USER-ID: "Duplicity Backups <email@example.com>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O You need a Passphrase to protect your secret key. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. ..+++++ .....+++++ We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. .....+++++ ..........+++++ gpg: /home/matt/.gnupg/trustdb.gpg: trustdb created gpg: key 5492263E marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u pub 2048R/5492263E 2020-08-21 Key fingerprint = 9F5D 33F3 D828 4D58 AF2C E7E0 1C68 23CF 5492 263E uid Duplicity Backups <firstname.lastname@example.org> sub 2048R/824055B1 2020-08-21
Then to backup to COS you set those variables in the environment can call Duplicity, e.g.
export PASSPHRASE="<passphrase from gpg above>" export AWS_ACCESS_KEY_ID="<access key from above>" export AWS_SECRET_ACCESS_KEY="<secret key from above>" /usr/local/bin/duplicity --asynchronous-upload \ --encrypt-key 5492263E \ --include /home --include /etc --exclude '**' \ --allow-source-mismatch \ / s3://s3.eu-gb.cloud-object-storage.appdomain.cloud/hamilton-backups-traci
That will then do a full backup including my
/etc directories and excluding everything else to a bucket called
hamilton-backups-traci on COS. I have this in a shell script that I then call from a cron entry every night.
traci# dobackup-full.sh Local and Remote metadata are synchronized, no sync needed. Last full backup date: Fri Aug 21 17:29:24 2020 --------------[ Backup Statistics ]-------------- StartTime 1598031880.95 (Fri Aug 21 18:44:40 2020) EndTime 1598034104.49 (Fri Aug 21 19:21:44 2020) ElapsedTime 2223.54 (37 minutes 3.54 seconds) SourceFiles 62261 SourceFileSize 8676319669 (8.08 GB) NewFiles 62261 NewFileSize 8676319669 (8.08 GB) DeletedFiles 0 ChangedFiles 0 ChangedFileSize 0 (0 bytes) ChangedDeltaSize 0 (0 bytes) DeltaEntries 62261 RawDeltaSize 8670476162 (8.08 GB) TotalDestinationSizeChange 5919687097 (5.51 GB) Errors 0 -------------------------------------------------
If you look in the console on IBM Cloud you will see the volumes in Cloud Object storage:
IBM Cloud gives you a fair amount of free storage, and the prices are pretty small if you go above that. You can sign up for a free account here