It’s 2019, and ransomware has become a thing.
Systems that interact with the public, like companies, educational institutions, and public services, are most susceptible. While delivery methods for ransomware vary from the physical realm to communication via social sites and email, all methods only require one person to make one mistake in order for ransomware to proliferate.
Ransomware, as you may have heard, is a malicious program that encrypts your files, rendering them unreadable and useless to you. It can include instructions for paying a ransom, usually by sending cryptocurrency, in order to obtain the decryption key. Successful ransomware attacks typically exploit vital, time-sensitive systems. Victims like public services and medical facilities are more likely to have poor or zero recovery processes, leaving governments or insurance providers to reward attackers with ransom payments.
Individuals, especially less-than-tech-savvy ones, are no less at risk. Ransomware can occlude personal documents and family photos that may only exist on one machine.
Thankfully, a fairly low-tech solution exists for rendering ransomware inept: back up your data!
You could achieve this with a straightforward system like plugging in an external hard drive and dragging files over once a day, but this method has a few hurdles. Manually transferring files may be slow or incomplete, and besides, you’ll first have to remember to do it.
In my constant pursuit of automating all the things, there’s one tool I often return to for its simplicity and reliability: cron. Cron does one thing, and does it well: it runs commands on a schedule.
I first used it a few months shy of three years ago (Have I really been blogging that long?!) to create custom desktop notifications on Linux. Using the crontab configuration file, which you can edit by running crontab -e, you can specify a schedule for running any commands you like. Here’s what the scheduling syntax looks like, from the Wikipedia cron page:
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6)
# │ │ │ │ │
# │ │ │ │ │
# │ │ │ │ │
# * * * * * command to execute
For example, a cron job that runs every day at 00:00 would look like:
0 0 * * *
To run a job every twelve hours, the syntax is:
0 */12 * * *
This great tool can help you wrap your head around the cron scheduling syntax.
What’s a scheduler have to do with backing up? By itself, not much. The simple beauty of cron is that it runs commands - any shell commands, and any scripts that you’d normally run on the command line. As you may have gleaned from my other posts, I’m of the strong opinion that you can do just about anything on the command line, including backing up your files. Options for storage in this area are plentiful, from near-to-free local and cloud options, as well as paid managed services too numerous to list. For CLI tooling, we have utilitarian classics like rsync, and CLI tools for specific cloud providers like AWS.
Backing up with rsync
The rsync utility is a classic choice, and can back up your files to an external hard drive or remote server while making intelligent determinations about which files to update. It uses file size and modification times to recognize file changes, and then only transfers changed files, saving time and bandwidth.
The rsync syntax can be a little nuanced; for example, a trailing forward slash will copy just the contents of the directory, instead of the directory itself. I found examples to be helpful in understanding the usage and syntax.
Here’s one for backing up a local directory to a local destination, such as an external hard drive:
rsync -a /home/user/directory /media/user/destination
The first argument is the source, and the second is the destination. Reversing these in the above example would copy files from the mounted drive to the local home directory.
The a flag for archive mode is one of rsync’s superpowers. Equivalent to flags -rlptgoD, it:
- Syncs files recursively through directories (
r); - Preserves symlinks (
l), permissions (p), modification times (t), groups (g), and owner (o); and - Copies device and special files (
D).
Here’s another example, this time for backing up the contents of a local directory to a directory on a remote server using SSH:
rsync -avze ssh /home/user/directory/ user@remote.host.net:home/user/directory
The v flag turns on verbose output, which is helpful if you like realtime feedback on which files are being transferred. During large transfers, however, it can tend to slow things down. The z flag can help with that, as it indicates that files should be compressed during transfer.
The e flag, followed by ssh, tells rsync to use SSH according to the destination instructions provided in the final argument.
Backing up with AWS CLI
Amazon Web Services offers a command line interface tool for doing just about everything with your AWS set up, including a straightforward s3 sync command for recursively copying new and updated files to your S3 storage buckets. As a storage method for back up data, S3 is a stable and inexpensive choice.
The syntax for interacting with directories is fairly straightforward, and you can directly indicate your S3 bucket as an S3Uri argument in the form of s3://mybucket/mykey. To back up a local directory to your S3 bucket, the command is:
aws s3 sync /home/user/directory s3://mybucket
Similar to rsync, reversing the source and destination would download files from the S3 bucket.
The sync command is intuitive by default. It will guess the mime type of uploaded files, as well as include files discovered by following symlinks. A variety of options exist to control these and other defaults, even including flags to specify the server-side encryption to be used.
Setting up your cronjob back up
You can edit your machine’s cron file by running:
crontab -e
Intuitive as it may be, it’s worth mentioning that your back up commands will only run when your computer is turned on and the cron daemon is running. With this in mind, choose a schedule for your cronjob that aligns with times when your machine is powered on, and maybe not overloaded with other work.
To back up to an S3 bucket every day at 8AM, for example, you’d put a line in your crontab that looks like:
0 8 * * * aws s3 sync /home/user/directory s3://mybucket
If you’re curious whether your cron job is currently running, find the PID of cron with:
pstree -ap | grep cron
Then run pstree -ap <PID>.
This rabbit hole goes deeper; a quick search can reveal different ways of organizing and scheduling cronjobs, or help you find different utilities to run cronjobs when your computer is asleep. However, this basic set up is all you really need to create a reliable, automatic back up system.
Don’t feed the trolls
Humans are fallible; that’s why cyberattacks work. The success of a ransomware attack depends on the victim having no choice but to pay up in order to return to business as usual. A highly accessible recent back up undermines attackers who depend on us being unprepared. By blowing away a system and restoring from yesterday’s back up, we may lose a day of progress; ransomers, however, gain nothing at all.
For further resources on ransomware defense for users and organizations, check out CISA’s advice on ransomware.
Top comments (38)
On the cloud side of things, I'm rather fond of rclone. As their own tagline says, it's quite literally rsync for cloud storage, and it supports a huge number of different storage providers (including all the big ones and quite a few smaller ones too).
Great article. Wanted to add that backing up is just one part of the story. Periodic restores must be tried out to ensure that everything works when restored.
Also, we should keep separate weekly and monthly snapshots in case we are slow to detect the ransomware as the cron can overwrite the earlier good backup with the backup tainted by ransomware.
Great points! Thank you!
I got caught up in my cron line and neglected to round out the practice. 😅
The next step in this process is to back up multiple versions of your files using incremental snapshots, which is also pretty easy with rsync. The key is using
cp -al(copy and make hard links) to prepare the latest version of the backup files, and then rsync into that directory. This way, files which have not been changed will only be stored once on the disk.Once that's working, you can add the
--deleteoption to rsync (removing individual files that have been deleted). You can also set up a process to remove older snapshots as needed. This is the kind of scheme I usually set up for my clients.More info about how and why this works: mikerubel.org/computers/rsync_snap...
I have to disagree with the --delete option in this particular use case.
It defeats the purpose of the article.
Ransomware will delete your files replacing them with their crypted version.
Then your backup process with --delete, will ... delete your sane files ...
I don't think it is what you want for your backup process to prevent ransomware ...
Same stuff for you snapshots. Sane backup could vanish ..
In ransomware case, a good indicator can be the percentage variation of changes in files.
I think you misunderstood my comment: you only add
--deleteafter you have snapshots working properly. Then the files only disappear from the latest backup, but they are still present in all previous snapshots.If you set this up correctly - and you understand what it is doing, which is always important - then it is a good system with no risk for data loss.
Great info on rsync! Thanks for that and the link, James!
Oof... Generally would not recommend using S3's sync subcommand. While, yeah, it will get all your files there, it gets them there at the cost of filesystem attributes (ownerships, xattrs, ACLs, SELinux labels, etc.). If you care about those types of things, then you want to encapsulate your filesystem data in something that's filesystem aware. Fortunately, the AWS S3 CLI does allow you to do piped inputs ...meaning you can use your filesystem's native incremental backup utility,
dumpto store incremental, attribute-aware backups of your data.Only down side is, if you don't know (roughly) how big your dump is going to be, the S3 CLI will tend to single-stream your data. For a day's worth of activity, this is probably OK, but if you're doing a full backup, it can make things painfully slow (particularly if backing up over links that have session-limits).
what happens when I am infected? Wouldn't rsync or equivalent simply sync the encrypted files to the cloud service?
Yup. Someone else pointed out that having separated, incremental backups gives you the ability to revert to an older, non-compromised one, should that happen. (Like a Git commit.)
Even better for gaurding against ransomware, a differential backup.
Run an
rdiff-backupin your cron and get daily/hourly snapshots so you can recover to a point in time before the encryption.Simly backing up (even using rsync) will backup the encrypted files over the good ones... no use at all.
Syncing files on a schedule will not protect against ransomeware unless it also maintains a history. I highly recommend duplicacy (CLI) for backups with history and de-duplication. It is not the simplest tools to use, but it is very powerful.
We replaced our rsync / rdiff-backup backups with duplicacy.
Another option for back-up is
backintime. It's a rdiff based backup solution which creates Apple's Time Machine style backups.It's and extremely efficient and sophisticated tool. Can work over network if desired. Can do backup merging, can store
nrevisions in multiple ways, can keep disk from completely filling up, useniceandionice, etc.Another nice feature is its profiles. I have some profiles with different settings with different source folders. Hence I got my backups the way I want.
It can work with or without the GUI, so it's worth checking up.
Great article 👏.
I guess I should start backing up my data more frequently.
If you're Windows or Mac, it's hard to beat Backblaze. Continuous backups with near-zero setup. Even better, you pay per backed-up system rather than by-the-byte. And, unlike some other cloud backup services, you can fine-tune how much bandwidth a given system's backups are allowed to consume.
Thanks a lot for the suggestion. I'm a Linux user though 😬. Also I would like to setup my own backup server at home instead of using a service.
Sounds like an
rsyncjob to me! Good luck with the project! Maybe you can write about how it went when you’re up and running!I used to do everything (quite literally) "in house". Then my electric bills got to be stoopid-big. Switched to using cloud-services. Also means I no longer worry about power outages, basement getting flooded (and my rack submerged), remote accessibility (and protecting the home network from getting owned via an exposed service/port), etc.
From a TCO price-point and reliability standpoint, using BackBlaze's B2 service with something like Restic is hard to beat for keeping your Linux system backed up. About the only thing that might be more cost effective (haven't looked, recently, would be one of the CSP's archival-oriented service – like AWS's Glacier offering).
If Server A is compromised, then any other servers or shares it has write access to may also be compromised. Your backup target should support snapshots, or should back itself up incrementally to another target that Server A doesn't have access to.
It's an extremely oft overlooked problem in backup strategies. The one time you get bit or see someone get bit, and their onsite and offsite backups are all encrypted, you'll die a little bit inside.
Great article! Very pedagogic! With that said, this approach could very well make you copy the ransomware encryption agent as well, spreading it to yet another host.
A safer approach would be to put the data you’re syncing in some kind of archive or similar before transferring it to make sure that the risk of accidentally retriggering the agent is as low as possible.
If the file is encrypted by ransomware, wouldn't this command back-up an encrypted file as well overwriting the good one?
A very good point and depending on what and how your method of backing up works, a very general method would be to have multiple backups. For example, traditional tape backup solutions followed daily/weekly/monthly (incremental/full) etc. routines, so you can restore from a last-known good backup i.e. before the file(s) were affected.
I'm a bit out of the loop on what's what in backup technology, but various tape/disk-based solutions etc. offer a myriad of options that would allow for point-in-time restores.
So I'm surprised no one mentioned just using the automated backup features of AWS defining a Lifecycle Manager policy. EC2 service will do periodic backups for you and store those for 15, 30, 45 days, etc. What I'm not sure about is why do this solution if you're running on EC2 when you can do this without modifying your instance through lifecycle policy snapshots?
Amazon has stressed trying to treat your EC2 instances like cattle and not like pets. Cattle are replaceable vs pets are uniquely special. By adding custom cron jobs to your EC2 instance when something dramatic happens to it you have to rebuild it back to its special pet-like state vs if you lost a box you just spin up a new image and recover it. I would consider this advice, while coming from a good place, is advocating for pet-like treatment.
Personally, I run my company's main server and I use Ubuntu Linux with ZFS on Linux filesystem to defend against ransomware. While ransomware "probably" doesn't run on Linux, it could, of course encrypt data on our Linux server which is served to our engineers' Windows machines. I address this threat by:
People have mentioned incremental backups. This is critical, and not only for ransomware!
Likewise, it is critical to also have off-site backups — otherwise, if your building suffers a disaster (fire, theft, earthquake, …) that destroys all of your data including all of your backups, you'll lose everything!
For on-site backups, I use rdiff-backup, which can be tailored to keep incremental backups for as long as you want. (You might know of an easier package than rdiff-backup.) These backups are important for easy access. Storing your backups on at least three different physical media is excellent practice. Of course, all of the media devices should be fully encrypted!
For off-site incremental backups, I use an online service. (I use SpiderOakONE because it's multi-platform and affordable, but I'm sure that there are other providers.)