loading...
Cover image for A cron job that could save you from a ransomware attack

A cron job that could save you from a ransomware attack

victoria profile image Victoria Drake Originally published at victoria.dev on ・5 min read

It’s 2019, and ransomware has become a thing.

Systems that interact with the public, like companies, educational institutions, and public services, are most susceptible. While delivery methods for ransomware vary from the physical realm to communication via social sites and email, all methods only require one person to make one mistake in order for ransomware to proliferate.

Ransomware, as you may have heard, is a malicious program that encrypts your files, rendering them unreadable and useless to you. It can include instructions for paying a ransom, usually by sending cryptocurrency, in order to obtain the decryption key. Successful ransomware attacks typically exploit vital, time-sensitive systems. Victims like public services and medical facilities are more likely to have poor or zero recovery processes, leaving governments or insurance providers to reward attackers with ransom payments.

Ransom note

Individuals, especially less-than-tech-savvy ones, are no less at risk. Ransomware can occlude personal documents and family photos that may only exist on one machine.

Thankfully, a fairly low-tech solution exists for rendering ransomware inept: back up your data!

You could achieve this with a straightforward system like plugging in an external hard drive and dragging files over once a day, but this method has a few hurdles. Manually transferring files may be slow or incomplete, and besides, you’ll first have to remember to do it.

In my constant pursuit of automating all the things, there’s one tool I often return to for its simplicity and reliability: cron. Cron does one thing, and does it well: it runs commands on a schedule.

I first used it a few months shy of three years ago (Have I really been blogging that long?!) to create custom desktop notifications on Linux. Using the crontab configuration file, which you can edit by running crontab -e, you can specify a schedule for running any commands you like. Here’s what the scheduling syntax looks like, from the Wikipedia cron page:

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6)
# │ │ │ │ │
# │ │ │ │ │
# │ │ │ │ │
# * * * * * command to execute

For example, a cron job that runs every day at 00:00 would look like:

0 0 * * *

To run a job every twelve hours, the syntax is:

0 */12 * * *

This great tool can help you wrap your head around the cron scheduling syntax.

What’s a scheduler have to do with backing up? By itself, not much. The simple beauty of cron is that it runs commands - any shell commands, and any scripts that you’d normally run on the command line. As you may have gleaned from my other posts, I’m of the strong opinion that you can do just about anything on the command line, including backing up your files. Options for storage in this area are plentiful, from near-to-free local and cloud options, as well as paid managed services too numerous to list. For CLI tooling, we have utilitarian classics like rsync, and CLI tools for specific cloud providers like AWS.

Backing up with rsync

The rsync utility is a classic choice, and can back up your files to an external hard drive or remote server while making intelligent determinations about which files to update. It uses file size and modification times to recognize file changes, and then only transfers changed files, saving time and bandwidth.

The rsync syntax can be a little nuanced; for example, a trailing forward slash will copy just the contents of the directory, instead of the directory itself. I found examples to be helpful in understanding the usage and syntax.

Here’s one for backing up a local directory to a local destination, such as an external hard drive:

rsync -a /home/user/directory /media/user/destination

The first argument is the source, and the second is the destination. Reversing these in the above example would copy files from the mounted drive to the local home directory.

The a flag for archive mode is one of rsync’s superpowers. Equivalent to flags -rlptgoD, it:

  • Syncs files recursively through directories (r);
  • Preserves symlinks (l), permissions (p), modification times (t), groups (g), and owner (o); and
  • Copies device and special files (D).

Here’s another example, this time for backing up the contents of a local directory to a directory on a remote server using SSH:

rsync -avze ssh /home/user/directory/ user@remote.host.net:home/user/directory

The v flag turns on verbose output, which is helpful if you like realtime feedback on which files are being transferred. During large transfers, however, it can tend to slow things down. The z flag can help with that, as it indicates that files should be compressed during transfer.

The e flag, followed by ssh, tells rsync to use SSH according to the destination instructions provided in the final argument.

Backing up with AWS CLI

Amazon Web Services offers a command line interface tool for doing just about everything with your AWS set up, including a straightforward s3 sync command for recursively copying new and updated files to your S3 storage buckets. As a storage method for back up data, S3 is a stable and inexpensive choice.

The syntax for interacting with directories is fairly straightforward, and you can directly indicate your S3 bucket as an S3Uri argument in the form of s3://mybucket/mykey. To back up a local directory to your S3 bucket, the command is:

aws s3 sync /home/user/directory s3://mybucket

Similar to rsync, reversing the source and destination would download files from the S3 bucket.

The sync command is intuitive by default. It will guess the mime type of uploaded files, as well as include files discovered by following symlinks. A variety of options exist to control these and other defaults, even including flags to specify the server-side encryption to be used.

Setting up your cronjob back up

You can edit your machine’s cron file by running:

crontab -e

Intuitive as it may be, it’s worth mentioning that your back up commands will only run when your computer is turned on and the cron daemon is running. With this in mind, choose a schedule for your cronjob that aligns with times when your machine is powered on, and maybe not overloaded with other work.

To back up to an S3 bucket every day at 8AM, for example, you’d put a line in your crontab that looks like:

0 8 * * * aws s3 sync /home/user/directory s3://mybucket

If you’re curious whether your cron job is currently running, find the PID of cron with:

pstree -ap | grep cron

Then run pstree -ap <PID>.

This rabbit hole goes deeper; a quick search can reveal different ways of organizing and scheduling cronjobs, or help you find different utilities to run cronjobs when your computer is asleep. However, this basic set up is all you really need to create a reliable, automatic back up system.

Don’t feed the trolls

Humans are fallible; that’s why cyberattacks work. The success of a ransomware attack depends on the victim having no choice but to pay up in order to return to business as usual. A highly accessible recent back up undermines attackers who depend on us being unprepared. By blowing away a system and restoring from yesterday’s back up, we may lose a day of progress; ransomers, however, gain nothing at all.

For further resources on ransomware defense for users and organizations, check out CISA’s advice on ransomware.

Posted on by:

victoria profile

Victoria Drake

@victoria

Senior software engineer, currently Director of Engineering. 💜 Cybersecurity, CI/CD. Co-author, core maintainer at OWASP Web Security Testing Guide. 👉 https://victoria.dev

Discussion

markdown guide
 

On the cloud side of things, I'm rather fond of rclone. As their own tagline says, it's quite literally rsync for cloud storage, and it supports a huge number of different storage providers (including all the big ones and quite a few smaller ones too).

 

Great article. Wanted to add that backing up is just one part of the story. Periodic restores must be tried out to ensure that everything works when restored.

Also, we should keep separate weekly and monthly snapshots in case we are slow to detect the ransomware as the cron can overwrite the earlier good backup with the backup tainted by ransomware.

 

Great points! Thank you!

I got caught up in my cron line and neglected to round out the practice. 😅

 

I'd rather use something like restic, which provides incremental ciphered snapshots for backups, and supports lots of cloud appliances (including S3).

Better reliability than risking having everything encrypted public.

 

I didn't know about this one. As it seems it can be used as a "client" for Backblaze, this includes what I think are the two most interesting tools for incremental backups.

 

Indeed. It's a very fine piece of software that is easy to automate.

 

The next step in this process is to back up multiple versions of your files using incremental snapshots, which is also pretty easy with rsync. The key is using cp -al (copy and make hard links) to prepare the latest version of the backup files, and then rsync into that directory. This way, files which have not been changed will only be stored once on the disk.

Once that's working, you can add the --delete option to rsync (removing individual files that have been deleted). You can also set up a process to remove older snapshots as needed. This is the kind of scheme I usually set up for my clients.

More info about how and why this works: mikerubel.org/computers/rsync_snap...

 

I have to disagree with the --delete option in this particular use case.
It defeats the purpose of the article.

Ransomware will delete your files replacing them with their crypted version.

Then your backup process with --delete, will ... delete your sane files ...
I don't think it is what you want for your backup process to prevent ransomware ...
Same stuff for you snapshots. Sane backup could vanish ..

In ransomware case, a good indicator can be the percentage variation of changes in files.

 

I think you misunderstood my comment: you only add --delete after you have snapshots working properly. Then the files only disappear from the latest backup, but they are still present in all previous snapshots.

If you set this up correctly - and you understand what it is doing, which is always important - then it is a good system with no risk for data loss.

 

Great info on rsync! Thanks for that and the link, James!

 

Oof... Generally would not recommend using S3's sync subcommand. While, yeah, it will get all your files there, it gets them there at the cost of filesystem attributes (ownerships, xattrs, ACLs, SELinux labels, etc.). If you care about those types of things, then you want to encapsulate your filesystem data in something that's filesystem aware. Fortunately, the AWS S3 CLI does allow you to do piped inputs ...meaning you can use your filesystem's native incremental backup utility, dump to store incremental, attribute-aware backups of your data.

Only down side is, if you don't know (roughly) how big your dump is going to be, the S3 CLI will tend to single-stream your data. For a day's worth of activity, this is probably OK, but if you're doing a full backup, it can make things painfully slow (particularly if backing up over links that have session-limits).

 

what happens when I am infected? Wouldn't rsync or equivalent simply sync the encrypted files to the cloud service?

 

Yup. Someone else pointed out that having separated, incremental backups gives you the ability to revert to an older, non-compromised one, should that happen. (Like a Git commit.)

 

Just my 2¢: As a rule of thumb you always want to pull and not push when doing backups. What that means is that you want your protected server doing the connecting part and downloading the files into the machine, and, here's the benefit: alerting if something is going wrong with the 'backuped' server. If there's any problem with your cronjob you would never know until maybe it is too late.

 

That’s interesting. Can you elaborate on what you’d use to achieve this set up? A remote with a job to SSH to the working machine? How would you go about creating the alert?

 

Yep, that is a simple and safe solution. The alert system could be based on the same crontab error email or any other way of proactively checking if everything is going fine. Another issue with going with the push style (that I forgot to mention on my previous comment) is that usually it is based on mount points and, what that leads to is that if you are infected with a ransomware all your backups are going to be encrypted as well.

 

Even better for gaurding against ransomware, a differential backup.
Run an rdiff-backup in your cron and get daily/hourly snapshots so you can recover to a point in time before the encryption.
Simly backing up (even using rsync) will backup the encrypted files over the good ones... no use at all.

 

Great article 👏.
I guess I should start backing up my data more frequently.

 

If you're Windows or Mac, it's hard to beat Backblaze. Continuous backups with near-zero setup. Even better, you pay per backed-up system rather than by-the-byte. And, unlike some other cloud backup services, you can fine-tune how much bandwidth a given system's backups are allowed to consume.

 

Thanks a lot for the suggestion. I'm a Linux user though 😬. Also I would like to setup my own backup server at home instead of using a service.

Sounds like an rsync job to me! Good luck with the project! Maybe you can write about how it went when you’re up and running!

I used to do everything (quite literally) "in house". Then my electric bills got to be stoopid-big. Switched to using cloud-services. Also means I no longer worry about power outages, basement getting flooded (and my rack submerged), remote accessibility (and protecting the home network from getting owned via an exposed service/port), etc.

From a TCO price-point and reliability standpoint, using BackBlaze's B2 service with something like Restic is hard to beat for keeping your Linux system backed up. About the only thing that might be more cost effective (haven't looked, recently, would be one of the CSP's archival-oriented service – like AWS's Glacier offering).

 

Syncing files on a schedule will not protect against ransomeware unless it also maintains a history. I highly recommend duplicacy (CLI) for backups with history and de-duplication. It is not the simplest tools to use, but it is very powerful.

We replaced our rsync / rdiff-backup backups with duplicacy.

 

Another option for back-up is backintime. It's a rdiff based backup solution which creates Apple's Time Machine style backups.

It's and extremely efficient and sophisticated tool. Can work over network if desired. Can do backup merging, can store n revisions in multiple ways, can keep disk from completely filling up, use nice and ionice, etc.

Another nice feature is its profiles. I have some profiles with different settings with different source folders. Hence I got my backups the way I want.

It can work with or without the GUI, so it's worth checking up.

 

If the file is encrypted by ransomware, wouldn't this command back-up an encrypted file as well overwriting the good one?

 

A very good point and depending on what and how your method of backing up works, a very general method would be to have multiple backups. For example, traditional tape backup solutions followed daily/weekly/monthly (incremental/full) etc. routines, so you can restore from a last-known good backup i.e. before the file(s) were affected.

I'm a bit out of the loop on what's what in backup technology, but various tape/disk-based solutions etc. offer a myriad of options that would allow for point-in-time restores.

 

If Server A is compromised, then any other servers or shares it has write access to may also be compromised. Your backup target should support snapshots, or should back itself up incrementally to another target that Server A doesn't have access to.

It's an extremely oft overlooked problem in backup strategies. The one time you get bit or see someone get bit, and their onsite and offsite backups are all encrypted, you'll die a little bit inside.

 

Great article! Very pedagogic! With that said, this approach could very well make you copy the ransomware encryption agent as well, spreading it to yet another host.

A safer approach would be to put the data you’re syncing in some kind of archive or similar before transferring it to make sure that the risk of accidentally retriggering the agent is as low as possible.

 

So I'm surprised no one mentioned just using the automated backup features of AWS defining a Lifecycle Manager policy. EC2 service will do periodic backups for you and store those for 15, 30, 45 days, etc. What I'm not sure about is why do this solution if you're running on EC2 when you can do this without modifying your instance through lifecycle policy snapshots?

Amazon has stressed trying to treat your EC2 instances like cattle and not like pets. Cattle are replaceable vs pets are uniquely special. By adding custom cron jobs to your EC2 instance when something dramatic happens to it you have to rebuild it back to its special pet-like state vs if you lost a box you just spin up a new image and recover it. I would consider this advice, while coming from a good place, is advocating for pet-like treatment.

 

Personally, I run my company's main server and I use Ubuntu Linux with ZFS on Linux filesystem to defend against ransomware. While ransomware "probably" doesn't run on Linux, it could, of course encrypt data on our Linux server which is served to our engineers' Windows machines. I address this threat by:

  1. Having automated snapshots running on ZFS. These are read-only incremental backups of our filesystem.
  2. Sending those snapshots to an external hard drive(s) for physical backup. Unless the ransomware knows ZFS AND can escalate to root privileges, the threat of this malware to ZFS data is minimal. Merely making backups can still expose you to risk because if the ransomware can access your backup drives, it will encrypt those too. So you would need to ensure that the backups are accessible as long as you're performing backups AND the mere act of performing a backup on such systems (without read-only snapshots) opens a window of vulnerability during the backup process - which can take hours. Moreover, I have our system set to perform backups every 15min - so you would lose at most, about 30min of work should ransomware strike - not a full day lost. Best
 

People have mentioned incremental backups. This is critical, and not only for ransomware!

Likewise, it is critical to also have off-site backups — otherwise, if your building suffers a disaster (fire, theft, earthquake, …) that destroys all of your data including all of your backups, you'll lose everything!

For on-site backups, I use rdiff-backup, which can be tailored to keep incremental backups for as long as you want. (You might know of an easier package than rdiff-backup.) These backups are important for easy access. Storing your backups on at least three different physical media is excellent practice. Of course, all of the media devices should be fully encrypted!

For off-site incremental backups, I use an online service. (I use SpiderOakONE because it's multi-platform and affordable, but I'm sure that there are other providers.)

 

Or simply use any filesystem with automatic history?

 

How does this protect you in any fashion if the entire filesystem is encrypted?

 

For this I'm not entirely sure, but could a ransomware totally encrypt a ZFS volume? That would mean elevating privileges up to the filesystem driver, which may not be in user-space

ZFS with snapshots on the targeted machine is a great mitigation but not a silver bullet. Snapshots can expire or be overwritten by new encrypted data until good data is lost (depending on the configuration). Or the ransomware attack could be block level, which is hitting a lot of people lately. Or the machine could explode :) I think snapshots should be considered as a way to restore point-in-time data locally, not as a backup per se.

 

If your backup versions are stored locally, they’re also susceptible to a ransomware lockdown. They’d need to be backed up in a similar fashion.

 

True, although even locally you can mitigate most of the effect by giving access to each program only to a pseudo-filesystem; your system will be as secure as your administrator access in that case.

Mixing both solution would probably work well, as you can transfer incremental backups from a filesystem history to a remote, then you only have to protect your remotes.

 

Since your article points out people with less technical awareness are at more risk (doesn't mean the computer literate are immune though 😁) - FreeFileSync is a UI based tool everyone could use to backup data from their devices to both local storage & cloud.

It's cross platform, open source, easily configurable (like an intelligent rsync with a front-end) & very reliable.

Here is a place to begin with it.

 

Really helpful article, thanks for describing rsync with examples!

 

Thank you Victoria for sharing valuable knowledge. Congrats from Brazil.

 

In another scenario, Objective-See for macOS has Ramsonwhere which detects background activities that tried to encrypt or zip in real-time, which you can halt before it's too late.

 

I'm lazy, so I just use Backblaze.

Could you share what things your automate? I'm in an automation spree!