It’s 2019, and ransomware has become a thing.
Systems that interact with the public, like companies, educational institutions, and public services...
For further actions, you may consider blocking this person and/or reporting abuse
On the cloud side of things, I'm rather fond of rclone. As their own tagline says, it's quite literally rsync for cloud storage, and it supports a huge number of different storage providers (including all the big ones and quite a few smaller ones too).
Great article. Wanted to add that backing up is just one part of the story. Periodic restores must be tried out to ensure that everything works when restored.
Also, we should keep separate weekly and monthly snapshots in case we are slow to detect the ransomware as the cron can overwrite the earlier good backup with the backup tainted by ransomware.
Great points! Thank you!
I got caught up in my cron line and neglected to round out the practice. 😅
The next step in this process is to back up multiple versions of your files using incremental snapshots, which is also pretty easy with rsync. The key is using
cp -al(copy and make hard links) to prepare the latest version of the backup files, and then rsync into that directory. This way, files which have not been changed will only be stored once on the disk.Once that's working, you can add the
--deleteoption to rsync (removing individual files that have been deleted). You can also set up a process to remove older snapshots as needed. This is the kind of scheme I usually set up for my clients.More info about how and why this works: mikerubel.org/computers/rsync_snap...
I have to disagree with the --delete option in this particular use case.
It defeats the purpose of the article.
Ransomware will delete your files replacing them with their crypted version.
Then your backup process with --delete, will ... delete your sane files ...
I don't think it is what you want for your backup process to prevent ransomware ...
Same stuff for you snapshots. Sane backup could vanish ..
In ransomware case, a good indicator can be the percentage variation of changes in files.
I think you misunderstood my comment: you only add
--deleteafter you have snapshots working properly. Then the files only disappear from the latest backup, but they are still present in all previous snapshots.If you set this up correctly - and you understand what it is doing, which is always important - then it is a good system with no risk for data loss.
Great info on rsync! Thanks for that and the link, James!
Oof... Generally would not recommend using S3's sync subcommand. While, yeah, it will get all your files there, it gets them there at the cost of filesystem attributes (ownerships, xattrs, ACLs, SELinux labels, etc.). If you care about those types of things, then you want to encapsulate your filesystem data in something that's filesystem aware. Fortunately, the AWS S3 CLI does allow you to do piped inputs ...meaning you can use your filesystem's native incremental backup utility,
dumpto store incremental, attribute-aware backups of your data.Only down side is, if you don't know (roughly) how big your dump is going to be, the S3 CLI will tend to single-stream your data. For a day's worth of activity, this is probably OK, but if you're doing a full backup, it can make things painfully slow (particularly if backing up over links that have session-limits).
what happens when I am infected? Wouldn't rsync or equivalent simply sync the encrypted files to the cloud service?
Yup. Someone else pointed out that having separated, incremental backups gives you the ability to revert to an older, non-compromised one, should that happen. (Like a Git commit.)
Even better for gaurding against ransomware, a differential backup.
Run an
rdiff-backupin your cron and get daily/hourly snapshots so you can recover to a point in time before the encryption.Simly backing up (even using rsync) will backup the encrypted files over the good ones... no use at all.
Syncing files on a schedule will not protect against ransomeware unless it also maintains a history. I highly recommend duplicacy (CLI) for backups with history and de-duplication. It is not the simplest tools to use, but it is very powerful.
We replaced our rsync / rdiff-backup backups with duplicacy.
Another option for back-up is
backintime. It's a rdiff based backup solution which creates Apple's Time Machine style backups.It's and extremely efficient and sophisticated tool. Can work over network if desired. Can do backup merging, can store
nrevisions in multiple ways, can keep disk from completely filling up, useniceandionice, etc.Another nice feature is its profiles. I have some profiles with different settings with different source folders. Hence I got my backups the way I want.
It can work with or without the GUI, so it's worth checking up.
Great article 👏.
I guess I should start backing up my data more frequently.
If you're Windows or Mac, it's hard to beat Backblaze. Continuous backups with near-zero setup. Even better, you pay per backed-up system rather than by-the-byte. And, unlike some other cloud backup services, you can fine-tune how much bandwidth a given system's backups are allowed to consume.
Thanks a lot for the suggestion. I'm a Linux user though 😬. Also I would like to setup my own backup server at home instead of using a service.
Sounds like an
rsyncjob to me! Good luck with the project! Maybe you can write about how it went when you’re up and running!I used to do everything (quite literally) "in house". Then my electric bills got to be stoopid-big. Switched to using cloud-services. Also means I no longer worry about power outages, basement getting flooded (and my rack submerged), remote accessibility (and protecting the home network from getting owned via an exposed service/port), etc.
From a TCO price-point and reliability standpoint, using BackBlaze's B2 service with something like Restic is hard to beat for keeping your Linux system backed up. About the only thing that might be more cost effective (haven't looked, recently, would be one of the CSP's archival-oriented service – like AWS's Glacier offering).
If Server A is compromised, then any other servers or shares it has write access to may also be compromised. Your backup target should support snapshots, or should back itself up incrementally to another target that Server A doesn't have access to.
It's an extremely oft overlooked problem in backup strategies. The one time you get bit or see someone get bit, and their onsite and offsite backups are all encrypted, you'll die a little bit inside.
Great article! Very pedagogic! With that said, this approach could very well make you copy the ransomware encryption agent as well, spreading it to yet another host.
A safer approach would be to put the data you’re syncing in some kind of archive or similar before transferring it to make sure that the risk of accidentally retriggering the agent is as low as possible.
If the file is encrypted by ransomware, wouldn't this command back-up an encrypted file as well overwriting the good one?
A very good point and depending on what and how your method of backing up works, a very general method would be to have multiple backups. For example, traditional tape backup solutions followed daily/weekly/monthly (incremental/full) etc. routines, so you can restore from a last-known good backup i.e. before the file(s) were affected.
I'm a bit out of the loop on what's what in backup technology, but various tape/disk-based solutions etc. offer a myriad of options that would allow for point-in-time restores.
So I'm surprised no one mentioned just using the automated backup features of AWS defining a Lifecycle Manager policy. EC2 service will do periodic backups for you and store those for 15, 30, 45 days, etc. What I'm not sure about is why do this solution if you're running on EC2 when you can do this without modifying your instance through lifecycle policy snapshots?
Amazon has stressed trying to treat your EC2 instances like cattle and not like pets. Cattle are replaceable vs pets are uniquely special. By adding custom cron jobs to your EC2 instance when something dramatic happens to it you have to rebuild it back to its special pet-like state vs if you lost a box you just spin up a new image and recover it. I would consider this advice, while coming from a good place, is advocating for pet-like treatment.
Personally, I run my company's main server and I use Ubuntu Linux with ZFS on Linux filesystem to defend against ransomware. While ransomware "probably" doesn't run on Linux, it could, of course encrypt data on our Linux server which is served to our engineers' Windows machines. I address this threat by:
People have mentioned incremental backups. This is critical, and not only for ransomware!
Likewise, it is critical to also have off-site backups — otherwise, if your building suffers a disaster (fire, theft, earthquake, …) that destroys all of your data including all of your backups, you'll lose everything!
For on-site backups, I use rdiff-backup, which can be tailored to keep incremental backups for as long as you want. (You might know of an easier package than rdiff-backup.) These backups are important for easy access. Storing your backups on at least three different physical media is excellent practice. Of course, all of the media devices should be fully encrypted!
For off-site incremental backups, I use an online service. (I use SpiderOakONE because it's multi-platform and affordable, but I'm sure that there are other providers.)
Or simply use any filesystem with automatic history?
How does this protect you in any fashion if the entire filesystem is encrypted?
For this I'm not entirely sure, but could a ransomware totally encrypt a ZFS volume? That would mean elevating privileges up to the filesystem driver, which may not be in user-space
ZFS with snapshots on the targeted machine is a great mitigation but not a silver bullet. Snapshots can expire or be overwritten by new encrypted data until good data is lost (depending on the configuration). Or the ransomware attack could be block level, which is hitting a lot of people lately. Or the machine could explode :) I think snapshots should be considered as a way to restore point-in-time data locally, not as a backup per se.
If your backup versions are stored locally, they’re also susceptible to a ransomware lockdown. They’d need to be backed up in a similar fashion.
True, although even locally you can mitigate most of the effect by giving access to each program only to a pseudo-filesystem; your system will be as secure as your administrator access in that case.
Mixing both solution would probably work well, as you can transfer incremental backups from a filesystem history to a remote, then you only have to protect your remotes.
Really helpful article, thanks for describing
rsyncwith examples!I didn't know about this one. As it seems it can be used as a "client" for Backblaze, this includes what I think are the two most interesting tools for incremental backups.
Thank you Victoria for sharing valuable knowledge. Congrats from Brazil.
Since your article points out people with less technical awareness are at more risk (doesn't mean the computer literate are immune though 😁) - FreeFileSync is a UI based tool everyone could use to backup data from their devices to both local storage & cloud.
It's cross platform, open source, easily configurable (like an intelligent rsync with a front-end) & very reliable.
Here is a place to begin with it.
That’s interesting. Can you elaborate on what you’d use to achieve this set up? A remote with a job to SSH to the working machine? How would you go about creating the alert?
In another scenario, Objective-See for macOS has Ramsonwhere which detects background activities that tried to encrypt or zip in real-time, which you can halt before it's too late.
I'm lazy, so I just use Backblaze.
Could you share what things your automate? I'm in an automation spree!