I got a task at my internship that sounded simple. Move one of our servers to a new AWS region, keep it the same as before, and spend as little money as possible.
That last part stuck with me. If you've spent any time reading about saving money on AWS, you've seen Spot instances everywhere. AWS says they can be up to 90% cheaper than normal on-demand instances. Ninety percent! I thought I'd found the easy win.
Then at standup, a teammate asked one question that killed the idea in about ten seconds:
"Where does this server keep its data?"
Here's what I learned, and the question I now ask before picking Spot for anything.
1. What a Spot Instance Actually Is
AWS has huge data centres full of servers, and not all of them are being used at any moment. Spot instances let you rent that spare capacity for a lot less money.
The catch is in the deal. AWS can take it back whenever it needs the capacity. You get a two minute warning, and then your instance is gone.
By default, an interrupted Spot instance is terminated, and unless you've set it up differently, the disk that came with it goes too.
Two minutes is enough to save a bit of work. It's not enough to save a server's whole memory.
2. Our Server Couldn't Be Thrown Away
This is where the standup question came in.
The server I was moving keeps its context on its own disk. Everything it knows lives right there on that one machine. There's no separate database or storage holding a copy.
So if AWS took it back, we wouldn't just lose a running server. We'd lose everything it had built up and have to start from zero.
The cheap option could have ended up being the most expensive one.
3. Pets vs Cattle
There's an old saying in DevOps that finally clicked for me after this.
Cattle are servers you don't get attached to. They're all the same, and if one dies, you replace it and nobody notices. Think of a web server behind a load balancer, where the real data lives in a database somewhere else.
Pets are servers you care about. They have their own name, their own data, and if one dies, you're having a bad day.
Spot instances are great for cattle. They're risky for pets.
Our server was very much a pet
4. The Question I Ask Now
Before choosing Spot for anything, I ask myself:
"If this server disappeared right now, what would we lose?"
If the answer is "nothing, we'd just start a new one", Spot is probably a great fit. Things like:
- Batch jobs that can be restarted
- CI runners
- Test environments
- Servers in an auto scaling group that keep their data somewhere else
If the answer is "a lot", stick with on-demand, at least until the data lives somewhere safer.
5. We Still Saved Money, Just Differently
The goal was still to keep costs down, so cheap wasn't off the table. It just had to be the right kind of cheap.
The plan is to run the server on an on/off schedule, so it only runs when it's needed. With on-demand, you don't pay for the compute while the instance is stopped, only a small amount for the disk. And because stopping an instance keeps its disk, nothing gets lost overnight.
Spot isn't ruled out forever either. Once the migration is done, we'll look at it again. If the data moves off the server, it stops being a pet, and Spot becomes an option.
Final Thoughts
I went in thinking the cheapest option was the smartest one. One question at standup showed me that cheap isn't cheap if it can lose your data.
Honestly, some of the best things I've learned in this internship didn't come from docs. They came from teammates asking simple questions I didn't think to ask. This is why I love working with people, you learn something new every day.
So I'm curious: what's a "cheap" decision that ended up costing you more?
Top comments (18)
I'd add one check after moving the data off the server: stop a worker halfway through a job and prove another worker can finish it without losing the task. External storage protects the files, but the handoff still needs a rule. This worker-failure exercise is a small way to practice that before treating an instance as disposable.
And the handover needs to be checkpoint durability tested. If it did half a task, it's unlikely it'll continue exactly where it left off, so it either restarts, or it needs to continue from a checkpoint.
That's usually the pricing question I run into when it comes to checkpoint intervals, because you want work done fast and constant write to disk is slow, but if it hits a wall and it drops the worker, that checkpoint can mean your 12h training run only lost 10 minutes, as opposed to the full 12 hours.
That checkpoint interval is the trade off I’d want to measure too. I’d include restoring after a checkpoint write gets interrupted, so the last usable checkpoint is known rather than assumed. Losing ten minutes is a much clearer recovery promise once the restore path has actually passed that test.
It's something I had to go into depth with, when I built V.E.L.O.C.I.T.Y. Workflows, which is basically a rust alternative to Temporal. part of durable workflows, is deciding how much loss is acceptable and how much work needs repeating. Temporal works with 'continue where you left off' losing all history, or 'repeat all history', which is a time consuming process. So I went the merkle root way for tracking continuation, with a WAL (write ahead ledger) and a db for more durable backing off-device, which is usually the case for ephemeral processes. The tradeoff is a hash over rerunning the entire history, which is alot more efficient and it preserves the entire run history. When looking at the different durable workflow solutions, ReState for instance, does batching, which means you lose more when it fails, but it doesnt slow down the actual running process as much (the single alternative that runs faster, though less durable as a result). Which is essentially the checkpointing, continue where the last process finished, not where the last process was at in the moment. Which for some instances is fine, eg. codebase edits, but for others, can cause duplicate entry, like ledger appends, which is why it's a measured tradeoff that is very much determined by the workflow type.
The ledger append case is the one I’d test next: the append succeeds, then the worker dies before its progress record commits. On retry, what identifies the already applied append? I’d want that external effect checked alongside the Merkle/WAL recovery proof.
Thanks Emily, that’s a really good point. I will note that down :)
Glad it helps! Even a small restart test can make that handoff much easier to reason about before relying on a disposable worker.
Thank you for the add on! :)
The standup question is the whole test. One mechanical addition worth having next to it: Spot's default interruption behavior is terminate, but AWS lets you set it to stop — the EBS volume survives the reclaim and the instance boots back up cold. That doesn't make the pet safe, it only decides whether an interruption loses state or just costs a restart. The other half of the answer is rebuild time — how long "just start a new one" actually takes. If that runs to an hour, even cattle need the on/off schedule you ended up with.
Thank you, this is a really good addition. The stop option is the part I only hinted at in the post, so I’m glad you spelled it out. One catch I found is that AWS only restarts it when capacity is back for the same instance type in the same zone, so a restart could turn into a long wait.
You’re right about rebuild time too. For our server that’s the bigger problem, since the data on that disk isn’t something you rebuild in a few minutes.
Great framing. The operational question is not whether Spot is cheap, but whether the workload has a restart contract. Stateless jobs, checkpoints, idempotency, and an observable handoff turn interruption into a normal event; a pet server turns it into an incident.
exactly Alex
Yeah, way I see spots, is basically 'ephemeral', they'll die in an instant and arent guaranteed to even run to completion, so for short-lived processes, they're fine (eg. generating an invoice).
Well, I got slapped with a 96 dollar Azure bill... For what you might ask? For a 26 dollar a month VM... How you might ask? Good question, I'm still trying to figure that 1 out myself, because it was a dead server I had started on trial credits, that I forgot to dispose of. So apparently an idle, unused VM can cost 3-4x what they advertise it at, for no apparent reason? Last time I'm trusting the price lists and last time I'm leaving any server running when not in active use or development. Dispose of your resources people, you might get a nasty bill.
Yes, exactly! Spot is great for anything that can just start again if it gets cut off, like your invoice example. It only goes wrong when you put a pet on it.
And ouch, sorry about that bill! A forgotten server is such an easy mistake to make, and you only notice when the bill shows up. “Dispose of your resources” is advice I’m definitely keeping in mind. Thanks for sharing this, it’s a really good reminder for everyone!
Great pet-vs-cattle framing. The two-minute warning is enough to run a graceful shutdown (detach an EBS snapshot, flush buffers, deregister from the load balancer) but not much else, so I'd add one rule to your checklist: if a machine's state can't be rebuilt from a source of truth in under two minutes, it's a pet. The on/off schedule is a nice middle ground too — stopping keeps the disk and you only pay EBS storage while you migrate state somewhere safer.
Thank you! I really like that rule, “if it can’t be rebuilt from a source of truth in under two minutes, it’s a pet.” That’s a much cleaner test than what I had, and I wish I’d thought of it before I started the migration. And yes, the on/off schedule turned out to be the safest middle ground for me. Paying a bit for storage while I moved things somewhere safer was worth it for the peace of mind.
The standup question is the whole post. Spot looks cheap right up until you price the interruption.
Exactly, that’s the bit I missed at first. I was only looking at the hourly price, not what one interruption would cost us. Glad that question came up at standup before we found out the hard way.