I'd add one check after moving the data off the server: stop a worker halfway through a job and prove another worker can finish it without losing the task. External storage protects the files, but the handoff still needs a rule. This worker-failure exercise is a small way to practice that before treating an instance as disposable.
And the handover needs to be checkpoint durability tested. If it did half a task, it's unlikely it'll continue exactly where it left off, so it either restarts, or it needs to continue from a checkpoint.
That's usually the pricing question I run into when it comes to checkpoint intervals, because you want work done fast and constant write to disk is slow, but if it hits a wall and it drops the worker, that checkpoint can mean your 12h training run only lost 10 minutes, as opposed to the full 12 hours.
That checkpoint interval is the trade off I’d want to measure too. I’d include restoring after a checkpoint write gets interrupted, so the last usable checkpoint is known rather than assumed. Losing ten minutes is a much clearer recovery promise once the restore path has actually passed that test.
It's something I had to go into depth with, when I built V.E.L.O.C.I.T.Y. Workflows, which is basically a rust alternative to Temporal. part of durable workflows, is deciding how much loss is acceptable and how much work needs repeating. Temporal works with 'continue where you left off' losing all history, or 'repeat all history', which is a time consuming process. So I went the merkle root way for tracking continuation, with a WAL (write ahead ledger) and a db for more durable backing off-device, which is usually the case for ephemeral processes. The tradeoff is a hash over rerunning the entire history, which is alot more efficient and it preserves the entire run history. When looking at the different durable workflow solutions, ReState for instance, does batching, which means you lose more when it fails, but it doesnt slow down the actual running process as much (the single alternative that runs faster, though less durable as a result). Which is essentially the checkpointing, continue where the last process finished, not where the last process was at in the moment. Which for some instances is fine, eg. codebase edits, but for others, can cause duplicate entry, like ledger appends, which is why it's a measured tradeoff that is very much determined by the workflow type.
The ledger append case is the one I’d test next: the append succeeds, then the worker dies before its progress record commits. On retry, what identifies the already applied append? I’d want that external effect checked alongside the Merkle/WAL recovery proof.
The standup question is the whole test. One mechanical addition worth having next to it: Spot's default interruption behavior is terminate, but AWS lets you set it to stop — the EBS volume survives the reclaim and the instance boots back up cold. That doesn't make the pet safe, it only decides whether an interruption loses state or just costs a restart. The other half of the answer is rebuild time — how long "just start a new one" actually takes. If that runs to an hour, even cattle need the on/off schedule you ended up with.
Thank you, this is a really good addition. The stop option is the part I only hinted at in the post, so I’m glad you spelled it out. One catch I found is that AWS only restarts it when capacity is back for the same instance type in the same zone, so a restart could turn into a long wait.
You’re right about rebuild time too. For our server that’s the bigger problem, since the data on that disk isn’t something you rebuild in a few minutes.
Great framing. The operational question is not whether Spot is cheap, but whether the workload has a restart contract. Stateless jobs, checkpoints, idempotency, and an observable handoff turn interruption into a normal event; a pet server turns it into an incident.
Yeah, way I see spots, is basically 'ephemeral', they'll die in an instant and arent guaranteed to even run to completion, so for short-lived processes, they're fine (eg. generating an invoice).
Well, I got slapped with a 96 dollar Azure bill... For what you might ask? For a 26 dollar a month VM... How you might ask? Good question, I'm still trying to figure that 1 out myself, because it was a dead server I had started on trial credits, that I forgot to dispose of. So apparently an idle, unused VM can cost 3-4x what they advertise it at, for no apparent reason? Last time I'm trusting the price lists and last time I'm leaving any server running when not in active use or development. Dispose of your resources people, you might get a nasty bill.
Yes, exactly! Spot is great for anything that can just start again if it gets cut off, like your invoice example. It only goes wrong when you put a pet on it.
And ouch, sorry about that bill! A forgotten server is such an easy mistake to make, and you only notice when the bill shows up. “Dispose of your resources” is advice I’m definitely keeping in mind. Thanks for sharing this, it’s a really good reminder for everyone!
Great pet-vs-cattle framing. The two-minute warning is enough to run a graceful shutdown (detach an EBS snapshot, flush buffers, deregister from the load balancer) but not much else, so I'd add one rule to your checklist: if a machine's state can't be rebuilt from a source of truth in under two minutes, it's a pet. The on/off schedule is a nice middle ground too — stopping keeps the disk and you only pay EBS storage while you migrate state somewhere safer.
Thank you! I really like that rule, “if it can’t be rebuilt from a source of truth in under two minutes, it’s a pet.” That’s a much cleaner test than what I had, and I wish I’d thought of it before I started the migration. And yes, the on/off schedule turned out to be the safest middle ground for me. Paying a bit for storage while I moved things somewhere safer was worth it for the peace of mind.
DevOps engineer in Stockholm. Most of my week is Kubernetes YAML, Terraform plans and making deploys so boring nobody talks about them. I write about CI/CD, cloud cost and small internal tools. Learni
Exactly, that’s the bit I missed at first. I was only looking at the hourly price, not what one interruption would cost us. Glad that question came up at standup before we found out the hard way.
We're a place where coders share, stay up-to-date and grow their careers.
I'd add one check after moving the data off the server: stop a worker halfway through a job and prove another worker can finish it without losing the task. External storage protects the files, but the handoff still needs a rule. This worker-failure exercise is a small way to practice that before treating an instance as disposable.
And the handover needs to be checkpoint durability tested. If it did half a task, it's unlikely it'll continue exactly where it left off, so it either restarts, or it needs to continue from a checkpoint.
That's usually the pricing question I run into when it comes to checkpoint intervals, because you want work done fast and constant write to disk is slow, but if it hits a wall and it drops the worker, that checkpoint can mean your 12h training run only lost 10 minutes, as opposed to the full 12 hours.
That checkpoint interval is the trade off I’d want to measure too. I’d include restoring after a checkpoint write gets interrupted, so the last usable checkpoint is known rather than assumed. Losing ten minutes is a much clearer recovery promise once the restore path has actually passed that test.
It's something I had to go into depth with, when I built V.E.L.O.C.I.T.Y. Workflows, which is basically a rust alternative to Temporal. part of durable workflows, is deciding how much loss is acceptable and how much work needs repeating. Temporal works with 'continue where you left off' losing all history, or 'repeat all history', which is a time consuming process. So I went the merkle root way for tracking continuation, with a WAL (write ahead ledger) and a db for more durable backing off-device, which is usually the case for ephemeral processes. The tradeoff is a hash over rerunning the entire history, which is alot more efficient and it preserves the entire run history. When looking at the different durable workflow solutions, ReState for instance, does batching, which means you lose more when it fails, but it doesnt slow down the actual running process as much (the single alternative that runs faster, though less durable as a result). Which is essentially the checkpointing, continue where the last process finished, not where the last process was at in the moment. Which for some instances is fine, eg. codebase edits, but for others, can cause duplicate entry, like ledger appends, which is why it's a measured tradeoff that is very much determined by the workflow type.
The ledger append case is the one I’d test next: the append succeeds, then the worker dies before its progress record commits. On retry, what identifies the already applied append? I’d want that external effect checked alongside the Merkle/WAL recovery proof.
Thanks Emily, that’s a really good point. I will note that down :)
The standup question is the whole test. One mechanical addition worth having next to it: Spot's default interruption behavior is terminate, but AWS lets you set it to stop — the EBS volume survives the reclaim and the instance boots back up cold. That doesn't make the pet safe, it only decides whether an interruption loses state or just costs a restart. The other half of the answer is rebuild time — how long "just start a new one" actually takes. If that runs to an hour, even cattle need the on/off schedule you ended up with.
Thank you, this is a really good addition. The stop option is the part I only hinted at in the post, so I’m glad you spelled it out. One catch I found is that AWS only restarts it when capacity is back for the same instance type in the same zone, so a restart could turn into a long wait.
You’re right about rebuild time too. For our server that’s the bigger problem, since the data on that disk isn’t something you rebuild in a few minutes.
Great framing. The operational question is not whether Spot is cheap, but whether the workload has a restart contract. Stateless jobs, checkpoints, idempotency, and an observable handoff turn interruption into a normal event; a pet server turns it into an incident.
exactly Alex
Yeah, way I see spots, is basically 'ephemeral', they'll die in an instant and arent guaranteed to even run to completion, so for short-lived processes, they're fine (eg. generating an invoice).
Well, I got slapped with a 96 dollar Azure bill... For what you might ask? For a 26 dollar a month VM... How you might ask? Good question, I'm still trying to figure that 1 out myself, because it was a dead server I had started on trial credits, that I forgot to dispose of. So apparently an idle, unused VM can cost 3-4x what they advertise it at, for no apparent reason? Last time I'm trusting the price lists and last time I'm leaving any server running when not in active use or development. Dispose of your resources people, you might get a nasty bill.
Yes, exactly! Spot is great for anything that can just start again if it gets cut off, like your invoice example. It only goes wrong when you put a pet on it.
And ouch, sorry about that bill! A forgotten server is such an easy mistake to make, and you only notice when the bill shows up. “Dispose of your resources” is advice I’m definitely keeping in mind. Thanks for sharing this, it’s a really good reminder for everyone!
Great pet-vs-cattle framing. The two-minute warning is enough to run a graceful shutdown (detach an EBS snapshot, flush buffers, deregister from the load balancer) but not much else, so I'd add one rule to your checklist: if a machine's state can't be rebuilt from a source of truth in under two minutes, it's a pet. The on/off schedule is a nice middle ground too — stopping keeps the disk and you only pay EBS storage while you migrate state somewhere safer.
Thank you! I really like that rule, “if it can’t be rebuilt from a source of truth in under two minutes, it’s a pet.” That’s a much cleaner test than what I had, and I wish I’d thought of it before I started the migration. And yes, the on/off schedule turned out to be the safest middle ground for me. Paying a bit for storage while I moved things somewhere safer was worth it for the peace of mind.
The standup question is the whole post. Spot looks cheap right up until you price the interruption.
Exactly, that’s the bit I missed at first. I was only looking at the hourly price, not what one interruption would cost us. Glad that question came up at standup before we found out the hard way.