A No-Kill Shelter, Petting Zoo, and U-Fish Trout Pond

Disclaimer: I am not a real sysadmin, but I do talk to many of them, and this humorous presentation is meant to highlight the problems of Going To The Cloud as experienced by legacy institutions.

The setting

Stateful Acres is a legacy institution. You know the kind — there are people here who have single-digit email address, websites dating back to the gopher protocol, and a bunch of databases, all of which are proprietary and locked in, sometimes with specific hardware dongles. There is not enough money, because there is never enough money to do maintenance.

Different parts of the organization have vastly different income streams, threat models, team structures, and sometimes they send younger co-workers to filch things out of networking closets. One person in the whole building understands the telephony system, and she’s retiring in 6 months. There is a team of alarmingly sharply dressed people doing business intelligence, and this organization is not even really a business, is it?

A new VP has come in. He would like to talk to the residents and caretakers of Stateful Acres about The Cloud. When he says The Cloud, there is sort of a little timpani effect behind the words. No one can figure out how that happens. In The Cloud, he promises, there is an environment for everyone, and there will never be weather. In the Cloud, it will be easy to manage all the animals because they’ll all have the same needs and be easy to replicate and automatically scale. A lot of the caretakers and BI people clap, because they have worked in The Cloud before. It’s way easier than shoveling hay, the way Stateful Acres works now.

The characters

The fish. No one asks the fish anything, or worries about them. They are easy to take care of, and you don’t name something that might get fished and eaten at any time. All you care about is that the pond is conducive to fish life and it’s no problem.

The poultry. There are 6 ducks and 6 chickens and one goose with a broken wing that never healed right. This is probably going to be ok. There’s enough general avian symmetry that modding the ducks to get chickens or vice versa won’t be a big problem. Even the singleton goose can get scripted based on the avian template, and she’s a scrapper, so it’s good to have a backup anyway. What’s a petting zoo without an alarm goose?

Chauncy the goat. Chauncy is a kind of evil animal. I mean, all goats look evil, but Chauncy will headbutt anyone who doesn’t come at him with exactly the right permissions. One of his horns is cracked, and he’s missing his back left leg. We tattooed all the accounting principles on his hide, and it’s a no-kill shelter, so until he dies, we’re stuck with making sure that he doesn’t run off or get into too much trouble. We could script him, but it would take hours, and probably it’s gonna be better if we just keep a cold-storage backup, because making all the database connections when a mean ol’ goat is giving you the evil eye and the system is complaining about the lack of accounting principles is just… let’s not ever do that again.

Dougal is also a three-legged ruminant, but that’s as close as he comes to resembling Chauncy. Dougal is the kind of petting zoo donkey who ends up wearing daisy garlands all the time, and tiny children try to feed him grass and he never minds. We need to keep Dougal around because he drives traffic to the farm. I guess we could script him, but we never need more than one donkey at a time, and we’re all kinda hoping that when he goes, we might find a shetland pony in need of love and care.

The dramatic tension

The man with the voice like a timpani has said we’re going to The Cloud. So we’re going. First are the fish, because they’re easy. Many people, especially ones who’ve never done a major migration, think this means that the problems will be straightforward and everything is going to continue smoothly. The fish are scripted, uploaded, and no one even notices the blip. The U-Fish guide goes with them to help kids bait their hooks without impaling themselves.

Next, the poultry goes up. Surprising everyone but the old-timers, the goose is A Problem. The ducks and chickens were pretty straightforward, but it turns out that the goose is actually a webserver that has a bunch of URLs that are in published textbooks. And that tenured professors are very attached to the goose and come around to visit her all the time. How, they ask, are they supposed to get such amazing goose-quill pens if the goose is in The Cloud? No one told the Cloud team that they would have to provide quills! That was not in the operational specs! The professors shrug and offer to keep the goose in the campus pond nearby, but we all know that the goose is needed on Stateful Acres to keep laying brass eggs (we got a cut-rate goose, that’s why she’s with the rest of us misfits). We can’t duplicate the goose. Well, we could, but then we would have two geese, one of them still not in The Cloud, and that does not solve any problems.

The team agrees to leave the goose alone until they can resolve the quill problem.

They look at Chauncy. Chauncy glares back. They look at Chauncy’s hide, so precious and so… attached to a goat. They look away. Later, they tell themselves that they are trying to gain enough experience in the transition to do the hard stuff properly.

The team looks at Dougal. Dougal looks back mildly. He’s perfectly prepared to get cloudified if it would be helpful. The team walk around him, taking pictures of him and his daisy chains. Someone says, “Wouldn’t it just be easier if we just built a Shetland pony in The Cloud, and left old Dougal here to amuse the few people who show up, even though most people will be in The Cloud?”. The team thinks about adorable, shaggy, four-legged Shetland ponies, and doesn’t move Dominic to the cloud. They leave him behind to “maintain legacy connections”. And he does. Everything is fine until audit time 6 months later when the auditors ask who is taking care of Dominic, and is he getting his shots and virus updates and hoof trims, and no one has an answer for it.

The farm staff is split between sites, and the people who aren’t in The Cloud never get to experience the easy parts of how it’s supposed to work, and the people who are in The Cloud don’t understand why the people back on the farm won’t stop whining about one stupid goat, how hard can it be compared to the thousands of fish the pond has scaled to? The nitrogen management problems alone are way harder than one manky animal!

This is a silly example story, but it illustrates a problem I see a lot of organizations encountering. There is a lot that you can loft up into the cloud, a lot of services and servers that can be cattle, easily-scriptable, ready to spin up or down without much fuss. But there are still many many many organizations that have 3-legged goats as servers.

They may live on VMWare now, but they are still too singular, too vendor-locked for anyone to be able to easily replicate them in a cloud, even a nice virtual machine in the cloud. If your licenses are by CPU, how do you translate that? If the program and the data are stateful, and cannot be separated, how do you manage that? How do you monitor an architecture that is multi-modal, some still in the data center and some in someone else’s data center? If a server is stateful, and writing to logs, even if it’s all virtualized, you still need some kind of alerting system to warn you when your logs are filling up disk space, and that’s not the kind of alerting the cloud vendors have been excited to build.

When someone tells you that they have a datacenter, or a legacy product, or that they can’t upgrade to the new version you’re selling, I promise you, it’s not because they haven’t thought about their options. They have, and this is the best option they have given their resources.

A while ago, there was a discussion on Twitter about how “bad code” was that way because whoever wrote it was operating under constraints we didn’t understand. The same is true of infrastructure. It is very easy for those of us working for cloud-native, SaaS-style companies to feel impatient about systems administration, but it’s never going away, no matter what it’s called and what the scope is. Someone, somewhere, is keeping the servers up and listening for alerts and being on call and performing backup and knowing the percentages on disk and cable failure.

Possibly while a donkey is glaring at them.