Be afraid developers, be very afraid.
Of zombies, knife-wielding slayers, or bloodthirsty sharks?
No! Of brutal bugs, demons from merge hell, and the dreaded Count Rollbackula!
Well maybe not afraid, just very, very tired.
What really haunts a developer’s or software engineer’s dreams? Just ask any developer and there’s a good chance they have a horror story to tell. A rollback while they’re on vacation. A nightmare hiding in the code, and staying up all night to solve it. Slack blowing up when something broke, and the big meeting with the CEO to explain what happened. Boo!
The following haunted tales are from current Splitters’ past lives, before they escaped into the open arms of Split. And they are all too real.
Do you have a developer horror story of your own? Share it with us on LinkedIn or twitter at #StopTheDevHorror
The Homegrown Feature Flag of Horrors
When I worked at a cloud-based content management company, we used to store feature flags in our main database. These feature flags were from a homegrown system we created internally. It was pretty flaky. Once, a developer added a feature flag in the code in a really high-traffic area.
The feature flag was for a piece of code that was also using the database. When the homegrown flag was turned on—all at once—the new code hit the DB so hard that it brought the DB down. Because the DB was down, we couldn’t turn off the feature. The monster couldn’t be killed.
We had to keep the app down, go manually in the DB to turn the feature off, and then restart our application. Lesson one: Homegrown feature flags are easy to mess up. Lesson two: FF systems should be disconnected fully from your application and its direct dependencies. Lesson three: Percentage rollouts are a good idea, even more so if you can have metrics on your DB in association with your rollout.
– Pierre-Alexandre Masse, VP Engineering
Night of the Living Dread
My last product used text files on disk to control features. These files were embedded into the EC2 images. When a new instance was launched, we needed to sync them, but some were region-specific or tests that were only running on certain servers.
As a result, we regularly found that three random machines had the wrong property set out. With a fleet of hundreds of machines, we had no idea which ones had the issues. That’s about as scary as it gets.
– Ron Pierce, Engineering Leader, Experimentation
Something Wicked This Way Comes
When I worked at a customer relationship management corporation, we decided to take the whole company to a theme park. It was supposed to be a day of mandatory fun. Upon landing, and as part of a release, we turned a blog page on. However, the homegrown feature flag solution didn’t have a good way of testing all cases or variations of a flag. So the application and world went down after switching the flag on. It was an integration issue that surfaced when turning the flag on.
As a result, unfortunately this team of developers had to spend most of their fun day in a coffee shop solving the problem.
Split localhost mode is a good way to enable earlier testing to avoid a similar fate.
– Patricio Echague, Founder and CTO
Nightmare on Dev Street
Do you have a developer horror story of your own? Share it with us on LinkedIn and Twitter using #StopTheDevHorror.
You might need a silver bullet to fend off any werewolves that may be lurking in your code. Split is here to keep you releasing features while mitigating risk. Our unique architecture is safe and secure. So you can focus on your customers instead of repairing a homegrown flag system with its own share of nightmares. Save yourself! Give Split a free spin at split.io/signup/
Top comments (0)