In security, there is a central concept called: least privilege. It states that every actor in a system should have only the minimal rights needed to fulfill their duties. As someone who used to code in the olden days, I can tell you that anyone advocating for "least privilege" is a boring, buttoned-down, anal-retentive, no-fun jerk. Because everyone with a little grey in their beard knows there's no prank quite like the prank you can pull when you've been granted god rights.
Coding in the "bad old days", when there were few tools and even fewer controls, wasn't without it's... charms. While I don't pine for the past, there was, at times, an extremely carefree nature to what we were doing. Consider my first "serious" coding job.
I'd been programming for a while before I managed to convince any "real" company to hire me in a "real" programming role. After I got out of the Air Force in 1997, I worked several roles that were "programming-adjacent". I was always programming on the side. But I couldn't convince any major employer to formally anoint me as a real programmer.
That all changed in early 2004 when I caught on with a local company that specialized in applicant tracking systems. It was a heady time. I was working as a full-fledged developer for the first time. For a software company, no less. Even better - they were a startup! (Back then, "startup" was an exciting term. It wasn't the danger-sign that I now perceive it to be.) The photo above was taken on my first day on that job.
The "production server" was a machine that sat on our CTO's desk. We could literally tell when people were accessing the app by looking at the little hard-drive light on the front of the box.
There was no such thing as "source control". We all had to grab local copies of the code via FTP. And then push it back up via the same method.
None of us had ever heard of "continuous integration" or "continuous delivery". When we had deployments (which could happen, quite literally, whenever), they were handled by those of us who were "senior" enough to do the manual FTP uploads. I'm pretty sure that I was considered to be in that category before I'd been there for 6 months.
Our "codebase" was a chaotic fractured nightmare - because there was no single codebase. We had about 100 clients. And every single one of them had their own unique copy of the code.
I originally worked on one of our maintenance teams. On those teams we practiced the patented Whack-A-Mole Methodology for quashing bugs. We had more than 100 different variants of the same codebase to maintain. We had no "core" codebase, no method of pushing bug fixes out to all clients, no way of tracking client versions. We just had a patchwork tribal knowledge of the clients that we'd individually worked on.
So when bugs came in - and they came in friggin fast - we jumped on them as quickly as possible. We fixed them in the most manual straightforward way possible. And then we swiftly moved on to the next ticket in the queue.
For those of us on the maintenance teams, our jobs came down to churning tickets. You tried to knock out as many tickets as you could each day. And you just prayed that the queue wouldn't be any longer tomorrow.
If this all sounds rather depressing... it's not. We were young and we all had this kinda naïve startup mindset. We were overworked and underpaid. But we generally had a lot of fun as we were playing fast-and-loose with every possible IT best practice.
This is why I simply had to, on occasion, make judicious use of my god rights to righteously f(*k with people. It was a stress reliever. It was a necessary aspect of my mental health in those days.
If this dev environment sounds like... like a challenge, well, imagine what it must have been like to be on the phones with our clients. Constant bugs. Bugs being resurrected like Lazarus. Bugs that were in the software for years - that no one previously noticed. Bugs, bugs, and more bugs. And through it all, customer service had to take all those calls. And maintain all those relationships.
Because CS had, by far, the worst jobs in the company, it was only natural that I would target them for endless abuse. I mean, how was I supposed to make myself feel more important if it wasn't by deviously ruining someone else's day?? And heaping that kinda abuse on CS was just a natural fit - like peanut butter and chocolate, or SpongeBob and Patrick.
This is where those god rights came in handy...
Working maintenance in those days was a bit like being on the floor of the New York Stock Exchange. Whenever you weren't nose-down in a coding task, you were monitoring the ticket queue. It would jitter around all day. New tickets coming in. Others getting reworked. Some getting closed, or reprioritized.
Watching the queue wasn't just entertainment. It was an act of efficiency. Because there were many times when I'd see a ticket come in that basically mirrored something I'd fixed, for another client, months ago. When that happened, I'd typically grab that ticket off the queue, regardless of priority, because I already knew how to fix it.
Remember when I said that we had 100+ clients, each with their own codebase? Well, each client's codebase was, on some level, unique. But it still shared many similarities with all the other clients.
There were many times when I'd get a bug ticket, I'd fix the bug, and I'd think to myself, "You know... this same bug probably exists in all of the other codebases." But there was no efficient way to test that. And even if I could confirm it, we had too many other pressing issues to embark on a project to fix a bug for all of the clients - when most of them hadn't even noticed that bug at all. So I'd fix the bug, close the ticket, file the fix away in my brain, and move onto the next task.
After enough time in that environment, many of the tickets started to look similar:
Client cannot [KNOWN ISSUE THAT WAS PREVIOUSLY FIXED FOR ANOTHER CLIENT IN A DIFFERENT CODEBASE].
Client reports that [KNOWN ISSUE THAT WAS PREVIOUSLY FIXED FOR ANOTHER CLIENT IN A DIFFERENT CODEBASE].
Client can't find [KNOWN ISSUE THAT WAS PREVIOUSLY FIXED FOR ANOTHER CLIENT IN A DIFFERENT CODEBASE].
It was not uncommon that some of these bugs came down to a single line of code. In fact, I could frequently remember the exact code file that contained the bug, as well as the general location in the file where the offending line of code existed. Those were my favorite tickets to grab.
None of us did straight FTPs to the production server. Instead, there was a homegrown utility that someone had written to automate the transfer of files for any given client.
Because I'd established myself as a very hard worker who knew the systems extremely well, I had god rights to that utility. I could promote code to production whenever I dang well pleased.
If that sounds like a rogue (or... cocky) statement, please know that the CS reps knew I had this "power". And they'd often come up to me in the middle of a day asking me to push a fix for any especially-cranky clients. So it's not as though I was operating undercover here. Everyone knew that I was one of the half-dozen or so devs in the company blessed/cursed with these magical abilities.
When a known (to me) issue would come through , and I knew exactly how to fix it, and I knew it was an incredibly fast-and-easy fix (typically, a single line of code), here's what I'd sometimes do:
- Fix the line of code in my local environment.
- Leverage our janky "promotion" tool to immediately push that fixed code file to the client's production site.
- Fill the ticket... with a status of
COULD NOT REPLICATE.
- Sit back and wait until the CS rep came over to my desk with a big ol' WTF? expression on their face.
They'd see the notice that the ticket had been filled. They'd also see that it was filled as
COULD NOT REPLICATE. And they'd march over, exasperated and incredulous. The conversation would usually go something like this:
CS Rep: You can't replicate the issue???
CS Rep: But I was just on the phone with the client. Like, minutes ago. And I saw the issue myself onscreen.
Me: Well, let's take a look.
At this point, I'd pull up the closed ticket and reference the "Steps to recreate" that the CS rep left on the ticket. Then I'd go through the steps, one-by-one, in production, and then I'd say:
Me: See? No issue.
CS Rep: But... but... it was broken! Like, a few minutes ago!
Me: I dunno, man. I did exactly what you said. And I'm not seeing any issue.
After a few minutes of blatant confusion, I'd watch them walk back to their desk. They be downtrodden. Or excessively perplexed.
A few of you may be chuckling at this little anecdote. Far more of you may be thinking that I'm a Grade A Certified Jerk. And you know what?? You'd be absolutely correct! But not necessarily for reasons outlined in this tale.
First, this isn't the kinda prank that I could've pulled frequently. It only works if the recipient (e.g., the CS Rep) actually thinks that you're giving them the god's-honest truth. I don't think I did this more than 3-or-4 times during my time at the company, and never to the same CS rep twice.
Second, I only did this with a handful of reps with whom I was actually pretty cool. In other words, we knew each other. And we messed with each frequently.
Third, I only did this with reps who I knew were professional enough to not simply run to the phone, call the client, and say, "We have no idea what caused the issue, but we can't find it anymore."
Fourth, I was never so cruel as to allow any particular rep to "simmer" on this. I don't think I ever went more than 10 minutes before walking over and explaining what I'd done.
I fear that I'm getting kinda old for this stuff now. It's been many years since I indulged in any kinda office pranks. They seem kinda pointless and hollow now.
Then again, maybe I've haven't pranked anyone because it's been a lonnnggg time since I've had "god rights" - to anything. Maybe that's because the devops guys read this article...?