What is the worst thing you've done directly in prod?

twitter logo github logo Updated on ・1 min read

People of my age might have been involved in some sort of situation where something needed to be done in a production environment. What is the worst thing you've done?

I start with myself, I remember changing the ASP classic files directly in the server to add form fields to a page (we didn't have source control) 😱🤷‍♂️

twitter logo DISCUSS (16)
markdown guide
 

I'm pretty sure at least half of us have done the old "update with no where clause" at some point. If you're lucky it happens on a test database...

Well, at that point I didn't even have a test database. Will never get back the hours of my life spent putting everything back in order.

 
 
 

rm -rf /

I was a teen in high school, while I had a small hosting venture on the side. My classmates were impressed with my skills, and one day I decided to show them the command that you should absolutely, under no circumstance, run as root on your server.

I signed in as root, typed the command, then I wanted to delete it again... but I accidentally pressed the return key 😬

 
 

I was once involved in a project where some test code had made it to a production box... someone wanted to test production and ran tests, first action was a purge database command. Ooops... millions of member accounts needed to be restored.

 

Oh that's something which has happened to many people 🙃

 

The really worst thing was during a go-live on site at the customer when the imported adresses needed to be fixed (we work with a big ERP system) and I quickly put together a statement to fix only one address and remap it to the correct customer
...
ending up remapping every single address of the million adresses to that one customer. 😆

 
 

I once updated every email to my email because I thought I was testing dev. So I locked out over 10000 accounts.

 
 

We did not test our algorithm on test server, so our production server started to send 1k6 emails in less than 30mins.

We did not noticed, untill we heard the phone ringing all around of customers that did not understood why we asked 16 times to renew their contract 1 month in advance 😂 "Hopefully" it was only a hundred of customers over our 10 000, but still stressful on the moment...

 
 

The same old 'update without where', I was manually setting up drug favorites for one practitioner on production in a healthcare system and ended up updating every practitioner's favorites. I was a junior dev and did not have the courage to ask anybody for the backup, so I went ahead to redis where the same data was cached, copied it from there for each practitioner (about a 100 of them) and set those manually back!
Stressful times 🙃

 

I once lost the SSH key to my server which had password login disabled. I had to set up my server from scratch, again.

Does that count? 😰

 
Classic DEV Post from Apr 3

How do you feel about chasing internet points, badges and the gamification of everything?

I'll admit up front, I'm chasing the dev.to 16-week streak posting badge and am...

Yaser Adel Mehraban  profile image
An almond croissant addict cleverly disguised as a web developer