DEV Community

What is the worst thing you've done directly in prod?

Yaser Adel Mehraban on June 12, 2019

People of my age might have been involved in some sort of situation where something needed to be done in a production environment. What is the worst thing you've done?

I start with myself, I remember changing the ASP classic files directly in the server to add form fields to a page (we didn't have source control) ๐Ÿ˜ฑ๐Ÿคทโ€โ™‚๏ธ

Collapse
 
tchaflich profile image
Thomas C. Haflich

I'm pretty sure at least half of us have done the old "update with no where clause" at some point. If you're lucky it happens on a test database...

Well, at that point I didn't even have a test database. Will never get back the hours of my life spent putting everything back in order.

Collapse
 
ryanrousseau profile image
Ryan Rousseau

Yep, this is mine

Collapse
 
yashints profile image
Yaser Adel Mehraban

Totally true

Collapse
 
mrrhodes profile image
Stephen Rhodes

I was once involved in a project where some test code had made it to a production box... someone wanted to test production and ran tests, first action was a purge database command. Ooops... millions of member accounts needed to be restored.

Collapse
 
yashints profile image
Yaser Adel Mehraban

Oh that's something which has happened to many people ๐Ÿ™ƒ

Collapse
 
egeriis profile image
Ronni Egeriis Persson

rm -rf /

I was a teen in high school, while I had a small hosting venture on the side. My classmates were impressed with my skills, and one day I decided to show them the command that you should absolutely, under no circumstance, run as root on your server.

I signed in as root, typed the command, then I wanted to delete it again... but I accidentally pressed the return key ๐Ÿ˜ฌ

Collapse
 
yashints profile image
Yaser Adel Mehraban

๐Ÿ˜ฑ

Collapse
 
anwar_nairi profile image
Anwar • Edited

We did not test our algorithm on test server, so our production server started to send 1k6 emails in less than 30mins.

We did not noticed, untill we heard the phone ringing all around of customers that did not understood why we asked 16 times to renew their contract 1 month in advance ๐Ÿ˜‚ "Hopefully" it was only a hundred of customers over our 10 000, but still stressful on the moment...

Collapse
 
yashints profile image
Yaser Adel Mehraban

๐Ÿ˜‚

Collapse
 
darksmile92 profile image
Robin Kretzschmar

The really worst thing was during a go-live on site at the customer when the imported adresses needed to be fixed (we work with a big ERP system) and I quickly put together a statement to fix only one address and remap it to the correct customer
...
ending up remapping every single address of the million adresses to that one customer. ๐Ÿ˜†

Collapse
 
yashints profile image
Yaser Adel Mehraban

๐Ÿ˜‚

Collapse
 
haseebelaahi profile image
Haseeb

The same old 'update without where', I was manually setting up drug favorites for one practitioner on production in a healthcare system and ended up updating every practitioner's favorites. I was a junior dev and did not have the courage to ask anybody for the backup, so I went ahead to redis where the same data was cached, copied it from there for each practitioner (about a 100 of them) and set those manually back!
Stressful times ๐Ÿ™ƒ

Collapse
 
dechamp profile image
DeChamp

I once updated every email to my email because I thought I was testing dev. So I locked out over 10000 accounts.

Collapse
 
yashints profile image
Yaser Adel Mehraban

Oops

Collapse
 
rahulchowdhury profile image
Rahul Chowdhury ๐Ÿ•ถ

I once lost the SSH key to my server which had password login disabled. I had to set up my server from scratch, again.

Does that count? ๐Ÿ˜ฐ

Collapse
 
yashints profile image
Yaser Adel Mehraban

It sure does