DEV Community

Ben Halpern
Ben Halpern Subscriber

Posted on

Tell me about a time you messed up

So I brought down the site for a little while this morning. Now I'm interested in hearing about when you messed everything up!

Latest comments (56)

Collapse
 
khrome83 profile image
Zane Milakovic

I use to be a game developer.

One time I corrupted the external hard drive by mistake. It ruined months of work that was unrecoverable.

This was a four person team project, we all failed the final.

The mistake was not backing it up. I also had dropped the hard drive which caused the corruption. Obviously this was a accident, and it was my personal drive. But I also was very bad about sharing the work in process, thinking I could wow people. This caused team mates not having a copy that was nearly as close.

Collapse
 
cwilby profile image
Cameron Wilby

We recently made a switch from environment foo to environment bar. After the migration to bar was completed, I assumed we could take foo down. For reasons, foo should have stayed up and I had to spin up foo within 15 minutes. I managed to do it, but that was not a fun time.

Takeaway was the situation could have been avoided if I had just asked "Hey, do we still need foo?".

Collapse
 
xjuanc profile image
xjuanc

The worst thing I've done so far was: uploading staging environment in production and production in staging. Damn CI was lacking by then

Collapse
 
jonnysmith1981 profile image
Jonathan Smith

A long time ago as a tech support guy I did an rm -rf on a customer site and forgot the "." before the "/". Whoopsie.

Collapse
 
bjorngrunde profile image
Bjรถrn Grunde

I was working on a quite easy and basic feature of an Economy system and the only advanced part was a really old legacy module that had two functions with really similar and bad names, something like cr_EDC_use_ETM20 and cr_EDC_use_ET20. My ide auto spelled me the wrong function and I managed to send several invoices to over 60k clients. Invoices with an infinite amount to pay. Luckily we had rollback systems, so clients never noticed. Later our old-school legacy programmer refactored the module with names that made sense to the younger generation :P

Collapse
 
theague profile image
Kody James Ague

I currently work in road design. Most of my career has been electrical building system design. So when I was new to road design I labeled a bunch of intersections with 20' radii. By a bunch I mean the whole project of 20+ intersections so over 80 corners, roughly. The problem is that they weren't actually drawn at 20', they were all drawn differently because I didn't know any better. Some were only 7', 13' etc. So when it was time to construct the curbs on each corner, nothing lined up. So now instead of a nice new consistent result these intersections all have random and different radii. It was embarrassing to say the least and I never was invited to visit the project during construction which I was thankful for!

Collapse
 
sebbdk profile image
Sebastian Vargr

I've brought down live sites numerously, messed up the master branch, ended up in merge conflict hell, grossly underestimate things, sent out e-mails to thousands of customers because i was using the production DB for developing, and all kinds of other sheit.

I've only messed up once of twice so bad a backup had to be used at the start of my career tho.

Developers should never test their own stuff or have access to production directly.

It always ends in chaos at some point in my experience. :)
Usually fixable chaos, but still.

Collapse
 
saurabhdaware profile image
Saurabh Daware ๐ŸŒป

In my internship while copy pasting code from other component I also copy pasted google analytics code and one of the variable was undefined.
It went unnoticed (since it was just part of analytics so it didn't exactly break anything ๐Ÿคท) and then they had bunch of undefined hits on their production analytics!

Collapse
 
omkarshirodkar profile image
OmkarShirodkar • Edited

New job. First time monitoring a Unix shell scripting job. Script errors out with memory issues. Go in the respective folder, find a couple of files ~ 2GB big, delete them. Re-run the script, it fails again. This time looking for the files yours truly deleted. Had to wake up the SME to fix the issue.

Collapse
 
itsmarkodonnell profile image
Mark O' Donnell • Edited

In my first job I managed a firewall configuration app for a large Telco. It allowed Fortune 500 companies to make firewall changes on their network. Canโ€™t remember the exact problem but long story short I changed something which stopped anyone logging in at all. No-one could make any changes in for 2 days until we found the issue. ๐Ÿ˜ณ

This was particularly bad because it affected bonuses that year, whooops! ๐Ÿ˜‚๐Ÿ˜‚