What's yours biggest IT (programming) fuckup?

#programming #fuckups #discuss #errors

Top comments (14)

I pushed an infinite loop into our deployment process which took down the site and got us rate limited and suspended by Heroku. It also caused elevated failure rates on their whole service enough that status.heroku.com had a yellow message about the situation.

Casey Brooks • Oct 2 '18

Two things, both on a large Wordpress installation:

While testing a migration of some custom tables, I deleted all our custom tables to do a clean run of those migrations. Unfortunately, I was in the SequelPro tab of the production database, not my local database.
I thought we were caching stuff in memcached, but the way you get Wordpress to use the cache is really hacky and I couldn't get it to work locally. So my typical workflow was to comment that file out, at which point Wordpress will start caching stuff to the database instead. Well, I accidentally pushed that commented file into production, so this large site with thousands of concurrent users started caching everything to the database instead of memcached. It didn't take long for MySQL queries to get exponentially longer due to millions of rows being queried on each request, bringing the entire site down.

Tiago Magalhães • Oct 1 '18

Once removed a Tomcat server while trying to remove symlink. Fortunately was a internal server for CI that was being implemented.
And other time deleted a production database. Fortunately there was a morning backup and lost 3 articles

Guido Vizoso • Sep 28 '18

First webmaster gig and first formal job. Learning FTP accidentally deleted a whole Wordpress site. Almost got fired on my first day.

James • Oct 2 '18

Sounds like their screw-up to me! First day on the job and they have you "learning" FTP on the server with important sites on it?

Accident waiting to happen.

Guido Vizoso • Oct 2 '18

Yeah the company management and infrastructure were missing hahahaha

Marcin Zajkowski • Sep 28 '18

Hope the company had the proper source control etc. and it was just a matter of redeploy? Or not...? :)

Guido Vizoso • Oct 1 '18

No it didn't. Infrastructure was not a word in that company hahaha. I got lucky because there was another copy of it in an old HDD

Marcin Zajkowski • Oct 1 '18

Oh man, so it was fun! ;)

Ross Henderson • Oct 1 '18

One of my calculations worked backwards. The internal testing was manually triggering the fake passage of time by increasing the number of days, and it worked perfectly. As soon as it went to live I realised it was supposed to be x - y, not y - x and the whole thing ran backwards. read: it didn't work.