Tell me about a time you messed up

When starting my current job, I was doing my first git rebase. But I didn't understand the command and wound up rebasing off the incorrect one. So my branch had dozens of extra commits that wound up getting pushed to master.

Thankfully the changes were reverted fast, but it also means I didn't see how spectacularly I screwed up the main site.

I don’t have an example (yet) but I was thinking how this goof could help others - especially newbies/beginners - realize they are definitely not alone when their first outage happens!

Three weeks into my new job, I deleted the marketing website sidebar including various signup widgets. Was not aware of it being used on every page, did not need it on the one I was editing and went „Nah, let‘s throw this out!“ There was no undo function for this. Luckily, a colleague noticed quite fast and she was able to insert the content again quickly 😅

Last night I forgot to stop all my docker containers before running yum update. Now all of my containers are corrupted. Yay!

Don't sweat it! Software wouldn't be software without bugs and some outages. It happens 😃

My most heinous incident involved multiple threads, hitting one shared API connection and, in turn, criss-crossing customer data 😳

I forked a GitHub repo my first week at work, then had to delete it when I realized I'd forked the wrong one. Well, despite GitHub's repeated warnings, I managed to delete the source repo, not the fork. There were at least 6 open PRs in active development against it, and nobody had a full local clone that we could use to restore it. Thankfully, GitHub support was incredibly helpful and restored it.

The worst part is that some sympathetic coworkers humorously explained that this happens to all GitHub n00bs at some point. The problem was that I'd been using GH for at least 5 years at that point, and I should have known better.

Once I used a PassThrough stream instead of an EventEmitter. Apparently PassThrough streams retain some state as the data goes through them, and so eventually it caused a memory leak.

This memory leak was in a multi-machine process, which led to two processes thinking they were responsible for updating the database.

That caused mongo queries from the affected nodes to randomly execute after about an hour of being slammed from lack of memory.

Eventually the affected node would restart and start to function normally. Which was worse, because it allowed this problem to go unnoticed, until some users started reporting duplicate or corrupted data.

Stuff of nightmares.

I was working on writing a shell script to delete some files that were installed in the Applications directory on a Mac. In my mind, I had run it through command line so many times that I had just forgotten to fully write out the path.

So I finish up, make it fully executable, and bam! it erases my entire application folder.

Thank goodness for Time Machine.

Today, I forgot to turn off a service which was indexing data in ElasticSearch and I've ran an command on my machine that was also indexing data, causing the system to go into a corrupt state. I had to turn of the service and re-import data into ElasticSearch from Mongo and then re-index all the data, luckily it didn't took that much

Last week I set up a load balancer to automatically forward http to https for a particular application. That part went fine. What didn't is that we use Slack's slash commands extensively to interact with the application, and I forgot I'd set those up before I'd even gotten an SSL cert. Slack does not like getting a 301 response. They were all broken for hours for our people in Europe before I woke up and figured out what had happened.