Recent 'Panic Moment'?

A few years ago, I built an A/B test for an enterprise level client. The test was making modifications to their SPA on one page, however, we were not aware that their checkout shared some of the same structure and classnames and we inadvertently removed their "Checkout" button, so user's couldn't complete their orders.

We quickly turned off the test, and I was able to fix it, but that was definitely one of the bigger panic moments because I had made a mistake and probably cost the company a good chunk of money...

Afterward, we had a long conversation about what had happened. There were many things that lead up to this, like:

• no time was baked into the onboarding for us to familiarize ourselves with the client's site.
• the PM had agreed to a deadline before talking with the team, and so we would meet that deadline, they had us skip QA where something like this would have most likely been caught.
• I had been so focused on the timeline, that I didn't take the time to test core functionality of the site.

Moral of the story, there is a process for a reason, and if you rush that process, or skip steps, eventually something will go wrong.

Most recent turned out to not be something I needed to sweat. Two of our most major clients called in reporting server not found errors on their site. Pull up both, sure enough, nothing. Start going back through all the changes we’d done recently, nothing would have caused this, much less in two unrelated sites.

Check the DNS records, and there’s NOTHING for either site…no A records, no MX, no cnames, nada…both sites completely wiped off the net. Panic mode sets in HARD through the entire department. Phone calls were made, no one has any clue what’s going on.

Turns out, the two “unrelated” sites were in face related. Both sites’ DNS were managed by the IT department of the provincial government…which was suffering a “network incident”, later determined to be a DDoS against their network, that took out the DNS servers.

We all went directly to the bar after work…

Most recent? Just now.

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "Could not find 'once.md'"', src/lenker/fileresolve.rs:71:66
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Facing one right now. Deployed project to QA. Without getting approval from the team members.
Now only now half of the features are moved. Dev team will be getting some amazing and interesting defects 😂😂

I pushed the EPO button in the server room, thinking it was the light switch...

Pushed a silly typo to prod.