Photo by Anna Shvets from Pexels
When you're new to something (like programming), you tend to look to those with more experience for guidance an...
For further actions, you may consider blocking this person and/or reporting abuse
So probably for me the time I tried an update at 4am, got the wrong branch merged and sent 100 text messages to every user of an app. The app was from a bank. So yeah, I woke 30,000 people up at 4am by having their bank ring them. This incident did make it to Twitter. Ugh. Didn't lose the client BTW and the app continued to be successful. Lucky.
“A rabbit would do” 🤣🤣🤣
This is a big reason why I always advocate for updates/deployments during normal business hours. I know, I know - a lotta businesses would still balk at such a request. But I'm now a firm believer that, if you've done proper QA/testing/best-practices, you should be able to deploy at almost any time.
I'm not naive. I realize that, even in the best of scenarios, you can still bork a deployment. But even in those cases, would you rather have an emergency when all of the potential support personnel are at work and immediately available to help? Or would you like to be trying to call around the support tree at 4AM trying to find someone who will pick up??
As long as you can do zero downtime I agree! That's often the driver for out of office hours. With this client they mandated an update window between 1am and 6am if there was downtime...
Ahh... yes. I agree. If your update/change/deployment requires downtime, then yeah, there's no way to get that kind of authorization during normal business hours.
💀
Do you remember what you were updating at 4am? I can't even imagine waking up to 100 text messages from my bank. I would immediately assume all of my money was gone.
Well yeah, ummm, I was updating the thing that stopped notification messages going out in quiet hours - at the time there was a problem with 3rd party services that if they would send in a window, but would retry out of hours. We'd written our own. I had been testing it with something that just sent lots of messages to known app installs to prove they didn't make it. I got my wires crossed on the release test, just stupid, and left in the "send test messages" code. I'm a bloody idiot.
Fortunately it was the marketing/voucher app for the bank, so while a bunch of people were very unhappy, and a bunch turned off notifications - it didn't actually suffer as much as I thought it would at around 5am, with a glass of whisky and a lot of cigarettes...
The irony of the fact that the feature was for not sending after hour texts makes me laugh 😂😂😂
My 2nd job, was working on a mid-range computer at the OS layer. They wanted me to add the ability to save files that had 3 character extensions like .txt, etc. I got it to work, tested it and sent it to distribution folks. 1 week later I found out that I regressed ability to save files without extensions. They had to pull back distribution to 115 countries. Ouch...
It really was the wild west before test automation...
I later became a test advocate, read testing books like crazy. At one point managed a team of 35 QA. People in 3 countries.
Best QA book was Software Testing Techniques. Boris Bezier It's still a good read but a bit dated.
I remember a moment of personal embarrassment that wasn't a huge deal, but early on with DEV, before we had much in the way of PR review or CI I was coding on the site by myself and I recall hearing the Mac "trashcan" sound in my headphones but wasn't sure what it was. I shrugged it off and moved on.
Later, after I'd deployed some code, I noticed that the logo was entirely missing. I'd accidentally put it in the trash somehow and deployed it with
git add .without noticing.It just felt embarrassing because I heard the trash can, and just didn't think much of it. Of course, things are much more careful these days, not that we don't push weird things up.
Surely someone has created the DEV logo in one div with just CSS by now. You could get rid of it entirely and save entire milliseconds of page load time! Younger Ben was just too radical for his own good.
It was 2015. My first job as a developer. I truncated a production table with no backup thinking I was on my local environment. But lucky for me, the table was small and I could retrieve all the data back by calling 20 customers. Bad day.
*shudder*
Back in the day, I were the sole active administrator for a private server in world of warcraft. Every wednesday morning, 6:30 am, I was obliged to update the game server with the weeks newest patches and fixes.
One wednesday morning, a morning like the rest of them, I woke up at 6 as usual and had my coffee while I prepared the deployment. The clock ticks and around 6:28 it was time to execute my prepared script on the server which would do everything in "screens" whilst I just supervise it.
Everything went well, green all the way, and off to university I go.
At the lunchbreak, I opened the community forums on my forums in mission to moderate its content while also replying to ingame tickets. I found to my surprise 2 separate forum threads and one ticket implying that "the thing that should be fixed is not fixed, therefore we as 25man guild cannot do this". At once I started looking into it whilst my staff went for damage control.
Only to find out, that the deployment I had done earlier in the morning, was the script for the stage server. Oh what an facepalm, oh so disappointed community members.
i was working with other programmers, and i had a task to implement a new webpage.
It was a big legacy web system made with webforms.
So I based the new implementation in another page already created and deployed. AND for some reason i commented out this entire page...
So when i deployed my new page to prod, I also disabled one of the pages that customers are used to access....
Well, after some boss yelling I fixed it...
And after 1 month i got fired :(
But for good, because of that i am on a very big computer enterprise!!
I was a fresher. I joined a team, which had build an event driven system. There was an event store ..
Unluckily, all senior devs left the team when i joined.. And event store disk space was increasing at rapid speed and business was behind me to keep the product running.
The ops guys were not willing to give more disk space.
I just deleted the events, and converted the architecture to plain CRUD.
It was bliss at that time. But now that I am experienced I realized, i could have done much better than dropping events table
That sounds super stressful. Just dropping everything in your lap, then leaving. It must have been rough :(
An acquisition happened. The Big company Ora--e to blame. All senio devs left
Managed 4 microservices.
Later left the company due to poor management and then joined a startup. Now i know DDD and event sourcing quite a bit :).
I must say, i was so naive as a fresh out of college guy. Simply deleted event store 😑. I thought how dumb it is to have it in the first place.
Currently i am working in a company where the architect of that design is CTO and when i told my story to him, his reaction was "Did you really just deleted my event store without knowing why it is used 🤦♂️"
I only recently started working with microservices and I think event sourcing is really cool. I could see how it would seem kind of weird and useless to a beginner, though.
I initially pushed the initially empty sql database of a recruitment test app for the company to git and then pulled on the server. When the database had received thousands of entries, I made a local change and then force pushed to git and then pulled the empty db on the server. I spent hours panicking and frantically searching on how to undo a force push as recruitment department began messaging me that they couldnt find any entry... luckily for me I had stashed before pulling, even though I didn't know what stash meant and I just did it because of the warning (thank you git deities). That day I immediately learnt not to ever push your db, even if empty... also how to connect postgres and set up automatic back ups
On GitHub, I think there's an option to forbid
forcepushing tomaster. I'm sure it's saved a lot of people 😅Copying so many peoples code and not understanding what I built....or how to use it. Then having to scrap it and start from scratch.
On the bright side I have learned alot about what not to do when starting to code haha
I remember doing this really early on, too. One thing I remember having trouble with when I was brand new was returning values from functions. Scoping rules and the fact that you could only return a single object seemed unintuitive and overly restrictive. Not sure when it finally clicked, but now I have new and exciting things to be confused about haha
Is it weird to say that I think I do not have an embarrassing story? I have made many mistakes, big and small, and still do. But I don't think any of them are embarrassing. Sure there were mistakes which, in hindsight, were stupid. But I never set out like "hold my beer, and watch this".
I think most people tend to make their biggest mistakes a few years into their career, where they have enough knowledge to break things but not enough wisdom to know how to avoid breaking things. If you've gotten past that hump, I'd say you're probably safe! (Hopefully!)
My most embarrassing one is likely when I took down our monitoring software when I was co-oping in college. I was building a new site and needed to add it to our monitoring, but I actually just broke the entire thing. I was petrified and didn't know what to do.
When I told me manager, he assured me that wouldn't let me actually configure the production monitoring system - I was after all still just co-op. ) I had been using the stage environment one the whole time. We figured out the problem and got our tools up and running again.
When i was a fresh grad, I accidentally deleted our client's source code from their production site (They were hosted in cpanel) . A few a minutes after, we already received a complaint from the client. Really thankful that my manager was there to retrieve the backup.
"No code" before it was trendy 😁
I once coded in PHP. I know... Was bad...I'm ok now.
Sounds like the first words of a "PHP Programmers Anonymous" meeting 😆
Biggest "oh my god we're idiots" moment was when me and a colleague (both several years of professional experience at that point) spent almost a full hour debugging, diving into the internals of our frameworks and libraries, trying to figure out why our Typescript class just wouldn't initialise correctly.
Only to realise that we had written "contructor" instead of "constructor"
I still do this sometimes. It just goes to show how helpful syntax highlighting / static code analysis / type-checking tools can be. Working in dynamic languages is so much more messy and difficult because of this.
I got a headache because my WordPress website inaccessible. Discuss with so many forum without any results. The next day I check back to my htaccess file and I found that error is caused by blank enter space on the first line.
Oh man, I know the feeling. I've gotten CI/CD pipeline failures because of a (missing or extra) blank line at the end of a file, etc. It's a pain.
Probably when I completely forgot how to do headings for an HTML page, until I realized that I had a sticky note literally 5 inches away from my hand. I was stuck for hours, until I realized that im absolute garbage at coding. Fun.
Killed a herd of cows with an extra 0.
I'm intrigued
It was a long time ago (~1985) and software development was very different back then. We wrote the code on coding sheets using pencil, then we went to site, log on to the mainframe, and type our code into the system.
Testing was done using a process called side-by-side testing. That is, the customer would run their existing system along side the new system, and cross check the results.
Anyway, back to killing the cows.
I was making a change to the feed formulation program, used to calculate the ingredients that go into animal feed. Specifically I was updating the part of the program that calculates the amount of active ingredients (aka drugs) to add to the feed.
This was a fairly complicated process, working from a recipe a feed might require a certain amount of an active ingredient at a certain potency, but the actual amount added depends on the potency of the available batches of the active ingredient in stock, each having their own potency value.
Anyway, as you may have guessed on keying the update to the software into the system, an extra 0 had been added (actually I think, it may have been omitted not added, can't really remember) resulting in a 10 fold increase in the potency of the active ingredient added to the feed.
First I knew about my coding error was when I was called into the managing directors office at the software company I worked for and told that the feed company was being sued by the farmer who had lost a herd of cows because they had been fed this feed which too high a concentrate of some drug. In turn the feed company was looking to sue the software company.
Bit of an oops moment I can tell you.
Anyway, you may wonder why I mentioned testing above. Well, because we were updating a live system, the only real way to test the changes made was to run the process and compare it to the old system, which was part of the contract between the software company and the feed company. The feed company, to save time, just assumed that the new system would be correct, and didn't bother running the process through the old system to verify the results.
So the feed company in the end were liable.
I tell this story from time to time to highlight the importance of testing.
That's a good story! But those poor cows :-(
When I started I used to do select * from table then filter the data in code.