DEV Community

Sherry
Sherry

Posted on

What's the most scared you've ever been as a developer?

Discussion (15)

Collapse
ben profile image
Ben Halpern

Ever deleted production data?

Collapse
sherrydays profile image
Sherry Author

😬

Collapse
ariprayoga66 profile image
Ari Dwi Prayogo

drop database in production πŸ˜‚

Collapse
imthedeveloper profile image
ImTheDeveloper • Edited on

Restarting a service which was reporting unhealthy, to then notice the restart caused a cascade of problems right through the stack. Cold sweat and panic, especially as end users started throwing ticket after ticket around x y z not working.

A literal, turn it all off and back on again until we could work through to find the issue. Turns out a memory leak caused by an update to a library was the problem with the unhealthy service, but by restarting we then had a lot of traffic hitting a couple of other areas of the stack which were unable to keep up, normally this would have been handled perfectly fine, but they also suffered from the same leak and the extra traffic sent them over the edge much quicker.

Collapse
nombrekeff profile image
Keff

We once had a bug on a financial application, where transaction amounts were calculated incorrectly. People were receiving more money than they should, hopefully it was a quick fix and no money was lost, but had me panicking!!!

Collapse
ricardovfreixo_84 profile image
Ricardo

Back in 2000, fixing the Y2K bug on a big bank while the country was adopting the Euro, we had a migration project. The goal was to fix the Y2K and the project managers decided to do the migration to the Euro at the same time since they had the resources in the field. What can go wrong, right? So we had this huge cobol scripts that was suposed to do the exchange. 3 step operation, collect all money from all accounts, exchange the values, and deposit afterwards. So it collects fine, exchanges fine and it breaks. For 4 days no one with account in the bank (and this was a big bank) had available funds to use.

Collapse
heatherw profile image
HeatherW

First time deploying to production, first time fixing an issue with prod without my more experienced colleagues available. Basically anything on prod makes me nervous still.
And realising the script you wrote is the reason the servers are down. Not a fun time but at least I learnt a lot from it.

Collapse
linaran profile image
Deni

I'm a tmux user and I always have 3 windows open: "local, production and random".
Thinking I was on "local", I was spamming some stuff (bash/apt installs) only to realize to my horror that I was on production and actually connected to a server.

Luckily nothing really happend but it was a very scary 20-minute confirmation check.

Collapse
gregorgonzalez profile image
Gregor Gonzalez

My story: I was told to update a system in production, we do it via SCP or SSH (we didn't know about git at the time), but this was the only client whose server was not local and was uploading files from our server to their server external, it took 1h: 40min (from local to local it was less than 5min) so they could not work during that time 😬. To workaround I started uploading to a different folder and then doing a local to local copy in production which took seconds without interrupting, then I learned git and thought "this could help a lot"

A story from a friend: He started at the same job as me, he was making a copy of a production database on our local test server, so he made a dump file and then ran the file. One day he did not check which server he was logged in to and inadvertently ran the file that dropped the entire production database 😳 fortunately it was a recent backup and the client lost only 1 hour of work, we called and informed them about the problem, so they had to re-record the missing information.

Collapse
arham_abiyan profile image
Arham Awal Abiyan

Changing staging config to production config

Collapse
patracio profile image
patracio

Deploy an ECS in production ... with develop properties.

Collapse
xanagandr profile image
Guillermo Saldivar

When you come back from your honeymoon break to the news that the lambda service you hooked up for the legal department has been erring out for, at least, a month, and no one noticed.

Collapse
weaponsforge profile image
Weapons Forge

Days of extreme mental block: the days when my hands refuse to code what my mind wants.

Collapse
riadhoq profile image
Riadul Hoque

AWS terminating the admin server where your build server lives for no apparent reason, and you're the sole developer/devops! That was a sleepless night.

Collapse
alumezi profile image
Arbnor Lumezi • Edited on

I used to be scared of any task, until I learned to enjoy it. Now if it does not work I know Im going to love it when it does :D