What have you crashed?

In my early days, I learned (...eventually...) that complicated SQL queries with incomplete JOIN conditions on large data sets in production = non-responsive server = angry customers.

Fast forward to today, I insist that the IT department provide a dev/staging server which mimics production knowing full well that a runaway query (or never-ending loop, or recursive function with a misplaced terminal condition, or ...) can happen to anyone from time to time. Even the best of us! :)

In the early 70s, I thought I was a hotshot programmer with 5 years of experience, I could do Assembly and Cobol. I did an optimization that impressed my supervisor, got permission to examine the computer. It was an IBM 370/148, not the small one fit in a large room partitioned into 4 but a big one, took up the entire floor at the bank. partitioned into 8. This was my first time on a big machine. I examine F0, F1, F2, F4, F5, F6, then paused. power was always on the last part F3, but this big machine power must be on F7. You never look at power, the machine would have scan every log file since it was built to fill the screen. So I go to F3, this big machine was actually a small machine that had been upgraded. First all the line printers quit, then all the tape drives quit, then nothing but the rattle of the giant HDD platters, then the phone rang, then every phone at the bank was ringing. 165 seconds after I pressed the wrong button, the screen filled up and everything went back to normal. I got to meet the bank president, the entire board, every manager, and the Federal Reserve guy. I said I was trying to speed up the computer. Seems every terminal in every branch also went down for 165 seconds. Federal Reserve banned me from any bank computer room, they took away my keyboard and terminal. I had to program with a plastic template and graph paper, get someone to keypunch it for me and then bring the stack of cards to the computer operated to be run after hours. making a change in the program and compiling it and seeing the results was about 60 hours. I would sweet talk the computer operated into fixing my code. I ended married to her, for years she kept fixing my code.

My body

Yeah...about 5 minutes before a demo at my company, I thought I could fix a simple SQL call that was causing so invalid data. It worked on my machine fine. But when I went to push it to the server, something went wrong and it broke the demo. The code was fine but the deployment failed. Took 15 minutes into the meeting to figure out what went wrong and fixed it. Whoops. haha.

I one time took down a Facebook game I worked on with over 4 million daily players. The combination of an off-by-one error and infinite retry with no back off turned all the game clients into a DDoS attack on our own servers. Even the Akamai web console was unresponsive for about 10 minutes.

Seeing as my primary area of coding is vulnerability scan automation I tend to crash a lot of things. Central web services, DNS servers, Document Repo's, Data taking systems. I've killed them all. Each time I do it I have to spend a lot of time trying to figure out how to be kinder and more gentle to the systems on the network. My Favorite is scanning for anonymously writable FTP daemons. I get a lot of complaints from users that their printers are spitting out lovely notes from me.

We've got an old records system on an AS/400 that needed the data migrated. Short of writing a fully custom app to allow editing (due to the legal need to provide redaction capability forever), we needed some kind of front end. We have a robust document management system that provides all of the retention, security, search, redaction, and managed access features that we need. We looked at a ton of options, and this was the best fit.

So, I set about turning the data into a series of reports and feeding them into the system using their SDK. The first iteration just about killed the report server. At that point I learned how to generate reports locally on my batch machine, and how to do it in parallel. The new process resulted in the queues on the server being filled up, and the resources being saturated. Everything ground to a halt. My next lesson was in tuning the import process to not be so hard on the servers. The next hurdle was, in importing a couple of hundred million files, the SAN hosting the application's storage ran out of inodes. Every server hitting that storage location had a fit. There wasn't a lot I could do about that, but after the operations team cleared the issue, add that to the list of things I know to think about for the future.

I learned about a lot of neat things during that project. It gave me a ton of great lessons that I wouldn't have learned if stuff hadn't have broken.

Brought down a couple warehouses for companies you've definitely heard of because our deployment strategy was to TeamViewer to customer site > Drop patched dlls. Not much QA there.

I am guilty of the old Update without a "Where" statement on a accounting production database (there was no staging environment at all).

And working previously on IT (more hardware stuff), I plugged a computer to the network, and somehow it caused an IP conflict between the computer and a software license server, which crashed the software on every computer in the network. That one caused lots of mayhem all through the building.

I saw this earlier and thought I was just going to read along and nod to some relatable content. Then visual studio hiccuped and I watched the entire bin folder start throwing errors because vs thought half the files were deleted. Somehow the project builds without them but it won't publish and I wasted enough time trying to fix it that I could have recreated the whole project