DEV Community

CarnivorousJoe
CarnivorousJoe

Posted on

How do you cope with a problem that you can't solve?

Hi everyone,

I've been lucky enough to get a full-stack web development position in my hometown (read: not a big city) and am approaching my 1st year marker. It's a great fit for my skillset and I love everything about the people and the work.

Today we deployed a semi-hefty feature release after doing our tests and QA, and everything was going swimmingly. Untill a few hours later when the calls starting coming in ... Fast.

This release consisted of a lot of work that I had been doing, and included some major refactoring of some legacy processes, but QA passed it and I had hands on experience in our staging environment to suggest green lights across the board, but wouldn't you know it, as I'm packing up to leave (about 3 hours post release, for context) a CX rep, measured panic in her eyes, lets me know something in the past half hour has gone terribly wrong. A major point of our flow is broken, and I'm the only dev left in the office.

I unpack and dig in to try to determine the issue. I mean, no forks or knives here, more like hands and feet, My thinking was that logic would dictate that a major issue would correlate to a major change, so I took ownership over the problem since the release earlier was mostly my doing. And I troubleshoot, and hotfix, and troubleshoot, and hotfix some more, but to no avail. Everything about my work seems to be pointing to 'it should work.'

Fast forward to about 4 hours after I had originally tried to pack up, the office is completely empty, and I'm no closer to solving the problem. I rollback the release in a desparate attempt to undo whatever happened, but the issue remains. I gained some valuable insight at that moment that I'm still wrestling with, which is that the issue might not have been my fault it may be somewhere else on the stack that I'm not involved in. Moreover, I feel like the blame game is a toxic sentiment, but I digress.

I had to leave any further investigation because I was exhausted and starving, and (insult to injury) have an operation the following day. But I feel so uneasy and worried about leaving a major loose end, despite knowing it's not something I could easily solve considering I might not actually know the circumstances around the root of the problem. But I also can't shake that feeling of responsibility over it.

To be honest, I feel jeopardized by the whole scenario.

So I want to know, all you dedicated problem solvers out there, what tips can you give someone to walk away from a mission critical problem when the solution seems completely out of your grasp and, to make matters worse, you physically can't be involved in it's solution?

Top comments (1)

Collapse
 
lbonanomi profile image
lbonanomi • Edited

First of all I hope your operation went well and I wish you a speedy recovery.

Everything you have described sounds like a formula for a whopping outage.

  • One developer on the tools post-install.
  • No escalation path.
  • Hotfixes in prod.
  • No systems folks around?
  • No hand-off to the next developer?

If you're a line developer some of this is probably out of your hands, and some of it you might be able to change through advocacy and setting good examples.

  • If there's no policy for incident response please ask about getting one drafted ASAP
  • Talk to the DBA(s) and admin(s) about availability during rollouts.
  • You stop work after a peer is logged-in and briefed on what's happening and what you've tried to sort things out.
  • Have a postmortem meeting and make the findings public.

Sticking to documented procedure isn't sexy or exciting, but it lets me sleep the sleep of the righteous.