Debugging is one of the hardest things you have to do as a dev. It pushes you to understand your creations at a whole new level. Your friendly Ruby...
For further actions, you may consider blocking this person and/or reporting abuse
thank you, Molly! I learnt valuable exp from your sharing! :D
Two thoughts...
1) the suspense in reading this almost killed me 😀
2) I’ve been but my similar issues with Sentry and am starting to think enabling real error reporting in development might be a good idea all the time.
Right?! Honeybadger has filters and ways to ignore tags and projects so its kinda like why not?!
In the past I’ve disabled production gems in development entirely, and ran into similar issues. That’s the main reason in “development mode” the honeybadger gem behaves as close to production as possible—it executes all code paths up to notifying the server (which it mocks). That was a super intentional decision; you don’t want code running in production that isn’t running in development/test. I also recommend testing exception reporting in your test suite for similar reasons. docs.honeybadger.io/lib/ruby/getti...
Unfortunately that doesn’t usually cover production states which result from heavy load or server interactions, so having a production-like dev environment for that could be a nice extra precaution. It’s still not guaranteed to find issues like this before they happen, but it could help with quicker debugging/troubleshooting.
Load testing in general is also a good idea, and may help surface some issues faster. I like the idea of your StandardErrorWorker, Molly. Putting your application into exceptional states (and other failed states) under load could be a useful practice to help spot issues across the entire monitoring stack. Maybe I’ll try adding that worker to our own Rails app. ;)
Great article! Engaging and well-written!!
Molly I think debugging on production sounds not right at all.
What we do is that commits should first pass to our staging env (that totally mimicks our production).
Then we try things there (automated and manually), and if everything seems right we hit the button to deploy on production too.
We also use that staging env for debugging like once we had a weird issue with Celery and AWS SQS, took us a bit of time but we never debugged on production.
Thanks for the thoughtful feedback!
We are hoping to get a staging environment up and running once we have containerized our application. Hopefully, that will cover some of the differences that still exist between our vagrants and production.
Wonderfully written! Thank you for sharing 🙏
My DevOps brain says why is there no staging environment which can mirror the production environment or just mirror the Sidekiq queue for debugging?
We have CodeServer (VSCode) which is packaged as CloudFormation template so we can launch and attach to an environment for live debugging in this case.
github.com/cdr/code-server
Another thought would be the nature of the tasks. I have a feeling that fargate could timeout a job just as lambda would timeout a function. I bet sidekiq could be powered by lambdas or fargate and in the case of timeout it would just requeue the job but having it isolated would mitigate unthread safe data from hanging the instance.
github.com/hoshinotsuyoshi/serverl...
Just some thoughts
Our vagrant setups, for the most part, cover all of the testing cases we need which is why we currently don't have a staging environment. We used to have one but it was so rarely used that it would get out of date and then on the off chance someone wanted to use it, it usually took them a day to get it back in working order. Because of this and the cost, we got rid of it a few years ago. We are currently working on containerization of our application and once that is done we hope to be able to spin up a staging environment with the push of a button.
Thanks for sharing those resources!
Debugging in Production is the mother of all problems. I won't suggest that at best use the logging level to log more information, but don't debug or connect it from your localhost.
thanks for sharing!!! great article!
Blimey, what a rollercoaster that was! The story is brilliant, and it's also one big subtle flex 💪 and I absolutely love it.
Uhh debugging on production title triggered my PTSD. I had quite the experience with production debugging 😅 nasty little thing.
😬Never fun, but sometimes it's necessary!
Thanks, useful