DEV Community

Cover image for 10 Tips For Debugging in Production

10 Tips For Debugging in Production

Molly Struve (she/her) on July 31, 2019

Debugging is one of the hardest things you have to do as a dev. It pushes you to understand your creations at a whole new level. Your friendly Ruby...
Collapse
 
capu profile image
HD

thank you, Molly! I learnt valuable exp from your sharing! :D

Collapse
 
phallstrom profile image
Philip Hallstrom

Two thoughts...

1) the suspense in reading this almost killed me 😀

2) I’ve been but my similar issues with Sentry and am starting to think enabling real error reporting in development might be a good idea all the time.

Collapse
 
molly profile image
Molly Struve (she/her)

Right?! Honeybadger has filters and ways to ignore tags and projects so its kinda like why not?!

Collapse
 
wood profile image
Joshua Wood

In the past I’ve disabled production gems in development entirely, and ran into similar issues. That’s the main reason in “development mode” the honeybadger gem behaves as close to production as possible—it executes all code paths up to notifying the server (which it mocks). That was a super intentional decision; you don’t want code running in production that isn’t running in development/test. I also recommend testing exception reporting in your test suite for similar reasons. docs.honeybadger.io/lib/ruby/getti...

Unfortunately that doesn’t usually cover production states which result from heavy load or server interactions, so having a production-like dev environment for that could be a nice extra precaution. It’s still not guaranteed to find issues like this before they happen, but it could help with quicker debugging/troubleshooting.

Load testing in general is also a good idea, and may help surface some issues faster. I like the idea of your StandardErrorWorker, Molly. Putting your application into exceptional states (and other failed states) under load could be a useful practice to help spot issues across the entire monitoring stack. Maybe I’ll try adding that worker to our own Rails app. ;)

Collapse
 
noiseeee profile image
JM

Great article! Engaging and well-written!!

Collapse
 
yaser profile image
Yaser Al-Najjar • Edited

Molly I think debugging on production sounds not right at all.

What we do is that commits should first pass to our staging env (that totally mimicks our production).

Then we try things there (automated and manually), and if everything seems right we hit the button to deploy on production too.

Yes, Docker and ElasticBeanstalk here helps a lot to create super simple maintainable pipeline

We also use that staging env for debugging like once we had a weird issue with Celery and AWS SQS, took us a bit of time but we never debugged on production.

Collapse
 
molly profile image
Molly Struve (she/her)

Thanks for the thoughtful feedback!

We are hoping to get a staging environment up and running once we have containerized our application. Hopefully, that will cover some of the differences that still exist between our vagrants and production.

Collapse
 
kaspermeyer profile image
Kasper Meyer

Wonderfully written! Thank you for sharing 🙏

Collapse
 
andrewbrown profile image
Andrew Brown 🇨🇦

My DevOps brain says why is there no staging environment which can mirror the production environment or just mirror the Sidekiq queue for debugging?

We have CodeServer (VSCode) which is packaged as CloudFormation template so we can launch and attach to an environment for live debugging in this case.

github.com/cdr/code-server

Another thought would be the nature of the tasks. I have a feeling that fargate could timeout a job just as lambda would timeout a function. I bet sidekiq could be powered by lambdas or fargate and in the case of timeout it would just requeue the job but having it isolated would mitigate unthread safe data from hanging the instance.

github.com/hoshinotsuyoshi/serverl...

Just some thoughts

Collapse
 
molly profile image
Molly Struve (she/her)

Our vagrant setups, for the most part, cover all of the testing cases we need which is why we currently don't have a staging environment. We used to have one but it was so rarely used that it would get out of date and then on the off chance someone wanted to use it, it usually took them a day to get it back in working order. Because of this and the cost, we got rid of it a few years ago. We are currently working on containerization of our application and once that is done we hope to be able to spin up a staging environment with the push of a button.

Thanks for sharing those resources!

Collapse
 
javinpaul profile image
javinpaul

Debugging in Production is the mother of all problems. I won't suggest that at best use the logging level to log more information, but don't debug or connect it from your localhost.

Collapse
 
aleksandrastolyar profile image
Aleksandra Stolyar

thanks for sharing!!! great article!

Collapse
 
lukegarrigan profile image
Luke Garrigan

Blimey, what a rollercoaster that was! The story is brilliant, and it's also one big subtle flex 💪 and I absolutely love it.

Collapse
 
kristijanfistrek profile image
KristijanFištrek

Uhh debugging on production title triggered my PTSD. I had quite the experience with production debugging 😅 nasty little thing.

Collapse
 
molly profile image
Molly Struve (she/her)

😬Never fun, but sometimes it's necessary!

Collapse
 
hiders90 profile image
Hiders90

Thanks, useful