The Problem
For the past 2+ years, I have had a Ruby on Rails application running in production without much issue (other then bugs I introduce on occasion). But one day recently, I started to see significant performance issues with Postgres & Redis timeouts and connection issues.
This is my first time seeing issues like these at such a large scale, and it left me scrambling to try all sorts of solutions to get things to work. In the end, any code tweak I made didn't work, and the issue kept coming back.
Shot in the Dark Solution
This left me scratching my head because all the EC2 metrics I can see on CloudWatch seemed perfectly fine. I had good CPU usage, storage isn't an issue, and network throughput is just fine. Since I didn't have any clue what was going on, that last thing I could do was just scale my application up and add another server, so I did.
To my expectation, an additional server did the trick. After all, throwing another server at the issue is the cliche thing to do to solve the problem. So it left me wondering, why?
After giving it some little thought, the only metric for some annoying reason you can't see on CloudWatch is memory! I think that's pretty important... So I set off on a quest to find out how to add memory tracking!
How To Track memory
Luckily my quest to find out how to track memory was a short one. After searching how to track memory usage for Elastic Beanstalk and came across the following article, How can I monitor memory in Elastic Beanstalk for Amazon Linux AMIs running on Amazon EC2 instances?
All I had to do was create a new file and add it to my ebextensions folder and deploy it. Once deployed, you should see a new option for metrics on CloudWatch under metrics called "System/Linux."
From there, you can track memory metrics for each instance you have running. You can then create a custom dashboard, and if your feeling frisky, set up alerts.
Looking Back
After taking a look at the memory usage with 3 servers now running, it a lot more obvious to me that was the root cause of my issues. From now on I am going to make it a point to track memory usage on any future applications so I can get ahead of this issue.
Top comments (0)