Do full cleanup for every completed courses by your users (regularly or on demand). They usually don't aware of your limited resources.
From my perspective, you may need to:
- Go to a Course settings > Reset course > Check all course module categories and options that considerably consume too many resources > Reset course
- Unenroll inactive users. If you have external database enrolments and/or authenticated users, you may want to delete all associated user's data that not currently active from those tables. Make sure the related Moodle CRON jobs are active for this case.
Be aware of these course resources! (your case may be different)