I have a problem where sometimes certain passenger spawned processes use too much memory. I've never been able to figure out why this is - it happens about once a month and debugging with GDB on a production server hasn't been so easy. I have not been able to reproduce the problem locally during testing. I can only assume some combination of software and hardware is causing some issues.
So, in order to minimise downtime when this occurs, I've written the following simple script. Pending an option in Phusion Passenger which allows one to set a memory limit, this script will terminate any passenger related process that exceeds a certain memory limit.
You can put this in
/etc/crontab to run every 10 minutes:
Just in case anyone is interested, this happens on a XEN VPS. From GDB inspection, I've often found that the Ruby process (version 1.8.7) is stuck in garbage collection code. Secondly, I've also noticed that within 30 minutes of the memory spike, there is typically some message like:
I wonder if there is some race condition/timing glitch in the garbage collection? I hope to upgrade to Ruby 1.9.3 but I'm very cautious about doing this on a production server without a lot of testing..