Samuel Williams Wednesday, 13 June 2012

I have a problem where sometimes certain passenger spawned processes use too much memory. I've never been able to figure out why this is - it happens about once a month and debugging with GDB on a production server hasn't been so easy. I have not been able to reproduce the problem locally during testing. I can only assume some combination of software and hardware is causing some issues.

So, in order to minimise downtime when this occurs, I've written the following simple script. Pending an option in Phusion Passenger which allows one to set a memory limit, this script will terminate any passenger related process that exceeds a certain memory limit.

#!/usr/bin/env ruby

LIMIT = 128 * 4

puts "Scanning memory stats..."
$lines = []
IO.popen("passenger-memory-stats") do |io|
	$lines = io.readlines
end

puts "Checking memory usage..."
$lines.each do |line|
	if line.match(/^(\d+)\s+([\d.]+)\s+MB/)
		pid = $1
		memory_usage = $2.to_f
		
		puts "Checking #{line}"
		if memory_usage > LIMIT
			puts "Killing process: #{pid}, memory usage #{memory_usage} > #{LIMIT}..."
			Process.kill("KILL", pid)
		end
	end
end

You can put this in /etc/crontab to run every 10 minutes:

*/10 * * * * root /srv/scripts/passenger-memory-check

Technical Details

Just in case anyone is interested, this happens on a XEN VPS. From GDB inspection, I've often found that the Ruby process (version 1.8.7) is stuck in garbage collection code. Secondly, I've also noticed that within 30 minutes of the memory spike, there is typically some message like:

Jun 13 05:01:57 ayako kernel: [1124672.119348] clocksource/0: Time went backwards: ret=f58a06913e397 delta=-38334276 shadow=f58a0581b591f offset=1341a9a7

I wonder if there is some race condition/timing glitch in the garbage collection? I hope to upgrade to Ruby 1.9.3 but I'm very cautious about doing this on a production server without a lot of testing..

Comments

Leave a comment

Please note, comments must be formatted using Markdown. Links can be enclosed in angle brackets, e.g. <www.codeotaku.com>.