I'm running an Apache server in a virtual Linux machine. It's running with mod_php and mod_python installed. At seemingly random intervals, from 10 minutes to several hours, one of the spawned apache2 processes will suddenly begin consuming all CPU time and rapidly increasing memory use. This quickly causes thrashing in the VM, which balloons VMWare's memory use, which causes thrashing in the physical server, which renders every virtual server running on that system unusable. If I don't notice the problem fast enough, logging into the physical server becomes impossible and I have to hit The Button.
My question has multiple parts:
How do I prevent this? The Apache processes are running as their own user and group (www-data). I would like to do something like limit this user to a maximum of 75MB per process. The
RLimitMEM directive looked promising, but it apparently only applies to external processes invoked by Apache such as CGI scripts. Does /etc/security/limits.conf apply to Apache's children? For now I've simply disabled swap on the virtual machine, but because of the odd intervals between problems I don't know whether this has done anything.
How do I diagnose this? I assume it's an unbounded recursion or loop somewhere in my code, but I can't think of any way to figure out where. If I could see the requests as they arrive at the Apache server, then it would help, but it seems nothing is written to the logs until the response is complete. However, I'm not even sure if the problem is related to requests or in my code at all. I don't even know how to see if the problem is happening within code executed by mod_php, mod_python, or other.
Posts