The new forums will be named Coin Return (based on the most recent vote)! You can check on the status and timeline of the transition to the new forums here.
The Guiding Principles and New Rules document is now in effect.

Apache flips out, kills web server [FIXED, sorta]

JaninJanin Registered User regular
edited December 2007 in Help / Advice Forum
I'm running an Apache server in a virtual Linux machine. It's running with mod_php and mod_python installed. At seemingly random intervals, from 10 minutes to several hours, one of the spawned apache2 processes will suddenly begin consuming all CPU time and rapidly increasing memory use. This quickly causes thrashing in the VM, which balloons VMWare's memory use, which causes thrashing in the physical server, which renders every virtual server running on that system unusable. If I don't notice the problem fast enough, logging into the physical server becomes impossible and I have to hit The Button.

My question has multiple parts:

How do I prevent this? The Apache processes are running as their own user and group (www-data). I would like to do something like limit this user to a maximum of 75MB per process. The RLimitMEM directive looked promising, but it apparently only applies to external processes invoked by Apache such as CGI scripts. Does /etc/security/limits.conf apply to Apache's children? For now I've simply disabled swap on the virtual machine, but because of the odd intervals between problems I don't know whether this has done anything.

How do I diagnose this? I assume it's an unbounded recursion or loop somewhere in my code, but I can't think of any way to figure out where. If I could see the requests as they arrive at the Apache server, then it would help, but it seems nothing is written to the logs until the response is complete. However, I'm not even sure if the problem is related to requests or in my code at all. I don't even know how to see if the problem is happening within code executed by mod_php, mod_python, or other.

[SIGPIC][/SIGPIC]
Janin on

Posts

  • DrFrylockDrFrylock Registered User regular
    edited December 2007
    Maybe you can figure out what's going on by attaching to the process with strace when it happens?

    DrFrylock on
  • JaninJanin Registered User regular
    edited December 2007
    This is sort of fixed - by sheer coincidence, an employee happened to visit the page that caused the problem using a terminal next to me, while I was watching the process list. Traced it down to a loop that was iterating "while month <= end_month". Guess what happens when end_month is 12 vOv.

    Janin on
    [SIGPIC][/SIGPIC]
Sign In or Register to comment.