As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

PHP and memory...

DisrupterDisrupter Registered User regular
Hey guys. I am working on a PHP project using Zend Framework and I am trying to track the bloated memory issues I am running into.

Long story short, I am making a business application that has an opportunity, each opportunity can have multiple quotes, each quote can have multiple items, each item has a product. Each product can have many other parameters.

As I step through the code, outputting memory, it seems as though the objects themselves are not too bad, but memory gets absolutely abused when I first create an object of any type.

For example, $product=new application_model_product_item(); $product->load($sku);

will increase the memory usage by 1 MB. Which is insane.

But if I clear out that object using a custom $product->clear(); function and then unset($product); I only gain back like 100kb. And, if I do

$product=new application_model_product_item(); $product->load($sku);
$product2=new application_model_product_item(); $product2->load($sku2);

my memory footprint only increases by 1.1MB. So its like the object itself is only 100kb, but the act of creating the first product uses a meg. I believe this may be due to the zend auto-loader. But I dont know for sure.

This would not be the end of the world. It sucks, but thats the biggest class, and if it takes up 1MB one time, then all other objects of it take up .1, I could deal.

But whats strange is, as I go through a quote, looping through each of its items, creating a product for each of those items has a footprint of 1MB. But creating TWO products for each of those items only costs 1.1MB. For some reason, creating that first product for each item is an entire meg.

This persists throughout. Components of the product each have a much larger footprint when first created, but subsequent objects of that class do not.

I can not figure out the leak. Ive gone through all the children and grand children, trying to tackle what is causing it, and each time, each child takes up a lot of memory when it is first created, but then it doesnt seem to actually use much as an object.

Basically it seems like

if parent one creates child one, tons of memory is used. But it does not use much to create child two.

But then if parent two creates child one, tons of memory is used again, but it does not use much to create child two.

I can not for the life of me figure this out. I am not doing anything that should require a ton of memory, I am simply loading data from a DB and storing the information in data model objects. But they are absolutely bloated.

616610-1.png
Disrupter on

Posts

  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    edited May 2012
    We have a programming thread for this stuff: http://forums.penny-arcade.com/discussion/155486/select-from-posts-where-tid-pa-programming-thread/p1

    And my short answer would be: PHP :rotate:

    GnomeTank on
    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • DisrupterDisrupter Registered User regular
    AH, yeah, for some reason it slipped my mind to post specifically in there. I blame it less on PHP and more on Zend. Ill post there, thanks.

    616610-1.png
  • bowenbowen How you doin'? Registered User regular
    edited May 2012
    @Disrupter I think it may have to do with PHP's garbage collection. It assumes you will be instantiating more objects and preallocates memory for it.

    Try gc_disable(); at the top of your script and see if that fixes it?

    bowen on
    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • DisrupterDisrupter Registered User regular
    I will give that a shot and let you know, thanks.

    616610-1.png
  • NightslyrNightslyr Registered User regular
    bowen wrote: »
    @Disrupter I think it may have to do with PHP being garbage.

    Fixed. :P

  • The AnonymousThe Anonymous Uh, uh, uhhhhhh... Uh, uh.Registered User regular
    Nightslyr wrote: »
    bowen wrote: »
    @Disrupter I think it may have to do with not using Django.

    Fixed. :P
    Re-fixed. :rotate:

  • bowenbowen How you doin'? Registered User regular
    At least he's not using Ajax.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • DisrupterDisrupter Registered User regular
    I...I do use ajax. What do you have against ajax?

    Anyway...turns out, our php version is 5.2.14 which is before the garbage collector was upgraded to handle cylical objects. So I can not do gc_disable();

    There seems to be a number of leaks, all coming form zend_framework. The Zend_Db_Row object for the product is 200kb in size, and even if I unset it, it doesnt seem to actually get removed. For now, when it comes to this project I am going to just give myself some memory leeway and increase the limit while processing a quote. There shouldnt be more then a few lines per quote, each with one product so the thing shouldnt get too big regardless. Its just painful to see such simple stuff absolutely eating up memory.

    Seems like zend_db and possibily the zend_auto_loader are the culprit. Unfortunately backs against the wall in regards to getting something to show to the higher ups, so for now I have no choice but to increase the memory limit like a chump and hopefully find time to optimize and investigate down the line (yeah right...). Thanks for the attempt at helping (and ridiculing PHP. Good news is, my new company is primarily .net so I should be getting away from PHP as time goes on.)

    616610-1.png
  • zeenyzeeny Registered User regular
    There is nothing abnormal about the behavior you describe, so I'm a bit confused about what the issue is.
    Also, I'd just like to point out that PHP garbage collection could happen at requirement, so if you are running close to the limit, but aren't encountering exhaustion errors you shouldn't assume you have a problem.

  • DisrupterDisrupter Registered User regular
    edited May 2012
    Im not sure how its normal. Usually when I set and then unset an object, the memory allocated for the object gets released. There is a couple of issues im experiencing.


    object 1= new class1();
    unset(object 1);

    I do not get back the memory allocated for object 1(); I get back a very small portion of it.

    If within a class, I have

    object 1a=new class1();
    object 1b= new class1();
    unset(object 1a);
    unset(object 1b);

    1a allocated a lot more memory than object 1b, and I will get back the amount of memory allocated for object 1b for both object 1a and object 1b.

    The other issue is that if I have two parent classes, and BOTH call

    object 1a=new class1();
    object 1b= new class1();
    unset(object 1a);
    unset(object 1b);

    they will both have the same issue described above. So within a class, each first instance of an object is taking up more memory than the object needs to be allocated and releases only what it actually needs.

    I actually believe I have tracked it down to zend_db. If I make a query to the database and get a zend_db_table_row object, and then unset that object, I do not get any more memory back from it. All the memory is still allocated. I have looked into this and apparently its because zend_db buffers the last result, so the result still has a pointer to it. Since all my objects have a data model associated with them which call a get($id) to populate the object, they are all leaving these buffered results behind, which seem to have large memory footprints. I need to find a way to release that buffered result. I am not sure why this occurs with only the first child object. Perhaps it has something to do with zend_db caching? I dont know.

    But looking through, the memory leak definately happens when I get the zend_db_table_row object and then release it. It does not free up the allocated memory at release.

    EDIT:
    Yep. If I do

    $db=new Zend_DB_Object();
    $results=$db->get($id);
    unset($results);

    I do not free up the memory from $results;

    But if I add $db=new Zend_DB_Object(); again, then the memory footprint is identical to before $resutls=$db->get($id); So $db is holding a reference to $results that I need to find a way to clear. You'd think itd be easy to find some sort of documentation on this issue, but so far no go.

    Disrupter on
    616610-1.png
  • zeenyzeeny Registered User regular
    edited May 2012
    object 1= new class1();
    unset(object 1);

    I do not get back the memory allocated for object 1(); I get back a very small portion of it.

    Nope. Unset does not force a garbage collection. You shouldn't concern yourself with memory management. It's not an issue that you don't get the memory back.
    Looking for memory leaks in a vm's memory management is a fools errand for a programmer who doesn't work on the VM.
    If within a class, I have

    object 1a=new class1();
    object 1b= new class1();
    unset(object 1a);
    unset(object 1b);
    1a allocated a lot more memory than object 1b, and I will get back the amount of memory allocated for object 1b for both object 1a and object 1b.

    Again, you don't care what memory you get back. It's not your job. However, it's perfectly normal for the first class instance to need more memory depending on what needs to be loaded and it's also perfectly normal not to get it back when all class objects are destroyed. Caching, delayed collection etc.
    Do you actually get an exhausted allocated memory exception?

    Edit: I'm trying to find a good article on PHP reference counting, but I"m striking out...

    zeeny on
  • DisrupterDisrupter Registered User regular
    I am running into exhausted allocated memory. Each of my objects is growing to be like 3 MB in size, which is insane. Its because there are a ton of child objects and each of those has children and all of them have a db_adapter.

    Basically, I have a product, then that product has a hard_product or a soft_product child, each of those have different children. The main product also contains an array of pricing for each different region it could be in, which consist of a cost, retail and shop_at_home object. And all of those have db_adapters to map to their table in the database.

    In each instance, the zend_db_result is much larger than id expect. It is not simply the size of an array containing 10-15 fields. For example the result from getting the main product info from the DB has like 30 fields but is 400kb in memory. When each product has such a leak, not to mention similar leaks for each child, it ends up being like 3MB an object. Which is quickly using up all my memory. I could get away with expanding the limit, since I wont have much more then like 10-15 products loaded at once (for this application) but it just seems...wrong.

    It shouldnt be using this much memory. And zend_db seems to be the culprit. I could just reset the adapter after I load, which would clear out the extra memory. But that is such a silly work around. Basically, ive tracked the problem and have an ugly hacky fix, but I find it hard to believe there isnt an actual way to tell the db adapter to not hold onto the last result. Or at least a way to clear it. I am looking at the framework now to try and find out exactly where its holding that result. It may actually be the PDO itself doing so, which I hope is not the case.

    616610-1.png
  • DisrupterDisrupter Registered User regular
    Well, I managed to fix the leak. I could go back through years of code and fix it elsewhere as well if I really want. But meh, it didnt really pop up as an issue much.

    The issue is that Zend_DB_Adapter_Abstract contains a Zend_Db_Select object which, when you execute a query contains a reference to Zend_DB_Adapter_Abstract. Thus a cyclical reference is created. And before 3.3 PHPs "garbage collector" did not play well with cyclical references.

    Normally I have models that contain their own db adapters.

    class Customer
    {
    private $db;

    public function __construct()
    {

    $this->db=new customer_db();
    }
    }

    Well the problem was, once a query was ran, the garbage collector no longer cleaned up the select object within the adapter which would be rather bloated. I have changed my code in this project to only create the database adapter when I need it, so it is no longer a property of the model. I then kill it when it is done. This has reduced the size of my "product" model from 3MB to 10KB.

    So I guess the thread is completed. I have beaten PHP! Im a winner.

    616610-1.png
  • GnomeTankGnomeTank What the what? Portland, OregonRegistered User regular
    Nightslyr wrote: »
    bowen wrote: »
    @Disrupter I think it may have to do with not using Rails.

    Fixed. :P
    Re-fixed. :rotate:

    Re-re-fixed :rotate:

    Sagroth wrote: »
    Oh c'mon FyreWulff, no one's gonna pay to visit Uranus.
    Steam: Brainling, XBL / PSN: GnomeTank, NintendoID: Brainling, FF14: Zillius Rosh SFV: Brainling
  • NightslyrNightslyr Registered User regular
    Yet another example where dependency injection saves the day.

  • zeenyzeeny Registered User regular
    edited May 2012
    Nightslyr wrote: »
    Yet another example where dependency injection saves the day.

    Most underrated pattern by far.
    The issue is that Zend_DB_Adapter_Abstract contains a Zend_Db_Select object which, when you execute a query contains a reference to Zend_DB_Adapter_Abstract. Thus a cyclical reference is created. And before 3.3 PHPs "garbage collector" did not play well with cyclical references.

    That doesn't sound right(particularly the part where the abstract gets the object), but I'm happy that your problem is resolved!!!!! Cyclical references can be a bitch.

    zeeny on
  • DisrupterDisrupter Registered User regular
    edited May 2012
    Eh, it may have been the Table abstract that contained a select which contained a refernece to said table, i was knee deep in the zend framework db library with like 10 files open all named abstract.php :)

    But regardless, cyclical reference + php version <5.3 meant big memory leak.

    dependency injection

    Tried googling this, got stuff about determining dependencies at runtime rather than compile time, so I wasnt sure how that applies to this situation. Can you explain the concept to me and how it applies here?

    Edit: Specifically looking at PHP examples it appears as though the concept is to not define the class of member but rather allow for that to be defined when the parent object is created by passing the member to it.

    So

    class employee
    {
    protected $person;
    public function __construct($person)
    {
    $this->person=$person;
    }

    }

    $person= new female();
    $employee=new employee($person);


    as opposed to

    class employee
    {
    protected $person;
    public function __construct()
    {
    $this->person=new female();
    }
    }

    Disrupter on
    616610-1.png
  • zeenyzeeny Registered User regular
    Dependency injection can cause circular dependency and often gets the blame for it. It shouldn't. Such a dependency is a design problem, not a pattern problem.

Sign In or Register to comment.