As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options
PHP and memory...
Disrupter
Registered User regular
Registered User regular
Hey guys. I am working on a PHP project using Zend Framework and I am trying to track the bloated memory issues I am running into.
Long story short, I am making a business application that has an opportunity, each opportunity can have multiple quotes, each quote can have multiple items, each item has a product. Each product can have many other parameters.
As I step through the code, outputting memory, it seems as though the objects themselves are not too bad, but memory gets absolutely abused when I first create an object of any type.
For example, $product=new application_model_product_item(); $product->load($sku);
will increase the memory usage by 1 MB. Which is insane.
But if I clear out that object using a custom $product->clear(); function and then unset($product); I only gain back like 100kb. And, if I do
$product=new application_model_product_item(); $product->load($sku);
$product2=new application_model_product_item(); $product2->load($sku2);
my memory footprint only increases by 1.1MB. So its like the object itself is only 100kb, but the act of creating the first product uses a meg. I believe this may be due to the zend auto-loader. But I dont know for sure.
This would not be the end of the world. It sucks, but thats the biggest class, and if it takes up 1MB one time, then all other objects of it take up .1, I could deal.
But whats strange is, as I go through a quote, looping through each of its items, creating a product for each of those items has a footprint of 1MB. But creating TWO products for each of those items only costs 1.1MB. For some reason, creating that first product for each item is an entire meg.
This persists throughout. Components of the product each have a much larger footprint when first created, but subsequent objects of that class do not.
I can not figure out the leak. Ive gone through all the children and grand children, trying to tackle what is causing it, and each time, each child takes up a lot of memory when it is first created, but then it doesnt seem to actually use much as an object.
Basically it seems like
if parent one creates child one, tons of memory is used. But it does not use much to create child two.
But then if parent two creates child one, tons of memory is used again, but it does not use much to create child two.
I can not for the life of me figure this out. I am not doing anything that should require a ton of memory, I am simply loading data from a DB and storing the information in data model objects. But they are absolutely bloated.
Long story short, I am making a business application that has an opportunity, each opportunity can have multiple quotes, each quote can have multiple items, each item has a product. Each product can have many other parameters.
As I step through the code, outputting memory, it seems as though the objects themselves are not too bad, but memory gets absolutely abused when I first create an object of any type.
For example, $product=new application_model_product_item(); $product->load($sku);
will increase the memory usage by 1 MB. Which is insane.
But if I clear out that object using a custom $product->clear(); function and then unset($product); I only gain back like 100kb. And, if I do
$product=new application_model_product_item(); $product->load($sku);
$product2=new application_model_product_item(); $product2->load($sku2);
my memory footprint only increases by 1.1MB. So its like the object itself is only 100kb, but the act of creating the first product uses a meg. I believe this may be due to the zend auto-loader. But I dont know for sure.
This would not be the end of the world. It sucks, but thats the biggest class, and if it takes up 1MB one time, then all other objects of it take up .1, I could deal.
But whats strange is, as I go through a quote, looping through each of its items, creating a product for each of those items has a footprint of 1MB. But creating TWO products for each of those items only costs 1.1MB. For some reason, creating that first product for each item is an entire meg.
This persists throughout. Components of the product each have a much larger footprint when first created, but subsequent objects of that class do not.
I can not figure out the leak. Ive gone through all the children and grand children, trying to tackle what is causing it, and each time, each child takes up a lot of memory when it is first created, but then it doesnt seem to actually use much as an object.
Basically it seems like
if parent one creates child one, tons of memory is used. But it does not use much to create child two.
But then if parent two creates child one, tons of memory is used again, but it does not use much to create child two.
I can not for the life of me figure this out. I am not doing anything that should require a ton of memory, I am simply loading data from a DB and storing the information in data model objects. But they are absolutely bloated.
Disrupter on
0
Posts
And my short answer would be: PHP :rotate:
Try gc_disable(); at the top of your script and see if that fixes it?
Fixed. :P
Anyway...turns out, our php version is 5.2.14 which is before the garbage collector was upgraded to handle cylical objects. So I can not do gc_disable();
There seems to be a number of leaks, all coming form zend_framework. The Zend_Db_Row object for the product is 200kb in size, and even if I unset it, it doesnt seem to actually get removed. For now, when it comes to this project I am going to just give myself some memory leeway and increase the limit while processing a quote. There shouldnt be more then a few lines per quote, each with one product so the thing shouldnt get too big regardless. Its just painful to see such simple stuff absolutely eating up memory.
Seems like zend_db and possibily the zend_auto_loader are the culprit. Unfortunately backs against the wall in regards to getting something to show to the higher ups, so for now I have no choice but to increase the memory limit like a chump and hopefully find time to optimize and investigate down the line (yeah right...). Thanks for the attempt at helping (and ridiculing PHP. Good news is, my new company is primarily .net so I should be getting away from PHP as time goes on.)
Also, I'd just like to point out that PHP garbage collection could happen at requirement, so if you are running close to the limit, but aren't encountering exhaustion errors you shouldn't assume you have a problem.
object 1= new class1();
unset(object 1);
I do not get back the memory allocated for object 1(); I get back a very small portion of it.
If within a class, I have
object 1a=new class1();
object 1b= new class1();
unset(object 1a);
unset(object 1b);
1a allocated a lot more memory than object 1b, and I will get back the amount of memory allocated for object 1b for both object 1a and object 1b.
The other issue is that if I have two parent classes, and BOTH call
object 1a=new class1();
object 1b= new class1();
unset(object 1a);
unset(object 1b);
they will both have the same issue described above. So within a class, each first instance of an object is taking up more memory than the object needs to be allocated and releases only what it actually needs.
I actually believe I have tracked it down to zend_db. If I make a query to the database and get a zend_db_table_row object, and then unset that object, I do not get any more memory back from it. All the memory is still allocated. I have looked into this and apparently its because zend_db buffers the last result, so the result still has a pointer to it. Since all my objects have a data model associated with them which call a get($id) to populate the object, they are all leaving these buffered results behind, which seem to have large memory footprints. I need to find a way to release that buffered result. I am not sure why this occurs with only the first child object. Perhaps it has something to do with zend_db caching? I dont know.
But looking through, the memory leak definately happens when I get the zend_db_table_row object and then release it. It does not free up the allocated memory at release.
EDIT:
Yep. If I do
$db=new Zend_DB_Object();
$results=$db->get($id);
unset($results);
I do not free up the memory from $results;
But if I add $db=new Zend_DB_Object(); again, then the memory footprint is identical to before $resutls=$db->get($id); So $db is holding a reference to $results that I need to find a way to clear. You'd think itd be easy to find some sort of documentation on this issue, but so far no go.
Nope. Unset does not force a garbage collection. You shouldn't concern yourself with memory management. It's not an issue that you don't get the memory back.
Looking for memory leaks in a vm's memory management is a fools errand for a programmer who doesn't work on the VM.
Again, you don't care what memory you get back. It's not your job. However, it's perfectly normal for the first class instance to need more memory depending on what needs to be loaded and it's also perfectly normal not to get it back when all class objects are destroyed. Caching, delayed collection etc.
Do you actually get an exhausted allocated memory exception?
Edit: I'm trying to find a good article on PHP reference counting, but I"m striking out...
Basically, I have a product, then that product has a hard_product or a soft_product child, each of those have different children. The main product also contains an array of pricing for each different region it could be in, which consist of a cost, retail and shop_at_home object. And all of those have db_adapters to map to their table in the database.
In each instance, the zend_db_result is much larger than id expect. It is not simply the size of an array containing 10-15 fields. For example the result from getting the main product info from the DB has like 30 fields but is 400kb in memory. When each product has such a leak, not to mention similar leaks for each child, it ends up being like 3MB an object. Which is quickly using up all my memory. I could get away with expanding the limit, since I wont have much more then like 10-15 products loaded at once (for this application) but it just seems...wrong.
It shouldnt be using this much memory. And zend_db seems to be the culprit. I could just reset the adapter after I load, which would clear out the extra memory. But that is such a silly work around. Basically, ive tracked the problem and have an ugly hacky fix, but I find it hard to believe there isnt an actual way to tell the db adapter to not hold onto the last result. Or at least a way to clear it. I am looking at the framework now to try and find out exactly where its holding that result. It may actually be the PDO itself doing so, which I hope is not the case.
The issue is that Zend_DB_Adapter_Abstract contains a Zend_Db_Select object which, when you execute a query contains a reference to Zend_DB_Adapter_Abstract. Thus a cyclical reference is created. And before 3.3 PHPs "garbage collector" did not play well with cyclical references.
Normally I have models that contain their own db adapters.
class Customer
{
private $db;
public function __construct()
{
$this->db=new customer_db();
}
}
Well the problem was, once a query was ran, the garbage collector no longer cleaned up the select object within the adapter which would be rather bloated. I have changed my code in this project to only create the database adapter when I need it, so it is no longer a property of the model. I then kill it when it is done. This has reduced the size of my "product" model from 3MB to 10KB.
So I guess the thread is completed. I have beaten PHP! Im a winner.
Re-re-fixed :rotate:
Most underrated pattern by far.
That doesn't sound right(particularly the part where the abstract gets the object), but I'm happy that your problem is resolved!!!!! Cyclical references can be a bitch.
But regardless, cyclical reference + php version <5.3 meant big memory leak.
Tried googling this, got stuff about determining dependencies at runtime rather than compile time, so I wasnt sure how that applies to this situation. Can you explain the concept to me and how it applies here?
Edit: Specifically looking at PHP examples it appears as though the concept is to not define the class of member but rather allow for that to be defined when the parent object is created by passing the member to it.
So
class employee
{
protected $person;
public function __construct($person)
{
$this->person=$person;
}
}
$person= new female();
$employee=new employee($person);
as opposed to
class employee
{
protected $person;
public function __construct()
{
$this->person=new female();
}
}