Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower
OK, this was was my experiment.... Image fresh with all my app and dependencies loades: 30MB After using it for some days/weeks: 160MB. SpaceTally new printSpaceAnalysis showed: Class code space # instances inst space percent inst average size ByteString 2785 413144 116244078 69.90 281.36 Array 3712 181772 8466668 5.10 46.58 ByteArray 8574 1319 8186802 4.90 6206.82 Bitmap 3653 303 6656340 4.00 21968.12 CompiledMethod 22467 90554 5685374 3.40 62.78 After executing ImageCleaner cleanUpForRelease: 36MB Then...I searched which part of #cleanUpForRelease: was making the difference, and it was: Smalltalk cleanUp: true except: #() confirming: false. So now it was time to know WHICH class did the diference, so I modified #cleanUp: aggressive except: exclusions confirming: aBool in these lines: "Run the cleanup code" classes do:[:aClass| Transcript show: 'Image size before cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. aClass cleanUp: aggressive. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. Transcript show: 'Image size after cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. ] displayingProgress: [:aClass| 'Cleaning up in ', aClass name]. I then opened a Transcript, and evaluated Smalltalk cleanUp: true except: #() confirming: false. I went to prepare Mate, and when I come back, the result was, of course: "Image size after cleaning MCFileBasedRepository : 39744008" That clean up ends up doing: flushAllCaches self allSubInstancesDo: [:ea | ea flushCache] So it sends #flushCache to all instances of MCHttpRepository and MCFileBasedRepository. Now what I wanted to see if it there was a particular repo that could take most of the space (like package-cache). And indeed, it was...I modified #flushCaches to: flushAllCaches | file | file := 'repos.txt' asFileReference writeStream text. self allSubInstancesDo: [:each | file nextPutAll: 'Image size before cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString; cr. each flushCache. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. file nextPutAll: 'Image size after cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString;cr. ]. file flush; close. And then I looked in the 'repos.txt' file. My package cache repo cleaned 60 MB. Glorp cleaned 35MB. Seaside30 cleaned 10MB. So...cleaning cache of just 3 repos frees approx 100MB. The question is....can we flush the cache safely? If they are called "cache", then I guess yes, we can. Thoughts? Thanks, On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
Guys, I have images also with seaside, magritte, glorp, postgresV2, etc and it is also around 200MB. I will try to do some research today and let you know.
Cheers,
On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker <marcus.denker@inria.fr>wrote:
On Jul 30, 2013, at 1:49 PM, "phil@highoctane.be" <phil@highoctane.be> wrote:
the changes file contained passwords and I replaced the text. So offsets may be wrong due to that.
Yes, the first thing I wanted to do is to recompile everything. Does not work.
Memorymonitor is not doing fanct stuff. It just counts instances.
Yes, but maybe it holds on to these instances?
Marcus
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it. Stef On Jul 31, 2013, at 8:53 PM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
OK, this was was my experiment....
Image fresh with all my app and dependencies loades: 30MB
After using it for some days/weeks: 160MB.
SpaceTally new printSpaceAnalysis showed:
Class code space # instances inst space percent inst average size ByteString 2785 413144 116244078 69.90 281.36 Array 3712 181772 8466668 5.10 46.58 ByteArray 8574 1319 8186802 4.90 6206.82 Bitmap 3653 303 6656340 4.00 21968.12 CompiledMethod 22467 90554 5685374 3.40 62.78
After executing ImageCleaner cleanUpForRelease: 36MB
Then...I searched which part of #cleanUpForRelease: was making the difference, and it was:
Smalltalk cleanUp: true except: #() confirming: false.
So now it was time to know WHICH class did the diference, so I modified #cleanUp: aggressive except: exclusions confirming: aBool
in these lines:
"Run the cleanup code" classes do:[:aClass| Transcript show: 'Image size before cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. aClass cleanUp: aggressive. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. Transcript show: 'Image size after cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. ] displayingProgress: [:aClass| 'Cleaning up in ', aClass name].
I then opened a Transcript, and evaluated
Smalltalk cleanUp: true except: #() confirming: false.
I went to prepare Mate, and when I come back, the result was, of course:
"Image size after cleaning MCFileBasedRepository : 39744008"
That clean up ends up doing:
flushAllCaches self allSubInstancesDo: [:ea | ea flushCache]
So it sends #flushCache to all instances of MCHttpRepository and MCFileBasedRepository.
Now what I wanted to see if it there was a particular repo that could take most of the space (like package-cache). And indeed, it was...I modified #flushCaches to:
flushAllCaches | file | file := 'repos.txt' asFileReference writeStream text. self allSubInstancesDo: [:each |
file nextPutAll: 'Image size before cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString; cr. each flushCache. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. file nextPutAll: 'Image size after cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString;cr.
]. file flush; close.
And then I looked in the 'repos.txt' file. My package cache repo cleaned 60 MB. Glorp cleaned 35MB. Seaside30 cleaned 10MB. So...cleaning cache of just 3 repos frees approx 100MB.
The question is....can we flush the cache safely? If they are called "cache", then I guess yes, we can.
Thoughts?
Thanks,
On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck <marianopeck@gmail.com> wrote: Guys, I have images also with seaside, magritte, glorp, postgresV2, etc and it is also around 200MB. I will try to do some research today and let you know.
Cheers,
On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Jul 30, 2013, at 1:49 PM, "phil@highoctane.be" <phil@highoctane.be> wrote:
the changes file contained passwords and I replaced the text. So offsets may be wrong due to that.
Yes, the first thing I wanted to do is to recompile everything. Does not work.
Memorymonitor is not doing fanct stuff. It just counts instances.
Yes, but maybe it holds on to these instances?
Marcus
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
This is yet another thing. Monticello keep all history data of all package in image, this is not a cache. When you delete it, your package loses it's history and you can not merge anymore. MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ]. We did that on Pharo2 release and it saved 8MB (nearly a third of the image). Marcus
This trimmed a couple of megs indeed. Then I did an ImageCleaner cleanUpForRelease. and saved. But nothing terrific as the size continues to grow. Need to see where those MorphExtensions are coming from... Phil On Thu, Aug 1, 2013 at 9:33 AM, Marcus Denker <marcus.denker@inria.fr>wrote:
On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
This is yet another thing.
Monticello keep all history data of all package in image, this is not a cache. When you delete it, your package loses it's history and you can not merge anymore.
MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].
We did that on Pharo2 release and it saved 8MB (nearly a third of the image).
Marcus
On Thu, Aug 1, 2013 at 4:33 AM, Marcus Denker <marcus.denker@inria.fr>wrote:
On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
This is yet another thing.
Yes, I was talking about the cache, not the ancestors. I think that removing the cache is indeed safe, isn't it?
Monticello keep all history data of all package in image, this is not a cache. When you delete it, your package loses it's history and you can not merge anymore.
MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].
We did that on Pharo2 release and it saved 8MB (nearly a third of the image).
Marcus
-- Mariano http://marianopeck.wordpress.com
On Aug 1, 2013, at 2:12 PM, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Aug 1, 2013 at 4:33 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
This is yet another thing.
Yes, I was talking about the cache, not the ancestors. I think that removing the cache is indeed safe, isn't it?
Yes! Marcus
Hi Stef, With Dale, we're working on a FileTree / GitFileTree without ancestors info in the repository [1] :) ... ... but recreated on the fly by GitFileTree for the Monticello history browsing tools [2] :( [1] is underway and ... [2] is already working in https://github.com/dalehenrich/filetree/tree/pharo2.0/repository/MonticelloF... Thierry Le 01/08/2013 08:56, Stéphane Ducasse a écrit :
since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
Stef
On Jul 31, 2013, at 8:53 PM, Mariano Martinez Peck <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote:
OK, this was was my experiment....
Image fresh with all my app and dependencies loades: 30MB
After using it for some days/weeks: 160MB.
SpaceTally new printSpaceAnalysis showed:
Class code space # instances inst space percent inst average size ByteString 2785 413144 116244078 69.90 281.36 Array 3712 181772 8466668 5.10 46.58 ByteArray 8574 1319 8186802 4.90 6206.82 Bitmap 3653 303 6656340 4.00 21968.12 CompiledMethod 22467 90554 5685374 3.40 62.78
After executing ImageCleaner cleanUpForRelease: 36MB
Then...I searched which part of #cleanUpForRelease: was making the difference, and it was:
Smalltalk cleanUp: true except: #() confirming: false.
So now it was time to know WHICH class did the diference, so I modified #cleanUp: aggressive except: exclusions confirming: aBool
in these lines:
"Run the cleanup code" classes do:[:aClass| Transcript show: 'Image size before cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. aClass cleanUp: aggressive. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. Transcript show: 'Image size after cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. ] displayingProgress: [:aClass| 'Cleaning up in ', aClass name]. I then opened a Transcript, and evaluated
Smalltalk cleanUp: true except: #() confirming: false.
I went to prepare Mate, and when I come back, the result was, of course:
"Image size after cleaning MCFileBasedRepository : 39744008"
That clean up ends up doing:
flushAllCaches self allSubInstancesDo: [:ea | ea flushCache] So it sends #flushCache to all instances of MCHttpRepository and MCFileBasedRepository.
Now what I wanted to see if it there was a particular repo that could take most of the space (like package-cache). And indeed, it was...I modified #flushCaches to:
flushAllCaches | file | file := 'repos.txt' asFileReference writeStream text. self allSubInstancesDo: [:each | file nextPutAll: 'Image size before cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString; cr. each flushCache. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. file nextPutAll: 'Image size after cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString;cr. ]. file flush; close. And then I looked in the 'repos.txt' file. My package cache repo cleaned 60 MB. Glorp cleaned 35MB. Seaside30 cleaned 10MB. So...cleaning cache of just 3 repos frees approx 100MB.
The question is....can we flush the cache safely? If they are called "cache", then I guess yes, we can.
Thoughts?
Thanks,
On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote:
Guys, I have images also with seaside, magritte, glorp, postgresV2, etc and it is also around 200MB. I will try to do some research today and let you know.
Cheers,
On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:
On Jul 30, 2013, at 1:49 PM, "phil@highoctane.be <mailto:phil@highoctane.be>" <phil@highoctane.be <mailto:phil@highoctane.be>> wrote:
> the changes file contained passwords and I replaced the text. So offsets may be wrong due to that. > Yes, the first thing I wanted to do is to recompile everything. Does not work.
> Memorymonitor is not doing fanct stuff. It just counts instances. > Yes, but maybe it holds on to these instances?
Marcus
-- Mariano http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
-- Mariano http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
participants (5)
-
Goubier Thierry -
Marcus Denker -
Mariano Martinez Peck -
phil＠highoctane.be -
Stéphane Ducasse