On Fri, 15 Jan 2010, Mariano Martinez Peck wrote:

On Fri, Jan 15, 2010 at 10:51 AM, Laval Jannik <jannik.laval@inria.fr>wrote:

...

Hi,

In some case I need to run my VM with more memory. For now, I use the parameter "-memory" when I run the VM.

Is it possible to define this value in the image ?

IIRC -memory allocates a fixed size memory chuck under unix and keeps it while the vm is running, -mmap is a better approach under unix.

I think you can do something like:

SmalltalkImage current vmParameterAt: 5 put: 100000; vmParameterAt: 6 put: 35000; vmParameterAt: 24 put: 16 * 1024 * 1024; vmParameterAt: 25 put: 8 * 1024 * 1024.

These are parameters for the GC, so are irrelevant here.

But...I am not sure how to know each parameter number what exactly is. Do you know where I can get that information?

See the comment in SmalltalkImage >> #vmParameterAt: Levente

To see your current data I think you can do:

SmalltalkImage current getVMParameters

Cheers

Mariano

Then I can use it in a script.

...

Cheers

--- Jannik Laval ---

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

right the memory logic is a moving target. In checking on unix -memory is malloc -mmap is mmap and on grow/shrink it grinds thru munmap/mmap. for os-x I just mmap the memory but I choose not to do the mmap/munmap since over the years I found various operating versions had funky responses to mmap/munmap On 2010-01-15, at 3:40 AM, Laval Jannik wrote:

Hi,

Thank you for your comment, I will try this.

Cheers Jannik

-- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

If it makes sense to have these parameters accessible for anyone more than a very hacker, then I strongly suggest we find the correct names for these and create meaningful name for getters/setters of them!

I haven't followed the discussion that closely, but I agree with you Cesar. Shall I add an entry in the bugtracker? Alexandre

Em 15/01/2010 07:57, Mariano Martinez Peck <marianopeck@gmail.com> escreveu:

On Fri, Jan 15, 2010 at 10:51 AM, Laval Jannik <jannik.laval@inria.fr> wrote:

Hi,

In some case I need to run my VM with more memory. For now, I use the parameter "-memory" when I run the VM.

Is it possible to define this value in the image ?

I think you can do something like:

SmalltalkImage current vmParameterAt: 5 put: 100000; vmParameterAt: 6 put: 35000; vmParameterAt: 24 put: 16 * 1024 * 1024; vmParameterAt: 25 put: 8 * 1024 * 1024.

But...I am not sure how to know each parameter number what exactly is. Do you know where I can get that information?

To see your current data I think you can do:

SmalltalkImage current getVMParameters

Cheers

Mariano

Then I can use it in a script.

Cheers

--- Jannik Laval ---

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Personally I think you need to add an entry. I am currently am doing a statistical study in this area for a client, of course not all of the data/solution will flow back to the community unless agreed to by the client, however I'll supply some numbers & thoughts to take us from the 1996 values into 2010. Also there will an active memory policy that will adjust some things to avoid some interesting edge cases as an example of the solution that lurks. On 2010-01-15, at 6:19 PM, csrabak@bol.com.br wrote:

I would only add the item if we could agree these parameters are to used with such frequency to make the addition worthwhile.

Do more users of Pharo feel in need to tweak these parameters before delivering their projects to users in production?

-- Cesar Rabak

-- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

I agree with cesar. I would also like to discuss the default parameters. Several people said that Pharo was slow for them. Maybe changing the default values may help in some point of view. On Sat, Jan 16, 2010 at 2:55 PM, Alexandre Bergel <alexandre@bergel.eu>wrote:

I added http://code.google.com/p/pharo/issues/detail?id=1836

I haven't found your emails in the Cc field.

Cheers, Alexandre

On 16 Jan 2010, at 01:17, John M McIntosh wrote:

...

Personally I think you need to add an entry.

I am currently am doing a statistical study in this area for a client, of course not all of the data/solution will flow back to the community unless agreed to by the client, however I'll supply some numbers & thoughts to take us from the 1996 values into 2010. Also there will an active memory policy that will adjust some things to avoid some interesting edge cases as an example of the solution that lurks.

On 2010-01-15, at 6:19 PM, csrabak@bol.com.br wrote:

...

I would only add the item if we could agree these parameters are to used with such frequency to make the addition worthwhile.

Do more users of Pharo feel in need to tweak these parameters before delivering their projects to users in production?

-- Cesar Rabak

-- = = = = = ====================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http:// www.smalltalkconsulting.com = = = = = ======================================================================

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On 2010-01-16, at 9:11 AM, csrabak@bol.com.br wrote:

I agree with the intent but cannot figure out a means for evaluating (testing) the tweaking of the parameters in 'scientific' way!!

-- Cesar Rabak

Well some clues are in http://www.smalltalkconsulting.com/papers/GCPaper/GCTalk%202001.htm See the "Squeak Memory Layout" page But really you have to have an understanding of how the GC behaves, and then the various parms (which aren't in the 2001 paper). then some Sunits, and generate charts like slide 92, then decide how to alter behaviour. Having Sunits is a important criteria since it then tells you if changes are impacting time to run in a meaningful way, since altering the values can have a negative effect. -- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

On 2010-01-16, at 4:08 PM, csrabak@bol.com.br wrote:

Is there a way to download a file of the presentation (be it powerpoint or PDF)?

A subset at http://www.esug.org/data/ESUG2006/pres/gctalkjohnmacintosh.pdf Generally I give a talk on the current paper to clients who pay for GC consulting. It takes about 3-4 hours to give at the moment.

...

But really you have to have an understanding of how the GC behaves, and then the various parms (which aren't in the 2001 paper).

OK. Is there other reference?

http://www.amazon.com/exec/obidos/ASIN/0471941484/corporatsmalltal http://www.amazon.com/exec/obidos/ASIN/0201116693/corporatsmalltal

...

then some Sunits, and generate charts like slide 92, then decide how to alter behaviour.

<thinking aloud> Given the multitude of platforms and configurations, perhaps instead (or in addition to) SUnit tests, we should create some script or some code that runs and do a diagnostic suggesting the best tuning for the user. </thinking aloud>

That would be helpful, usual complaints focus on opening and using browser windows as an example. More data from a note to the squeak list a year ago, you'd have to cross check some of the discussion of setting values from that era to today. Ok, the headings would be: statGCTime oldSpaceEnd youngSpaceEnd memoryEnd fullGCs fullGCTime incrGCs incrGCTime tenureCount statMarkCount statSweepCount statMkFwdCount statCompMoveCount statGrowMemory statShrinkMemory statRootTableCount statAllocationCount statSurvivorCount statSpecialMarkCount statIGCDeltaTime pendingFinalizationSignals Most of this data comes from the primitiveVMParameter 0 args: return an Array of VM parameter values; 1 arg: return the indicated VM parameter; 2 args: set the VM indicated parameter. VM parameters are numbered as follows: 1 end of old-space (0-based, read-only) 2 end of young-space (read-only) 3 end of memory (read-only) 4 allocationCount (read-only) 5 allocations between GCs (read-write) 6 survivor count tenuring threshold (read-write) 7 full GCs since startup (read-only) 8 total milliseconds in full GCs since startup (read-only) 9 incremental GCs since startup (read-only) 10 total milliseconds in incremental GCs since startup (read-only) 11 tenures of surving objects since startup (read-only) 12-20 specific to the translating VM 21 root table size (read-only) 22 root table overflows since startup (read-only) 23 bytes of extra memory to reserve for VM buffers, plugins, etc. 24 memory threshold above which shrinking object memory (rw) 25 memory headroom when growing object memory (rw) 26 interruptChecksEveryNms - force an ioProcessEvents every N milliseconds, in case the image is not calling getNextEvent often (rw) 27 number of times mark loop iterated for current IGC/FGC (read-only) includes ALL marking 28 number of times sweep loop iterated for current IGC/FGC (read-only) 29 number of times make forward loop iterated for current IGC/FGC (read-only) 30 number of times compact move loop iterated for current IGC/FGC (read-only) 31 number of grow memory requests (read-only) 32 number of shrink memory requests (read-only) 33 number of root table entries used for current IGC/FGC (read-only) 34 number of allocations done before current IGC/FGC (read-only) 35 number of survivor objects after current IGC/FGC (read-only) 36 millisecond clock when current IGC/FGC completed (read-only) 37 number of marked objects for Roots of the world, not including Root Table entries for current IGC/FGC (read-only) 38 milliseconds taken by current IGC (read-only) 39 Number of finalization signals for Weak Objects pending when current IGC/FGC completed (read-only) 40 BytesPerWord for this image Let's look at the values, so allow me to ramble... statGCTime 123904 This is the millisecond clock time when the sample was taken. The SqueakVMMemoryPolicy will run in two modes, active or passive. In passive mode the SqueakVMMemoryPolicy process wakes up every N (5?) Seconds and takes a reading of the GC statistics. In active mode a semaphore is used to signal the SqueakVMMemoryPolicy process to collect data on every incremental GC event. oldSpaceEnd 26725864 The byte offset of old space end. We don't have the start of memory here, it's a number elsewhere, but using that and this number we would know how many bytes oldspace uses, this boundary moves higher, or lower. youngSpaceEnd 27365792 Memory is broken into old space, youngspace, free, forwarding block reserve. This is the end of allocated young space, the start of free space. memoryEnd 30183488 This byte offset is the end of memory before the forwarding blocks, so let me talk about that space where since we have some other stats on forwarding blocks later. Note: The amount of space reserved for forwarding blocks should be chosen to ensure that incremental compactions can usually be done in a single pass. However, there should be enough forwarding blocks so a full compaction can be done in a reasonable number of passes, say ten. (A full compaction requires N object-moving passes, where N = number of non-garbage objects / number of forwarding blocks). di 11/18/2000 Re totalObjectCount: Provide a margin of one byte per object to be used for forwarding pointers at GC time. Since fwd blocks are 8 bytes, this means an absolute worst case of 8 passes to compact memory. In most cases it will be adequate to do compaction in a single pass. fullGCs 1 The number of full garbage collection count from zero at image startup, on a full GC we do a full GC against all objects in the image. Doing full GC frequently is a bad thing, but perhaps it cannot be avoid, depends on the app. fullGCTime 224 The number of milliseconds accumulative from zero at image startup take in full GC work. incrGCs 1561 Number of incremental garbage collection count from zero at image startup, on a incremental GC we scan objects in the root table and in the young space. incrGCTime 1259 The number of milliseconds accumulative from zero at image startup take in incremental GC work, on machines today you'll note we can do some GC work in less than 1 milliscond, thus this count of 1,259 is less than the total taken, which is 1,561. In order to ensure events that are NEEDED to be triggered with millisecond accuracy we would like to complete a incremental GC (IGC) in 1 ms or less. If for example a IGC took say 25ms, then you would have a 25 error in average on a Delay termination. tenureCount 12 By default we allocation 4,000 objects,then do a IGC, then see if the number of survivors is greater than the survivor count of 2000. If so then we tenure all the objects to old space by moving the oldSpaceEnd higher. This then ages the objects into oldspace and they are not looked at again when doing a IGC. The 4,000/2,000 numbers where picked in the 90's, 16Mhz machines. Some Squeakers report better GC behavior by making the allocation count & threashold count higher. How high is a number that requires some statistical calculations, again you want to ensure an IGC over N objects will take about 1 ms. SqueakVMMemoryPolicy>>calculateGoals contains some sample not used code for consideration and experimentation. Also see SmalltalkImage current vmParameterAt: 5 put: 8000. "do an incremental GC after this many allocations" SmalltalkImage current vmParameterAt: 6 put: 4000. "tenure when more than this many objects survive the GC" statMarkCount 13331 The Squeak GC algorithm uses a mark algorithm to mark which objects are accessible, then a sweep to mark objects that are free versus used, then a move and compacting event that consolidates the free space into one chunk at the end of young space. In this case we know we reached the 4,000 limit of allocation by the statAllocationCount and using the root table and youngspace marked 13,331 as used. statSweepCount 5579 At the end of the IGC we swept 5,579 object, which means 5,579 objects (free or allocated where in young space). The other data statSurvivorCount at 1,591 and the allocation count of 4,000 means on the last IGC there were 1,579 object alive, we allocated 4,000 more, and on the IGC only found 1,591 alive. statMkFwdCount 109 This is a count of the number times thru the forwarding block loop. statCompMoveCount 109 This is a count of the number times thru the compression move block loop. statGrowMemory 1 memoryEnd is the limit of memory, but it might not be the limit of available memory, if so the memoryEnd can move by providing more memory. This counts the number times this happens. This happens when free memory becomes too low, or a large object cannot be allocated. statShrinkMemory 1 When free space becomes too large, then we give some back to the operating system. This counts how many times this happens. The low memory and too much free memory thresholds are arbitrarily set values. 24 memory threshold above which shrinking object memory (rw) 25 memory headroom when growing object memory (rw) Excessive statGrowMemory/statShrinkMemory incrrementing points to a problem where you are cycling between a high/low value exceeding the limits, this is expensive and the thresholds should be adjust to avoid thi. s statRootTableCount 331 "Record that the given oop in the old object area points to an object in the young area. HeaderLoc is usually = oop, but may be an addr in a forwarding block." This is the number of roots, the number of objects in oldspace (usually) that point to some object in young space. The IGC needs this information to determine which objects are live in young space. Other Smalltalk might call this the remember table. Now a bit about Squeak's implementation, some other smalltalk record what the object is and what the slot is, this can require a lot of memory and require memory to be allocated when you can't allocate memory (crash). Squeak records the object, but now if this is a Collection then later when we mark we must visit ALL the slots in the object to determine which object is in young space. How this issue shows up in Squeak is stories about people allocating large collections then for some reason squeak becomes slow, but if they do a full GC, squeak becomes fast. The SqueakVMMemoryPolicy>>calculateGoals attempts to solve this by doing: | target | ... (statMarkCount ) > (statAllocationCount*2) ifTrue: [[Smalltalk forceTenure] on: Error do: [:ex | ]]. "Tenure if we think too much root table marking is going on" If we think took much marking is going on because the root table counting is too high then we invoke Smalltalk forceTenure which tenures everything to oldspace, and hopefully removing the problem object from the root table. Although this seem excessive too much mark scanning means less CPU time for byte code interpretation. statAllocationCount 4000 At the start of the IGC, this is the number of objects that were allocated, this helps determine if the IGC was done as a result of object allocation, or because some other allocation issue required the IGC to run to fix an issue, say a large object being allocated statSurvivorCount 1591 At the end of the IGC this is the number of objects that still were alive, using the statAllocationCount, and statSurvivorCount you might be able to determine values that ensure a IGC doesn't tenure objects, yet finishes in about 1 ms. statSpecialMarkCount 633 count the number of objects we mark which are accessible by the special objects, receiver, method, activecontext, message selector, method class, receiver class,etc, and the internal remapping objects array. statIGCDeltaTime 1 For this IGC, it took 1 ms pendingFinalizationSignals 0 This is the number of weak objects that have "died" in this GC cycle. ------------------------------------------------------- I'll note Squeak out of the box has a bias to compact memory, there is a condition that Squeak can enter into where when you allocate that next object memory is full, then you do a IGC and get back a reasonal number of bytes, you allocate the object, then fail on the next one, then a IGC, get back some bytes, allocate the object, fail on the next one... This then triggers 10 of thousands of IGC events a second, Squeak appears to hang. The [Smalltalk setGCBiasToGrowGCLimit: 8*1024*1024] on: Error do: [:ex | ^self]. [Smalltalk setGCBiasToGrow: 1] on: Error do: [:ex | ^self]. in SqueakVMMemoryPolicy startUp Alters the allocation behaviour to grow memory versus doing just another IGC, so bias to grow memory. However we can't grow memory forever so the setGCBiasToGrowGCLimit to 8MB to indicate where a full GC should occur after 8MB of biased growth. This avoid the bug in the memory allocation logic. On 10-Jan-09, at 3:24 PM, Aidan Gauland wrote:

John M McIntosh <johnmci <at> smalltalkconsulting.com> writes:

...

Ok, well why don't you post the first 5 lines of the file and I'll explain them.

Ok, here they are...

123904 26725864 27365792 30183488 1 224 1561 1259 12 13331 5579 109 109 1 1 331 4000 1591 633 1 0 129430 27191724 27961664 30183488 1 224 1567 1265 13 1530 4126 192 192 1 1 36 4000 142 351 2 0 133846 27191724 29113412 30183488 1 224 1572 1270 13 3441 4341 318 318 1 1 44 4000 370 350 2 0 139371 27191724 29181436 30183488 1 224 1578 1282 13 3793 4411 270 270 1 1 44 4000 418 350 2 0 143936 27191724 29757020 30183488 1 224 1583 1294 13 4263 4474 333 333 1 1 44 4000 479 314 2 0

Oh, and I had to do "SqueakVMMemoryPolicy new; collectStatistics: true; run" to make it create a file; just "SqueakVMMemoryPolicy run" didn't create a file.

-- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

On 2010-01-17, at 9:08 AM, Hernán Morales Durand wrote:

Hi John,

2010/1/16 John M McIntosh <johnmci@smalltalkconsulting.com>:

...

On 2010-01-16, at 4:08 PM, csrabak@bol.com.br wrote:

...

Is there a way to download a file of the presentation (be it powerpoint or PDF)?

A subset at http://www.esug.org/data/ESUG2006/pres/gctalkjohnmacintosh.pdf

Generally I give a talk on the current paper to clients who pay for GC consulting. It takes about 3-4 hours to give at the moment.

Have you started experiments or do you plan to include information about (dead) timestamped garbage collection mechanisms?

I don't understand some of the statistical graphics (pages 80, 82-86),

There aren't any graphs on pages 80,83, do yo have the page numbers correct?

have you used MMU or BMU curves to display the GC responsiveness? (just thinking, it would be cool to show application-centric GC metrics)

do you have some links, google is not helpful this morning

Best regards,

Hernán

-- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================

2010/1/17 John M McIntosh <johnmci@smalltalkconsulting.com>:

On 2010-01-17, at 9:08 AM, Hernán Morales Durand wrote:

...

Have you started experiments or do you plan to include information about (dead) timestamped garbage collection mechanisms?

Herman, I didn't understand this question at all, can you explain further?

IIRC the main reference paper about Stampede GC algorithms is http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.58.3285 Cheers, Hernán