- Pharo-dev - lists.pharo.org

Esteban's ChangeLog week of 27 November 2017

by estebanlm＠gmail.com

Hello! This is my weekly ChangeLog, from 27 November 2017 to 3 December 2017. You can see it in a better format by going here: http://log.smallworks.eu/web/search?from=27/11/2017&to=3/12/2017 ChangeLog ========= 28 November 2017: ----------------- * I just released [Iceberg 0.6.4](https://github.com/pharo-vcs/iceberg/releases/tag/v0.6.4). This is a bugfix release :) cheers! Esteban

Dec. 4, 2017

Re: [Pharo-dev] Alternative bytecode sets demo request

by Eliot Miranda

Hi Ben, On Sun, Dec 3, 2017 at 6:08 PM, Ben Coman <btc(a)openinworld.com> wrote: > I'm shooting in the dark here since its a bit hard to grasp... > a. The implications of support alternative bytecode sets > * Do the alternative bytecode sets run in parallel to normal Pharo > bytecodes? > Yes. There's a single bit in the header of a compiled method that selects between two bytecode sets. In current Squeak and Pharoi VMs, if the bit is unset then the normal bytecode set is used, and if the nit is set, the Sista bytecode set is used. * Can the bytecode sets be loaded adhoc/dynamically into an Image? > or only compiled into the VM? > Only compiled into the VM. > b. How to implement/work with alternative bytecodes > In Squeak there are class-side accessors in CompiledCode. See CompiledCode class>>installSecondaryBytecodeSet:. Note that the support isn't quite finished yet. I hope to have the Sista vytecde set ready for the next release of Squeak. ClÃ©ment is providing it in Pharo. In Squeak the BytecodeEncoder hierarchy implements the necessary support for multiple bytecode sets. Via CompiledCode class>>installSecondaryBytecodeSet: the system is informed as to which bytecode set to use. This must match the bytecode set(s) that are in the VM. Classes beneath BytecodeEncoder, in particular EncoderForV3PlusClosures and EncoderForSistaV1 (in package BytecodeSets at http://source.squeak.org/VMMaker) encode the current and the Sista sets respectively. For (a.) I guess an exciting opportunity is facilitating an Ethereum > virtual machine > running as a Process inside a Pharo image, managed by tools written in > Pharo. > Other languages are already getting in a party which would be good for > Pharo to join. > In such a revolutionary domain I expect people are more willing to > experiment > with alternative systems like Pharo, and I bet Pharo could excel in that > domain. > * https://github.com/ethereum/go-ethereum/blob/master/core/vm/opcodes.go > * https://medium.com/@jeff.ethereum/optimising-the- > ethereum-virtual-machine-58457e61ca15 > * https://github.com/pirapira/awesome-ethereum-virtual- > machine#evm-implementations > > > To help (b.), would it be feasible to blog a mini-demo showing how to > install/use alternative bytecodes? maybe for something like this simple RPN > calculator substitute for "arith.c"... > * https://github.com/philipaconrad/mini-vm/tree/master/examples/arith > * https://github.com/philipaconrad/mini-vm/blob/ > master/examples/arith/arith.c > That's not how this will work. Essentially it is transparent. Once the bytecode set support is installed in the compiler one simply compiles Smalltalk (or anything else that produces parse trees that can be output to compiled methods via the BytecodeEncoder hierarchy) and the method will be output in the currently selected bytecode set. So it's not something one plays around with, unless that is, one is developing the bytecode set in the Vm sim,letter. > > (a calculator demo could be an interesting adjunct to Sven's calculator > tutorial...) > https://medium.com/concerning-pharo/rediscovering-the-ux-of- > the-legendary-hp-35-scientific-pocket-calculator-d1d497ece999 > > > cheers -ben > -- _,,,^..^,,,_ best, Eliot

Dec. 4, 2017

Alternative bytecode sets demo request

by Ben Coman

I'm shooting in the dark here since its a bit hard to grasp... a. The implications of support alternative bytecode sets * Do the alternative bytecode sets run in parallel to normal Pharo bytecodes? * Can the bytecode sets be loaded adhoc/dynamically into an Image? or only compiled into the VM? b. How to implement/work with alternative bytecodes For (a.) I guess an exciting opportunity is facilitating an Ethereum virtual machine running as a Process inside a Pharo image, managed by tools written in Pharo. Other languages are already getting in a party which would be good for Pharo to join. In such a revolutionary domain I expect people are more willing to experiment with alternative systems like Pharo, and I bet Pharo could excel in that domain. * https://github.com/ethereum/go-ethereum/blob/master/core/vm/opcodes.go * https://medium.com/@jeff.ethereum/optimising-the-ethereum-virtual-machine-5… * https://github.com/pirapira/awesome-ethereum-virtual-machine#evm-implementa… To help (b.), would it be feasible to blog a mini-demo showing how to install/use alternative bytecodes? maybe for something like this simple RPN calculator substitute for "arith.c"... * https://github.com/philipaconrad/mini-vm/tree/master/examples/arith * https://github.com/philipaconrad/mini-vm/blob/master/examples/arith/arith.c (a calculator demo could be an interesting adjunct to Sven's calculator tutorial...) https://medium.com/concerning-pharo/rediscovering-the-ux-of-the-legendary-h… cheers -ben

Dec. 4, 2017

Re: [Pharo-dev] Design pattern for wrapper/proxy objects?

by Ben Coman

On 4 December 2017 at 00:10, Luke Gorrie <luke(a)snabb.co> wrote: > Howdy, > > I have some Smalltalk objects that represent external data and I am looking > for a neat way to access them. > > The data that I am dealing with is C binary objects described by DWARF > schemas. So I will load a type declaration like this: > > struct outer { > struct inner { > int a, b; > }; > }; > > and then I will want to be able to write convenient expressions like: > > aStruct outer inner a + aStruct outer inner b > > I have implemented this already but I am not sure that I took the right > approach. Seems like a reasonable approach, except difficulties with the broadness of your DSL domain (as you've found.) > I have implemented these accessors like #outer #inner #a #b via a > messageNotUnderstood: method that looks into the object, so that 'aStruct > outer' is equivalent to 'aStruct field: #outer'. My concern is that the > names of the C struct fields can collide with existing Smalltalk methods and > then I will get totally unexpected results. > > Is there a canonical solution to this problem? > > The first idea I have is to add a bit of syntatic sugar, like to write > 'aStruct _outer _inner _a' where the leading underscore avoids method name > collisions and is then stripped off in messageNotUnderstood. That doesn't > seem especially elegant though. > > I can't quite bring myself to write verbose things like > > ((aStruct child: #outer) child: #inner) child: #a > > An easy improvement might be to subclass your DwarfStruct from ProtoObject rather than Object to significantly reduce the surface area of name clashes. * https://github.com/pharo-project/pharo/blob/development/src/Kernel/ProtoObj… But I've not worked much with ProtoObject so I'm not sure what difficulties that might bring. In particular, can anyone comment on extra requirements to support object display in GT Inspector? A naming clash with the remaining ProtoObject methods seems extremely unlikely, and the easiest path would be to just guard against loading such structs. DwarfStruct>>loadStructure ... nameClashes := ProtoObject methodNames. self fieldnames do: [ :f | (nameClashes includes: f) ifTrue: [ self error: 'Seriously!? Your trying awful hard to break me' ] ] Just for the fun of it(!?) you might even try reducing that surface area by renaming that just in your personal Images by renaming ProtoObject methods. (Actually thats any interesting experiment I'll have to try later when I've got Pharo in front of me.) The only other thing I can think of is, presuming you've got a single class (e.g. DwarfStruct) holding the structure as data. for each loaded structure, generate "DwarfStruct subclass: CStruct1" and CStruct1>>fieldname so that method lookup stops at the first class and doesn't run all the way up the class hierarchy. But that would still be susceptible to breaking the system by overloading messages critical to system operation. cheers -ben

Dec. 3, 2017

Design pattern for wrapper/proxy objects?

by Luke Gorrie

Howdy, I have some Smalltalk objects that represent external data and I am looking for a neat way to access them. The data that I am dealing with is C binary objects described by DWARF schemas. So I will load a type declaration like this: struct outer { struct inner { int a, b; }; }; and then I will want to be able to write convenient expressions like: aStruct outer inner a + aStruct outer inner b I have implemented this already but I am not sure that I took the right approach. I have implemented these accessors like #outer #inner #a #b via a messageNotUnderstood: method that looks into the object, so that 'aStruct outer' is equivalent to 'aStruct field: #outer'. My concern is that the names of the C struct fields can collide with existing Smalltalk methods and then I will get totally unexpected results. Is there a canonical solution to this problem? The first idea I have is to add a bit of syntatic sugar, like to write 'aStruct _outer _inner _a' where the leading underscore avoids method name collisions and is then stripped off in messageNotUnderstood. That doesn't seem especially elegant though. I can't quite bring myself to write verbose things like ((aStruct child: #outer) child: #inner) child: #a

Dec. 3, 2017

Re: [Pharo-dev] [Vm-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

by Ben Coman

On 27 November 2017 at 00:24, Todd Blanchard <tblanchard(a)mac.com> wrote: > > i'm getting the idea that we should probably write a test suite/library > for FFI > I noticed these... * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/ platforms/unix/plugins/SqueakFFIPrims/ffi-test-main.c * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/ Cog/platforms/Cross/plugins/SqueakFFIPrims/sqFFITestFuncs.c * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/ Cog/platforms/Cross/plugins/IA32ABI/AlienSUnitTestProcedures.c These look like c-code test frames for FFI. Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them? And also some general info... * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/ Cog/platforms/unix/plugins/SqueakFFIPrims/00README > > On Nov 24, 2017, at 12:54 AM, Ben Coman <btc(a)openinworld.com> wrote: > > > On 24 November 2017 at 13:16, Ben Coman <btc(a)openinworld.com> wrote: > >> >> >> On 22 November 2017 at 21:59, Ben Coman <btc(a)openinworld.com> wrote: >> >>> >>> >>> On 22 November 2017 at 13:38, Todd Blanchard <tblanchard(a)mac.com> wrote: >>> >>>> >>>> I've been trying to track this down for a couple weeks now. >>>> >>> >>>> I have concluded that structs passed by value to functions on the 64 >>>> bit VM are not properly populated. The struct's memory is all zero'd. >>>> >>>> I found this while trying to work with LibClang and found that >>>> functions that fetched code locations from code ranges always returned >>>> invalid zero'd locations. After spending some time with lldb I have traced >>>> the problem into the native code and found that the argument is not correct. >>>> >>>> I've carved out the wee bit of clang to reproduce this in a tiny >>>> library. >>>> >>>> The gist of it is below and the entire file is included. Basically the >>>> struct passed to the function clang_getRangeStart is zero'd memory >>>> regardless of the data I send from the image side. >>>> >>> My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole. I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases. The attached zipfile containing libstruct.c a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers. There is a Makefile with three main targets: 1. make layout - statically compiles libstruct.c and runs produced a.out to display structure layouts 2. make run - downloads Pharo, starts it loading LibStruct.st , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console 3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs. Note Pharo will freeze and you need to move to LLDB. Try these commands... frame variable call print_struct(&GoodStruct1_fmt, &aStruct) continue That works on Ubuntu 16.04 64 bit. You will need to tune it for OSX. clang and lldb are required. ================================ The offsets configured in the class variables of all ExternalStructs correctly matches that reported by the C code test frame results here... $ make layout clang -g libstruct.c ./a.out GoodStruct1: uint32_t:int1: 01 4 uint32_t:int2: 05 4 GoodStruct2: uint32_t:int1: 01 4 uint32_t:int2: 05 4 void*:ptr_data: 09 8 GoodStruct3: void*:ptr_data: 01 8 uint32_t:int1: 09 4 uint32_t:int2: 13 4 BadStruct: void*:ptr_data1: 01 8 void*:ptr_data2: 09 8 uint32_t:int1: 17 4 uint32_t:int2: 21 4 ================================ $ make run clang -g -o libstruct.so -shared -fPIC libstruct.c getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st Now manually browse to and run LibStruct>>>LibStructTest>>#testStructs LibStructTest>>testStructs |good1Struct good2Struct good3Struct badStruct | good1Struct := GoodStruct1 new int1: 2; int2: 3. good2Struct := GoodStruct2 new int1: 2; int2: 3; ptr_force_int: 4. good3Struct := GoodStruct3 new int1: 2; int2: 3; ptr_force_int: 4. badStruct := BadStruct new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5. self assert: (LibStruct tryGood1: good1Struct) equals: 6. self assert: (LibStruct tryGood2: good2Struct) equals: 6. self assert: (LibStruct tryGood3: good3Struct) equals: 6. "Problem exposed in next line" self assert: (LibStruct tryBad: badStruct) equals: 6. which on console produces... GoodStruct1: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 GoodStruct2: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 void*:ptr_data: 09 8 = 04 00 00 00 00 00 00 00 GoodStruct3: void*:ptr_data: 01 8 = 04 00 00 00 00 00 00 00 uint32_t:int1: 09 4 = 02 00 00 00 uint32_t:int2: 13 4 = 03 00 00 00 BadStruct: void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00 void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 17 4 = 29 03 03 00 uint32_t:int2: 21 4 = 00 00 00 00 Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two. The first time this is run after Image boots seems like BadStruct gets some random data. ================================ In same image, subsequent runs of LibStruct>>>LibStructTest>>testStructs give only all zeros for BadStruct, as Todd observed. GoodStruct1: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 GoodStruct2: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 void*:ptr_data: 09 8 = 00 00 00 00 00 00 00 00 GoodStruct3: void*:ptr_data: 01 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 09 4 = 02 00 00 00 uint32_t:int2: 13 4 = 03 00 00 00 BadStruct: void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00 void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 17 4 = 00 00 00 00 uint32_t:int2: 21 4 = 00 00 00 00 cheers -ben

Dec. 3, 2017

Re: [Pharo-dev] Where is the latest linux vm?

by Nicolas Cellier

It seems that the .travis_tests.sh all stalled for recent squeak versions of linux32x86 The build are then marked as failed and nothing uploaded on bintray. https://travis-ci.org/OpenSmalltalk/opensmalltalk-vm/jobs/309909223 For pharo-vm it seems to be a different problem when compiling libgit2 https://travis-ci.org/OpenSmalltalk/opensmalltalk-vm/jobs/309909228 opensmalltalk / linux32x86 deserves some more care... Too many failures for too much time! 2017-12-03 11:29 GMT+01:00 stephan <stephan(a)stack.nl>: > On 02-12-17 11:18, Thierry Goubier wrote: > >> Le 02/12/2017 Ã 11:00, stephan a Ã©crit : >> >>> On 01-12-17 21:21, Thierry Goubier wrote: >>> >>>> To avoid segfaults, I'm using one from November. >>>> >>> >>> Where does that come from? Or do you compile it yourself? >>> >> >> It comes from the bintray opensmalltalk folder. For example, >> >> https://bintray.com/opensmalltalk/vm/cog/201711061254#files >> >> I've never had much success compiling it myself. >> > > Ha. I found the latest in > https://bintray.com/opensmalltalk/vm/cog/201712010755#files > > But there is no 32 bit linux x86 pharo build there, only the 64 bit. > > Stephan > > >

Dec. 3, 2017

Re: [Pharo-dev] Where is the latest linux vm?

by Thierry Goubier

Le 03/12/2017 Ã 11:29, stephan a Ã©critÂ : > On 02-12-17 11:18, Thierry Goubier wrote: >> Le 02/12/2017 Ã 11:00, stephan a Ã©critÂ : >>> On 01-12-17 21:21, Thierry Goubier wrote: >>>> To avoid segfaults, I'm using one from November. >>> >>> Where does that come from? Or do you compile it yourself? >> >> It comes from the bintray opensmalltalk folder. For example, >> >> https://bintray.com/opensmalltalk/vm/cog/201711061254#files >> >> I've never had much success compiling it myself. > > Ha. I found the latest in > https://bintray.com/opensmalltalk/vm/cog/201712010755#files > > But there is no 32 bit linux x86 pharo build there, only the 64 bit. Interesting: one only find the newspeak build there in linux x86 32bits (no pharo, no squeak builds). And it seems to have been the case for a little while. Thierry > Stephan > > >

Dec. 3, 2017

Re: [Pharo-dev] Where is the latest linux vm?

by stephan

On 02-12-17 11:18, Thierry Goubier wrote: > Le 02/12/2017 Ã 11:00, stephan a Ã©critÂ : >> On 01-12-17 21:21, Thierry Goubier wrote: >>> To avoid segfaults, I'm using one from November. >> >> Where does that come from? Or do you compile it yourself? > > It comes from the bintray opensmalltalk folder. For example, > > https://bintray.com/opensmalltalk/vm/cog/201711061254#files > > I've never had much success compiling it myself. Ha. I found the latest in https://bintray.com/opensmalltalk/vm/cog/201712010755#files But there is no 32 bit linux x86 pharo build there, only the 64 bit. Stephan

Dec. 3, 2017

Garbage Collection (was Re: Discussing the roadmap)

by Ben Coman

Hi Eliot, Clement, On 7 July 2017 at 00:41, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > > > - Better support for large heaps (GC tuning API, incremental GC). > > Pharo 64 bit is now able to manage large heap. However better > > performance can be proposed by offering better settings for the > > different GC zone. > > The most important thing here is the incremental GC. Spur has a generation > scavenger that collects garbage in newly created objects (new space), > and a mark-compact collector that collects and compacts garbage in old space. > > Right now on my 2.3GHz MacMini doing normal development, the generation > scavenger causes pauses of 1ms or less, and the mark-compact collector > than causes pauses of around 200ms. Both account for about 0.75% of > entire execution time (1.5% total), so the scavenger is invoked frequently > and the pauses that it creates are not noticeable. But the occasional > pauses created by the mark-compact collector /are/ noticeable, > especially in games and music. > > The idea for the incremental collector is to implement a mark-sweep or > mark-sweep-compact collector for old space that works incrementally, > doing a little bit of work on each invocation, probably immediately after a scavenge. > It is intended to avoid the long pauses caused by the non-incremental > mark-compact collector and make the system more suitable for games, music, etc. Reading http://www.mirandabanda.org/cogblog/2013/09/13/lazy-become-and-a-partial-re… "An alternative implementation, oft-used in Lisp systems, is to add a read barrier to all object access, and mark objects as forwarders. This can be used to implement a >>>lazy copying garbage collection<<<< where objects are copied from one semi-space to another in parallel to the main program (the âmutatorâ). To become, or move an object one replaces the objectâs header or first field with a forwarding pointer to the desired target or copy in a new location, marking the âcorpseâ as forwarded. The program checks the forwarded flag on each access. If there is hardware support, as in a Lisp machine, this can work well. But without hardware support, and like the object table representation, it has costs, slowing down program execution due to the scattering of forwarding checks and forwarding pointer follows throughout program execution." I'm curious... Given we now have forwarders with Spur, are we already sufficiently paying the cost of forwarding checks that a lazy copying garbage collector might be a feasible form of incremental garbage collection? I presume "parallel to the main program" means garbage collection occuring in a separate thread to the main vm thread, potentially resulting in very low main program pause times for garbage collection. I found this a useful summary of the terminology... * https://www.dynatrace.com/resources/ebooks/javabook/reduce-garbage-collecti… and I'm curious how our planned Incremental CG fits those categories. That article got me contemplating our performance constraint of the VM only operating in only in a single native thread. Even though GC is a only a few percent of performance, I wondered what a concurrent GC might look like for us. I found this video describing concurrent GC in Go (ignore first 11:20 and the second half was not so interesting) * https://pusher.com/sessions/meetup/the-realtime-guild/golangs-realtime-garb… where they present some interesting charts of their concern with latency pauses. (btw they reference a multi-language GC latency benchmark https://github.com/WillSewell/gc-latency-experiment) And for balance of that I found... https://blog.plan99.net/modern-garbage-collection-911ef4f8bd8e And then my mind wandered around implementation details of concurrent garbage collection. To organise and quiet my thoughts I needed to put pen to paper, so I thought sharing that might stir thoughts for others. Probably naive and please excuse the brain dump format... Considering two threads... * Main program thread "MP" * Garbage collection thread "GC" with object-space shared between them, consisting of objects split in object-header & object-body... 1. Only "MP" mutates the object-body, updating slots creating new edges in the object-graph, and relocating objects in memory using forwarders. This rule avoids potential race conditions without needing to add synchronisation code affecting performance of "MP". 2. Concurrently "GC" performs a Marking Phase by following the object-graph tricolour tagging objects gray & black. it needs to mutate the object-header, in a synchronised way something like... a. Load object-header from shared object-space into local variable H <== object-header b. Modify header into another local variable H' <== H + updated GC color bits c. Atomic compare-and-swap H' back into object-space object-header <== if H then H' "MP" gets priority. Conflicts presumed rare. Then considering object mutation by "MP" concurrent/overlapped with "GC" marking... 3. TLDR; see 4. In old-space if 'mutated' object is linked to a 'target' object a. if 'mutated' isBlack, "MP" marks 'target' gray (to be later processed by "GC"). b. if 'mutated' isGray, overlapped 'mutated' gray->black by "GC" while 'target' remains white would break tricolor invariant (is it a credible case?), options... i. mark 'target' gray, the state anyway in case 'mutated' gray -> black ii. re-mark 'mutated' gray, i.e. normally gray->gray, and rarely black->gray iii. optimisticly just check after mutation if color changed (cached reads cheaper than writes and conflicts presumed rare) then set color to gray. c. if 'mutated' isWhite, options... i. do nothing and let "GC" reach 'target' normally, as long as overlapped white->black cannot occur ii. if white->black possible, do same as (3.b) "MP" marks 'target' gray, to be later processed by "GC". 4. Overall simplification of (3.) might be... "MP" checks for any color change during mutation, and only then marks 'mutated' object gray. How expensive would such a check be? Presumed marking is infrequent and can be done safely like (2.) 5. Eden/survivor space is ignored by "GC" thread. No point in adding work to the gray set until survivors filtered. Stop the world scavenging done as normal by "MP", only marking objects gray when they are moved to old-space. Post scavenging options... a. Resume the world and leave it for "GC" to process these recently grayed objects. b. Keep world stopped for "MP" to complete marking, emptying gray set to transition to Sweep Phase, at which point "MP" resumes the world. It doesn't matter if subsequently objects are added to the gray set, since existing white objects can never again be referenced by "MP". 6. Sweeping can be safely done by "GC" since white-objects are unreachable from "MP". "GC" can also take time to determine an optimum page P to compact and then (per 1.) passes to "MP" via a relocation-queue the objects to be relocated using forwarding pointers. "GC" could even spend extra time to determine the optimum relocation-destination without impacting the performance of "MP". When "MP" empties the relocation-queue, "GC" starts on the next Marking Phase. 7. Now a question remains about multi-threaded flattening of forwarding pointers. If two threads simultaneously perform an identical transform from... someObject-slot --> forwarder-b --> finalObject-c to... someObject-slot --> finalObject-c does it matter that these operation may be done twice overlapping? Options... a. One mitigation could be for "GC" to identify forwarders to be flattened and queue them for "MP" to process (reuse the compaction-queue). This is work that "MP" would need to do anyway, but brings it forward to be dealt with at a convenient time. b. I guess "MP" and "GC" could play nice together if when encountering a slot containing with a forwarding pointer they both do something similar to (2.) like... i. Load object-slot from shared object-space into local variable S <== slot ii. Set local variable S' <== flattened/followed pointer iii. Atomic compare-and-swap S' back into object-space, (object-slot <== if S then S') - "MP" gets priority but presume conflicts rare anyway. - While this violates rule (1.) the presumed frequency of encountering forwarding pointers is low for "MP", so performance should not be affected. Indeed by "GC" pre-emptively flattening forwarders the frequency "MP" sees reduces. 8. The release of the compacted page back to the OS is held up by forwarding pointers. Forwarders are part of the graph followed in the Marking Phase they get marked gray/black just like objects if they are referenced. After forwarders are fully flattened they are skipped by Marking and end up marked white and released just like any other object. Once all forwarders are released, the page is released to the OS. If "GC" can effectively flatten pointers concurrently with "MP" during its normal Marking Phase, then pages would be released back to the OS in a timely manner. Now one thing I am curious about how the GC tri-color marking is implemented. At https://clementbera.wordpress.com/2014/01/16/spurs-new-object-format it describes how Spur's object header has three bits for GC. * isGray * isRemembered * isMarked (I presume this means marked "black") Do the bits imply the gray set is not stored in a separate data structure on the heap, but rather distributed in-place, which I guess would require multiple passes through memory to grow the gray set? So this is not an area I'm set up to seriously work on, but I remain curious and hopefully its a useful seed discussion for others. cheers -ben P.S. At https://clementbera.wordpress.com/2017/09/19/vm-learning-memory-management/ nice to hear that you have Sophie (I presume) continuing with the VM.

Dec. 3, 2017