��
Hi all,
On 19.08.2015, at 23:09, Eliot Miranda <eliot.miranda@gmail.com> wrote:
> Hi All, but especially Tobias,
just for reference, I only had time to skim this mail.
I'll work through it more in-depth in September (vacation upcoming, sorry).
Also, I like Fuel as a way forward. I'd also like to deprecate DataStreams in
favor of Fuel[1], I also had a prototype of Fuel-based Monticello packages
which loaded much faster for large packages���
Best regards
�� �� �� �� -Tobias
[1]: PS: Hi Pharo folks, MCDataStream will break with Spur,
�� �� ��Have a look at System-topa.741 unil System-topa.473 in the
�� �� ��Squeak trunk or the attached change set, YMMV tho.
>
>�� �� ��as we know, ImageSegments are broken in Spur. They work occasionally, but when stressed the VM ends up crashing.�� The fundamental issue is that the ImageSegment code makes assumptions about object ordering that Spur violates.�� For example, here's ImageSegment's install:
>
> install
>�� �� �� ��"This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."
>
>�� �� �� ��| newRoots |
>�� �� �� ��state = #onFile ifTrue: [self readFromFile].
>�� �� �� ��state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
>�� �� �� �� �� �� �� ��endMarker := segment nextObject.�� �� �� �� "for enumeration of objects"
>�� �� �� �� �� �� �� ��endMarker == 0 ifTrue: [endMarker := 'End' clone]].
>�� �� �� ��(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>�� �� �� ��newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>�� �� �� ��state = #imported
>�� �� �� �� �� �� �� ��ifTrue: ["just came in from exported file"
>�� �� �� �� �� �� �� �� �� �� �� ��arrayOfRoots := newRoots]
>�� �� �� �� �� �� �� ��ifFalse: [
>�� �� �� �� �� �� �� �� �� �� �� ��arrayOfRoots elementsForwardIdentityTo: newRoots].
>�� �� �� ��state := #inactive.
>�� �� �� ��Beeper beepPrimitive
>
> So before the image segment bytes (the segment inst var) is loaded, the object after it is assigned to endMarker, and if there isn't an object after segment, a new object ('End' clone) is assigned to endMarker.
>
> This makes the assumption that objects are allocated in a strict order, and therefore endMarker will always be the object after segment.
>
> Loading the segment via "newRoots := self loadSegmentFrom: segment outPointers: outPointers" then turns segment into a zero-length WordArray and its contents into the objects loaded by the segment.�� Therefore, in the V3 system, the objects loaded from segment can be enumerated starting at segment nextObject and repeating until endMarker is found:
>
> allObjectsDo: aBlock
>�� �� �� ��"Enumerate all objects that came from this segment.�� NOTE this assumes that the segment was created (and extracted).�� After the segment has been installed (install), this method allows you to enumerate its objects."
>�� �� �� ��| obj |
>
>�� �� �� ��endMarker == nil ifTrue: [
>�� �� �� �� �� �� �� ��^ self error: 'Just extract and install, don''t writeToFile:'].
>�� �� �� ��segment size ~= 1 ifTrue: [
>�� �� �� �� �� �� �� ��^ self error: 'Vestigial segment size must be 1 (version word)'].
>
>�� �� �� ��obj := segment nextObject.�� "Start with the next object after the vestigial header"
>�� �� �� ��[obj == endMarker] whileFalse:�� "Stop at the next object after the full segment"
>�� �� �� �� �� �� �� ��[aBlock value: obj.
>�� �� �� �� �� �� �� ��obj := obj nextObject].�� "Step through the objects installed from the segment."
>
> Now, as written, this just won't work in Spur.
>
> a) the only place where there is any kind of stable order to objects is in oldSpace, so segment /has/ to be forced to old space to have any chance of its objects being in order when it gets converted from bytes to objects.
>
> b) 'End' clone will be in newSpace and so endMarker isn't reliable unless it was obtained by segment nextObject when segment was already in oldSpace.
>
> So it is perhaps possible to fix ImageSegments in Spur by forcing segment to oldSpace and being more careful with endMarker.�� But I think there is a better way.
>
> If the set of objects the segment contains can be obtained some how then this set can be simply enumerated, not depending on nextObject.�� The primitive has to answer the array of roots, so its result can't be changed to be the entire array.�� But segment could be becomed into an Array of all the objects in segment prior to it being loaded, in which case the above would become
>
>
> install
>�� �� �� ��"This operation retrieves the segment if necessary from file storage, installs it in memory, and replaces (using become:) all the root stubs with the reconstructed roots of the segment."
>
>�� �� �� ��| newRoots |
>�� �� �� ��state = #onFile ifTrue: [self readFromFile].
>�� �� �� ��state = #onFileWithSymbols ifTrue:
>�� �� �� �� �� �� �� ��[self readFromFileWithSymbols].
>�� �� �� ��(state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>�� �� �� ��newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>�� �� �� ��state = #imported
>�� �� �� �� �� �� �� ��ifTrue: "just came in from exported file"
>�� �� �� �� �� �� �� �� �� �� �� ��[arrayOfRoots := newRoots]
>�� �� �� �� �� �� �� ��ifFalse:
>�� �� �� �� �� �� �� �� �� �� �� ��[arrayOfRoots elementsForwardIdentityTo: newRoots].
>�� �� �� ��state := #inactive.
>�� �� �� ��Beeper beepPrimitive
>
> allObjectsDo: aBlock
>�� �� �� ��"Enumerate all objects that came from this segment.�� NOTE this assumes that the segment was created (and extracted).�� After the segment has been installed (install), this method allows you to enumerate its objects."
>�� �� �� ��| obj |
>
>�� �� �� ��segment isArray ifFalse:
>�� �� �� �� �� �� �� ��[^ self error: 'Segment hasn''t been loaded?'].
>
>�� �� �� ��segment do: aBlock
>
> and the endMarker instVar would be deleted.
>
> I am willing and ready to modify the primitive to convert the segment correctly.�� Who will volunteer to rewrite the image-level ImageSegment code to use the new primitive?
>
>
>
> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
> Hi Tobias,
>
> On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <Das.Linux@gmx.de> wrote:
> Hi all
> On 13.08.2015, at 02:15, Eliot Miranda <eliot.miranda@gmail.com> wrote:
>
> > Hi Tobias,
> >
> >�� �� forget that. I found it.�� THis right?
> >
> > Trunk test suite for Spur
> > Using existing cogspur r.3410
> > cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410 /tmp/d20150812-28620-etbikj
> >�� ��image test suite
> > VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
> > /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak -version
> > /var/lib/jenkins/workspace/Trunk/default/tests.st
> > spawning command 0 with timeout 1800 seconds: "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak" "-vm-sound-null" "-vm-display-null" "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image" "../tests.st"
> > (Command started with PID 28643)
> > 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
> > 2015-08-12T23:31:48.388+01:00: Running tests...
> > setsockopt: Protocol not available
> > setsockopt: Protocol not available
> > 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:612:
> >
> > Recursive not understood error encountered
> >
> >
> > I bet this is ImageSegment related.
>
> It sure is.
>
>�� �� �� �� ��BitmapStreamTests>testMatrixTransform2x3WithImageSegment
> sends
>�� �� �� �� ��BitmapStreamTests>validateImageSegment
>
>�� Basically the ImageSegment code has been working in Spur on a hope and a prayer.�� The code assumes objects are allocated in chronological order and that this order is preserved, not so with Spur.�� So post image segment loading someObject/nextObject is used to enumerate the objects loaded.�� This can't work reliably in Spur.�� I *think* (ok I hope) that I've implemented the segment load primitive in Spur to answer an Array of the objects loaded, so that these can be explicitly enumerated.
>
> So the job is a) to check that I have indeed implemented the primitive to do this and b) to rewrite the image segment loading code in the light of this.
>
> David, this is an example of something that isn't back portable and should not be back-ported.
>
> The most strange thing is that I cannot reproduce this on my Mac���
> There the test does not crash the image���
>
> Well then it may be a signed/unsigned bug in image loading instead.�� On linux the image typically gets loaded quite high in the address space and so a good portion of the heap ends up above 0x7FFFFFFF, or negative territory if misinterpreted as signed integers.�� On Mac and Windows the image tends to get loaded quite low and so has to be pretty big to fall foul of signedness issues.
>
>
> Busy right now but will check tomorrow.
>
>
> Best regards
>�� �� �� �� ��-Tobias
>
>
> >
> > On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
> > Hi Tobias,
> >
> > On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <Das.Linux@gmx.de> wrote:
> >
> > On 12.08.2015, at 20:55, Eliot Miranda <eliot.miranda@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > >
> > >�� �� ��Fabio's kindly done most of the changes.�� But some questions remain for more general discussion.�� See below.
> > >
> > > Fabio,�� thanks so much for doing this, and so quickly!
> > >
> > > On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
> > > Hi All,
> > >
> > >�� �� ��who will update http://squeak.org/downloads/ to include the 5.0 release?
> > >
> > > a) I suggest that the 5.0 all-in-one have a line in the left-hand table that includes the 4.6 release and that it precede the 4.6 release in the list.
> > >
> > > Done.
> > >
> > >
> > > b) the Trunk link points to TrunkImage.zip which contains a non-Spur image.
> > >
> > > So this is the biggie.�� What should we do?
> > >
> > > - add a Trunk 5.0 build with a different link?�� (Noooooo)
> > >
> > > - change�� http://build.squeak.org/job/SqueakTrunk to build what it says (Yessss please, who can do this?)
> >
> > Done: build.squeak.org/job/Trunk/
> > But as I said several times, Spur/trunk test just crash for month���
> >
> > can you point me to the crash?�� I'm looking at http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink and see 8 test failures but no crash.
> >
> >
> > >
> > > - add a Squeak 4.6 build job?�� Is that even worth it any more considering 4.6 is released and stable?�� If it is then what should it build?�� David?
> > >
> > >
> > > c) Spur VMs need to be linked to by an added line in the Virtual Machines list.
> > >
> > > Not applicable.�� the VM links point to the root of my site so they effectively point to both old and Spur VMs.
> > >
> > >
> > > d) Spur imagers need to be included in the Image and Changes list under Custom Installation
> > >
> > > Done.
> > >
> > >
> > >
> > > e) The SqueakV50.sources file also needs to be included in the Current Sources list.
> > >
> > > Done.
> > >
> > >
> > > f) we could start a History list for the 5.0 line
> > >
> > > This is probably quite a big reorg on the page and not urgent.
> > >
> > >
> > > So there are a few things to fix before 5.0 is freely downloadable.
> > >
> > > On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <ma.chris.m@gmail.com> wrote:
> > > In the 17 months since Squeak 4.5 was released, a huge development
> > > effort took place to create the next generation virtual-machine for
> > > the Squeak / Pharo / Newspeak family of programming systems.�� Squeak
> > > is the modern incarnation of the Smalltalk-80 programming environment
> > > originally developed at the Xerox PARC.
> > >
> > > "Squeak 5" introduces this new VM and associated new memory model,
> > > collectively referred to as "Spur".�� Presented [1] by Eliot Miranda
> > > and Cl��ment B��ra at the 2015 International Symposium on Memory
> > > Management, this new VM affords Squeak applications a significant
> > > boost in performance and memory management.�� Among other
> > > optimizations, the #become operation no longer requires a memory scan.
> > > Object pinning and ephemerons are also now supported.�� The release
> > > notes [2] provide more details.
> > >
> > > The new memory model requires a new image file format.�� Although this
> > > new format results in about a 15% increased memory requirement for the
> > > same number of 4.x objects, a new segmented heap allows memory to be
> > > given back to the OS when its no longer needed, a great benefit for
> > > application servers.
> > >
> > > As forward compatibility is as important to the Squeak community as
> > > backward compatibility, Squeak 5 is delivers an image with identical
> > > content as the recent 4.6 release.�� Although this new Squeak 5 VM
> > > cannot open images saved under the prior 4.x Cog format, objects and
> > > code can be easily exported from the 4.x image and then imported into
> > > Squeak 5.�� Applications whose code runs strictly above the Smalltalk
> > > meta layer will prove remarkably compatible with the new format, most
> > > applications will require no changes whatsotever.
> > >
> > > Squeak 5 is the result of monumental effort by a tiny group of very
> > > talented people, but its also just the beginning of yet a new effort;
> > > Spur is just a stepping stone to a more ambitious goals planned over
> > > the next five years.
> > >
> > > [1] -- A Partial Read Barrier for Efficient Support of Live
> > > Object-oriented Programming
> > > http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
> > >
> > > [2] -- Squeak 5 Release Notes
> > > http://wiki.squeak.org/squeak/6207