June 2011 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] [GS/SS Beta] serializing object graphs / ordering graph anchors

by Henrik Sperre Johansen

On 03.06.2011 11:00, Henrik Sperre Johansen wrote: > On 03.06.2011 10:38, Norbert Hartl wrote: >> >> Am 03.06.2011 um 10:20 schrieb Mariano Martinez Peck: >> >>> >>> >>>> If I look at the serializers I know the mechanics are simple: >>>> >>>> 1. take an object and look it up in a table. >>>> 2. if the object is in the table it has already been >>>> serialized. We take a reference from the table and write it >>>> down >>>> 3. if it isn't in the table it means the object has not >>>> been serialized and the contents need to be written. After >>>> that the object with a reference representation is written >>>> to the table >>>> 4. while writing the content of an object (inst vars) I >>>> encounter simple objects (direct string representation) or >>>> complex objects (composite). In the latter case we start >>>> over from 1. >>>> >>>> If the object graph is well interconnected than the odds >>>> are high that a stack will grow high until the first >>>> reference can be written. With a small pier kernel I >>>> already matched a stack size of 1000. >>>> >>>> >>>> In Fuel we traverse the graph but instead of using a recursion >>>> we use our own simple stack implementation where we push and >>>> pop objects. >>>> But Martin may explaint better than me. >>> >>> I'm interested in details how it works and if it avoids the deep >>> stack. Is there additional information about that? >>> >>> >>> In 4. you said "while writing the content of an object (inst vars) I >>> encounter simple objects (direct string representation) or complex >>> objects (composite). In the latter case we start over from 1.". >>> So... Imagine object X with two instVars Y and Z. Imagine the method >>> of the traverse is called #analye:. So you do #analize: X. Then, >>> in such method you check whether X has regular pointers to regular >>> objects, and if true, you analize them. So in this case you would >>> send #analyze: Y and #analize: Z, generating the recursion. Right ? >> >> yes, the time between analyse: Y and analyse: Z is dependent on the >> graph that is attached to Y. Y can have objects that have further >> objects etc. So leaving X aside, Y can be the first object to >> serialize and Z the last. >>> >>> Ok, instead of sending #analyze: Y and #analize: Z what we do is >>> just to do a push on a stack: #push: Y and #push: Z. And then we >>> are done with X, so we pop from the stack and we continue with the >>> next object (at some point in the future we will pop Y and Z) When >>> there are no more objects in the stack it means we are done. >> >> How do you write X? I can see that you turn a recursive trace into >> depth-by-depth trace but the difficult part is to write X. In X you >> need a reference representation of any sort. How do you derive that >> from the stack? >> >>> For performance (I think) we did FLSimpleStack. You can check also >>> FLSimpleStackTest. Both are in Fuel repo. >>> >>> Would something like that help for the stack in GemStone? >>> >> Yes. With this approach the resulting stack size is much smaller. To >> exceed the stack setting you need a graph containing an object which >> has a shortest route that is larger than the stack limit. And I think >> this is very unlikely :) >> I would need to change Sixx in order to have it work like this. I >> need to think about it. >> >> Thanks, >> >> Norbert > Yes, using a specific stack in the way Fuel does relies on there being > a separate analyze-phase before the actual serialization of the objects. > (Due to wanting to group objects of the same type together when > serializing) > > You could do the same in a "raw" serializer like SIXX: > 1) pop first object from stack > 2) if any of the instvars need to be serialized, and their ids can not > be found in cache, push self + missing objects > 3) if not, serialize self, then goto 1 > > The key difference is you can not considering an object "done" during > an analysis phase, but rather push it back on the stack for > reevaluation after its dependencies have been handled in the same way. > > So the resulting stack will be much larger than what you see in Fuel, > but still kept outside the call stack. > > Cheers, > Henry Note: This never grows the call stack. You're basically left with: [stack isEmpty ] whileFalse: [self serialize: stack pop] where serialize: either pushes objects and exits, or does actual serialization. Cheers, Henry

June 3, 2011

Re: [Pharo-project] [GS/SS Beta] serializing object graphs / ordering graph anchors

by Henrik Sperre Johansen

On 03.06.2011 10:38, Norbert Hartl wrote: > > Am 03.06.2011 um 10:20 schrieb Mariano Martinez Peck: > >> >> >>> If I look at the serializers I know the mechanics are simple: >>> >>> 1. take an object and look it up in a table. >>> 2. if the object is in the table it has already been >>> serialized. We take a reference from the table and write it down >>> 3. if it isn't in the table it means the object has not been >>> serialized and the contents need to be written. After that >>> the object with a reference representation is written to the >>> table >>> 4. while writing the content of an object (inst vars) I >>> encounter simple objects (direct string representation) or >>> complex objects (composite). In the latter case we start >>> over from 1. >>> >>> If the object graph is well interconnected than the odds are >>> high that a stack will grow high until the first reference >>> can be written. With a small pier kernel I already matched a >>> stack size of 1000. >>> >>> >>> In Fuel we traverse the graph but instead of using a recursion >>> we use our own simple stack implementation where we push and pop >>> objects. >>> But Martin may explaint better than me. >> >> I'm interested in details how it works and if it avoids the deep >> stack. Is there additional information about that? >> >> >> In 4. you said "while writing the content of an object (inst vars) I >> encounter simple objects (direct string representation) or complex >> objects (composite). In the latter case we start over from 1.". >> So... Imagine object X with two instVars Y and Z. Imagine the method >> of the traverse is called #analye:. So you do #analize: X. Then, in >> such method you check whether X has regular pointers to regular >> objects, and if true, you analize them. So in this case you would >> send #analyze: Y and #analize: Z, generating the recursion. Right ? > > yes, the time between analyse: Y and analyse: Z is dependent on the > graph that is attached to Y. Y can have objects that have further > objects etc. So leaving X aside, Y can be the first object to > serialize and Z the last. >> >> Ok, instead of sending #analyze: Y and #analize: Z what we do is >> just to do a push on a stack: #push: Y and #push: Z. And then we >> are done with X, so we pop from the stack and we continue with the >> next object (at some point in the future we will pop Y and Z) When >> there are no more objects in the stack it means we are done. > > How do you write X? I can see that you turn a recursive trace into > depth-by-depth trace but the difficult part is to write X. In X you > need a reference representation of any sort. How do you derive that > from the stack? > >> For performance (I think) we did FLSimpleStack. You can check also >> FLSimpleStackTest. Both are in Fuel repo. >> >> Would something like that help for the stack in GemStone? >> > Yes. With this approach the resulting stack size is much smaller. To > exceed the stack setting you need a graph containing an object which > has a shortest route that is larger than the stack limit. And I think > this is very unlikely :) > I would need to change Sixx in order to have it work like this. I need > to think about it. > > Thanks, > > Norbert Yes, using a specific stack in the way Fuel does relies on there being a separate analyze-phase before the actual serialization of the objects. (Due to wanting to group objects of the same type together when serializing) You could do the same in a "raw" serializer like SIXX: 1) pop first object from stack 2) if any of the instvars need to be serialized, and their ids can not be found in cache, push self + missing objects 3) if not, serialize self, then goto 1 The key difference is you can not considering an object "done" during an analysis phase, but rather push it back on the stack for reevaluation after its dependencies have been handled in the same way. So the resulting stack will be much larger than what you see in Fuel, but still kept outside the call stack. Cheers, Henry

June 3, 2011

Re: [Pharo-project] [GS/SS Beta] serializing object graphs / ordering graph anchors

by Norbert Hartl

Am 03.06.2011 um 10:20 schrieb Mariano Martinez Peck: > > >> If I look at the serializers I know the mechanics are simple: >> >> 1. take an object and look it up in a table. >> 2. if the object is in the table it has already been serialized. We take a reference from the table and write it down >> 3. if it isn't in the table it means the object has not been serialized and the contents need to be written. After that the object with a reference representation is written to the table >> 4. while writing the content of an object (inst vars) I encounter simple objects (direct string representation) or complex objects (composite). In the latter case we start over from 1. >> >> If the object graph is well interconnected than the odds are high that a stack will grow high until the first reference can be written. With a small pier kernel I already matched a stack size of 1000. >> >> >> In Fuel we traverse the graph but instead of using a recursion we use our own simple stack implementation where we push and pop objects. >> But Martin may explaint better than me. > > I'm interested in details how it works and if it avoids the deep stack. Is there additional information about that? > > > In 4. you said "while writing the content of an object (inst vars) I encounter simple objects (direct string representation) or complex objects (composite). In the latter case we start over from 1.". > So... Imagine object X with two instVars Y and Z. Imagine the method of the traverse is called #analye:. So you do #analize: X. Then, in such method you check whether X has regular pointers to regular objects, and if true, you analize them. So in this case you would send #analyze: Y and #analize: Z, generating the recursion. Right ? yes, the time between analyse: Y and analyse: Z is dependent on the graph that is attached to Y. Y can have objects that have further objects etc. So leaving X aside, Y can be the first object to serialize and Z the last. > > Ok, instead of sending #analyze: Y and #analize: Z what we do is just to do a push on a stack: #push: Y and #push: Z. And then we are done with X, so we pop from the stack and we continue with the next object (at some point in the future we will pop Y and Z) When there are no more objects in the stack it means we are done. How do you write X? I can see that you turn a recursive trace into depth-by-depth trace but the difficult part is to write X. In X you need a reference representation of any sort. How do you derive that from the stack? > For performance (I think) we did FLSimpleStack. You can check also FLSimpleStackTest. Both are in Fuel repo. > > Would something like that help for the stack in GemStone? > Yes. With this approach the resulting stack size is much smaller. To exceed the stack setting you need a graph containing an object which has a shortest route that is larger than the stack limit. And I think this is very unlikely :) I would need to change Sixx in order to have it work like this. I need to think about it. Thanks, Norbert > > > Norbert > >> >> One solution I can think of is to collect and order objects in the graph that cover smaller segments from the overall graph. This way I would construct a sorted collection that is sorted by stack depth (lowest first). The collection will then be serialized leading to a much smaller possible stack sizes. It might be there are cases it can't work but I think for most cases it will do its job. And it will be slow of course. >> >> Does anybody know if such a tool exists? >> >> Norbert >> >> >> >> >> >> -- >> Mariano >> http://marianopeck.wordpress.com >> > > > > > -- > Mariano > http://marianopeck.wordpress.com >

June 3, 2011

Re: [Pharo-project] [GS/SS Beta] serializing object graphs / ordering graph anchors

by Mariano Martinez Peck

> > If I look at the serializers I know the mechanics are simple: >> >> 1. take an object and look it up in a table. >> 2. if the object is in the table it has already been serialized. We take a >> reference from the table and write it down >> 3. if it isn't in the table it means the object has not been serialized >> and the contents need to be written. After that the object with a reference >> representation is written to the table >> 4. while writing the content of an object (inst vars) I encounter simple >> objects (direct string representation) or complex objects (composite). In >> the latter case we start over from 1. >> >> If the object graph is well interconnected than the odds are high that a >> stack will grow high until the first reference can be written. With a small >> pier kernel I already matched a stack size of 1000. >> >> > In Fuel we traverse the graph but instead of using a recursion we use our > own simple stack implementation where we push and pop objects. > But Martin may explaint better than me. > > > I'm interested in details how it works and if it avoids the deep stack. Is > there additional information about that? > > In 4. you said "while writing the content of an object (inst vars) I encounter simple objects (direct string representation) or complex objects (composite). In the latter case we start over from 1.". So... Imagine object X with two instVars Y and Z. Imagine the method of the traverse is called #analye:. So you do #analize: X. Then, in such method you check whether X has regular pointers to regular objects, and if true, you analize them. So in this case you would send #analyze: Y and #analize: Z, generating the recursion. Right ? Ok, instead of sending #analyze: Y and #analize: Z what we do is just to do a push on a stack: #push: Y and #push: Z. And then we are done with X, so we pop from the stack and we continue with the next object (at some point in the future we will pop Y and Z) When there are no more objects in the stack it means we are done. For performance (I think) we did FLSimpleStack. You can check also FLSimpleStackTest. Both are in Fuel repo. Would something like that help for the stack in GemStone? Cheers > Norbert > > > >> One solution I can think of is to collect and order objects in the graph >> that cover smaller segments from the overall graph. This way I would >> construct a sorted collection that is sorted by stack depth (lowest first). >> The collection will then be serialized leading to a much smaller possible >> stack sizes. It might be there are cases it can't work but I think for most >> cases it will do its job. And it will be slow of course. >> >> Does anybody know if such a tool exists? >> >> Norbert >> >> >> > > > -- > Mariano > http://marianopeck.wordpress.com > > > -- Mariano http://marianopeck.wordpress.com

June 3, 2011

Re: [Pharo-project] Do we freeze 1.3?

by Marcus Denker

On May 25, 2011, at 11:29 AM, Marcus Denker wrote: > Hello, > > 1.3 seems to stabilize... > > http://code.google.com/p/pharo/issues/list?can=2&q=milestone=1.3 > > just 12 issues left. So we have 5 issues left... We started to tag issues 1.4 for things that are not show-stoppers: http://code.google.com/p/pharo/issues/list?can=2&q=milestone=1.4 My idea would be -> try to fix the 5 issues next week -> as soon as they are fixed, set the version number to 1.3 (and open 1.4 so we don't get drowned in open issues that are easy to add yet not for 1.3) -> wait 3 weeks, while fixing bugs people find and then announce the release formally. The idea is to not have any release canditate, nor 1.3.x releases. 1.3 will just have updates as soon as somone fixes something important. Marcus -- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.

June 3, 2011

Re: [Pharo-project] [GS/SS Beta] serializing object graphs / ordering graph anchors

by Norbert Hartl

Am 02.06.2011 um 18:35 schrieb Mariano Martinez Peck: > > > On Thu, Jun 2, 2011 at 12:38 PM, Norbert Hartl <norbert(a)hartl.name> wrote: > Before I start a utility myself I want to ask if there is such a tool already. > > I'm using sixx for object graph serialization. It works quite good but it has some severe limits. > > I am interesting in listening which ones apart from performance/speed. > The limit I described below. > > Well, the limits are not really sixx dependent but a problem of usual serializers. My problem is the size of the stack such a tool generates. > > The stack is a problem??? By stack you mean stack of MethodContexts ? yes > why do you say a "usual" problem of serializers? do you mean because of the fact the the graph is usually traversed with a recursion? > yes > Especially in gemstone this appears to be a problem because the stack is limited per gem. > > > In which sense the stack is limited? > It is limited in size. It is a configured maximum where a stack size of 1000 is default AFAIK. > If I look at the serializers I know the mechanics are simple: > > 1. take an object and look it up in a table. > 2. if the object is in the table it has already been serialized. We take a reference from the table and write it down > 3. if it isn't in the table it means the object has not been serialized and the contents need to be written. After that the object with a reference representation is written to the table > 4. while writing the content of an object (inst vars) I encounter simple objects (direct string representation) or complex objects (composite). In the latter case we start over from 1. > > If the object graph is well interconnected than the odds are high that a stack will grow high until the first reference can be written. With a small pier kernel I already matched a stack size of 1000. > > > In Fuel we traverse the graph but instead of using a recursion we use our own simple stack implementation where we push and pop objects. > But Martin may explaint better than me. I'm interested in details how it works and if it avoids the deep stack. Is there additional information about that? Norbert > > One solution I can think of is to collect and order objects in the graph that cover smaller segments from the overall graph. This way I would construct a sorted collection that is sorted by stack depth (lowest first). The collection will then be serialized leading to a much smaller possible stack sizes. It might be there are cases it can't work but I think for most cases it will do its job. And it will be slow of course. > > Does anybody know if such a tool exists? > > Norbert > > > > > > -- > Mariano > http://marianopeck.wordpress.com >

June 3, 2011

Re: [Pharo-project] usability of Pharo and Squeak

by Geert Claes

I really like the idea Dale! I do realize you are still in the early stages but from a UI design point of view its a shame the typical back, forward, home and bookmark Internet browser buttons had to duplicated in the tODE. Since it is browser based I assume the idea is to allow developers to open as many (Internet browser) tabs as they want? Anyhow good stuff Dale :) -- View this message in context: http://forum.world.st/usability-of-Pharo-and-Squeak-tp3562378p3570224.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

June 3, 2011

[Pharo-project] [TODO] 1.3 (please help)

by Marcus Denker

Core ==== http://code.google.com/p/pharo/issues/list?can=2&q=milestone=1.3 4319 VM parameter handling problems 4326 Connection timeout problem 4328 Transcript: 5 ) Update problem when evaluating 1 to: 30 do: [: ... 4329 Transcript: 4) clear on cmd-s 4338 testAllDictionariesAreHealthy sometimes fails Full 4 Issues =========== 4245 Add file ArchiveViewer not working (fixed, needs to be added) 4303 completion and tab key 4318 Test for LedMorph (fixed, needs to be added) 4337 Fix OB Metacello Config (This is actualy 1.2-dev) http://code.google.com/p/pharo/issues/list?can=2&q=Milestone%3D1.3-DevImage Here it would be nice to have *someone* (not me) who is willing to look at the Metacello configs and include the work that is already finished. Marcus -- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.

June 3, 2011

Re: [Pharo-project] 1.2.2 First Impression

by Pat Maddox

On Jun 2, 2011, at 6:33 PM, Sean P. DeNigris wrote: > I moved from 1.1.1 to 1.2.2 for my development work and wanted to share some > things I noticed... > > Overall, 1.2.2 feels cleaner, snappier, and prettier. > > My Favorite Things: > * OCompletion is really fast!!! +100

June 3, 2011

Re: [Pharo-project] 1.2.2 First Impression

by Levente Uzonyi

On Thu, 2 Jun 2011, Sean P. DeNigris wrote: > I moved from 1.1.1 to 1.2.2 for my development work and wanted to share some > things I noticed... > > Overall, 1.2.2 feels cleaner, snappier, and prettier. > > My Favorite Things: > * OCompletion is really fast!!! I'm glad you like it. :) Levente > * Creating new method protocols - you can type spaces > * Help System starting to fill out, and easily accessible from the World > Menu > > Bugs / Regressions (I don't have time to check if each is still true in > trunk) > * Debugger > * the current statement is highlighted in "invisible blue". It's > impossible to see. > * Creating an accessor - "should be implemented" is highlighted, which is > good; but the window doesn't have focus, and clicking on the window's title > bar doesn't give the source pane focus > * Choosing a protocol for a created method - the selection doesn't jump to > the first matching category name like it did in 1.1.1 > * OB > * Syntax errors - "Nothing more expected ->" is not highlighted, making > you have to first select it before you delete it and correct your mistake > * Surrounding selected text (e.g. with quotes) - the surrounded text > doesn't stay highlighted (the 1.1.1 behavior, which I found very useful) > * Monticello - After changing a category name to a sub-category of a > different package (e.g. "MyLibrary" -> "MyOtherProject-MyLibrary"), the > package is not marked dirty in the MC browser until changes are viewed. > > I'm really excited to see what comes next!! We're creating a great system. > > Sean > > -- > View this message in context: http://forum.world.st/1-2-2-First-Impression-tp3569795p3569795.html > Sent from the Pharo Smalltalk mailing list archive at Nabble.com. > >

June 3, 2011