- Pharo-dev - lists.pharo.org

Re: [Pharo-project] pharo closures

by Toon Verwaest

> No I can't. Since I did it, I naturally think it's a good idea. > Perhaps, instead of denigrating it without substantiating your claims > you could propose (and then implement, and then get adopted) a better > idea? Sure. My own VM will take a lot longer to get done! ;) I don't want to blemish any of your credit for building a cool VM. I was rather just wondering why you decided to go for this particular implementation which seems unobvious to me. Hence the question. I guess I should've formulated it slightly differently :) More info below. > > I can see why it would pay off for Lisp programmers to have > closures that run like the Pharo closures, since it has O(1) > access performance. However, this performance boost only starts > paying off once you have at least more than 4 levels of nested > closures, something which, unlike in LISP, almost never happens in > Pharo. Or at least shouldn't happen (if it does, it's probably ok > to punish the people by giving them slower performance). > > > Slower performance than what? BTW, I think you have things backwards. > I modelled the pharo closure implementation on lisp closures, not the > other way around. This is exactly what I meant. The closures seem like a very good idea for languages with very deeply nested closures. Lisp is such a language with all the macros ... I don't really see this being so in Pharo. > > This implementation is pretty hard to understand, and it makes > decompilation semi-impossible unless you make very strong > assumptions about how the bytecodes are used. This then again > reduces the reusability of the new bytecodes and probably of the > decompiler once people start actually using the pushNewArray: > bytecodes. > > > Um, the decompiler works, and in fact works better now than it did a > couple of years ago. So how does your claim stand up? For example when I just use the InstructionClient I get in pushNewArray: and then later popIntoTemp. This combination is supposed to make clear that you are storing a remote array. This is not what the bytecode says however. And this bytecode can easily be reused for something else; what if I use the bytecode to make my own arrays? What if this array is created in a different way? I can think of a lot of ways the temparray could come to be using lots of variations of bytecodes, from which I would never (...) be able to figure out that it's actually making the tempvector. Somehow I just feel there's a bigger disconnect between the bytecodes and the Smalltalk code and I'm unsure if this isn't harmful. But ok, I am working on the Opal decompiler of course. Are you building an IR out with your decompiler? If so I'd like to have a look since I'm spending the whole day already trying to get the Opal compiler to somehow do what I want... getting one that works and builds a reusable IR would be useful. (I'm implementing your field-index-updating through bytecode transformation btw). > > You might save a teeny tiny bit of memory by having stuff garbage > collected when it's not needed anymore ... but I doubt that the > whole design is based on that? Especially since it just penalizes > the performance in almost all possible ways for standard methods. > And it even wastes memory in general cases. I don't get it. > > > What has garbage collection got to do with anything? What precisely > are you talking about? Indirection vectors? To understand the > rationale for indirection vectors you have to understand the rationale > for implementing closures on a conventional machine stack. For lisp > that's clear; compile to a conventional stack as that's an easy model, > in which case one has to store values that outlive LIFO discipline on > the heap, hence indirection vectors. Why you might want to do that in > a Smalltalk implementation when you could just access the outer > context directly has a lot to do with VM internals. Basically its the > same argument. If one can map Smalltalk execution to a conventional > stack organization then the JIT can produce a more efficient execution > engine. Not doing this causes significant problems in context management. With the garbage collection I meant the fact that you can already collect part of the stack frames and leave other parts (the remote temps) and only get them GCd later on when possible. I do understand why you want to keep them on the stack as long as possible. The stack-frame marriage stuff for optimizations is very neat indeed. What I'm more worried about myself is the fact that stackframes aren't just linked to each other and share memory that way. This means that you only have 1 indirection to access the method-frame (via the homeContext), and 1 for the outer context. You can directly access yourself. So only the 4rd context will have 2 indirections (what all contexts have now for remotes). From the 5th on it gets worse... but I can't really see this happening in real world situations. Then you have the problem that since you don't just link the frames and don't look up values via the frames, you have to copy over part of your frame for activation. This isn't necessarily -that- slow (although it is an overhead); but it's slightly clumsy and uses more memory. And that's where my problem lies I guess ... There's such a straightforward implementation possible, by just linking up stackframes (well... they are already linked up anyway), and traversing them. You'll have to do some rewriting whenever you leave a context that's still needed, but you do that anyway for the remote temps right? > The explanation is all on my blog > <http://www.mirandabanda.org/cogblog/2009/01/14/under-cover-contexts-and-the…> > and in my Context Management in VisualWorks 5i > <http://www.esug.org/data/Articles/misc/oopsla99-contexts.pdf> paper. > > But does a bright buy like yourself find this /really/ hard to > understand? It's not that hard a transformation, and compared to what > goes on in the JIT (e.g. in bytecode to machine-code pc mapping) its > pretty trivial. I guess I just like to really see what's going on by having a decent model around. When I look at the bytecodes; in the end I can reconstruct what it's doing ... as long as they are aligned in the way that the compiler currently generates them. But I can easily see how slight permutations would already throw me off completely. > > > But probably I'm missing something? > > > It's me who's missing something. I did the simplest thing I knew > could possibly work re getting an efficient JIT and a Squeak with > closures (there's huge similarity between the above scheme and the > changes I made to VisualWorks that resulted in a VM that was 2 to 3 > times faster depending on platform than VW 1.0). But you can see a > far more efficient and simple scheme. What is it? Basically my scheme isn't necessarily far more efficient. It's just more understandable I think. I can understand scopes that point to their outer scope; and I can follow these scopes to see how the lookup works. And the fact that it does some pointer dereferencing and copying of data less is just something that makes me think it wouldn't be less efficient than what you have now. My problem is not that your implementation is slow, rather that it's complex. And I don't really see why this complexity is needed. Obviously playing on my ego by telling me I should be clever enough to understand it makes me say I do! But I feel it's not the easiest; and probably less people understand this than the general model of just linking contexts together. > > best, > Eliot cheers, Toon

March 25, 2011

Re: [Pharo-project] pharo closures

by Eliot Miranda

Hi Toon, On Thu, Mar 24, 2011 at 4:46 PM, Toon Verwaest <toon.verwaest(a)gmail.com>wrote: > I've just been working the whole day on the OPAL decompiler, getting it > almost to run (already works for various methods). Now ... > > could someone please explain to me why the current implementation of > closures isn't a terribly bad idea? > No I can't. Since I did it, I naturally think it's a good idea. Perhaps, instead of denigrating it without substantiating your claims you could propose (and then implement, and then get adopted) a better idea? > I can see why it would pay off for Lisp programmers to have closures that > run like the Pharo closures, since it has O(1) access performance. However, > this performance boost only starts paying off once you have at least more > than 4 levels of nested closures, something which, unlike in LISP, almost > never happens in Pharo. Or at least shouldn't happen (if it does, it's > probably ok to punish the people by giving them slower performance). > Slower performance than what? BTW, I think you have things backwards. I modelled the pharo closure implementation on lisp closures, not the other way around. > This implementation is pretty hard to understand, and it makes > decompilation semi-impossible unless you make very strong assumptions about > how the bytecodes are used. This then again reduces the reusability of the > new bytecodes and probably of the decompiler once people start actually > using the pushNewArray: bytecodes. > Um, the decompiler works, and in fact works better now than it did a couple of years ago. So how does your claim stand up? > > You might save a teeny tiny bit of memory by having stuff garbage collected > when it's not needed anymore ... but I doubt that the whole design is based > on that? Especially since it just penalizes the performance in almost all > possible ways for standard methods. And it even wastes memory in general > cases. I don't get it. > What has garbage collection got to do with anything? What precisely are you talking about? Indirection vectors? To understand the rationale for indirection vectors you have to understand the rationale for implementing closures on a conventional machine stack. For lisp that's clear; compile to a conventional stack as that's an easy model, in which case one has to store values that outlive LIFO discipline on the heap, hence indirection vectors. Why you might want to do that in a Smalltalk implementation when you could just access the outer context directly has a lot to do with VM internals. Basically its the same argument. If one can map Smalltalk execution to a conventional stack organization then the JIT can produce a more efficient execution engine. Not doing this causes significant problems in context management. The explanation is all on my blog<http://www.mirandabanda.org/cogblog/2009/01/14/under-cover-contexts-and-the…>and in my Context Management in VisualWorks 5i<http://www.esug.org/data/Articles/misc/oopsla99-contexts.pdf>paper. But does a bright buy like yourself find this /really/ hard to understand? It's not that hard a transformation, and compared to what goes on in the JIT (e.g. in bytecode to machine-code pc mapping) its pretty trivial. > But probably I'm missing something? > It's me who's missing something. I did the simplest thing I knew could possibly work re getting an efficient JIT and a Squeak with closures (there's huge similarity between the above scheme and the changes I made to VisualWorks that resulted in a VM that was 2 to 3 times faster depending on platform than VW 1.0). But you can see a far more efficient and simple scheme. What is it? best, Eliot > > cheers, > Toon > >

March 25, 2011

[Pharo-project] pharo closures

by Toon Verwaest

I've just been working the whole day on the OPAL decompiler, getting it almost to run (already works for various methods). Now ... could someone please explain to me why the current implementation of closures isn't a terribly bad idea? I can see why it would pay off for Lisp programmers to have closures that run like the Pharo closures, since it has O(1) access performance. However, this performance boost only starts paying off once you have at least more than 4 levels of nested closures, something which, unlike in LISP, almost never happens in Pharo. Or at least shouldn't happen (if it does, it's probably ok to punish the people by giving them slower performance). This implementation is pretty hard to understand, and it makes decompilation semi-impossible unless you make very strong assumptions about how the bytecodes are used. This then again reduces the reusability of the new bytecodes and probably of the decompiler once people start actually using the pushNewArray: bytecodes. You might save a teeny tiny bit of memory by having stuff garbage collected when it's not needed anymore ... but I doubt that the whole design is based on that? Especially since it just penalizes the performance in almost all possible ways for standard methods. And it even wastes memory in general cases. I don't get it. But probably I'm missing something? cheers, Toon

March 24, 2011

Re: [Pharo-project] [squeak-dev] positionalMesssge breaks literal array parsing

by Igor Stasenko

On 24 March 2011 22:12, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > Hi All, > Â Â Â the following won't parse because of support for positional messages > (whatever they are; I'll come to that later). > Â Â Â Â Â #(word()) should be equal to { #word. #() } > but because of theÂ type == #leftParenthesis clause in the below it causes a > syntax error: > Scanner methods for multi-character scans > xLetter > "Form a word or keyword." > | type | > buffer reset. > [(type := self typeTableAt: hereChar) == #xLetter > or: [type == #xDigit > or: [type == #xUnderscore and:[self allowUnderscoreSelectors]]]] whileTrue: > ["open code step for speed" > buffer nextPut: hereChar. > hereChar := aheadChar. > aheadChar := source atEnd > ifTrue: [30 asCharacter "doit"] > ifFalse: [source next]]. > tokenType := (type == #colon or: [type == #xColon and: [aheadChar ~~ $=]]) > ifTrue: > [buffer nextPut: self step. > "Allow any number of embedded colons in literal symbols" > [(self typeTableAt: hereChar) == #xColon] whileTrue: > [buffer nextPut: self step]. > #keyword] > ifFalse: > [type == #leftParenthesis > ifTrue: > [buffer nextPut: self step; nextPut: $). > #positionalMessage] > ifFalse:[#word]]. > token := buffer contents > e.g. evaluate any of the following: > Â Â Â #(word()) > Â Â Â Compiler evaluate: '#(word())' > Â Â Â Scanner new scanTokens: '#(word())' > What is this support for? Â Is it for those funky ffi calls in the OpenGL > code? Â Can it not be hoisted out of the scanner and into the parser proper? My IMO: just wipe it out. It is broken anyways, because expressions like: self foo(4). are not parsed correctly and producing syntax error instead of recognizing it as a message with positional arguments . > [and optimistically anyone have a fix already?] > best, > Eliot > > > -- Best regards, Igor Stasenko AKA sig.

March 24, 2011

Re: [Pharo-project] [squeak-dev] positionalMesssge breaks literal array parsing

by Igor Stasenko

On 24 March 2011 22:12, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > Hi All, > Â Â Â the following won't parse because of support for positional messages > (whatever they are; I'll come to that later). http://www.lazarevic.de/PositionalArgs.html > Â Â Â Â Â #(word()) should be equal to { #word. #() } > but because of theÂ type == #leftParenthesis clause in the below it causes a > syntax error: > Scanner methods for multi-character scans > xLetter > "Form a word or keyword." > | type | > buffer reset. > [(type := self typeTableAt: hereChar) == #xLetter > or: [type == #xDigit > or: [type == #xUnderscore and:[self allowUnderscoreSelectors]]]] whileTrue: > ["open code step for speed" > buffer nextPut: hereChar. > hereChar := aheadChar. > aheadChar := source atEnd > ifTrue: [30 asCharacter "doit"] > ifFalse: [source next]]. > tokenType := (type == #colon or: [type == #xColon and: [aheadChar ~~ $=]]) > ifTrue: > [buffer nextPut: self step. > "Allow any number of embedded colons in literal symbols" > [(self typeTableAt: hereChar) == #xColon] whileTrue: > [buffer nextPut: self step]. > #keyword] > ifFalse: > [type == #leftParenthesis > ifTrue: > [buffer nextPut: self step; nextPut: $). > #positionalMessage] > ifFalse:[#word]]. > token := buffer contents > e.g. evaluate any of the following: > Â Â Â #(word()) > Â Â Â Compiler evaluate: '#(word())' > Â Â Â Scanner new scanTokens: '#(word())' > What is this support for? Â Is it for those funky ffi calls in the OpenGL > code? Â Can it not be hoisted out of the scanner and into the parser proper? > [and optimistically anyone have a fix already?] > best, > Eliot > > > -- Best regards, Igor Stasenko AKA sig.

March 24, 2011

Re: [Pharo-project] "Does Committing Code to an Open Source Project Mean Committing Career Suicide?"

by Igor Stasenko

On 24 March 2011 22:41, <csrabak(a)bol.com.br> wrote: > Hi Folks, > > In the interminable stream of conflicting information I've received this link: > > http://www.softwarequalityconnection.com/2011/02/does-committing-code-to-an… > > I will not say that the author's resume (shown in the page linked) increase my level of credibility on her posting, but given the popularity of the site from where it was posted, and IMNHO echoes a kind of FUD typically seen w.r.t. Open Source I thought you would like to get to know about. > > If any of the concerns described in the article should be considered correct, perhaps we should have the letters we sign for participating updated with the appropriate provisos? > Why? If you got hired by asshole, who thinks that he owns all of your free time, then my advice - just quit. Changing the letters, you sign, won't turn an asshole into a nice guy :) > Regards, > > -- > Cesar Rabak > > -- Best regards, Igor Stasenko AKA sig.

March 24, 2011

[Pharo-project] Issue 3876 in pharo: SqNumberParserTest>>testFloatGradualUnderflow has an odd condition

by pharo＠googlecode.com

Status: Accepted Owner: da...(a)teleport.com Labels: Importance-Low New issue 3876 by da...(a)teleport.com: SqNumberParserTest>>testFloatGradualUnderflow has an odd condition http://code.google.com/p/pharo/issues/detail?id=3876 SqNumberParserTest>>testFloatGradualUnderflow has an odd condition: In the following code, note that #areLowercaseDigitsAllowed is expected to return nil or notNil: "Parse the number in base 16 if possible - it is impossible if lowercase letter are allowed digits due to exponent letter ambiguity." float := self areLowercaseDigitsAllowed ifNil: [SqNumberParser parse: '16r2.D2593D58B4FC4e-256'] ifNotNil: [trueFraction asFloat]. but #areLowercaseDigitsAllowed returns true or false and in the 1.2 image it returns true ... implying that is impossible to parse? As it stands the `trueFraction asFloat` expression is always executed ... I'm trying to port SqNumberParser to GemStone and this little section of code is misleading, especially considering that GemStone doesn't pass the test:)

March 24, 2011

Re: [Pharo-project] Issue 3861 in pharo: Image freeze when changing system time

by pharo＠googlecode.com

Comment #3 on issue 3861 by siguc...(a)gmail.com: Image freeze when changing system time http://code.google.com/p/pharo/issues/detail?id=3861 this is known issue. but thanks for filing it on tracker. it should be fixed :)

March 24, 2011

[Pharo-project] "Does Committing Code to an Open Source Project Mean Committing Career Suicide?"

by csrabak＠bol.com.br

Hi Folks, In the interminable stream of conflicting information I've received this link: http://www.softwarequalityconnection.com/2011/02/does-committing-code-to-an… I will not say that the author's resume (shown in the page linked) increase my level of credibility on her posting, but given the popularity of the site from where it was posted, and IMNHO echoes a kind of FUD typically seen w.r.t. Open Source I thought you would like to get to know about. If any of the concerns described in the article should be considered correct, perhaps we should have the letters we sign for participating updated with the appropriate provisos? Regards, -- Cesar Rabak

March 24, 2011

Re: [Pharo-project] custom code on squeaksource

by Camillo Bruni

well the bitfield stuff would already work, thats fairly easy to implement. The only issue right now is that we rely on helvetia for the source code transformation. And I urge not to introduce custom syntax for slot creation, that doesn't make sense. Right now the class definition has always been hardcoded, which IMO is quite wrong. But by relying on first-class layouts in slots we can simply ask these objects what properties a class has. Just as a hint, first-class layouts render the format integer on classes basically useless. Since you can always query the layout and ask it for these properties. Thus the format should be used only at VM level to speed things up (the same for compact classes ;)). On 2011-03-24, at 22:04, StÃ©phane Ducasse wrote: >> yes now we should probably avoid to have syntax in the class definition but in a description of the slots. >> >> What means exactly: #identityHash => 20 => #Align => 32. >> >> This is just an example > > Yes I see that but I value real examples I cannot imagine :) > > So we could have a nice BitFieldSlot class with nice message and creation interface (no magic intepretation needed > and we would write something like > > HeapField subclass: #ObjectHeaderLittleEndian > instanceVar: > 'identityHash <BitFieldSlot length: 20 aligned: 32> ' > > or something like that and it will be quite cool. > > >> but it means a 20-bit unsigned integer field aligned on a 32-bit boundary. In the above #isPointers => 1 => #Boolean would mean a 1 bit-field accessed as a boolean (a flag). The above is close to the object header in the new GC I want to build. >> >> For slots we did a first experience with marcus in 2008/9 where we show that we could get first class instanceVariable >> without speed penalties. We postpone our experience because it required a working new compiler and a complete rewrite >> of classBuilder. >> We want to be able to define >> active value >> hashTable >> relationship >> magritte like description >> and a lot more >> >> So if you have ideas or wishes let us know. >> >> Stef

March 24, 2011