[Pharo-project] [Pharo-users] RB AST serialization
Hi, I'm working on a serialization of a RB AST, is in a very very earlier state, it can encode simple ASTs into a string but today i changed the encode methods, so now you can't decode the generated string :-P .Some node from AST-Core can't be encoded yet (RBPragmaNode, RBArrayNode, RBLiteralArrayNode) because I'm not sure what to encode, and the nodes aren't documented, at least in Pharo 1.0. Any comments and suggestions will be appreciated!! :-) I uploaded the code in squeaksource: http://www.squeaksource.com/AstSerializer.html I'm a bit new to Smalltalk and Pharo, so feel free to comment... Greetings, Daniel Galdames G.
Hi Daniel, I haven't looked at your code yet, but wonder what your goal is? How is your serialization different from source := aProgramNode source ? How is your deserialization different from RBParser parseMethod: source ? Cheers, Lukas 2010/4/27 Daniel Galdames <lemuus@gmail.com>:
Hi, I'm working on a serialization of a RB AST, is in a very very earlier state, it can encode simple ASTs into a string but today i changed the encode methods, so now you can't decode the generated string :-P .Some node from AST-Core can't be encoded yet (RBPragmaNode, RBArrayNode, RBLiteralArrayNode) because I'm not sure what to encode, and the nodes aren't documented, at least in Pharo 1.0. Any comments and suggestions will be appreciated!! :-) I uploaded the code in squeaksource:
http://www.squeaksource.com/AstSerializer.html
I'm a bit new to Smalltalk and Pharo, so feel free to comment...
Greetings,
  Daniel Galdames G.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
On Apr 27, 2010, at 7:48 AM, Lukas Renggli wrote:
Hi Daniel,
I haven't looked at your code yet, but wonder what your goal is?
How is your serialization different from
source := aProgramNode source
?
How is your deserialization different from
RBParser parseMethod: source
?
In Reflectivity, we have an AST for every method. In addition, the AST can be annotated with non-source visible annotations. So what I want to explore is to have a very compact representations of ASTs, but one that does not lose information *and* that allows the AST to be annotated with non-textual objects even when it's compressed. Marcus -- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
So what I want to explore is to have a very compact representations of ASTs, but one that does not lose information *and* that allows the AST to be annotated with non-textual objects even when it's compressed.
Indeed, annotations (or properties) get lost when serializing an AST as source code. However I doubt that a different encoding of the AST as a string is any more efficient than just storing the (compressed) original source code (see the tests of AstSerializer). Also having a "custom pretty printer" and a "custom parser" makes it kind of difficult to have code that cannot be represented with the standard AST, as this is for example the case in Helvetia. What about storing the annotations (or properties) separate the the source string and map them back to the AST after parsing? Lukas
    Marcus
-- Marcus Denker  -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
what theo d hondt told us a while ago is that pascal or oberon they did not store bytecode but compressed ast and it was shorter than bytecode Stef On Apr 27, 2010, at 1:44 PM, Lukas Renggli wrote:
So what I want to explore is to have a very compact representations of ASTs, but one that does not lose information *and* that allows the AST to be annotated with non-textual objects even when it's compressed.
Indeed, annotations (or properties) get lost when serializing an AST as source code. However I doubt that a different encoding of the AST as a string is any more efficient than just storing the (compressed) original source code (see the tests of AstSerializer). Also having a "custom pretty printer" and a "custom parser" makes it kind of difficult to have code that cannot be represented with the standard AST, as this is for example the case in Helvetia.
What about storing the annotations (or properties) separate the the source string and map them back to the AST after parsing?
Lukas
Marcus
-- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
what theo d hondt told us a while ago is that pascal or oberon they did not store bytecode but compressed ast and it was shorter than bytecode
Sounds strange (imagine if you have comments in your code). Are there any references on this? Lukas -- Lukas Renggli www.lukas-renggli.ch
On Apr 27, 2010, at 2:33 PM, Lukas Renggli wrote:
what theo d hondt told us a while ago is that pascal or oberon they did not store bytecode but compressed ast and it was shorter than bytecode
Sounds strange (imagine if you have comments in your code). Are there any references on this? No I know that marcus read some papers on slim binaries
Stef
Lukas
-- Lukas Renggli www.lukas-renggli.ch
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Apr 27, 2010, at 2:33 PM, Lukas Renggli wrote:
what theo d hondt told us a while ago is that pascal or oberon they did not store bytecode but compressed ast and it was shorter than bytecode
Sounds strange
Why?
(imagine if you have comments in your code).
There are no comments in bytecode, either. The idea is *not* to replace source code.
Are there any references on this?
There is a lot of work on this, e.g. slim binaries: http://portal.acm.org/citation.cfm?id=265576 or lots of work on grammar-driven compression of code, and much more. Not that it's exactly slim binaries that we want to implement here. We want to learn about compact serialization of AST like structures. Just learn. First serialization. Than compression. And the nice thing is that I am payed for exploring strange things :-) And I think we should pospone discussion after we have learned. It's so easy to kill all energy on a mailinglist in a minute. Marcus -- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.
Daniel what would be really cool is to see if we can compact/zip AST because marcus is really interested into slimbinary libraries and having a really strong compaction of AST would be an interesting basis for a lot of experience On Apr 27, 2010, at 6:45 AM, Daniel Galdames wrote:
Hi, I'm working on a serialization of a RB AST, is in a very very earlier state, it can encode simple ASTs into a string but today i changed the encode methods, so now you can't decode the generated string :-P .Some node from AST-Core can't be encoded yet (RBPragmaNode, RBArrayNode, RBLiteralArrayNode) because I'm not sure what to encode, and the nodes aren't documented, at least in Pharo 1.0. Any comments and suggestions will be appreciated!! :-) I uploaded the code in squeaksource: http://www.squeaksource.com/AstSerializer.html
I'm a bit new to Smalltalk and Pharo, so feel free to comment... Greetings, Daniel Galdames G. _______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
participants (4)
-
Daniel Galdames -
Lukas Renggli -
Marcus Denker -
Stéphane Ducasse