Well, not really.
Technically, tokens are used to drive a parser from a scanner.
If an AST node knows how to relate itself to its original source code chunk and is able to print itself correctly, then tokens are redundant.
In short, if you work with parsers, you'd better know what tokens are. If you're only working with the AST, tokens are redundant and noise (i.e. they often have a type (or more than one) which is only understood by the parser).
I'm working with ASTs sourceInterval. Trying to calculate it after method replaceWith:. You see, my proble was that each node of AST doesn't hold its start and stop position in same place. So I thought that token is such a place, however, eventually I understood that RBValueNodes don't have tokens:)
Example of how it is done:
RBPragmaNode accessing-tokens gives access to left and right, which are positions, not tokens.
Yes, I know.
Thierry
Mark
2014-10-28 11:59 GMT+02:00 Nicolai Hess <nicolaihess@web.de <mailto:nicolaihess@web.de>>:
2014-10-27 19:36 GMT+01:00 Mark Rizun <mrizun@gmail.com <mailto:mrizun@gmail.com>>:
Hi all,
Trying to understand here how tokens are used in AST. So far I can not see any order in usage of tokens. For instance, why RBValueNode doesn't have token? Is it haow it's supposed to be?
RBValueNode is an abstract class.
Cheers, Mark