Code generation for ASTs with syntax errors
We can parse code with syntax errors: it generates partial ASTs with the faulty part of the input as a RBParseErrorNode. Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification is raised at runtime instead of compile time. testEvalSimpleMethodWithError | ast cm | ast := OpalCompiler new source: 'method 3+'; useFaultyForParsing: true; parse. self assert: ast isMethod. self assert: ast isFaulty. cm := ast compiledMethod. self should: [cm valueWithReceiver: nil arguments: #()] raise: SyntaxErrorNotification The syntax error instance is compiled in via a literal variable, in the end the only thing needed was: visitParseErrorNode: anErrorNode methodBuilder pushLiteralVariable: #error -> anErrorNode asSyntaxErrorNotification; send: #signal. This is in #50044. Future work: introduce a RuntimeSyntaxError that can be turned off per error and globally. Marcus
On 13 May 2015 at 08:35, Marcus Denker <marcus.denker@inria.fr> wrote:
Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification is raised at runtime instead of compile time.
I can hear the noise of eyebrows raising in the compiler community :) Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" âwhatever that means for us. For instance, faulty methods could have a flashy icon in the browser; I think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods. -- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet
On 13 May 2015, at 10:41, Damien Pollet <damien.pollet@gmail.com> wrote:
On 13 May 2015 at 08:35, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote: Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification is raised at runtime instead of compile time.
I can hear the noise of eyebrows raising in the compiler community :)
Yes! :-)
Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" âwhatever that means for us. For instance, faulty methods could have a flashy icon in the browser; I think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods.
I think that GT does it already right: it does not insert the error message in the text, but instead shows it using a tiny window undefendend of the text. Combined with syntax highlighting, this should be very close to what we need (the devil is in the detail, of course). As for finding them: the nice property of syntax errors is that they are trivial to detect statically⦠so e.g. we can add that people never can commit code like that, and of course code critics can find it trivially⦠the rule just needs to check for #isFaulty (maybe we should range that to #hasSyntaxErrors). Marcus
On 18 May 2015 at 21:33, Marcus Denker <marcus.denker@inria.fr> wrote:
As for finding them: the nice property of syntax errors is that they are trivial to detect statically⦠so e.g. we can add that people never can commit code like that
NOPE. The problem I had was that NBExternalStructure was changed from a variable bytes class to a normal one. So subclasses were committed without error but still couldn't be loaded with a more recent NB. Errors WILL happen, and fixing those is where plain text beats objects⦠-- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet
On Wed, May 13, 2015 at 4:41 PM, Damien Pollet <damien.pollet@gmail.com> wrote:
On 13 May 2015 at 08:35, Marcus Denker <marcus.denker@inria.fr> wrote:
Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification is raised at runtime instead of compile time.
What is the advantage of allowing syntax errors into the running system, when they can be trapped statically before that ? It seems to me the advantage of this change is that it allows compilation to complete rather than bombing out part way through. But just because compilation finishes, doesn't mean it needs to be committed into the runtime system.
Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" âwhatever that means for us.
For instance, faulty methods could have a flashy icon in the browser; I
think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods.
I guess it *should* mean that the faulty-compiled-method is not committed but stored in a temporary buffer similar to how editing source is effectively done in a temporary buffer. Indication of the fault would only be needed in the window that compiled the fault. cheers -ben
On 19 May 2015, at 01:51, Ben Coman <btc@openInWorld.com> wrote:
On Wed, May 13, 2015 at 4:41 PM, Damien Pollet <damien.pollet@gmail.com <mailto:damien.pollet@gmail.com>> wrote: On 13 May 2015 at 08:35, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote: Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification is raised at runtime instead of compile time.
What is the advantage of allowing syntax errors into the running system, when they can be trapped statically before that ? It seems to me the advantage of this change is that it allows compilation to complete rather than bombing out part way through. But just because compilation finishes, doesn't mean it needs to be committed into the runtime system. Syntax errors shouldn't require immediate fixing during compilation per se, but there should be a way to deal with them "at compile time" âwhatever that means for us. For instance, faulty methods could have a flashy icon in the browser; I think I'd even color the whole package-class-protocol-selector in red if either of them contain faulty methods.
I guess it *should* mean that the faulty-compiled-method is not committed but stored in a temporary buffer similar to how editing source is effectively done in a temporary buffer. Indication of the fault would only be needed in the window that compiled the fault.
No, we once experimented with adding support to Nautilus for that. But the result is that it is very easy to loose code. And you never know what is âreally savedâ and what not. But in general: I know that this is one of those things that I will not be able to convince anyone that it is useful before it is done⦠which is good: I like to explore directions that everyone else rejects immediately. Marcus
Thanks's Marcus. It is beautiful thing. Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one Best regards, Denis 2015-05-13 9:35 GMT+03:00 Marcus Denker <marcus.denker@inria.fr>:
We can parse code with syntax errors: it generates partial ASTs with the faulty part of the input as a RBParseErrorNode.
Only three methods are needed to allow generating executable methods from such an AST: the SyntaxErrorNotification is raised at runtime instead of compile time.
testEvalSimpleMethodWithError | ast cm | ast := OpalCompiler new source: 'method 3+'; useFaultyForParsing: true; parse.
self assert: ast isMethod. self assert: ast isFaulty.
cm := ast compiledMethod. self should: [cm valueWithReceiver: nil arguments: #()] raise: SyntaxErrorNotification
The syntax error instance is compiled in via a literal variable, in the end the only thing needed was:
visitParseErrorNode: anErrorNode methodBuilder pushLiteralVariable: #error -> anErrorNode asSyntaxErrorNotification; send: #signal.
This is in #50044.
Future work: introduce a RuntimeSyntaxError that can be turned off per error and globally.
Marcus
On 13 May 2015, at 11:21, Denis Kudriashov <dionisiydk@gmail.com> wrote:
Thanks's Marcus. It is beautiful thing.
Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one
You can just look at OpalCompiler new source: 'method 3+. 1+2'; useFaultyForParsing: true; parse. with the GT inspector, it has a very nice AST view:
2015-05-19 7:49 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
On 13 May 2015, at 11:21, Denis Kudriashov <dionisiydk@gmail.com> wrote:
Thanks's Marcus. It is beautiful thing.
Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one
You can just look at
OpalCompiler new source: 'method 3+. 1+2'; useFaultyForParsing: true; parse.
Impressive !
with the GT inspector, it has a very nice AST view:
The AST view of the GT Inspector is useless for serious work. I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code). Thierry
On Tue, May 19, 2015 at 10:02 AM, Thierry Goubier <thierry.goubier@gmail.com
wrote:
2015-05-19 7:49 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
On 13 May 2015, at 11:21, Denis Kudriashov <dionisiydk@gmail.com> wrote:
Thanks's Marcus. It is beautiful thing.
Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one
You can just look at
OpalCompiler new source: 'method 3+. 1+2'; useFaultyForParsing: true; parse.
Impressive !
with the GT inspector, it has a very nice AST view:
The AST view of the GT Inspector is useless for serious work.
I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).
Switching to the 'Source code' view in the second pane gives you exactly that [image: Inline image 1]
Thierry
2015-05-19 10:16 GMT+02:00 Andrei Chis <chisvasileandrei@gmail.com>:
On Tue, May 19, 2015 at 10:02 AM, Thierry Goubier < thierry.goubier@gmail.com> wrote:
2015-05-19 7:49 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
On 13 May 2015, at 11:21, Denis Kudriashov <dionisiydk@gmail.com> wrote:
Thanks's Marcus. It is beautiful thing.
Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one
You can just look at
OpalCompiler new source: 'method 3+. 1+2'; useFaultyForParsing: true; parse.
Impressive !
with the GT inspector, it has a very nice AST view:
The AST view of the GT Inspector is useless for serious work.
I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).
Switching to the 'Source code' view in the second pane gives you exactly that
[image: Inline image 1]
And that view get the selection wrong ;) Thierry
2015-05-19 10:27 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
And that view get the selection wrong ;)
No, the ErrorNode has "all the rest" as it's extend for Shout so it can color the error just the same as people are used to.
Ok. But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :) Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was? Thierry
Marcus
On 19 May 2015, at 10:32, Thierry Goubier <thierry.goubier@gmail.com> wrote:
2015-05-19 10:27 GMT+02:00 Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>>:
And that view get the selection wrong ;)
No, the ErrorNode has âall the restâ as itâs extend for Shout so it can color the error just the same as people are used to.
Ok.
But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)
Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?
I am not entirely sure⦠it sets the code of the ErrorNode to be just from where we are to the end of the to be parsed text. At that point it is hard to know the real extend of the error⦠needs to be checked how to do it nicer. I think we should first get it in a state that we can replace the shout parser (both for syntax highlight and code completion). Marcus
2015-05-19 11:22 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
On 19 May 2015, at 10:32, Thierry Goubier <thierry.goubier@gmail.com> wrote:
2015-05-19 10:27 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
And that view get the selection wrong ;)
No, the ErrorNode has "all the rest" as it's extend for Shout so it can color the error just the same as people are used to.
Ok.
But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)
Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?
I am not entirely sure... it sets the code of the ErrorNode to be just from where we are to the end of the to be parsed text. At that point it is hard to know the real extend of the error... needs to be checked how to do it nicer.
In the example you give, the parser is able to trigger an error, go up the ast tree, and restart correct parsing further down the source code, no? Is that the way you are doing it? I'm interested in a general parsing theory for that kind of error detection / recovery. Thierry
I think we should first get it in a state that we can replace the shout parser (both for syntax highlight and code completion).
Marcus
But statement after error is parsed and ast node exists for them. Is it not correct to make end of error equal to start of next ast node? 19 Ð¼Ð°Ñ 2015 г. 11:22 полÑзоваÑÐµÐ»Ñ "Marcus Denker" <marcus.denker@inria.fr> напиÑал:
On 19 May 2015, at 10:32, Thierry Goubier <thierry.goubier@gmail.com> wrote:
2015-05-19 10:27 GMT+02:00 Marcus Denker <marcus.denker@inria.fr>:
And that view get the selection wrong ;)
No, the ErrorNode has âall the restâ as itâs extend for Shout so it can color the error just the same as people are used to.
Ok.
But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)
Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?
I am not entirely sure⦠it sets the code of the ErrorNode to be just from where we are to the end of the to be parsed text. At that point it is hard to know the real extend of the error⦠needs to be checked how to do it nicer.
I think we should first get it in a state that we can replace the shout parser (both for syntax highlight and code completion).
Marcus
Yes, we should fix itâ¦
On 21 May 2015, at 09:22, Denis Kudriashov <dionisiydk@gmail.com> wrote:
But statement after error is parsed and ast node exists for them. Is it not correct to make end of error equal to start of next ast node?
19 Ð¼Ð°Ñ 2015 г. 11:22 полÑзоваÑÐµÐ»Ñ "Marcus Denker" <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> напиÑал:
On 19 May 2015, at 10:32, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
2015-05-19 10:27 GMT+02:00 Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>>:
And that view get the selection wrong ;)
No, the ErrorNode has âall the restâ as itâs extend for Shout so it can color the error just the same as people are used to.
Ok.
But I'd really like Shout not to color red the rest of the code when I start to modify code half way through a method :)
Is that inherent to the way ErrorNode instances are generated or is it just done to maintain as it was?
I am not entirely sure⦠it sets the code of the ErrorNode to be just from where we are to the end of the to be parsed text. At that point it is hard to know the real extend of the error⦠needs to be checked how to do it nicer.
I think we should first get it in a state that we can replace the shout parser (both for syntax highlight and code completion).
Marcus
with the GT inspector, it has a very nice AST view:
The AST view of the GT Inspector is useless for serious work.
I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).
Agree, something like this would be even nicer... Jan
It's already available :). [image: Inline image 1] Doru On Tue, May 19, 2015 at 10:18 AM, Jan Vrany <jan.vrany@fit.cvut.cz> wrote:
with the GT inspector, it has a very nice AST view:
The AST view of the GT Inspector is useless for serious work.
I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).
Agree, something like this would be even nicer...
Jan
-- www.tudorgirba.com "Every thing has its own flow"
Ok. Now I need to evaluate if the time spent writing a GT presentation is worth the performance impact of using the GT Inspector use compared to writing a dedicated version of a faster tree viewer ;) Thierry 2015-05-19 10:18 GMT+02:00 Tudor Girba <tudor@tudorgirba.com>:
It's already available :).
[image: Inline image 1]
Doru
On Tue, May 19, 2015 at 10:18 AM, Jan Vrany <jan.vrany@fit.cvut.cz> wrote:
with the GT inspector, it has a very nice AST view:
The AST view of the GT Inspector is useless for serious work.
I've been spending the past weeks and months looking at ASTs, SSA and CFGs of code and I allways need a twin view: node + source code (and selecting one of those node select the relevant portion of the code).
Agree, something like this would be even nicer...
Jan
-- www.tudorgirba.com
"Every thing has its own flow"
Perfect! 19 Ð¼Ð°Ñ 2015 г. 7:49 полÑзоваÑÐµÐ»Ñ "Marcus Denker" <marcus.denker@inria.fr> напиÑал:
On 13 May 2015, at 11:21, Denis Kudriashov <dionisiydk@gmail.com> wrote:
Thanks's Marcus. It is beautiful thing.
Is parsing locate errors by "dot"s? I mean how it will parse "3+. 1+2"? It will be super cool if we have only "3+." as error but remains code as good one
You can just look at
OpalCompiler new source: 'method 3+. 1+2'; useFaultyForParsing: true; parse.
with the GT inspector, it has a very nice AST view:
participants (8)
-
Andrei Chis -
Ben Coman -
Damien Pollet -
Denis Kudriashov -
Jan Vrany -
Marcus Denker -
Thierry Goubier -
Tudor Girba