#parseMethod:onError: isn't as obvious as expected - is it me?
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:. I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go. Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etc⦠So this is where Iâve hit a few issues which strike me as confusing (and I will freely admit that Iâm not a mega compiler, parser guy - but Iâd love to beâ¦). The following code blows up strangely - and I think its very confusing⦠as it turns out that #parseMethod:onError: doesnât work like I would expect (and this is where Iâm after feedback). It seems that the onError block doesât actually exit parsing - it takes the result and then keeps running with something in an intermediate state - thus below, when I try to parse with a subset of the code (and maybe this is a naive approach to handle the case when you start typing code in the middle of a complete method, but want to parse it before youâve entered a final â.â or â;â) - you actually end up with an AST that is using all of the source code because the result of the error block is then applied inside the executing method. I find this very confusing? There are no comments in this code, and for me the convention of onError: is to execute in isolation unless there is a parameter that the error block can interact with. Iâm wondering if those with more knowledge agree - or if it is my naiveté? This was my original code (and yes it needs refactoring a bit) - but anyway: bestNodeInTextAreaOnError: aBlock "Find the best node in the editor text area at the current pointer location" | ast start stop node source | start := self textArea startIndex. stop := self textArea stopIndex. source := self textArea string trimRight. ast := RBParser parseMethod: source onError: [ RBParser parseExpression: source onError: [ source := (source truncateTo: stop) trimRight. RBParser parseMethod: source onError: [ RBParser parseExpression: source onError: aBlock ]]]. stop > source size ifTrue: [ start := stop := source size]. node := ast bestNodeFor: (start to: stop). node ifNil: aBlock. ^node But this doesnât work, as âastâ is incorrect when it parses truncated source. To work around what Iâm seeing I have to instead extract the ast assignment to another method that looks as follows - but to me it seems very unexpected to have the explicit ^ statements? agressivelyParse: source at: loc "Find the best node in the editor text area at the current pointer location" ^RBParser parseMethod: source onError: [ ^RBParser parseExpression: source onError: [ | reducedSrc | reducedSrc := (source truncateTo: loc) trimRight. ^RBParser parseMethod: reducedSrc onError: [ ^RBParser parseExpression: reducedSrc onError: [^nil]]]]. This forces the ast to be exactly what I would expect - but Iâm wondering if I am missing something obvious? Tim
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
So this is where Iâve hit a few issues which strike me as confusing (and I will freely admit that Iâm not a mega compiler, parser guy - but Iâd love to beâ¦).
The following code blows up strangely - and I think its very confusing⦠as it turns out that #parseMethod:onError: doesnât work like I would expect (and this is where Iâm after feedback). It seems that the onError block doesât actually exit parsing - it takes the result and then keeps running with something in an intermediate state - thus below, when I try to parse with a subset of the code (and maybe this is a naive approach to handle the case when you start typing code in the middle of a complete method, but want to parse it before youâve entered a final â.â or â;â) - you actually end up with an AST that is using all of the source code because the result of the error block is then applied inside the executing method.
I find this very confusing? There are no comments in this code, and for me the convention of onError: is to execute in isolation unless there is a parameter that the error block can interact with. Iâm wondering if those with more knowledge agree - or if it is my naiveté?
This was my original code (and yes it needs refactoring a bit) - but anyway:
bestNodeInTextAreaOnError: aBlock "Find the best node in the editor text area at the current pointer location"
| ast start stop node source |
start := self textArea startIndex. stop := self textArea stopIndex.
source := self textArea string trimRight.
ast := RBParser parseMethod: source onError: [ RBParser parseExpression: source onError: [ source := (source truncateTo: stop) trimRight.
RBParser parseMethod: source onError: [ RBParser parseExpression: source onError: aBlock ]]].
stop > source size ifTrue: [ start := stop := source size].
node := ast bestNodeFor: (start to: stop).
node ifNil: aBlock.
^node
But this doesnât work, as âastâ is incorrect when it parses truncated source. To work around what Iâm seeing I have to instead extract the ast assignment to another method that looks as follows - but to me it seems very unexpected to have the explicit ^ statements?
agressivelyParse: source at: loc "Find the best node in the editor text area at the current pointer location"
^RBParser parseMethod: source onError: [ ^RBParser parseExpression: source onError: [ | reducedSrc | reducedSrc := (source truncateTo: loc) trimRight.
^RBParser parseMethod: reducedSrc onError: [ ^RBParser parseExpression: reducedSrc onError: [^nil]]]].
This forces the ast to be exactly what I would expect - but Iâm wondering if I am missing something obvious?
Tim
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUG⦠Marcus
Marcus - I'd be interested in how you thought to solve this. My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact). Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results? Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy. I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do. Tim Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Hi Tim, The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval. For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should). Regards, Thierry 2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod: onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Thanks Thierry - this is proving an interesting problem domain ... I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit. I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way. All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me. Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring. I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what. Tim Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Tim Oh yes. There are still part of Pharo that we did not clean. Stef On Fri, Sep 1, 2017 at 1:39 PM, Tim Mackinnon <tim@testit.works> wrote:
Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Hi Tim, 2017-09-01 13:39 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way). The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible. This would make for a very nice internship subject...
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API). Now, most Pharo developpers just jump into hardcoding keyboard handling instead. Thierry
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Thanks Thierry - Iâve been noodling in the background and next I hit the question of âcommentsâ - which I found RB does handle⦠and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project. I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when Iâm coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction. Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ. Tim
On 7 Sep 2017, at 08:02, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
2017-09-01 13:39 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).
The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.
This would make for a very nice internship subject...
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).
Now, most Pharo developpers just jump into hardcoding keyboard handling instead.
Thierry
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com <mailto:stepharo.self@gmail.com>> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Damn you smalltalk⦠you suck me in to little side problemsâ¦. RBParser is actually a lot more flexible than I had realised - and not so difficult to understand for a mere mortal⦠It naively seems that (what I still consider a confusingly named method) is quite simple to extend to handle at least medium sized errors just by doing something like: ast := [RBParser parseMethod: source onError: [ :msg :pos :parser | |seqNode errorNode| errorNode := RBParseErrorNode errorMessage: msg value: parser currentToken value printString at: pos. seqNode := parser sequenceNodeClass new. parser step; parseStatements: true into: seqNode. seqNode addNodeFirst: errorNode; yourself ]] onDNU: #body: do: [ nil âHandle non method source fragments"]. NOTE: I did have to add >>#currentToken as a method to RBParser as it doesnât expose very much for the onError blocks, which I think it should to make them a bit more useful. Tim
On 7 Sep 2017, at 14:39, Tim Mackinnon <tim@testit.works> wrote:
Thanks Thierry - Iâve been noodling in the background and next I hit the question of âcommentsâ - which I found RB does handle⦠and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project.
I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when Iâm coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction.
Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ.
Tim
On 7 Sep 2017, at 08:02, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Hi Tim,
2017-09-01 13:39 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).
The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.
This would make for a very nice internship subject...
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).
Now, most Pharo developpers just jump into hardcoding keyboard handling instead.
Thierry
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com <mailto:stepharo.self@gmail.com>> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
there is #parseFaultyMethod:, too. We use RBParser for syntax highlighting. We do already have âsendersâ and âimplementors working in the AST when using Suggestions. See SugsMenuBuilder>>#findBestNodeFor: in the case of non-accepted code, it uses the compiler chain with #optionParseErrors (which uses parseFaultyMethod: to parse). context hasUnacceptedEdits ifTrue: [ root := Smalltalk compiler source: context code; options: #(+ optionParseErrors); parse. context selectedClass ifNotNil: [ :theClass | root methodNode methodClass: theClass ]. root doSemanticAnalysis ] This works well but fails when you actually are in an error node.. for the same code the âdumbâ way we use with standard âsenders ofâ just takes line, looks for spaces and feeds that word into the âsenders ofâ.. So to make this work better we need to combine both approaches and deal better with the Error Node content. (e.g. trying to fix it and re-parse or just use the dumb âget the work under the cursorâ way as a first step). Marcus
On 8 Sep 2017, at 03:43, Tim Mackinnon <tim@testit.works> wrote:
Damn you smalltalk⦠you suck me in to little side problemsâ¦.
RBParser is actually a lot more flexible than I had realised - and not so difficult to understand for a mere mortalâ¦
It naively seems that (what I still consider a confusingly named method) is quite simple to extend to handle at least medium sized errors just by doing something like:
ast := [RBParser parseMethod: source onError: [ :msg :pos :parser | |seqNode errorNode|
errorNode := RBParseErrorNode errorMessage: msg value: parser currentToken value printString at: pos.
seqNode := parser sequenceNodeClass new. parser step; parseStatements: true into: seqNode.
seqNode addNodeFirst: errorNode; yourself
]] onDNU: #body: do: [ nil âHandle non method source fragments"].
NOTE: I did have to add >>#currentToken as a method to RBParser as it doesnât expose very much for the onError blocks, which I think it should to make them a bit more useful.
Tim
On 7 Sep 2017, at 14:39, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Thanks Thierry - Iâve been noodling in the background and next I hit the question of âcommentsâ - which I found RB does handle⦠and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project.
I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when Iâm coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction.
Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ.
Tim
On 7 Sep 2017, at 08:02, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Hi Tim,
2017-09-01 13:39 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).
The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.
This would make for a very nice internship subject...
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).
Now, most Pharo developpers just jump into hardcoding keyboard handling instead.
Thierry
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com <mailto:stepharo.self@gmail.com>> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Thanks for the pointer Marcus, I'll have a look - it's a fascinating area that I didn't know much about until now ;) All I wanted was to not have to select specific code to do senders (& refactoring), as well as have better statement selection so I can more easily put brackets around things... But now it dawns on me I'd like better code completion after I've already completed but maybe want the fuller keyword method.... Even the refactoring of local method changes shouldn't need you to save your method to rename a local variable or extract a statement to a variable. And I hate having to declare local variables painfully one by one - and I miss the option to detect a simple typo and correct it. These are all things that I love in IntelliJ - and I think we have fantastic infrastructure to do these things too. So I'll keep plugging away at lunch times to see what I can do. I've rewritten it about 3 times already ;) Tim Sent from my iPhone
On 9 Sep 2017, at 08:42, Marcus Denker <marcus.denker@inria.fr> wrote:
there is #parseFaultyMethod:, too. We use RBParser for syntax highlighting.
We do already have âsendersâ and âimplementors working in the AST when using Suggestions.
See SugsMenuBuilder>>#findBestNodeFor:
in the case of non-accepted code, it uses the compiler chain with #optionParseErrors (which uses parseFaultyMethod: to parse).
context hasUnacceptedEdits ifTrue: [ root := Smalltalk compiler source: context code; options: #(+ optionParseErrors); parse. context selectedClass ifNotNil: [ :theClass | root methodNode methodClass: theClass ]. root doSemanticAnalysis ]
This works well but fails when you actually are in an error node.. for the same code the âdumbâ way we use with standard âsenders ofâ just takes line, looks for spaces and feeds that word into the âsenders ofâ..
So to make this work better we need to combine both approaches and deal better with the Error Node content. (e.g. trying to fix it and re-parse or just use the dumb âget the work under the cursorâ way as a first step).
Marcus
On 8 Sep 2017, at 03:43, Tim Mackinnon <tim@testit.works> wrote:
Damn you smalltalk⦠you suck me in to little side problemsâ¦.
RBParser is actually a lot more flexible than I had realised - and not so difficult to understand for a mere mortalâ¦
It naively seems that (what I still consider a confusingly named method) is quite simple to extend to handle at least medium sized errors just by doing something like:
ast := [RBParser parseMethod: source onError: [ :msg :pos :parser | |seqNode errorNode|
errorNode := RBParseErrorNode errorMessage: msg value: parser currentToken value printString at: pos.
seqNode := parser sequenceNodeClass new. parser step; parseStatements: true into: seqNode.
seqNode addNodeFirst: errorNode; yourself
]] onDNU: #body: do: [ nil âHandle non method source fragments"].
NOTE: I did have to add >>#currentToken as a method to RBParser as it doesnât expose very much for the onError blocks, which I think it should to make them a bit more useful.
Tim
On 7 Sep 2017, at 14:39, Tim Mackinnon <tim@testit.works> wrote:
Thanks Thierry - Iâve been noodling in the background and next I hit the question of âcommentsâ - which I found RB does handle⦠and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project.
I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when Iâm coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction.
Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ.
Tim
On 7 Sep 2017, at 08:02, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
2017-09-01 13:39 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).
The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.
This would make for a very nice internship subject...
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).
Now, most Pharo developpers just jump into hardcoding keyboard handling instead.
Thierry
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
Hey Marcus - I've kept noodling on this as a little sub interesting thing I can play with in spare moments. I think I'm getting close to an interesting solution using #parseFaulty (although I wasn't sure how to distinguish between a method and playground without asking the editor for its edittingMode - which maybe is ok?) I've had to add a new visitor to run through parseErrorNodes (I'm surprised there isn't a visitor for this already) (I also added one for comments too) - as #findBestNode doesn't try hard enough - and I've also had a crack at reparsing an error if it seems simple enough (like when you are working on statement in the middle of code and have just broken that one line). I'll try it for a few days and then maybe you might be interested in having a look? Tim Sent from my iPhone
On 9 Sep 2017, at 12:53, Tim Mackinnon <tim@testit.works> wrote:
Thanks for the pointer Marcus, I'll have a look - it's a fascinating area that I didn't know much about until now ;)
All I wanted was to not have to select specific code to do senders (& refactoring), as well as have better statement selection so I can more easily put brackets around things...
But now it dawns on me I'd like better code completion after I've already completed but maybe want the fuller keyword method....
Even the refactoring of local method changes shouldn't need you to save your method to rename a local variable or extract a statement to a variable. And I hate having to declare local variables painfully one by one - and I miss the option to detect a simple typo and correct it.
These are all things that I love in IntelliJ - and I think we have fantastic infrastructure to do these things too. So I'll keep plugging away at lunch times to see what I can do.
I've rewritten it about 3 times already ;)
Tim
Sent from my iPhone
On 9 Sep 2017, at 08:42, Marcus Denker <marcus.denker@inria.fr> wrote:
there is #parseFaultyMethod:, too. We use RBParser for syntax highlighting.
We do already have âsendersâ and âimplementors working in the AST when using Suggestions.
See SugsMenuBuilder>>#findBestNodeFor:
in the case of non-accepted code, it uses the compiler chain with #optionParseErrors (which uses parseFaultyMethod: to parse).
context hasUnacceptedEdits ifTrue: [ root := Smalltalk compiler source: context code; options: #(+ optionParseErrors); parse. context selectedClass ifNotNil: [ :theClass | root methodNode methodClass: theClass ]. root doSemanticAnalysis ]
This works well but fails when you actually are in an error node.. for the same code the âdumbâ way we use with standard âsenders ofâ just takes line, looks for spaces and feeds that word into the âsenders ofâ..
So to make this work better we need to combine both approaches and deal better with the Error Node content. (e.g. trying to fix it and re-parse or just use the dumb âget the work under the cursorâ way as a first step).
Marcus
On 8 Sep 2017, at 03:43, Tim Mackinnon <tim@testit.works> wrote:
Damn you smalltalk⦠you suck me in to little side problemsâ¦.
RBParser is actually a lot more flexible than I had realised - and not so difficult to understand for a mere mortalâ¦
It naively seems that (what I still consider a confusingly named method) is quite simple to extend to handle at least medium sized errors just by doing something like:
ast := [RBParser parseMethod: source onError: [ :msg :pos :parser | |seqNode errorNode|
errorNode := RBParseErrorNode errorMessage: msg value: parser currentToken value printString at: pos.
seqNode := parser sequenceNodeClass new. parser step; parseStatements: true into: seqNode.
seqNode addNodeFirst: errorNode; yourself
]] onDNU: #body: do: [ nil âHandle non method source fragments"].
NOTE: I did have to add >>#currentToken as a method to RBParser as it doesnât expose very much for the onError blocks, which I think it should to make them a bit more useful.
Tim
On 7 Sep 2017, at 14:39, Tim Mackinnon <tim@testit.works> wrote:
Thanks Thierry - Iâve been noodling in the background and next I hit the question of âcommentsâ - which I found RB does handle⦠and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project.
I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when Iâm coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction.
Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ.
Tim
On 7 Sep 2017, at 08:02, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
2017-09-01 13:39 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).
The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.
This would make for a very nice internship subject...
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).
Now, most Pharo developpers just jump into hardcoding keyboard handling instead.
Thierry
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works>:
Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
On 11 Sep 2017, at 02:30, Tim Mackinnon <tim@testit.works> wrote:
Hey Marcus - I've kept noodling on this as a little sub interesting thing I can play with in spare moments.
I think I'm getting close to an interesting solution using #parseFaulty (although I wasn't sure how to distinguish between a method and playground without asking the editor for its edittingMode - which maybe is ok?)
I've had to add a new visitor to run through parseErrorNodes (I'm surprised there isn't a visitor for this already) (I also added one for comments too) - as #findBestNode doesn't try hard enough - and I've also had a crack at reparsing an error if it seems simple enough (like when you are working on statement in the middle of code and have just broken that one line).
Great!
I'll try it for a few days and then maybe you might be interested in having a look?
Yes!
Tim
Sent from my iPhone
On 9 Sep 2017, at 12:53, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Thanks for the pointer Marcus, I'll have a look - it's a fascinating area that I didn't know much about until now ;)
All I wanted was to not have to select specific code to do senders (& refactoring), as well as have better statement selection so I can more easily put brackets around things...
But now it dawns on me I'd like better code completion after I've already completed but maybe want the fuller keyword method....
Even the refactoring of local method changes shouldn't need you to save your method to rename a local variable or extract a statement to a variable. And I hate having to declare local variables painfully one by one - and I miss the option to detect a simple typo and correct it.
These are all things that I love in IntelliJ - and I think we have fantastic infrastructure to do these things too. So I'll keep plugging away at lunch times to see what I can do.
I've rewritten it about 3 times already ;)
Tim
Sent from my iPhone
On 9 Sep 2017, at 08:42, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:
there is #parseFaultyMethod:, too. We use RBParser for syntax highlighting.
We do already have âsendersâ and âimplementors working in the AST when using Suggestions.
See SugsMenuBuilder>>#findBestNodeFor:
in the case of non-accepted code, it uses the compiler chain with #optionParseErrors (which uses parseFaultyMethod: to parse).
context hasUnacceptedEdits ifTrue: [ root := Smalltalk compiler source: context code; options: #(+ optionParseErrors); parse. context selectedClass ifNotNil: [ :theClass | root methodNode methodClass: theClass ]. root doSemanticAnalysis ]
This works well but fails when you actually are in an error node.. for the same code the âdumbâ way we use with standard âsenders ofâ just takes line, looks for spaces and feeds that word into the âsenders ofâ..
So to make this work better we need to combine both approaches and deal better with the Error Node content. (e.g. trying to fix it and re-parse or just use the dumb âget the work under the cursorâ way as a first step).
Marcus
On 8 Sep 2017, at 03:43, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Damn you smalltalk⦠you suck me in to little side problemsâ¦.
RBParser is actually a lot more flexible than I had realised - and not so difficult to understand for a mere mortalâ¦
It naively seems that (what I still consider a confusingly named method) is quite simple to extend to handle at least medium sized errors just by doing something like:
ast := [RBParser parseMethod: source onError: [ :msg :pos :parser | |seqNode errorNode|
errorNode := RBParseErrorNode errorMessage: msg value: parser currentToken value printString at: pos.
seqNode := parser sequenceNodeClass new. parser step; parseStatements: true into: seqNode.
seqNode addNodeFirst: errorNode; yourself
]] onDNU: #body: do: [ nil âHandle non method source fragments"].
NOTE: I did have to add >>#currentToken as a method to RBParser as it doesnât expose very much for the onError blocks, which I think it should to make them a bit more useful.
Tim
On 7 Sep 2017, at 14:39, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Thanks Thierry - Iâve been noodling in the background and next I hit the question of âcommentsâ - which I found RB does handle⦠and then as per your point - how to skip past an error and try further down (which it seems that the scanner might have the info to let me skip simple things). You seem to get sucked in more and more when trying to to this properly - which does make it sound like a very good internship or GSoc project.
I think I may have enough to get something polished enough to release to the wider world. I certainly love having better keyboard support when Iâm coding - as we type a lot (even though we have simple syntax) - and having efficient tools takes away the friction.
Next on my hit list, it add the same support to the refactoring tools - which while sophisticated are so clunky in use when compared to IntelliJ.
Tim
On 7 Sep 2017, at 08:02, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Hi Tim,
2017-09-01 13:39 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Thanks Thierry - this is proving an interesting problem domain ...
I too am using #bestNodeFor: although I don't find it always gives the best node (it does if you are clearly in the target range, but if you are on the outer boundary like the next position after a selector or next to a "." or ";" it favours the parent and not the closest node. So in usage I find I need to tweak it a bit.
Yes, when you are on a boundary, then finding the right node may be difficult. When developping the ancestor to smart suggestion, I considered #bestNodeFor: as good enough, but I considered looking into "up" and "down" ast navigation at a time, and I wasn't alone (I think it was active in the Pharo editor at a point).
I'll look at smacc though - also there is that experimental setting to allow parse errors, I don't know if it helps in any way.
There is a general question there, which is how you try to parse as much as possible while jumping over the error, making the error area as small as possible (that would be really great for syntax highlighting, by the way).
The problem is that in hand-written parsers, you need to hard-code the error management in the parser. With a table-driven parser (as is SmaCC), then you can explore in a systematic way what are the possible states that would allow the parsing to continue correctly after making the error area as small as possible.
This would make for a very nice internship subject...
All this said, I think I have a workable suggestion that is not much code that might be open to scrutiny from those wiser than me.
Looking at the keyboard handling in the system - it's quite sprawling and tricky to follow. I think there is lots of room for refactoring.
It is very easy in Pharo to implement a nice and clean keyboard handling; all the necessary components have been in place for years, and we already have implementations (using the KM API).
Now, most Pharo developpers just jump into hardcoding keyboard handling instead.
Thierry
I'm also not sure if I like the pragma usage either - I find it quite difficult to follow what's calling what.
Tim
Sent from my iPhone
On 1 Sep 2017, at 09:18, Thierry Goubier <thierry.goubier@gmail.com <mailto:thierry.goubier@gmail.com>> wrote:
Hi Tim,
The RB ast has a specific method for finding the right node: in AltBrowser, I use ast bestNodeFor: aTarget selectionInterval.
For the second case (parse what is selected), you may want to look into SmaCC: it has specific entry points (used for refactoring) to try all possible starts in a parser for a given piece of text and returning all the possible correct asts. As a hand-writen parser, the RBParser can't do that unless you add the necessary entry points by hand and hack around the error handling, because most of the parse attempts end on an error (as they should).
Regards,
Thierry
2017-09-01 8:50 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Marcus - I'd be interested in how you thought to solve this.
My solution seems to work - but it's not the most elegant. Retrying lots of different ways like I show, smacks a bit of desperation (this said, it is quite compact).
Apart from that - I was actually interested in thoughts on the parse method - the onError naming is a bit misleading compared to other methods in the image. I wonder if it should be called something like "onErrorMerge:" to signal that it's going to reuse the results?
Stef - with regards to broken source, my proposed solution reuses what is already there - thus if you highlight specific code, it reverts to the original strategy and tries to compile just that code. This bugs me however - so if you highlight nothing, it tries to compile the source multiple ways, I drop down to chopping the source up to where your cursor is - which seems like a decent cheap strategy.
I don't know if our parser is able to recover from simple errors and just have error nodes in the ast, and whether the #bestNodeFor method then ignored error nodes... this is what I think Dolphin used to do.
Tim
Sent from my iPhone
On 31 Aug 2017, at 19:11, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:
On 31 Aug 2017, at 19:07, Stephane Ducasse <stepharo.self@gmail.com <mailto:stepharo.self@gmail.com>> wrote:
On Wed, Aug 30, 2017 at 9:50 PM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Iâm looking for some feedback on the usage of RBParser>>#parseMethod:onError:.
I am looking at ways of improving the way we edit code - and making things work (as least for me - but maybe for everyone) a bit more like IntelliJ where they have some great code operations and very good keyboard support. We are certainly close, but at times feel a bit clumsy - which is a shame as we have far better infrastructure to support equivalent features. So I thought I would have a go.
Excellent!!!
Anyway - step one (which I think Iâve come close on), was to teach senders/implementors to look at the AST so you donât have to highlight exactly what you want to search on - instead it can infer from your cursor⦠however my next step was to improve how we can select code to ease refactoring, bracketing etcâ¦
Yes but we have to handle broken code then. Did you check how smart suggestions did it for code that compiled?
I looked some weeks ago and made some notes what to do⦠but I fear this has to wait for next week, there is no way that I can do anything till ESUGâ¦
Marcus
participants (4)
-
Marcus Denker -
Stephane Ducasse -
Thierry Goubier -
Tim Mackinnon