[Pharo-dev] Encoding method source code in Tonel

Stephane Ducasse stepharo.self at gmail.com
Fri Nov 10 13:26:32 EST 2017


I would prefer the last one.
Regularity over exception.
Now the parser should just be good :)

Stef

On Fri, Nov 10, 2017 at 7:20 PM, Sven Van Caekenberghe <sven at stfx.eu> wrote:
>
>
>> On 10 Nov 2017, at 19:12, Stephane Ducasse <stepharo.self at gmail.com> wrote:
>>
>> Hi sven
>>
>> Do you have an example because I do not understand your doubling.
>
> Sure,
>
> Bla { bla:bla } [
> self isFoo ifTrue: [ #foo
> ]
> ifFalse: [ [ "block" #bar
> ]]
> ]
>
> would have to become
>
> Bla { bla:bla } [
> self isFoo ifTrue: [ #foo
> ]]
> ifFalse: [ [ "block" #bar
> ]]]
> ]
>
> but this very rare (because usually code is intended).
>
> Nicolas' latest idea is cool too (use as much ['s as needed to avoid conflict). So the example would then become
>
> Bla { bla:bla } [[[
> self isFoo ifTrue: [ #foo
> ]
> ifFalse: [ [ "block" #bar
> ]]
> ]]]
>
>> stef
>>
>>
>> On Fri, Nov 10, 2017 at 11:40 AM, Sven Van Caekenberghe <sven at stfx.eu> wrote:
>>> Hi,
>>>
>>> Right now, TonelParser>>#methodBody looks a bit dangerous (too complex), and Nicolas already found some breaking cases as well a one possible solution. I was thinking a bit along the same lines, so here is a proposal.
>>>
>>> Overall I think we should keep the current readability of Tonel as much as possible (it is very important for diffs), while still being correct - i.e. no method source, even convoluted ones, possibly in different languages should break the encoding.
>>>
>>> The problem is how to quickly scan a stream for the closing $] (which occurs as a single character on a line by its own). Especially in Pharo source code, that situation (a conflict between the proper marker and actual source) is very rare, since code is usually intended. But is can occur and must be dealt with.
>>>
>>> The proposal is to just double the $] but only on lines that could conflict (which are lines consisting of only $]'s). The code itself becomes quite simple and remains fast enough.
>>>
>>> Here is some code to play with (EOL fixed to #cr for demo purposes):
>>>
>>> writer := [ :stream :methodSourceString |
>>>        stream nextPut: $[; cr.
>>>        methodSourceString linesDo: [ :line |
>>>                (line notEmpty and: [ line allSatisfy: [ :ch | ch = $] ] ])
>>>                        ifTrue: [ stream << $] ].
>>>                stream << line; cr ].
>>>        stream nextPut: $]; cr ].
>>>
>>> reader := [ :stream | | line |
>>>        (stream peekFor: $[) ifTrue: [ stream nextLine ] ifFalse: [ self error ].
>>>        String streamContents: [ :out |
>>>                [ (line := stream nextLine) = ']' ] whileFalse: [
>>>                        (line notEmpty and: [ line allSatisfy: [ :ch | ch = $] ] ])
>>>                                ifTrue: [ out << line allButFirst ]
>>>                                ifFalse: [ out << line ].
>>>                        out cr ] ] ].
>>>
>>> encoded := String streamContents: [ :out | writer value: out value: (Collection>>#inject:into:) sourceCode ].
>>>
>>> (reader value: encoded readStream) = ((Collection>>#inject:into:) sourceCode, String cr).
>>>
>>> escapedEncoded := String streamContents: [ :out | writer value: out value: 'test
>>> "Make sure there is no whitespace at line start to force an escape"
>>> self isFoo ifTrue: [ #foo
>>> ]
>>> ifFalse: [ [ "block" #bar
>>> ]]
>>> ' ].
>>>
>>> (reader value: escapedEncoded readStream).
>>>
>>> What do you think ?
>>> Did I miss any special cases ?
>>> Is the price (very occasional insertion of an extra #] in the source code) worth the advantage (a simple, safe scheme).
>>>
>>> Sven
>>>
>>> PS: Note that several tools line syntax highlighting and double-click next to brackets have problems with weird delimiters such as in the above code ;-)
>>>
>>>
>>
>
>




More information about the Pharo-dev mailing list