Hi Kris, I fixed the problem that you reported. It was a regression caused when adding support for parsing mailto: URLs which don't have a // in them. There is now a new explicit test for parsing '/foo:bar' asZnUrl. Thanks again for reporting this problem! Sven In #bleedingEdge: === Name: Zinc-Resource-Meta-Core-SvenVanCaekenberghe.31 Author: SvenVanCaekenberghe Time: 13 February 2014, 10:41:58.821615 pm UUID: dee515f8-aff3-4930-8b90-da2b01e1ee33 Ancestors: Zinc-Resource-Meta-Core-SvenVanCaekenberghe.30 Patch ZnUrl>>#parseFrom:defaultScheme: to parse schemes more carefully so that a colon in a path does not mess things up - see also the regression test ZnUrlTests>>#testNoSchemeColonInPath parsing /foo:bar (thanks Kris Gijbels) === Name: Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.19 Author: SvenVanCaekenberghe Time: 13 February 2014, 10:42:42.63619 pm UUID: 264c99fe-52ef-4f4a-8a25-fc870f34cfb2 Ancestors: Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.18 Patch ZnUrl>>#parseFrom:defaultScheme: to parse schemes more carefully so that a colon in a path does not mess things up - see also the regression test ZnUrlTests>>#testNoSchemeColonInPath parsing /foo:bar (thanks Kris Gijbels) ===
Hey Sven, Thanks for fixing that. We're close to getting Yesplan working on Pharo3 now. However, yesterday I stumbled upon another problem. It seems that a percent-encoded '+' (i.e. %2B) get eaten by Zinc and converted into a space (ie. %20). The following test fails in latest Zinc but works in a Zinc on Pharo 1.4: | input request | input := 'GET http://localhost/foo+%2B+bar HTTP/1.1', String crlf, String crlf. request := ZnRequest readFrom: input readStream. self assert: (request url pathSegments hasEqualElements: #('foo%20%2B%20bar')). I'm tracing if I can find the actual source of the problem. regards Johan On 13 Feb 2014, at 22:47, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Kris,
I fixed the problem that you reported. It was a regression caused when adding support for parsing mailto: URLs which don't have a // in them. There is now a new explicit test for parsing '/foo:bar' asZnUrl.
Thanks again for reporting this problem!
Sven
In #bleedingEdge:
=== Name: Zinc-Resource-Meta-Core-SvenVanCaekenberghe.31 Author: SvenVanCaekenberghe Time: 13 February 2014, 10:41:58.821615 pm UUID: dee515f8-aff3-4930-8b90-da2b01e1ee33 Ancestors: Zinc-Resource-Meta-Core-SvenVanCaekenberghe.30
Patch ZnUrl>>#parseFrom:defaultScheme: to parse schemes more carefully so that a colon in a path does not mess things up - see also the regression test ZnUrlTests>>#testNoSchemeColonInPath parsing /foo:bar (thanks Kris Gijbels) === Name: Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.19 Author: SvenVanCaekenberghe Time: 13 February 2014, 10:42:42.63619 pm UUID: 264c99fe-52ef-4f4a-8a25-fc870f34cfb2 Ancestors: Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.18
Patch ZnUrl>>#parseFrom:defaultScheme: to parse schemes more carefully so that a colon in a path does not mess things up - see also the regression test ZnUrlTests>>#testNoSchemeColonInPath parsing /foo:bar (thanks Kris Gijbels) ===
Johan, On 14 Feb 2014, at 09:43, Johan Brichau <johan@inceptive.be> wrote:
Hey Sven,
Thanks for fixing that. We're close to getting Yesplan working on Pharo3 now.
That is very good to hear, keep on pushing ! It would be a super cool reference for Pharo 3. I am working with the latest Seaside (Bootstrap) on Pharo 3 this week and I am quite pleased as well, thanks for all the hard working getting the codebase and configs working.
However, yesterday I stumbled upon another problem. It seems that a percent-encoded '+' (i.e. %2B) get eaten by Zinc and converted into a space (ie. %20). The following test fails in latest Zinc but works in a Zinc on Pharo 1.4:
| input request |
input := 'GET http://localhost/foo+%2B+bar HTTP/1.1', String crlf, String crlf. request := ZnRequest readFrom: input readStream. self assert: (request url pathSegments hasEqualElements: #('foo%20%2B%20bar')).
I'm tracing if I can find the actual source of the problem.
Again, I think this can be simplified to URL parsing. 'http://localhost/foo+%2B+bar' asZnUrl. => 'http://localhost/foo%20+%20bar' 'http://localhost/foo+%2B+bar' asZnUrl firstPathSegment. => 'foo + bar' But I think it is correct, unless you read the specs differently. There were indeed some recent , under the hood changes to which characters are allowed in which part of URLs, I did that together with Jan van de Sandt, and we thought we had everything covered, but it is hard to be sure. See ZnUrl>>#encodePath:on: ZnUrl>>#encodeQuery:on: to get started. Sven
regards Johan
On 13 Feb 2014, at 22:47, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Hi Kris,
I fixed the problem that you reported. It was a regression caused when adding support for parsing mailto: URLs which don't have a // in them. There is now a new explicit test for parsing '/foo:bar' asZnUrl.
Thanks again for reporting this problem!
Sven
In #bleedingEdge:
=== Name: Zinc-Resource-Meta-Core-SvenVanCaekenberghe.31 Author: SvenVanCaekenberghe Time: 13 February 2014, 10:41:58.821615 pm UUID: dee515f8-aff3-4930-8b90-da2b01e1ee33 Ancestors: Zinc-Resource-Meta-Core-SvenVanCaekenberghe.30
Patch ZnUrl>>#parseFrom:defaultScheme: to parse schemes more carefully so that a colon in a path does not mess things up - see also the regression test ZnUrlTests>>#testNoSchemeColonInPath parsing /foo:bar (thanks Kris Gijbels) === Name: Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.19 Author: SvenVanCaekenberghe Time: 13 February 2014, 10:42:42.63619 pm UUID: 264c99fe-52ef-4f4a-8a25-fc870f34cfb2 Ancestors: Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.18
Patch ZnUrl>>#parseFrom:defaultScheme: to parse schemes more carefully so that a colon in a path does not mess things up - see also the regression test ZnUrlTests>>#testNoSchemeColonInPath parsing /foo:bar (thanks Kris Gijbels) ===
Hey Sven,
From what I traced already, the path segments are correctly stored. The parsing works fine. It's the printing of a ZnUrl that seems to ignore it and produce a 'foo%20+%20bar'
Diving into it... Johan On 14 Feb 2014, at 10:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Again, I think this can be simplified to URL parsing.
'http://localhost/foo+%2B+bar' asZnUrl.
=> 'http://localhost/foo%20+%20bar'
'http://localhost/foo+%2B+bar' asZnUrl firstPathSegment.
=> 'foo + bar'
But I think it is correct, unless you read the specs differently. There were indeed some recent , under the hood changes to which characters are allowed in which part of URLs, I did that together with Jan van de Sandt, and we thought we had everything covered, but it is hard to be sure. See
ZnUrl>>#encodePath:on: ZnUrl>>#encodeQuery:on:
to get started.
On 14 Feb 2014, at 10:25, Johan Brichau <johan@inceptive.be> wrote:
Hey Sven,
From what I traced already, the path segments are correctly stored. The parsing works fine. It's the printing of a ZnUrl that seems to ignore it and produce a 'foo%20+%20bar'
But that is correct, no ? ZnResourceMetaUtils urlPathSafeSet includes: $+ + is a normal URL path character, Character space is not. It is possible that Zn did something else earlier, but now it is (more) correct ;-)
Diving into it...
Johan
On 14 Feb 2014, at 10:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Again, I think this can be simplified to URL parsing.
'http://localhost/foo+%2B+bar' asZnUrl.
=> 'http://localhost/foo%20+%20bar'
'http://localhost/foo+%2B+bar' asZnUrl firstPathSegment.
=> 'foo + bar'
But I think it is correct, unless you read the specs differently. There were indeed some recent , under the hood changes to which characters are allowed in which part of URLs, I did that together with Jan van de Sandt, and we thought we had everything covered, but it is hard to be sure. See
ZnUrl>>#encodePath:on: ZnUrl>>#encodeQuery:on:
to get started.
No, because the original request was 'foo%20%2B%20' If it prints it, the + should be encoded again. On 14 Feb 2014, at 10:31, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 14 Feb 2014, at 10:25, Johan Brichau <johan@inceptive.be> wrote:
Hey Sven,
From what I traced already, the path segments are correctly stored. The parsing works fine. It's the printing of a ZnUrl that seems to ignore it and produce a 'foo%20+%20bar'
But that is correct, no ?
ZnResourceMetaUtils urlPathSafeSet includes: $+
+ is a normal URL path character, Character space is not.
It is possible that Zn did something else earlier, but now it is (more) correct ;-)
Diving into it...
Johan
On 14 Feb 2014, at 10:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Again, I think this can be simplified to URL parsing.
'http://localhost/foo+%2B+bar' asZnUrl.
=> 'http://localhost/foo%20+%20bar'
'http://localhost/foo+%2B+bar' asZnUrl firstPathSegment.
=> 'foo + bar'
But I think it is correct, unless you read the specs differently. There were indeed some recent , under the hood changes to which characters are allowed in which part of URLs, I did that together with Jan van de Sandt, and we thought we had everything covered, but it is hard to be sure. See
ZnUrl>>#encodePath:on: ZnUrl>>#encodeQuery:on:
to get started.
On 14 Feb 2014, at 10:34, Johan Brichau <johan@inceptive.be> wrote:
No, because the original request was 'foo%20%2B%20'
If it prints it, the + should be encoded again.
Why ? Here, but also in most of my other packages, the parsers are more lenient than the specs, to support common practice, but stricter when generating output. So any percent-encoded character is accepted [ but being encoded means that its meaning as structuring element is lost, consider $/ ], not necessarily encoded on output. WAUrl does it indeed, but that is because it considers $+ unsafe, which is not correct (Zn did that as well before, in the 'better safe than sorry' philosophy). How exactly is this creating problems ?
On 14 Feb 2014, at 10:31, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 14 Feb 2014, at 10:25, Johan Brichau <johan@inceptive.be> wrote:
Hey Sven,
From what I traced already, the path segments are correctly stored. The parsing works fine. It's the printing of a ZnUrl that seems to ignore it and produce a 'foo%20+%20bar'
But that is correct, no ?
ZnResourceMetaUtils urlPathSafeSet includes: $+
+ is a normal URL path character, Character space is not.
It is possible that Zn did something else earlier, but now it is (more) correct ;-)
Diving into it...
Johan
On 14 Feb 2014, at 10:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Again, I think this can be simplified to URL parsing.
'http://localhost/foo+%2B+bar' asZnUrl.
=> 'http://localhost/foo%20+%20bar'
'http://localhost/foo+%2B+bar' asZnUrl firstPathSegment.
=> 'foo + bar'
But I think it is correct, unless you read the specs differently. There were indeed some recent , under the hood changes to which characters are allowed in which part of URLs, I did that together with Jan van de Sandt, and we thought we had everything covered, but it is hard to be sure. See
ZnUrl>>#encodePath:on: ZnUrl>>#encodeQuery:on:
to get started.
On 14 Feb 2014, at 10:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 14 Feb 2014, at 10:34, Johan Brichau <johan@inceptive.be> wrote:
No, because the original request was 'foo%20%2B%20'
If it prints it, the + should be encoded again.
Why ?
The + character in a url is decoded as a space: self assert: (ZnPercentEncoder new decode: '+' = ' ') So, because I notice that the incoming url is decoded when parsing to a ZnUrl's path segments, recoded when printed and then again decoded, the following happens: - '%2B' (encoding of +) gets decoded to '+' (correct) - '+' gets encoded as '+' (incorrect) - '+' gets decoded as ' ' (correct)
How exactly is this creating problems ?
In Seaside, when you serialize an input form using a jquery ajax request and you enter a '+' in the input form, the browser correctly encodes the '+' character to a '%2B' in the request url. However, because the Zinc server performs the above transformation, the '+' becomes a ' ' in the incoming value of the Seaside callback. The + is not a safe character because it is an encoding of a space. Johan
On 14 Feb 2014, at 10:53, Johan Brichau <johan@inceptive.be> wrote:
On 14 Feb 2014, at 10:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 14 Feb 2014, at 10:34, Johan Brichau <johan@inceptive.be> wrote:
No, because the original request was 'foo%20%2B%20'
If it prints it, the + should be encoded again.
Why ?
The + character in a url is decoded as a space:
self assert: (ZnPercentEncoder new decode: '+' = ' ')
Hmm, but like I said before, there are different safe sets, depending on the situation. That is was recently added.
So, because I notice that the incoming url is decoded when parsing to a ZnUrl's path segments, recoded when printed and then again decoded, the following happens: - '%2B' (encoding of +) gets decoded to '+' (correct) - '+' gets encoded as '+' (incorrect) - '+' gets decoded as ' ' (correct)
Where/why/how is it re-encoded and decoded ? Maybe it should happen differently in that particular case.
How exactly is this creating problems ?
In Seaside, when you serialize an input form using a jquery ajax request and you enter a '+' in the input form, the browser correctly encodes the '+' character to a '%2B' in the request url. However, because the Zinc server performs the above transformation, the '+' becomes a ' ' in the incoming value of the Seaside callback.
The + is not a safe character because it is an encoding of a space.
Is there a standard functional Seaside Ajax test that I can use to see this for myself ? Sven
On 14 Feb 2014, at 11:12, Johan Brichau <johan@inceptive.be> wrote:
On 14 Feb 2014, at 11:06, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Is there a standard functional Seaside Ajax test that I can use to see this for myself ?
localhost:8080/tests/functional-ajaxified
WAUrlEncodingFunctionalTest
enter a + in the form
That is not in my image (I got started with ConfigurationOfBootstrap which seems to load plain ConfigurationOfSeaside3). But I have seen these tests in the past. What groups should I load additionally ? Sorry...
On 14 Feb 2014, at 11:33, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Is there a standard functional Seaside Ajax test that I can use to see this for myself ?
localhost:8080/tests/functional-ajaxified
WAUrlEncodingFunctionalTest
enter a + in the form
That is not in my image (I got started with ConfigurationOfBootstrap which seems to load plain ConfigurationOfSeaside3). But I have seen these tests in the past.
What groups should I load additionally ?
Sorry...
No worries, thank _you_ for taking a look into this! The 'Tests' group contains those tests. On the page, it's the second-to-last element in the dropdown. Johan
On 14 Feb 2014, at 11:54, Johan Brichau <johan@inceptive.be> wrote:
On 14 Feb 2014, at 11:33, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Is there a standard functional Seaside Ajax test that I can use to see this for myself ?
localhost:8080/tests/functional-ajaxified
WAUrlEncodingFunctionalTest
enter a + in the form
That is not in my image (I got started with ConfigurationOfBootstrap which seems to load plain ConfigurationOfSeaside3). But I have seen these tests in the past.
What groups should I load additionally ?
Sorry...
No worries, thank _you_ for taking a look into this! The 'Tests' group contains those tests. On the page, it's the second-to-last element in the drop down.
Yes, I see the function tests, but not the functional-ajaxified ones...
On 14 Feb 2014, at 11:57, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Yes, I see the function tests, but not the functional-ajaxified ones...
Sorry, I now realize this is present in the non-ajaxified too. So, you can test it there. If you enter a text with a '+' inside, you will see it gets replaced with a space too. Johan
On 14 Feb 2014, at 10:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:
So any percent-encoded character is accepted [ but being encoded means that its meaning as structuring element is lost, consider $/ ], not necessarily encoded on output.
Right, but here the exact opposite happens. A reserved character (+) was included as an encoded character (%2B), gets correctly decoded by Zinc but then never gets encoded again when printing the url, hence become a structuring element representing a space in the printed url. Because a WAUrl is constructed from incoming ZnUrl's printString, the WAUrl gets to read the '+' instead of the '%2B'. The following code snippet is now true, but should be false imho: (ZnUrl fromString: (ZnUrl fromString: 'http://localhost/?param=foo%2Bbar') printString) printString = 'http://localhost/?param=foo%20bar' Johan
I think we are getting somewhere (it is about time). $+ is handled in any usage of ZnPercentEncoder, independent of context and safeSet. That is not right, it is only valid in the context of application/x-www-form-urlencoded get/post requests: http://en.wikipedia.org/wiki/Percent-encoding#The_application.2Fx-www-form-u... Outside of that specific use case, $+ becomes space, should not be done, I think. OK, now I have something concrete to work with. I still would like to be able to run the Seaside Ajax functional test though. On 14 Feb 2014, at 11:42, Johan Brichau <johan@inceptive.be> wrote:
On 14 Feb 2014, at 10:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:
So any percent-encoded character is accepted [ but being encoded means that its meaning as structuring element is lost, consider $/ ], not necessarily encoded on output.
Right, but here the exact opposite happens. A reserved character (+) was included as an encoded character (%2B), gets correctly decoded by Zinc but then never gets encoded again when printing the url, hence become a structuring element representing a space in the printed url.
Because a WAUrl is constructed from incoming ZnUrl's printString, the WAUrl gets to read the '+' instead of the '%2B'.
The following code snippet is now true, but should be false imho:
(ZnUrl fromString: (ZnUrl fromString: 'http://localhost/?param=foo%2Bbar') printString) printString = 'http://localhost/?param=foo%20bar'
Johan
On 14 Feb 2014, at 12:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
$+ is handled in any usage of ZnPercentEncoder, independent of context and safeSet. That is not right, it is only valid in the context of application/x-www-form-urlencoded get/post requests:
http://en.wikipedia.org/wiki/Percent-encoding#The_application.2Fx-www-form-u...
Outside of that specific use case, $+ becomes space, should not be done, I think.
OK, now I have something concrete to work with.
Well... WAUrl does exactly the same: it always replaces a $+ with a space. If what you say regarding $+ being a safe character (I'm not the expert), then it's part of the problem because a WAUrl instance is constructed from a ZnUrl printString. If a ZnUrl prints a $+ where it originally was constructed from a %2B, then it will still fail. I might we wrong, but I keep having the impression it's the (non-)encoding of a $+ that is wrong (rather than the decoding of $+ to space). If you create a url as follows: (ZnUrl fromString: 'http://localhost:8080/foo%20%2B%20bar') The internal pathSegments variable of the ZnUrl instance contains a (I think correctly) decoded string 'foo + bar'. When this url is printed (more exactly in #printPath:), it prints it as 'foo%20+%20bar', which is strange for me. But, as I said, I am not the expert. Though I will have to see where this impacts Seaside on working with other adaptors... Johan
Hey Sven, I'm catching up on some reading on the topic [1]. It starts making sense to me :-) So, it does seem that both WAUrl and ZnUrl treat the '+' incorrectly (as you mentioned), leading to this: (ZnUrl fromString: (ZnUrl fromString: 'http://localhost/?param=foo%2Bbar') printString) printString = 'http://localhost/?param=foo%20bar'. (WAUrl decodePercent: (ZnUrl fromString: 'http://localhost/?param=foo%2Bbar') printString) printString = 'http://localhost/?param=foo bar' [1] http://blog.lunatech.com/2009/02/03/what-every-web-developer-must-know-about... I learned something new today ;-) Johan On 14 Feb 2014, at 16:47, Johan Brichau <johan@inceptive.be> wrote:
On 14 Feb 2014, at 12:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
$+ is handled in any usage of ZnPercentEncoder, independent of context and safeSet. That is not right, it is only valid in the context of application/x-www-form-urlencoded get/post requests:
http://en.wikipedia.org/wiki/Percent-encoding#The_application.2Fx-www-form-u...
Outside of that specific use case, $+ becomes space, should not be done, I think.
OK, now I have something concrete to work with.
Well... WAUrl does exactly the same: it always replaces a $+ with a space. If what you say regarding $+ being a safe character (I'm not the expert), then it's part of the problem because a WAUrl instance is constructed from a ZnUrl printString. If a ZnUrl prints a $+ where it originally was constructed from a %2B, then it will still fail.
I might we wrong, but I keep having the impression it's the (non-)encoding of a $+ that is wrong (rather than the decoding of $+ to space). If you create a url as follows: (ZnUrl fromString: 'http://localhost:8080/foo%20%2B%20bar') The internal pathSegments variable of the ZnUrl instance contains a (I think correctly) decoded string 'foo + bar'. When this url is printed (more exactly in #printPath:), it prints it as 'foo%20+%20bar', which is strange for me.
But, as I said, I am not the expert. Though I will have to see where this impacts Seaside on working with other adaptors...
Johan
participants (2)
-
Johan Brichau -
Sven Van Caekenberghe