On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com> escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works Not silly and no need to suspect, but absolutely correct !

Sven

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find. Inspect this in a workspace/playground. ZnClient new https; host: 'google.com'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error. Of course Google won't understand this and in this snippet won't receive it. And please let me know if I am doing something wrong. Any help greatly appreciated. Jimmie

That's because the comma does not need to be escaped in the query part of the uri. Norbert

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com>:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com> escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: 'google.com'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Just to clarify: "Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt <https://www.ietf.org/rfc/rfc2396.txt> The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded <http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded>

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com>:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> <mailto:stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com> escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works Not silly and no need to suspect, but absolutely correct !

Sven My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: 'google.com'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Thanks, that worked great. That's why I said I am not an expert. I can pull up a page or two. But that doesn't mean I understand all the legal minutiae involved in what is and what isn't legal in an URI or query or path or ... That is why I noted my requirements regardless of legal minutiae and right or wrong. Thanks for a practical solution. Jimmie On 06/10/2015 06:22 PM, PBKResearch wrote:

There is a simpler way than Norbert’s suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your ‘Google’ example generates the query line with the ‘%2C’ encoding of the commas.

Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem – earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving.

Hope this helps

Peter Kenny

*From:*Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] *On Behalf Of *Norbert Hartl *Sent:* 10 June 2015 23:56 *To:* Pharo users users *Subject:* Re: [Pharo-users] ZnClient and percent characters

Just to clarify:

"Characters in the "reserved" set are not reserved in all contexts.

The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com <mailto:jlhouchin@gmail.com>>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt

The plus "+", dollar "$", and comma "," characters have been added to

those in the "reserved" set, since they are treated as reserved

within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin<jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com>:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

On 10 Jun 2015, at 17:24, David<stormbyte@gmail.com> <mailto:stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500

Jimmie Houchin<jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com>

escribió:

Hello,

I am attempting to use ZnClient to request data. The request requires

a %2C (comma) delimited string as part of the query. Below is a

snippet.

znClient

addPath: '/v1/instruments';

queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip';

get ;

contents)

The string 'displayName%2Cinstrument%2Cpip'

is being converted to 'displayName%252Cinstrument%252Cpip'

which causes the request to fail.

The query needs to be

fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly.

Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to

make itself encode that? Like

znClient

addPath: '/v1/instruments';

queryAt: 'fields' putAll: 'displayName,instrument,pip';

get ;

contents)

I suspect it is using encoding internally, that is why % is also

encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version.

Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new

https;

host: 'google.com <http://google.com>';

addPath: '/commaTest';

queryAt: 'fields' put: 'displayName,instrument,pip';

yourself

View the request / requestLine / uri. The commas are still present in the URI.

So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

I don’t quite understand Norbert’s comment. Does ‘monkey’ apply to me or to what I have done? Either way, it seems unnecessary and abusive. Thanks to Sven’s careful use of informative method names, the effect of my change is quite clear. Any comma in the value part of a query line will be encoded. Nothing else. I did my best to trace any side effects, and didn’t find any. What is not ‘reliable’ about that? Peter Kenny I was thinking of a reliable solution. Of course if it only needs to be tested than monkey patching foreign packages is ok. Norbert Am 11.06.2015 um 01:22 schrieb PBKResearch <peter@pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk> >: There is a simpler way than Norbert’s suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your ‘Google’ example generates the query line with the ‘%2C’ encoding of the commas. Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem – earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving. Hope this helps Peter Kenny From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Norbert Hartl Sent: 10 June 2015 23:56 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters Just to clarify: "Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert Am 11.06.2015 um 00:26 schrieb Jimmie Houchin < <mailto:jlhouchin@gmail.com> jlhouchin@gmail.com>: I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas. As far as commas needing to be escaped, it seems from other sources that they should be.

From <https://www.ietf.org/rfc/rfc2396.txt> https://www.ietf.org/rfc/rfc2396.txt

The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component. States that commas are reserved within the query component. <http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states. fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values. Which will look like this or something similar. fields=displayName%2Cinstrument%2Cpip Thanks. Jimmie On 06/10/2015 03:27 PM, Norbert Hartl wrote: That's because the comma does not need to be escaped in the query part of the uri. Norbert Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <mailto:jlhouchin@gmail.com> <jlhouchin@gmail.com>: On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: On 10 Jun 2015, at 17:24, David <mailto:stormbyte@gmail.com> <stormbyte@gmail.com> wrote: El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <mailto:jlhouchin@gmail.com> <jlhouchin@gmail.com> escribió: Hello, I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet. znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents) The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail. The query needs to be fields=displayName%2Cinstrument%2Cpip I have not found how to do this correctly. Any help greatly appreciated. Thanks. Jimmie Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents) I suspect it is using encoding internally, that is why % is also encoded if you try to put it. I hope that works Not silly and no need to suspect, but absolutely correct ! Sven My apologies for not having full disclosure. Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04 That is what I thought would happen and what I tried first. But it is not being encoded from what I can find. Inspect this in a workspace/playground. ZnClient new https; host: ' <http://google.com/> google.com'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error. Of course Google won't understand this and in this snippet won't receive it. And please let me know if I am doing something wrong. Any help greatly appreciated. Jimmie

No offense meant. I assumed you would know the term. Sorry! Henry explained it quite good what I meant. Norbert

Am 11.06.2015 um 09:51 schrieb PBKResearch <peter@pbkresearch.co.uk>:

I don’t quite understand Norbert’s comment. Does ‘monkey’ apply to me or to what I have done? Either way, it seems unnecessary and abusive.

Thanks to Sven’s careful use of informative method names, the effect of my change is quite clear. Any comma in the value part of a query line will be encoded. Nothing else. I did my best to trace any side effects, and didn’t find any. What is not ‘reliable’ about that?

Peter Kenny

I was thinking of a reliable solution. Of course if it only needs to be tested than monkey patching foreign packages is ok.

Norbert

...

Am 11.06.2015 um 01:22 schrieb PBKResearch <peter@pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>:

There is a simpler way than Norbert’s suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your ‘Google’ example generates the query line with the ‘%2C’ encoding of the commas.

Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem – earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving.

Hope this helps

Peter Kenny

From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org>] On Behalf Of Norbert Hartl Sent: 10 June 2015 23:56 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters

Just to clarify:

"Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com <mailto:jlhouchin@gmail.com>>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt <https://www.ietf.org/rfc/rfc2396.txt>

The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded <http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded>

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com>:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> <mailto:stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com> escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works Not silly and no need to suspect, but absolutely correct !

Sven My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: 'google.com <http://google.com/>'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Am 11.06.2015 um 16:51 schrieb PBKResearch <peter@pbkresearch.co.uk>:

Norbert

Apology accepted – I had never heard this term before. However, having read the Wikipedia article Henry mentioned, I do not think it applies to what I had done. If Sven accepts the argument that commas should always be encoded in query lines – and I can see a strong argument for that from what I have read – then the change I made is just what he will need to do. I simply anticipated the decision. If it goes the other way, then it was just a hack to solve Jimmie’s problem – but it might be better to say ‘hack’ rather than use a term which is potentially offensive.

I just wrote because you told Jimmie to alter a method of a third-party library. And that is a no-go. The way to change it and have Sven it commited is doable. As is my proposal that you can change the code in a reliable way without changing a third-party library. That's all I wanted to say. And yes I think it applies to monkey patching. Choosing weaker terms just to avoid problems is not the way to go IMHO.

I also note Jimmie’s argument that, as long as there are some sites which insist on encoded commas, it must be possible to generate such lines if Zinc is to be generally useful. Whether this should be a user-controlled option, as Sven seems to suggest, is open to discussion – but if so I would argue for encoding of commas as the safe default, which can be overridden if necessary.

Sure. Zinc should behave correctly and it should provide functionalities to make it "less correct". And these functionalities shouldn't be too easy to use ;) Norbert

Peter Kenny

From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Norbert Hartl Sent: 11 June 2015 11:25 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters

No offense meant. I assumed you would know the term. Sorry! Henry explained it quite good what I meant.

Norbert

...

Am 11.06.2015 um 09:51 schrieb PBKResearch <peter@pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>:

I don’t quite understand Norbert’s comment. Does ‘monkey’ apply to me or to what I have done? Either way, it seems unnecessary and abusive.

Thanks to Sven’s careful use of informative method names, the effect of my change is quite clear. Any comma in the value part of a query line will be encoded. Nothing else. I did my best to trace any side effects, and didn’t find any. What is not ‘reliable’ about that?

Peter Kenny

I was thinking of a reliable solution. Of course if it only needs to be tested than monkey patching foreign packages is ok.

Norbert

...

Am 11.06.2015 um 01:22 schrieb PBKResearch <peter@pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>:

There is a simpler way than Norbert’s suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your ‘Google’ example generates the query line with the ‘%2C’ encoding of the commas.

Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem – earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving.

Hope this helps

Peter Kenny

From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org>] On Behalf Of Norbert Hartl Sent: 10 June 2015 23:56 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters

Just to clarify:

"Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com <mailto:jlhouchin@gmail.com>>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt <https://www.ietf.org/rfc/rfc2396.txt>

The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded <http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded>

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com>:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> <mailto:stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com> escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works Not silly and no need to suspect, but absolutely correct !

Sven My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: 'google.com <http://google.com/>'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Am 12.06.2015 um 12:03 schrieb PBKResearch <peter@pbkresearch.co.uk>:

Norbert

Some comments on your latest post below.

I just wrote because you told Jimmie to alter a method of a third-party library. And that is a no-go.

Zinc, like almost all of Pharo, is MIT licenced, “including without limitation the rights to use, copy, modify, merge, publish…” Once the code is on my machine, I am entitled to alter it for my own use in any way I see fit; so is Jimmie. It is clear from Sven’s comments that not encoding commas is a design decision, and a departure from earlier Zinc working. Sven himself says “ maybe we made a mistake, maybe not.” I am entitled to disagree with that decision, and to act on that disagreement; so is Jimmie. I just told him how to do it quickly and safely.

So, the misunderstanding is because we seem to have different views on the topic. I have always projects in mind that are built together by a build system and they produce something deployable. Your proposal doesn't fit in this view for me. If you alter a third-party function you need to make that a repeatable task which is having an expression to patch the third-party library again if you build the project again.

The way to change it and have Sven it commited is doable.

Obviously it would be ideal if Sven made a change which allowed encoding of commas. However, that does not seem likely to happen any time soon. Meanwhile, Jimmie was stuck with a program which would not carry out a function that should be quite normal.

As is my proposal that you can change the code in a reliable way without changing a third-party library. That's all I wanted to say.

No doubt your subclassing approach would work, though not as easily as you imply. The method that encodes the key-value pairs of a query is not #encodeQuery:on:, it is #printQueryOn: (see ZnUrl>>#printPathQueryFragmentOn:); changing that would involve having variant versions of ZnResourceMetaUtils>>#writeQueryFields:on: and >>writeQueryFields:withEncoder:on:, as well as a new safe list.

Yes, but it is still the thing you can do immediately without having to patch code that is not under your control. That's my whole point. I wouldn't consider my approach good or even nice.

And yes I think it applies to monkey patching. Choosing weaker terms just to avoid problems is not the way to go IMHO.

I was not asking for a weaker term, I was asking for a less offensive one. I think this term is offensive in all contexts, and should not be used. Henry’s comment implies that it is widely used, and everyone knows what it means. In my youth, 60-70 years ago, the n-word was widely used to refer to people of African origin, and everyone knew what it meant. Now, thank goodness, we see how offensive it was. To me, this expression is equally offensive. You used it, in a derogatory sense, to criticise what I had done in helping Jimmie. Frankly, I do not think you were sincere in saying “No offense meant.”

I like the term monkey patching because it is as funny as it might be offensive. The rest I don't want to comment because I could easily be offended by your paragraph above…if I only had those kinds of sensitivities. And btw. don't hide behind something like not using a certain word. You can still be a racist without using the n-word. What is needed is to change attitude not words.

I had thought of just ignoring your comment, thinking that we will have to agree to differ. However, your reiteration of the m-word has got me sufficiently riled to make another protest. I may be a lone voice here, but that does not make my views any less valid.

dito. Let's close this case. To me it is a little misunderstanding that took the wrong route. Norbert

Peter Kenny

From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Norbert Hartl Sent: 11 June 2015 17:07 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters

...

Am 11.06.2015 um 16:51 schrieb PBKResearch <peter@pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>:

Norbert

Apology accepted – I had never heard this term before. However, having read the Wikipedia article Henry mentioned, I do not think it applies to what I had done. If Sven accepts the argument that commas should always be encoded in query lines – and I can see a strong argument for that from what I have read – then the change I made is just what he will need to do. I simply anticipated the decision. If it goes the other way, then it was just a hack to solve Jimmie’s problem – but it might be better to say ‘hack’ rather than use a term which is potentially offensive.

I just wrote because you told Jimmie to alter a method of a third-party library. And that is a no-go. The way to change it and have Sven it commited is doable. As is my proposal that you can change the code in a reliable way without changing a third-party library. That's all I wanted to say. And yes I think it applies to monkey patching. Choosing weaker terms just to avoid problems is not the way to go IMHO.

...

I also note Jimmie’s argument that, as long as there are some sites which insist on encoded commas, it must be possible to generate such lines if Zinc is to be generally useful. Whether this should be a user-controlled option, as Sven seems to suggest, is open to discussion – but if so I would argue for encoding of commas as the safe default, which can be overridden if necessary.

Sure. Zinc should behave correctly and it should provide functionalities to make it "less correct". And these functionalities shouldn't be too easy to use ;)

Norbert

...

Peter Kenny

From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org>] On Behalf Of Norbert Hartl Sent: 11 June 2015 11:25 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters

No offense meant. I assumed you would know the term. Sorry! Henry explained it quite good what I meant.

Norbert

...

Am 11.06.2015 um 09:51 schrieb PBKResearch <peter@pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>:

I don’t quite understand Norbert’s comment. Does ‘monkey’ apply to me or to what I have done? Either way, it seems unnecessary and abusive.

Thanks to Sven’s careful use of informative method names, the effect of my change is quite clear. Any comma in the value part of a query line will be encoded. Nothing else. I did my best to trace any side effects, and didn’t find any. What is not ‘reliable’ about that?

Peter Kenny

I was thinking of a reliable solution. Of course if it only needs to be tested than monkey patching foreign packages is ok.

Norbert

...

Am 11.06.2015 um 01:22 schrieb PBKResearch <peter@pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>:

There is a simpler way than Norbert’s suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your ‘Google’ example generates the query line with the ‘%2C’ encoding of the commas.

Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem – earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving.

Hope this helps

Peter Kenny

From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org>] On Behalf Of Norbert Hartl Sent: 10 June 2015 23:56 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters

Just to clarify:

"Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com <mailto:jlhouchin@gmail.com>>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt <https://www.ietf.org/rfc/rfc2396.txt>

The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded <http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded>

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com>:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> <mailto:stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com> <mailto:jlhouchin@gmail.com> escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works Not silly and no need to suspect, but absolutely correct !

Sven My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: 'google.com <http://google.com/>'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Thanks for the reply. I implemented Peter's suggestion as an easy keep moving solution. As I said, I am not expert in what is or is not legal according to the standards. However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default. _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-') https://docs.python.org/3/library/urllib.parse.html https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py That's at least how one major language understands the standard. And Python 2.7 is the same. According to Wikipedia http://en.wikipedia.org/wiki/Query_string # Characters that cannot be converted to the correct charset are replaced with HTMLnumeric character references <http://en.wikipedia.org/wiki/Numeric_character_reference>^[9] <http://en.wikipedia.org/wiki/Query_string#cite_note-html5_urlencoded-9> # SPACE is encoded as '|+|' # Letters (|A|–|Z|and|a|–|z|), numbers (|0|–|9|) and the characters '|*|','|-|','|.|' and '|_|' are left as-is It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded Just a little more information to add to the discussion. Thanks. Jimmie On 06/10/2015 05:56 PM, Norbert Hartl wrote:

Just to clarify:

"Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com <mailto:jlhouchin@gmail.com>>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin<jlhouchin@gmail.com>:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David<stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin<jlhouchin@gmail.com> escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works Not silly and no need to suspect, but absolutely correct !

Sven My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: 'google.com <http://google.com>'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Sven As I read the code, there are two methods in ZnMetaResourceUtils class which specify percent encoding of queries, #queryKeyValueSafeSet and #querySafeSet. The first is used in encoding the (key, value) pairs, the second in encoding the fragment after the #. It is not clear whether we need separate safe sets for the two cases, but in any case the first one was obviously the one to change to fix Jimmie's original problem. Having been accused of monkeying with your code in an unreliable way, I shall keep my head down in future. It's a pity - I find it quite enjoyable trying to help people, and trying to prove that my elderly brain is not completely addled. Peter Kenny -----Original Message----- From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Sven Van Caekenberghe Sent: 11 June 2015 07:35 To: Any question about pharo is welcome Subject: Re: [Pharo-users] ZnClient and percent characters @everybody The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts. Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets. Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations. The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context. ZnUrl was written against RFC 3986 mostly. Now, maybe we made a mistake, maybe not. But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis.

On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouchin@gmail.com> wrote:

Thanks for the reply.

I implemented Peter's suggestion as an easy keep moving solution.

As I said, I am not expert in what is or is not legal according to the standards. However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default.

_ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-')

https://docs.python.org/3/library/urllib.parse.html https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py

That's at least how one major language understands the standard. And Python 2.7 is the same.

According to Wikipedia http://en.wikipedia.org/wiki/Query_string • Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[9] • SPACE is encoded as '+' • Letters (A–Z and a–z), numbers (0–9) and the characters '*','-','.' and '_' are left as-is

It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encode d Just a little more information to add to the discussion.

Thanks.

Jimmie

On 06/10/2015 05:56 PM, Norbert Hartl wrote:

...

Just to clarify:

" Characters in the "reserved" set are not reserved in all contexts.

The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding."

If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-enco ded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> :

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com>

escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: ' google.com '; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Here is one older discussion that let to Zn going away from the better safe than sorry approach to encoding: http://forum.world.st/Zinc-How-to-use-the-character-in-an-URL-td4716386.html... You have to read it up to the end.

On 11 Jun 2015, at 16:19, Jimmie Houchin <jlhouchin@gmail.com> wrote:

This is exactly why I expressly state I am not a language lawyer and explicitly do not know what is and is not expressly forbidden or allowed with regards to a comma.

You are correct about the Wikipedia article.

Is it every wrong or illegal to use a complete safely encoded request? Is it just simply not required? So not fully encoded is still valid and legal and also the fully encode is also fully valid and legal.

Yes, any server should accept all encodings. However, the following are different: http://foo.com/bar?x=1 http://foo.com/bar?x%3D1 In the second case, you take away the meaning of = by encoding it. So you just can't encode everything everywhere.

eg: http://yourserver.com/path?options=eggs,toast,coffee is fully valid and legal, but may encounter problems depending to whom the request is made and their implementation? Encoding is not required but is at the discretion of the server implementation?

http://yourserver.com/path?options=eggs%2Ctoast%2Ccoffee is fully valid and legal and is always usable and will never encounter problems? All valid server side implementations will handle this properly?

Since I am sure that the API that I am writing for is probably not the only server implementation which requires the comma to be encoded.

Actually, this is the first time I hear of a problem with , So, from that perspective, the server might be in error.

And regardless of legality of the use of comma it appears that some implementers are on the "to be safe we encode everything" side of things. It would be nice to have some option which allows us to encode all to be safe option.

Yes, maybe such an option would be good, but only if it is really needed.

Thanks for listening and your help.

Jimmie

On 06/11/2015 01:35 AM, Sven Van Caekenberghe wrote:

...

@everybody

The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet

Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts.

Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets.

Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations.

The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context.

ZnUrl was written against RFC 3986 mostly.

Now, maybe we made a mistake, maybe not.

But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis.

...

On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouchin@gmail.com> wrote:

Thanks for the reply.

I implemented Peter's suggestion as an easy keep moving solution.

As I said, I am not expert in what is or is not legal according to the standards. However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default.

_ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-')

https://docs.python.org/3/library/urllib.parse.html https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py

That's at least how one major language understands the standard. And Python 2.7 is the same.

According to Wikipedia http://en.wikipedia.org/wiki/Query_string • Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[9] • SPACE is encoded as '+' • Letters (A–Z and a–z), numbers (0–9) and the characters '*','-','.' and '_' are left as-is

It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded Just a little more information to add to the discussion.

Thanks.

Jimmie

On 06/10/2015 05:56 PM, Norbert Hartl wrote:

...

Just to clarify:

" Characters in the "reserved" set are not reserved in all contexts.

The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding."

If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> :

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com>

escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: ' google.com '; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

Sven One thing which has not been looked at in this discussion is why Jimmie's first attempt to solve the problem failed. He replaced every comma in the query by its percent encoding, '%2C'. This failed because ZnClient replaced this by '%252C', i.e. it percent encoded the percent sign. In my first attempt to help him, I tried replacing the query line by: queryAt: 'fields' put: 'displayName,instrument,pip' urlEncoded; but this produced the same result. The puzzling thing is why ZnClient is doing this. In http://www.w3.org/Addressing/URL/uri-spec.html it is stated that: " The percent sign ("%", ASCII 25 hex) is used as the escape character in the encoding scheme and is never allowed for anything else." Further on, there are two examples of strings which are illegal as URIs because they contain percent signs which are not followed by two decodable hex characters. Hence, if ZnClient encounters a percent sign in a query string, either it is the beginning of a percent encoded character or it is illegal. In neither case is it right to percent encode the percent sign. Presumably the correct action for ZnClient is to examine the two characters following the percent sign and see if together they can be decoded to a legal character. Of course, if this were strictly enforced, it would still be possible for some perverse user to smuggle a percent sign through as part of a literal string by percent encoding it first - but this would not involve any illegality. On the other hand, anyone in Jimmie's position would be able to pre-encode a comma to %2C and have it accepted, while still allowing commas to be used where the site is happy with them. I may have misunderstood something, but anyway that's my 2p worth. Peter Kenny -----Original Message----- From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Sven Van Caekenberghe Sent: 11 June 2015 15:49 To: Any question about pharo is welcome Subject: Re: [Pharo-users] ZnClient and percent characters Here is one older discussion that let to Zn going away from the better safe than sorry approach to encoding: http://forum.world.st/Zinc-How-to-use-the-character-in-an-URL-td4716386.html... You have to read it up to the end.

On 11 Jun 2015, at 16:19, Jimmie Houchin <jlhouchin@gmail.com> wrote:

This is exactly why I expressly state I am not a language lawyer and explicitly do not know what is and is not expressly forbidden or allowed with regards to a comma.

You are correct about the Wikipedia article.

Is it every wrong or illegal to use a complete safely encoded request? Is it just simply not required? So not fully encoded is still valid and legal and also the fully encode is also fully valid and legal.

Yes, any server should accept all encodings. However, the following are different: http://foo.com/bar?x=1 http://foo.com/bar?x%3D1 In the second case, you take away the meaning of = by encoding it. So you just can't encode everything everywhere.

eg: http://yourserver.com/path?options=eggs,toast,coffee is fully valid and legal, but may encounter problems depending to whom the request is made and their implementation? Encoding is not required but is at the discretion of the server implementation?

http://yourserver.com/path?options=eggs%2Ctoast%2Ccoffee is fully valid and legal and is always usable and will never encounter problems? All valid server side implementations will handle this properly?

Since I am sure that the API that I am writing for is probably not the only server implementation which requires the comma to be encoded.

Actually, this is the first time I hear of a problem with , So, from that perspective, the server might be in error.

And regardless of legality of the use of comma it appears that some implementers are on the "to be safe we encode everything" side of things. It would be nice to have some option which allows us to encode all to be safe option.

Yes, maybe such an option would be good, but only if it is really needed.

Thanks for listening and your help.

Jimmie

On 06/11/2015 01:35 AM, Sven Van Caekenberghe wrote:

...

@everybody

The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet

Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts.

Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets.

Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations.

The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context.

ZnUrl was written against RFC 3986 mostly.

Now, maybe we made a mistake, maybe not.

But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis.

...

On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouchin@gmail.com> wrote:

Thanks for the reply.

I implemented Peter's suggestion as an easy keep moving solution.

As I said, I am not expert in what is or is not legal according to the standards. However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default.

_ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-')

https://docs.python.org/3/library/urllib.parse.html https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py

That's at least how one major language understands the standard. And Python 2.7 is the same.

According to Wikipedia http://en.wikipedia.org/wiki/Query_string • Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[9] • SPACE is encoded as '+' • Letters (A–Z and a–z), numbers (0–9) and the characters '*','-','.' and '_' are left as-is

It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. http://stackoverflow.com/questions/8828702/why-is-the-comma-url-enco ded Just a little more information to add to the discussion.

Thanks.

Jimmie

On 06/10/2015 05:56 PM, Norbert Hartl wrote:

...

Just to clarify:

" Characters in the "reserved" set are not reserved in all contexts.

The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding."

If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-en coded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> :

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com>

escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: ' google.com '; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

On 11 Jun 2015, at 08:35, Sven Van Caekenberghe <sven@stfx.eu> wrote:

@everybody

The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet

Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts.

Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets.

Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations.

The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context.

ZnUrl was written against RFC 3986 mostly.

Now, maybe we made a mistake, maybe not.

I looked into this a bit more, and I am confused. My most strict reading of RFC 3986 (which obsoletes RFC 2396) says in section 3.4 Query: query = *( pchar / "/" / "?" ) where pchar = unreserved / pct-encoded / sub-delims / ":" / "@" unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~" sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" which I understand to allow , The description above is what is in ZnMetaResourceUtils class>>#querySafeSet which the noted exceptions (=, & and + because we interpret the query as key-value pairs). In http://www.w3.org/Addressing/URL/uri-spec.html is read the same. That being said, there are counter examples, like when you search for foo,bar in Google using Google Chrome, which then results in the URL: https://www.google.be/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=... Or when you do $ curl -G -v --data-urlencode "foo=one,two" "http://zn.stfx.eu/echo?q=1,2,3&x=a,b" * Hostname was NOT found in DNS cache * Trying 46.137.113.215... * Connected to zn.stfx.eu (46.137.113.215) port 80 (#0)

GET /echo?q=1,2,3&x=a,b&foo=one%2Ctwo HTTP/1.1 User-Agent: curl/7.37.1 Host: zn.stfx.eu Accept: */*

< HTTP/1.1 200 OK < Date: Thu, 11 Jun 2015 14:23:06 GMT * Server Zinc HTTP Components 1.0 is not blacklisted < Server: Zinc HTTP Components 1.0 < Content-Type: text/plain;charset=utf-8 < Content-Length: 421 < Vary: Accept-Encoding < This is Zinc HTTP Components echoing your request ! Running a ZnManagingMultiThreadedServer(running 8083) GET request for /echo?q=1,2,3&x=a,b&foo=one,two with headers X-Forwarded-Server: ip-10-226-6-28.eu-west-1.compute.internal X-Forwarded-Host: zn.stfx.eu X-Zinc-Remote-Address: 127.0.0.1 User-Agent: curl/7.37.1 Host: localhost:8083 Connection: Keep-Alive Accept: */* X-Forwarded-For: 81.83.7.35 * Connection #0 to host zn.stfx.eu left intact Reading about JavaScripts' encodeURI and encodeURIComponent functions does not help either (the first one keeps the comma, the latter one encodes it). I know there are some other people on this list that might have an opinion, so let's try to figure this out together.

But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis.

...

On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouchin@gmail.com> wrote:

Thanks for the reply.

I implemented Peter's suggestion as an easy keep moving solution.

As I said, I am not expert in what is or is not legal according to the standards. However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default.

_ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' b'abcdefghijklmnopqrstuvwxyz' b'0123456789' b'_.-')

https://docs.python.org/3/library/urllib.parse.html https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py

That's at least how one major language understands the standard. And Python 2.7 is the same.

According to Wikipedia http://en.wikipedia.org/wiki/Query_string • Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[9] • SPACE is encoded as '+' • Letters (A–Z and a–z), numbers (0–9) and the characters '*','-','.' and '_' are left as-is

It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded Just a little more information to add to the discussion.

Thanks.

Jimmie

On 06/10/2015 05:56 PM, Norbert Hartl wrote:

...

Just to clarify:

" Characters in the "reserved" set are not reserved in all contexts.

The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding."

If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert

...

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin@gmail.com>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

...

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

...

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin@gmail.com> :

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

...

...

On 10 Jun 2015, at 17:24, David <stormbyte@gmail.com> wrote:

El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <jlhouchin@gmail.com>

escribió:

...

Hello,

I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet.

znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents)

The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail.

The query needs to be fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly. Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents)

I suspect it is using encoding internally, that is why % is also encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new https; host: ' google.com '; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself

View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie