June 2015 - Pharo-users - lists.pharo.org

Re: [Pharo-users] ZnClient and percent characters

by Jimmie Houchin

On 06/11/2015 09:48 AM, Sven Van Caekenberghe wrote: > Here is one older discussion that let to Zn going away from the better safe than sorry approach to encoding: > > http://forum.world.st/Zinc-How-to-use-the-character-in-an-URL-td4716386.htm… > > You have to read it up to the end. > >> On 11 Jun 2015, at 16:19, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >> >> This is exactly why I expressly state I am not a language lawyer and explicitly do not know what is and is not expressly forbidden or allowed with regards to a comma. >> >> You are correct about the Wikipedia article. >> >> Is it every wrong or illegal to use a complete safely encoded request? Is it just simply not required? >> So not fully encoded is still valid and legal and also the fully encode is also fully valid and legal. > Yes, any server should accept all encodings. > > However, the following are different: > > http://foo.com/bar?x=1 > > http://foo.com/bar?x%3D1 > > In the second case, you take away the meaning of = by encoding it. So you just can't encode everything everywhere. > >> eg: >> http://yourserver.com/path?options=eggs,toast,coffee >> is fully valid and legal, but may encounter problems depending to whom the request is made and their implementation? >> Encoding is not required but is at the discretion of the server implementation? >> >> http://yourserver.com/path?options=eggs%2Ctoast%2Ccoffee >> is fully valid and legal and is always usable and will never encounter problems? >> All valid server side implementations will handle this properly? >> >> Since I am sure that the API that I am writing for is probably not the only server implementation which requires the comma to be encoded. > Actually, this is the first time I hear of a problem with , > > So, from that perspective, the server might be in error. > >> And regardless of legality of the use of comma it appears that some implementers are on the "to be safe we encode everything" side of things. It would be nice to have some option which allows us to encode all to be safe option. > Yes, maybe such an option would be good, but only if it is really needed. > >> Thanks for listening and your help. >> >> Jimmie I am not a domain expert. This my first attempt to do anything like this, so I have not encountered it either. However, it seems that there is a lot of ambiguity, discussion and variety in implementations as to whether or not to encode the comma. As ZnClient is the client and user of somebody else's provided service. It bears the burden of being able to communicate with the server whether the server is right or wrong. And are we actually able to say the encoding the comma is wrong even if unencoded may be legal. In other words can to take the counter argument, could I unambiguously prove to someone that they must allow for unencoded commas in the query? That would force upon them the requirement of handling both encoded and unencoded commas in queries. The simpler implementation may be simply to require encoded commas as this API does. Regardless. We can't control other peoples implementations. Though we may find ourselves in a situation to write apps which use their services. Is this option really needed. In my opinion, yes. As we are availing ourselves to other peoples services and implementations. And as the requirements of the implementation isn't exactly clear. And as can be demonstrated amply encoding commas in the URI query is the default for Python and other web client languages and tools. So, Python, one of the largest web tool languages encodes by default. Implementers may often understand that as meaning it should be so. Or at least according to one communities understanding of the specs. Or at least their seeking to be practical in face of ambiguity. Regarding encoding the = equal sign. The same spec says in general don't, that it may cause problems. I was simply seeking understanding and a solution. I have received both. Thanks. Jimmie

June 11, 2015

Minimal delay in Pharo...

by Thierry Goubier

Hi all, anybody knows if, when doing something like: (Delay forMilliseconds: n) wait It is possible for the Delay to return immediately if n is small enough or do we have a guaranteed behavior of at least a 'yield' equivalent? I'm looking into active polling for parallel stuff and I noticed that a busy loop with Processor yield may result in more or less locking the vm since it only gives scheduling to processes which have a higher or equal priority, so this is why I'm looking into using Delay wait instead. But I'd like the delay to be as short as possible for performance reason while allowing lower priority processes a chance to run. Thanks, Thierry

June 11, 2015

Re: [Pharo-users] ZnClient and percent characters

by Norbert Hartl

> Am 11.06.2015 um 16:51 schrieb PBKResearch <peter(a)pbkresearch.co.uk>: > > Norbert > > Apology accepted â I had never heard this term before. However, having read the Wikipedia article Henry mentioned, I do not think it applies to what I had done. If Sven accepts the argument that commas should always be encoded in query lines â and I can see a strong argument for that from what I have read â then the change I made is just what he will need to do. I simply anticipated the decision. If it goes the other way, then it was just a hack to solve Jimmieâs problem â but it might be better to say âhackâ rather than use a term which is potentially offensive. > I just wrote because you told Jimmie to alter a method of a third-party library. And that is a no-go. The way to change it and have Sven it commited is doable. As is my proposal that you can change the code in a reliable way without changing a third-party library. That's all I wanted to say. And yes I think it applies to monkey patching. Choosing weaker terms just to avoid problems is not the way to go IMHO. > I also note Jimmieâs argument that, as long as there are some sites which insist on encoded commas, it must be possible to generate such lines if Zinc is to be generally useful. Whether this should be a user-controlled option, as Sven seems to suggest, is open to discussion â but if so I would argue for encoding of commas as the safe default, which can be overridden if necessary. > Sure. Zinc should behave correctly and it should provide functionalities to make it "less correct". And these functionalities shouldn't be too easy to use ;) Norbert > Peter Kenny > > From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Norbert Hartl > Sent: 11 June 2015 11:25 > To: Pharo users users > Subject: Re: [Pharo-users] ZnClient and percent characters > > No offense meant. I assumed you would know the term. Sorry! Henry explained it quite good what I meant. > > Norbert > >> Am 11.06.2015 um 09:51 schrieb PBKResearch <peter(a)pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>: >> >> I donât quite understand Norbertâs comment. Does âmonkeyâ apply to me or to what I have done? Either way, it seems unnecessary and abusive. >> >> Thanks to Svenâs careful use of informative method names, the effect of my change is quite clear. Any comma in the value part of a query line will be encoded. Nothing else. I did my best to trace any side effects, and didnât find any. What is not âreliableâ about that? >> >> Peter Kenny >> >> >> I was thinking of a reliable solution. Of course if it only needs to be tested than monkey patching foreign packages is ok. >> >> Norbert >> >>> Am 11.06.2015 um 01:22 schrieb PBKResearch <peter(a)pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk>>: >>> >>> There is a simpler way than Norbertâs suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your âGoogleâ example generates the query line with the â%2Câ encoding of the commas. >>> >>> Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem â earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving. >>> >>> Hope this helps >>> >>> Peter Kenny >>> >>> From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org>] On Behalf Of Norbert Hartl >>> Sent: 10 June 2015 23:56 >>> To: Pharo users users >>> Subject: Re: [Pharo-users] ZnClient and percent characters >>> >>> Just to clarify: >>> >>> "Characters in the "reserved" set are not reserved in all contexts. >>> The set of characters actually reserved within any given URI >>> component is defined by that component. In general, a character is >>> reserved if the semantics of the URI changes if the character is >>> replaced with its escaped US-ASCII encoding." >>> If I were you I'd subclass ZnUrl and implement >>> #encodeQuery:on: >>> on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: >>> Cannot think of anything better for a quick resolve of your problem. >>> Norbert >>>> Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin(a)gmail.com <mailto:jlhouchin@gmail.com>>: >>>> >>>> I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas. >>>> >>>> As far as commas needing to be escaped, it seems from other sources that they should be. >>>> >>>> From https://www.ietf.org/rfc/rfc2396.txt <https://www.ietf.org/rfc/rfc2396.txt> >>>> >>>> >>>> >>>> The plus "+", dollar "$", and comma "," characters have been added to >>>> those in the "reserved" set, since they are treated as reserved >>>> within the query component. >>>> >>>> States that commas are reserved within the query component. >>>> >>>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded <http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded> >>>> >>>> Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states. >>>> >>>> fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values. >>>> >>>> Which will look like this or something similar. >>>> >>>> fields=displayName%2Cinstrument%2Cpip >>>> >>>> >>>> Thanks. >>>> >>>> Jimmie >>>> >>>> On 06/10/2015 03:27 PM, Norbert Hartl wrote: >>>>> That's because the comma does not need to be escaped in the query part of the uri. >>>>> >>>>> Norbert >>>>> >>>>>> Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin(a)gmail.com> <mailto:jlhouchin@gmail.com>: >>>>>> >>>>>> On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: >>>>>>>> On 10 Jun 2015, at 17:24, David <stormbyte(a)gmail.com> <mailto:stormbyte@gmail.com> wrote: >>>>>>>> >>>>>>>> El Wed, 10 Jun 2015 10:14:37 -0500 >>>>>>>> Jimmie Houchin <jlhouchin(a)gmail.com> <mailto:jlhouchin@gmail.com> >>>>>>>> escribiÃ³: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I am attempting to use ZnClient to request data. The request requires >>>>>>>>> a %2C (comma) delimited string as part of the query. Below is a >>>>>>>>> snippet. >>>>>>>>> >>>>>>>>> znClient >>>>>>>>> addPath: '/v1/instruments'; >>>>>>>>> queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; >>>>>>>>> get ; >>>>>>>>> contents) >>>>>>>>> >>>>>>>>> The string 'displayName%2Cinstrument%2Cpip' >>>>>>>>> is being converted to 'displayName%252Cinstrument%252Cpip' >>>>>>>>> which causes the request to fail. >>>>>>>>> >>>>>>>>> The query needs to be >>>>>>>>> fields=displayName%2Cinstrument%2Cpip >>>>>>>>> >>>>>>>>> I have not found how to do this correctly. >>>>>>>>> Any help greatly appreciated. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Jimmie >>>>>>>>> >>>>>>>>> >>>>>>>> Maybe a silly thing, but since %2C = , ... Did you tried already to >>>>>>>> make itself encode that? Like >>>>>>>> znClient >>>>>>>> addPath: '/v1/instruments'; >>>>>>>> queryAt: 'fields' putAll: 'displayName,instrument,pip'; >>>>>>>> get ; >>>>>>>> contents) >>>>>>>> >>>>>>>> I suspect it is using encoding internally, that is why % is also >>>>>>>> encoded if you try to put it. >>>>>>>> >>>>>>>> I hope that works >>>>>>> Not silly and no need to suspect, but absolutely correct ! >>>>>>> >>>>>>> Sven >>>>>> My apologies for not having full disclosure. >>>>>> >>>>>> Pharo 4, new image, freshly installed Zinc stable version. >>>>>> Xubuntu 15.04 >>>>>> >>>>>> >>>>>> >>>>>> That is what I thought would happen and what I tried first. But it is not being encoded from what I can find. >>>>>> >>>>>> Inspect this in a workspace/playground. >>>>>> >>>>>> ZnClient new >>>>>> https; >>>>>> host: 'google.com <http://google.com/>'; >>>>>> addPath: '/commaTest'; >>>>>> queryAt: 'fields' put: 'displayName,instrument,pip'; >>>>>> yourself >>>>>> >>>>>> View the request / requestLine / uri. The commas are still present in the URI. >>>>>> So I tried encoding myself and get the other error. >>>>>> >>>>>> Of course Google won't understand this and in this snippet won't receive it. >>>>>> >>>>>> And please let me know if I am doing something wrong. >>>>>> >>>>>> Any help greatly appreciated. >>>>>> >>>>>> Jimmie

June 11, 2015

Re: [Pharo-users] Using GTSpotter how do I find an implementor of #accept ?

by Tudor Girba

Hi Paul, Let's take it from the beginning. Spotter is a completely new tool whose goal is to help you find all sorts of objects. See here some more detailed explanations: http://www.humane-assessment.com/blog/introducing-gtspotter/ http://www.humane-assessment.com/blog/boosting-gtspotter-with-preview/ http://www.humane-assessment.com/blog/finding-asclass-usages-in-glamour-usi… http://www.humane-assessment.com/blog/searching-file-system-with-gtspotter/ http://www.humane-assessment.com/blog/scoping-for-specific-search-category-… Although the tool looks simple, it implied a rather extensive effort. As a consequence some features did not make it until the Pharo4 release. One of these features is a proper results sorting mechanism. So, it's not that someone decided that the order should now be different. It's more that proper ordering did not make it in the release, and this impacts the implementors case more often. But, if you want to play with a different strategy, here is an example to start from. One thing that the current Implementors processor does is to show the actual methods. This works well in many cases, but there were people that preferred to search for selectors instead and then get the list of all methods. To achieve this, try the following: Compile this method: GTSpotter>>spotterSelectorsFor: aStep <spotterOrder: 29> aStep listProcessor title: 'Selectors'; filter: GTFilterSubstring item: [ :filter :context | SystemNavigation default allBehaviorsDo: [ :class | class selectorsDo: filter ] ]; actLogic: [ :each | self systemNavigation browseAllImplementorsOf: each ] Then type accept, and click on a result and you will get the implementors browser. This is just an example to help you get started with customizing Spotter, and it is not the experience we want to promote. Instead, we should be able to dive in a selector and see the concrete implementors within Spotter. Cheers, Doru On Thu, Jun 11, 2015 at 7:05 AM, Paul DeBruicker <pdebruic(a)gmail.com> wrote: > Hi Doru, > > By "the old version of Spotter" I meant whatever was in Pharo 3 that would > produce a list of search results with exact matches at the top when I'd hit > shift+enter and then type e.g. 'accept.' Sorry to have said "Pharo 4" & > been unnecessarily confusing. > > > I'm just going to use 'accept' for this example but why show the mixed list > of 637 implementors of accept* and not lead with "accept". Why was it > decided that inexact matches to the typed input be privileged above exact > matches in the new tool? Is it a bad Levenshtein distance algorithm or > something? > > > Thanks for helping me figure it out > > > Paul > > > > > > Tudor Girba-2 wrote > > Hi, > > > > What do you mean by the old version of Spotter? > > > > Just in case: you should know that Spotter is made to be extensible. This > > means that if you want to play with your own way of searching for > objects, > > you can just do it. Let me know if you to try and if you need help in > this > > direction. > > > > Cheers, > > Doru > > > > > > > > On Wed, Jun 10, 2015 at 9:10 PM, Paul DeBruicker < > > > pdebruic@ > > > > wrote: > > > >> > >> Is there any way to change back to the old version of Spotter in Pharo > 4? > >> > >> > >> > >> > >> > >> Nicolai Hess wrote > >> > 2015-06-10 16:24 GMT+02:00 Paul DeBruicker < > >> > >> > pdebruic@ > >> > >> > >: > >> > > >> >> So by default the search tool is only guaranteed to return an exact > >> term > >> >> match if there are only less than 5 non-exact match results? > >> >> > >> >> > >> > Yes > >> > > >> > > >> > > >> > > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> Nicolai Hess wrote > >> >> > 2015-06-10 7:39 GMT+02:00 Paul DeBruicker < > >> >> > >> >> > pdebruic@ > >> >> > >> >> > >: > >> >> > > >> >> >> when I hit shift+enter and type 'accept' I get things that are not > >> >> >> #accept, e.g. #accept: and AbstractAcceptor. > >> >> >> > >> >> >> If I add a space after accept it doesn't help. > >> >> >> > >> >> >> > >> >> >> What do I not understand? > >> >> >> > >> >> > > >> >> > the result list is not sorted and the result list is built by all > >> >> methods > >> >> > having the query string as part > >> >> > of its selector name. > >> >> > > >> >> > Yes this can be improved and it is not difficult, for example you > >> can > >> >> add > >> >> > this method to > >> >> > > >> >> > GTFilterImplementor>>applyFilterWithQuery > >> >> > super applyFilterWithQuery. > >> >> > items sort: [ :a :b | (self itemFilterNameFor: a) size < (self > >> >> > itemFilterNameFor: b) size ] > >> >> > > >> >> > this will sort the result list by the size of the selector name. > So, > >> if > >> >> > there is a perfect match, > >> >> > it will be listed first. > >> >> > (BUT only in the implementors category if you "dive-in", not in the > >> >> > 5-elements-result-preview-list). > >> >> > > >> >> > Maybe there is a better way without sorting. (We can modify > >> >> > applyFilterWithQuery for the implementors > >> >> > filter, to put perfect matches at the begining of the list). > >> >> > > >> >> > But all this is not easy to discover. Spotter classes make some > >> heavy > >> >> use > >> >> > of delegation, many operations > >> >> > are split and delgated to subclasses (GOOD!) > >> >> > many classes aren't documented (BAD!) and this makes it really > >> >> difficult > >> >> > to > >> >> > catch how all this is supposed to work together. > >> >> > > >> >> > > >> >> > nicolai > >> >> > > >> >> > > >> >> > > >> >> > > >> >> >> > >> >> >> > >> >> >> Thanks > >> >> >> > >> >> >> > >> >> >> Paul > >> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> View this message in context: > >> >> > >> > http://forum.world.st/Using-GTSpotter-how-do-I-find-an-implementor-of-accep… > >> >> Sent from the Pharo Smalltalk Users mailing list archive at > >> Nabble.com. > >> >> > >> >> > >> > >> > >> > >> > >> > >> -- > >> View this message in context: > >> > http://forum.world.st/Using-GTSpotter-how-do-I-find-an-implementor-of-accep… > >> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > >> > >> > > > > > > -- > > www.tudorgirba.com > > > > "Every thing has its own flow" > > > > > > -- > View this message in context: > http://forum.world.st/Using-GTSpotter-how-do-I-find-an-implementor-of-accep… > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > -- www.tudorgirba.com "Every thing has its own flow"

June 11, 2015

Re: [Pharo-users] Mac Squeak binary virtual machine for Squeak 3.10.2

by Tudor Girba

Hi, On Wed, Jun 10, 2015 at 8:36 PM, Trygve Reenskaug <trygver(a)ifi.uio.no> wrote: > Stef, > Why be sorry? It's great that you have a stable kernel in Pharo. Where do > I find the definition of the Pharo public API? > That is an interesting request coming from someone that > In which way is the Pharo technology that underlies Moose more complex > than BabyIDE? Porting BabyIDE from Squeak 3.10 to 4.5 was hard because it > extends the Squeak Parser and debugger and that this is unknown territory > to me. There remains, of course, minorproblems and bugs caused by changes > in the various services offered by the Squeak kernel. > It's clear that one port went without a hitch for you. > It is not one port. We are developing Moose on top of Pharo since 7 years. Even though Moose depends quite deeply on the inner workings of Pharo (package model, code model, RB AST, inspector, debugger, canvas, text editor), we could consistently move it between successive versions with limited effort (typically measured in days). > That does not mean that porting BabyIDE to Pharo will be equally simple. > The comparison is not quite fair :). The point of Stef was about moving between versions of Pharo, not between Squeak and Pharo. > And may be the people who did your port do not share my extensive > ignorance of the Pharo innards and the nature of the changes in the > release? I am not a Pharo creator. I was considering to become a Pharo user > but reconsidered when I read what you say below. This was partly because I > believe you underestimate the work needed to port BabyIDE to Pharo and > partly because you do not appear to appreciate the need that programmers > (your customers) have for a stable programming language. > The choice belongs to you, but I do not quite understand your line of reasoning. What Stef said is that even if Pharo changes fast, there is evidence to show that even in larger projects moving between successive versions of Pharo is a handle-able undertaking. Now, about BabyIDE. I think it is certainly an interesting project, and it would be interesting to have it in Pharo. If you need help, you can always ask on the Pharo lists and you typically get the expected answers. Cheers, Doru > > Trygve > > On 09.06.2015 19:59, stepharo wrote: > > I'm sorry to say that Pharo public API does not change that much. > We could port all Moose in one afternoon and Moose is certainly more > complex :). > > Stef > > -- www.tudorgirba.com "Every thing has its own flow"

June 11, 2015

Re: [Pharo-users] Mac Squeak binary virtual machine for Squeak 3.10.2

by Stephan Eggermont

On 10/06/15 20:36, Trygve Reenskaug wrote: > In which way is the Pharo technology that underlies Moose more complex > than BabyIDE? Porting BabyIDE from Squeak 3.10 to 4.5 was hard because > it extends the Squeak Parser and debugger and that this is unknown > territory to me. There remains, of course, minor problems and bugs caused > by changes in the various services offered by the Squeak kernel. Porting BabyIDE to Pharo is hard because: - we now have a different compiler and debugger - BabyIDE uses parts of Morphic that we changed and/or removed. - Nautilus doesn't easily support changing the syntax highlighter As key developers of Moose are also key developers of Pharo, a smooth transformation is something that should be expected or at least hoped for. On the other hand, BabyIDE is much smaller than Moose. > you do not appear to appreciate the need that programmers (your > customers) have for a stable programming language. We try to provide for both the need to have a stable environment and the need to innovate and try new things. There is an inherent conflict there that we try to resolve by providing a continuous integration environment where we can get fast feedback on breaking changes. It works pretty well for the projects that have an active maintainer. Stephan Eggermont

June 11, 2015

Re: [Pharo-users] ZnClient and percent characters

by PBKResearch

Norbert Apology accepted â I had never heard this term before. However, having read the Wikipedia article Henry mentioned, I do not think it applies to what I had done. If Sven accepts the argument that commas should always be encoded in query lines â and I can see a strong argument for that from what I have read â then the change I made is just what he will need to do. I simply anticipated the decision. If it goes the other way, then it was just a hack to solve Jimmieâs problem â but it might be better to say âhackâ rather than use a term which is potentially offensive. I also note Jimmieâs argument that, as long as there are some sites which insist on encoded commas, it must be possible to generate such lines if Zinc is to be generally useful. Whether this should be a user-controlled option, as Sven seems to suggest, is open to discussion â but if so I would argue for encoding of commas as the safe default, which can be overridden if necessary. Peter Kenny From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Norbert Hartl Sent: 11 June 2015 11:25 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters No offense meant. I assumed you would know the term. Sorry! Henry explained it quite good what I meant. Norbert Am 11.06.2015 um 09:51 schrieb PBKResearch <peter(a)pbkresearch.co.uk <mailto:peter@pbkresearch.co.uk> >: I donât quite understand Norbertâs comment. Does âmonkeyâ apply to me or to what I have done? Either way, it seems unnecessary and abusive. Thanks to Svenâs careful use of informative method names, the effect of my change is quite clear. Any comma in the value part of a query line will be encoded. Nothing else. I did my best to trace any side effects, and didnât find any. What is not âreliableâ about that? Peter Kenny I was thinking of a reliable solution. Of course if it only needs to be tested than monkey patching foreign packages is ok. Norbert Am 11.06.2015 um 01:22 schrieb PBKResearch < <mailto:peter@pbkresearch.co.uk> peter(a)pbkresearch.co.uk>: There is a simpler way than Norbertâs suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your âGoogleâ example generates the query line with the â%2Câ encoding of the commas. Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem â earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving. Hope this helps Peter Kenny From: Pharo-users [ <mailto:pharo-users-bounces@lists.pharo.org> mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Norbert Hartl Sent: 10 June 2015 23:56 To: Pharo users users Subject: Re: [Pharo-users] ZnClient and percent characters Just to clarify: "Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding." If I were you I'd subclass ZnUrl and implement #encodeQuery:on: on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: Cannot think of anything better for a quick resolve of your problem. Norbert Am 11.06.2015 um 00:26 schrieb Jimmie Houchin < <mailto:jlhouchin@gmail.com> jlhouchin(a)gmail.com>: I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas. As far as commas needing to be escaped, it seems from other sources that they should be. >From <https://www.ietf.org/rfc/rfc2396.txt> https://www.ietf.org/rfc/rfc2396.txt The plus "+", dollar "$", and comma "," characters have been added to those in the "reserved" set, since they are treated as reserved within the query component. States that commas are reserved within the query component. <http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states. fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values. Which will look like this or something similar. fields=displayName%2Cinstrument%2Cpip Thanks. Jimmie On 06/10/2015 03:27 PM, Norbert Hartl wrote: That's because the comma does not need to be escaped in the query part of the uri. Norbert Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <mailto:jlhouchin@gmail.com> <jlhouchin(a)gmail.com>: On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: On 10 Jun 2015, at 17:24, David <mailto:stormbyte@gmail.com> <stormbyte(a)gmail.com> wrote: El Wed, 10 Jun 2015 10:14:37 -0500 Jimmie Houchin <mailto:jlhouchin@gmail.com> <jlhouchin(a)gmail.com> escribiÃ³: Hello, I am attempting to use ZnClient to request data. The request requires a %2C (comma) delimited string as part of the query. Below is a snippet. znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; get ; contents) The string 'displayName%2Cinstrument%2Cpip' is being converted to 'displayName%252Cinstrument%252Cpip' which causes the request to fail. The query needs to be fields=displayName%2Cinstrument%2Cpip I have not found how to do this correctly. Any help greatly appreciated. Thanks. Jimmie Maybe a silly thing, but since %2C = , ... Did you tried already to make itself encode that? Like znClient addPath: '/v1/instruments'; queryAt: 'fields' putAll: 'displayName,instrument,pip'; get ; contents) I suspect it is using encoding internally, that is why % is also encoded if you try to put it. I hope that works Not silly and no need to suspect, but absolutely correct ! Sven My apologies for not having full disclosure. Pharo 4, new image, freshly installed Zinc stable version. Xubuntu 15.04 That is what I thought would happen and what I tried first. But it is not being encoded from what I can find. Inspect this in a workspace/playground. ZnClient new https; host: ' <http://google.com/> google.com'; addPath: '/commaTest'; queryAt: 'fields' put: 'displayName,instrument,pip'; yourself View the request / requestLine / uri. The commas are still present in the URI. So I tried encoding myself and get the other error. Of course Google won't understand this and in this snippet won't receive it. And please let me know if I am doing something wrong. Any help greatly appreciated. Jimmie

June 11, 2015

Re: [Pharo-users] ZnClient and percent characters

by Sven Van Caekenberghe

Here is one older discussion that let to Zn going away from the better safe than sorry approach to encoding: http://forum.world.st/Zinc-How-to-use-the-character-in-an-URL-td4716386.htm… You have to read it up to the end. > On 11 Jun 2015, at 16:19, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: > > This is exactly why I expressly state I am not a language lawyer and explicitly do not know what is and is not expressly forbidden or allowed with regards to a comma. > > You are correct about the Wikipedia article. > > Is it every wrong or illegal to use a complete safely encoded request? Is it just simply not required? > So not fully encoded is still valid and legal and also the fully encode is also fully valid and legal. Yes, any server should accept all encodings. However, the following are different: http://foo.com/bar?x=1 http://foo.com/bar?x%3D1 In the second case, you take away the meaning of = by encoding it. So you just can't encode everything everywhere. > eg: > http://yourserver.com/path?options=eggs,toast,coffee > is fully valid and legal, but may encounter problems depending to whom the request is made and their implementation? > Encoding is not required but is at the discretion of the server implementation? > > http://yourserver.com/path?options=eggs%2Ctoast%2Ccoffee > is fully valid and legal and is always usable and will never encounter problems? > All valid server side implementations will handle this properly? > > Since I am sure that the API that I am writing for is probably not the only server implementation which requires the comma to be encoded. Actually, this is the first time I hear of a problem with , So, from that perspective, the server might be in error. > And regardless of legality of the use of comma it appears that some implementers are on the "to be safe we encode everything" side of things. It would be nice to have some option which allows us to encode all to be safe option. Yes, maybe such an option would be good, but only if it is really needed. > Thanks for listening and your help. > > Jimmie > > > > > On 06/11/2015 01:35 AM, Sven Van Caekenberghe wrote: >> @everybody >> >> The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet >> >> Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts. >> >> Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets. >> >> Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations. >> >> The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context. >> >> ZnUrl was written against RFC 3986 mostly. >> >> Now, maybe we made a mistake, maybe not. >> >> But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis. >> >>> On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >>> >>> Thanks for the reply. >>> >>> I implemented Peter's suggestion as an easy keep moving solution. >>> >>> As I said, I am not expert in what is or is not legal according to the standards. >>> However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default. >>> >>> _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' >>> b'abcdefghijklmnopqrstuvwxyz' >>> b'0123456789' >>> b'_.-') >>> >>> https://docs.python.org/3/library/urllib.parse.html >>> https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py >>> >>> That's at least how one major language understands the standard. And Python 2.7 is the same. >>> >>> According to Wikipedia >>> http://en.wikipedia.org/wiki/Query_string >>> â¢ Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[9] >>> â¢ SPACE is encoded as '+' >>> â¢ Letters (AâZ and aâz), numbers (0â9) and the characters '*','-','.' and '_' are left as-is >>> >>> It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. >>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded >>> Just a little more information to add to the discussion. >>> >>> Thanks. >>> >>> Jimmie >>> >>> >>> >>> >>> On 06/10/2015 05:56 PM, Norbert Hartl wrote: >>>> Just to clarify: >>>> >>>> " >>>> Characters in the "reserved" set are not reserved in >>>> all contexts. >>>> >>>> The set of characters actually reserved within any given URI >>>> component is defined by that component. In general, a character is >>>> reserved if the semantics of the URI changes if the character is >>>> replaced with its escaped US-ASCII encoding." >>>> >>>> If I were you I'd subclass ZnUrl and implement >>>> #encodeQuery:on: >>>> on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: >>>> Cannot think of anything better for a quick resolve of your problem. >>>> Norbert >>>>> Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin(a)gmail.com>: >>>>> >>>>> I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas. >>>>> >>>>> As far as commas needing to be escaped, it seems from other sources that they should be. >>>>> >>>>> From https://www.ietf.org/rfc/rfc2396.txt >>>>> The plus "+", dollar "$", and comma "," characters have been added to >>>>> those in the "reserved" set, since they are treated as reserved >>>>> within the query component. >>>>> >>>>> States that commas are reserved within the query component. >>>>> >>>>> >>>>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded >>>>> >>>>> >>>>> Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states. >>>>> >>>>> fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values. >>>>> >>>>> Which will look like this or something similar. >>>>> >>>>> fields=displayName%2Cinstrument%2Cpip >>>>> >>>>> >>>>> Thanks. >>>>> >>>>> Jimmie >>>>> >>>>> >>>>> On 06/10/2015 03:27 PM, Norbert Hartl wrote: >>>>>> That's because the comma does not need to be escaped in the query part of the uri. >>>>>> >>>>>> Norbert >>>>>> >>>>>> >>>>>>> Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin(a)gmail.com> >>>>>>> : >>>>>>> >>>>>>> On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: >>>>>>> >>>>>>>>> On 10 Jun 2015, at 17:24, David <stormbyte(a)gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> El Wed, 10 Jun 2015 10:14:37 -0500 >>>>>>>>> Jimmie Houchin >>>>>>>>> <jlhouchin(a)gmail.com> >>>>>>>>> >>>>>>>>> escribiÃ³: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> I am attempting to use ZnClient to request data. The request requires >>>>>>>>>> a %2C (comma) delimited string as part of the query. Below is a >>>>>>>>>> snippet. >>>>>>>>>> >>>>>>>>>> znClient >>>>>>>>>> addPath: '/v1/instruments'; >>>>>>>>>> queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; >>>>>>>>>> get ; >>>>>>>>>> contents) >>>>>>>>>> >>>>>>>>>> The string 'displayName%2Cinstrument%2Cpip' >>>>>>>>>> is being converted to 'displayName%252Cinstrument%252Cpip' >>>>>>>>>> which causes the request to fail. >>>>>>>>>> >>>>>>>>>> The query needs to be >>>>>>>>>> fields=displayName%2Cinstrument%2Cpip >>>>>>>>>> >>>>>>>>>> I have not found how to do this correctly. >>>>>>>>>> Any help greatly appreciated. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> Jimmie >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Maybe a silly thing, but since %2C = , ... Did you tried already to >>>>>>>>> make itself encode that? Like >>>>>>>>> znClient >>>>>>>>> addPath: '/v1/instruments'; >>>>>>>>> queryAt: 'fields' putAll: 'displayName,instrument,pip'; >>>>>>>>> get ; >>>>>>>>> contents) >>>>>>>>> >>>>>>>>> I suspect it is using encoding internally, that is why % is also >>>>>>>>> encoded if you try to put it. >>>>>>>>> >>>>>>>>> I hope that works >>>>>>>>> >>>>>>>> Not silly and no need to suspect, but absolutely correct ! >>>>>>>> >>>>>>>> Sven >>>>>>>> >>>>>>> My apologies for not having full disclosure. >>>>>>> >>>>>>> Pharo 4, new image, freshly installed Zinc stable version. >>>>>>> Xubuntu 15.04 >>>>>>> >>>>>>> >>>>>>> >>>>>>> That is what I thought would happen and what I tried first. But it is not being encoded from what I can find. >>>>>>> >>>>>>> Inspect this in a workspace/playground. >>>>>>> >>>>>>> ZnClient new >>>>>>> https; >>>>>>> host: ' >>>>>>> google.com >>>>>>> '; >>>>>>> addPath: '/commaTest'; >>>>>>> queryAt: 'fields' put: 'displayName,instrument,pip'; >>>>>>> yourself >>>>>>> >>>>>>> View the request / requestLine / uri. The commas are still present in the URI. >>>>>>> So I tried encoding myself and get the other error. >>>>>>> >>>>>>> Of course Google won't understand this and in this snippet won't receive it. >>>>>>> >>>>>>> And please let me know if I am doing something wrong. >>>>>>> >>>>>>> Any help greatly appreciated. >>>>>>> >>>>>>> Jimmie >>>>>>> >>>>>>> >>>>>>> >>>>>>> >> > >

June 11, 2015

Re: [Pharo-users] ZnClient and percent characters

by Sven Van Caekenberghe

> On 11 Jun 2015, at 08:35, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > @everybody > > The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet > > Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts. > > Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets. > > Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations. > > The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context. > > ZnUrl was written against RFC 3986 mostly. > > Now, maybe we made a mistake, maybe not. I looked into this a bit more, and I am confused. My most strict reading of RFC 3986 (which obsoletes RFC 2396) says in section 3.4 Query: query = *( pchar / "/" / "?" ) where pchar = unreserved / pct-encoded / sub-delims / ":" / "@" unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~" sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" which I understand to allow , The description above is what is in ZnMetaResourceUtils class>>#querySafeSet which the noted exceptions (=, & and + because we interpret the query as key-value pairs). In http://www.w3.org/Addressing/URL/uri-spec.html is read the same. That being said, there are counter examples, like when you search for foo,bar in Google using Google Chrome, which then results in the URL: https://www.google.be/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q… Or when you do $ curl -G -v --data-urlencode "foo=one,two" "http://zn.stfx.eu/echo?q=1,2,3&x=a,b" * Hostname was NOT found in DNS cache * Trying 46.137.113.215... * Connected to zn.stfx.eu (46.137.113.215) port 80 (#0) > GET /echo?q=1,2,3&x=a,b&foo=one%2Ctwo HTTP/1.1 > User-Agent: curl/7.37.1 > Host: zn.stfx.eu > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 11 Jun 2015 14:23:06 GMT * Server Zinc HTTP Components 1.0 is not blacklisted < Server: Zinc HTTP Components 1.0 < Content-Type: text/plain;charset=utf-8 < Content-Length: 421 < Vary: Accept-Encoding < This is Zinc HTTP Components echoing your request ! Running a ZnManagingMultiThreadedServer(running 8083) GET request for /echo?q=1,2,3&x=a,b&foo=one,two with headers X-Forwarded-Server: ip-10-226-6-28.eu-west-1.compute.internal X-Forwarded-Host: zn.stfx.eu X-Zinc-Remote-Address: 127.0.0.1 User-Agent: curl/7.37.1 Host: localhost:8083 Connection: Keep-Alive Accept: */* X-Forwarded-For: 81.83.7.35 * Connection #0 to host zn.stfx.eu left intact Reading about JavaScripts' encodeURI and encodeURIComponent functions does not help either (the first one keeps the comma, the latter one encodes it). I know there are some other people on this list that might have an opinion, so let's try to figure this out together. > But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis. > >> On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >> >> Thanks for the reply. >> >> I implemented Peter's suggestion as an easy keep moving solution. >> >> As I said, I am not expert in what is or is not legal according to the standards. >> However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default. >> >> _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' >> b'abcdefghijklmnopqrstuvwxyz' >> b'0123456789' >> b'_.-') >> >> https://docs.python.org/3/library/urllib.parse.html >> https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py >> >> That's at least how one major language understands the standard. And Python 2.7 is the same. >> >> According to Wikipedia >> http://en.wikipedia.org/wiki/Query_string >> â¢ Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[9] >> â¢ SPACE is encoded as '+' >> â¢ Letters (AâZ and aâz), numbers (0â9) and the characters '*','-','.' and '_' are left as-is >> >> It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. >> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded >> Just a little more information to add to the discussion. >> >> Thanks. >> >> Jimmie >> >> >> >> >> On 06/10/2015 05:56 PM, Norbert Hartl wrote: >>> Just to clarify: >>> >>> " >>> Characters in the "reserved" set are not reserved in >>> all contexts. >>> >>> The set of characters actually reserved within any given URI >>> component is defined by that component. In general, a character is >>> reserved if the semantics of the URI changes if the character is >>> replaced with its escaped US-ASCII encoding." >>> >>> If I were you I'd subclass ZnUrl and implement >>> #encodeQuery:on: >>> on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: >>> Cannot think of anything better for a quick resolve of your problem. >>> Norbert >>>> Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin(a)gmail.com>: >>>> >>>> I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas. >>>> >>>> As far as commas needing to be escaped, it seems from other sources that they should be. >>>> >>>> From https://www.ietf.org/rfc/rfc2396.txt >>>> The plus "+", dollar "$", and comma "," characters have been added to >>>> those in the "reserved" set, since they are treated as reserved >>>> within the query component. >>>> >>>> States that commas are reserved within the query component. >>>> >>>> >>>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded >>>> >>>> >>>> Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states. >>>> >>>> fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values. >>>> >>>> Which will look like this or something similar. >>>> >>>> fields=displayName%2Cinstrument%2Cpip >>>> >>>> >>>> Thanks. >>>> >>>> Jimmie >>>> >>>> >>>> On 06/10/2015 03:27 PM, Norbert Hartl wrote: >>>>> That's because the comma does not need to be escaped in the query part of the uri. >>>>> >>>>> Norbert >>>>> >>>>> >>>>>> Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin(a)gmail.com> >>>>>> : >>>>>> >>>>>> On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: >>>>>> >>>>>>>> On 10 Jun 2015, at 17:24, David <stormbyte(a)gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> El Wed, 10 Jun 2015 10:14:37 -0500 >>>>>>>> Jimmie Houchin >>>>>>>> <jlhouchin(a)gmail.com> >>>>>>>> >>>>>>>> escribiÃ³: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I am attempting to use ZnClient to request data. The request requires >>>>>>>>> a %2C (comma) delimited string as part of the query. Below is a >>>>>>>>> snippet. >>>>>>>>> >>>>>>>>> znClient >>>>>>>>> addPath: '/v1/instruments'; >>>>>>>>> queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; >>>>>>>>> get ; >>>>>>>>> contents) >>>>>>>>> >>>>>>>>> The string 'displayName%2Cinstrument%2Cpip' >>>>>>>>> is being converted to 'displayName%252Cinstrument%252Cpip' >>>>>>>>> which causes the request to fail. >>>>>>>>> >>>>>>>>> The query needs to be >>>>>>>>> fields=displayName%2Cinstrument%2Cpip >>>>>>>>> >>>>>>>>> I have not found how to do this correctly. >>>>>>>>> Any help greatly appreciated. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Jimmie >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Maybe a silly thing, but since %2C = , ... Did you tried already to >>>>>>>> make itself encode that? Like >>>>>>>> znClient >>>>>>>> addPath: '/v1/instruments'; >>>>>>>> queryAt: 'fields' putAll: 'displayName,instrument,pip'; >>>>>>>> get ; >>>>>>>> contents) >>>>>>>> >>>>>>>> I suspect it is using encoding internally, that is why % is also >>>>>>>> encoded if you try to put it. >>>>>>>> >>>>>>>> I hope that works >>>>>>>> >>>>>>> Not silly and no need to suspect, but absolutely correct ! >>>>>>> >>>>>>> Sven >>>>>>> >>>>>> My apologies for not having full disclosure. >>>>>> >>>>>> Pharo 4, new image, freshly installed Zinc stable version. >>>>>> Xubuntu 15.04 >>>>>> >>>>>> >>>>>> >>>>>> That is what I thought would happen and what I tried first. But it is not being encoded from what I can find. >>>>>> >>>>>> Inspect this in a workspace/playground. >>>>>> >>>>>> ZnClient new >>>>>> https; >>>>>> host: ' >>>>>> google.com >>>>>> '; >>>>>> addPath: '/commaTest'; >>>>>> queryAt: 'fields' put: 'displayName,instrument,pip'; >>>>>> yourself >>>>>> >>>>>> View the request / requestLine / uri. The commas are still present in the URI. >>>>>> So I tried encoding myself and get the other error. >>>>>> >>>>>> Of course Google won't understand this and in this snippet won't receive it. >>>>>> >>>>>> And please let me know if I am doing something wrong. >>>>>> >>>>>> Any help greatly appreciated. >>>>>> >>>>>> Jimmie >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>> >> >

June 11, 2015

Re: [Pharo-users] ZnClient and percent characters

by Jimmie Houchin

This is exactly why I expressly state I am not a language lawyer and explicitly do not know what is and is not expressly forbidden or allowed with regards to a comma. You are correct about the Wikipedia article. Is it every wrong or illegal to use a complete safely encoded request? Is it just simply not required? So not fully encoded is still valid and legal and also the fully encode is also fully valid and legal. eg: http://yourserver.com/path?options=eggs,toast,coffee is fully valid and legal, but may encounter problems depending to whom the request is made and their implementation? Encoding is not required but is at the discretion of the server implementation? http://yourserver.com/path?options=eggs%2Ctoast%2Ccoffee is fully valid and legal and is always usable and will never encounter problems? All valid server side implementations will handle this properly? Since I am sure that the API that I am writing for is probably not the only server implementation which requires the comma to be encoded. And regardless of legality of the use of comma it appears that some implementers are on the "to be safe we encode everything" side of things. It would be nice to have some option which allows us to encode all to be safe option. Thanks for listening and your help. Jimmie On 06/11/2015 01:35 AM, Sven Van Caekenberghe wrote: > @everybody > > The key method that defines how the query part of a URL is percent encoded is ZnMetaResourceUtils class>>#querySafeSet > > Years ago, Zinc HTTP Components followed the better safe than sorry approach of encoding almost every character except for the ones that are safe in all contexts. > > Later on, we began reading the specs better and decided to follow them more closely, that is why there are now different safe sets. > > Now, we can (and should) all read the different specs, and try to learn from things in the wild as well from other implementations. > > The quote from http://en.wikipedia.org/wiki/Query_string was incomplete, it said 'for HTML 5 when submitting a form using GET', which is a very specific context. > > ZnUrl was written against RFC 3986 mostly. > > Now, maybe we made a mistake, maybe not. > > But maybe it also would be a good idea to allow users to decide this for themselves on a case by case basis. > >> On 11 Jun 2015, at 05:18, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >> >> Thanks for the reply. >> >> I implemented Peter's suggestion as an easy keep moving solution. >> >> As I said, I am not expert in what is or is not legal according to the standards. >> However, looking at Python, their urllib library in the quote and urlencode methods they encode the commas by default. >> >> _ALWAYS_SAFE = frozenset(b'ABCDEFGHIJKLMNOPQRSTUVWXYZ' >> b'abcdefghijklmnopqrstuvwxyz' >> b'0123456789' >> b'_.-') >> >> https://docs.python.org/3/library/urllib.parse.html >> https://hg.python.org/cpython/file/3.4/Lib/urllib/parse.py >> >> That's at least how one major language understands the standard. And Python 2.7 is the same. >> >> According to Wikipedia >> http://en.wikipedia.org/wiki/Query_string >> â¢ Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[9] >> â¢ SPACE is encoded as '+' >> â¢ Letters (AâZ and aâz), numbers (0â9) and the characters '*','-','.' and '_' are left as-is >> >> It appeared in the stackoverflow article I quoted previously that ASP.NET encodes commas. I could misunderstand or be reading into it. >> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded >> Just a little more information to add to the discussion. >> >> Thanks. >> >> Jimmie >> >> >> >> >> On 06/10/2015 05:56 PM, Norbert Hartl wrote: >>> Just to clarify: >>> >>> " >>> Characters in the "reserved" set are not reserved in >>> all contexts. >>> >>> The set of characters actually reserved within any given URI >>> component is defined by that component. In general, a character is >>> reserved if the semantics of the URI changes if the character is >>> replaced with its escaped US-ASCII encoding." >>> >>> If I were you I'd subclass ZnUrl and implement >>> #encodeQuery:on: >>> on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url: >>> Cannot think of anything better for a quick resolve of your problem. >>> Norbert >>>> Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <jlhouchin(a)gmail.com>: >>>> >>>> I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas. >>>> >>>> As far as commas needing to be escaped, it seems from other sources that they should be. >>>> >>>> From https://www.ietf.org/rfc/rfc2396.txt >>>> The plus "+", dollar "$", and comma "," characters have been added to >>>> those in the "reserved" set, since they are treated as reserved >>>> within the query component. >>>> >>>> States that commas are reserved within the query component. >>>> >>>> >>>> http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded >>>> >>>> >>>> Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states. >>>> >>>> fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values. >>>> >>>> Which will look like this or something similar. >>>> >>>> fields=displayName%2Cinstrument%2Cpip >>>> >>>> >>>> Thanks. >>>> >>>> Jimmie >>>> >>>> >>>> On 06/10/2015 03:27 PM, Norbert Hartl wrote: >>>>> That's because the comma does not need to be escaped in the query part of the uri. >>>>> >>>>> Norbert >>>>> >>>>> >>>>>> Am 10.06.2015 um 22:00 schrieb Jimmie Houchin <jlhouchin(a)gmail.com> >>>>>> : >>>>>> >>>>>> On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote: >>>>>> >>>>>>>> On 10 Jun 2015, at 17:24, David <stormbyte(a)gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> El Wed, 10 Jun 2015 10:14:37 -0500 >>>>>>>> Jimmie Houchin >>>>>>>> <jlhouchin(a)gmail.com> >>>>>>>> >>>>>>>> escribiÃ³: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I am attempting to use ZnClient to request data. The request requires >>>>>>>>> a %2C (comma) delimited string as part of the query. Below is a >>>>>>>>> snippet. >>>>>>>>> >>>>>>>>> znClient >>>>>>>>> addPath: '/v1/instruments'; >>>>>>>>> queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip'; >>>>>>>>> get ; >>>>>>>>> contents) >>>>>>>>> >>>>>>>>> The string 'displayName%2Cinstrument%2Cpip' >>>>>>>>> is being converted to 'displayName%252Cinstrument%252Cpip' >>>>>>>>> which causes the request to fail. >>>>>>>>> >>>>>>>>> The query needs to be >>>>>>>>> fields=displayName%2Cinstrument%2Cpip >>>>>>>>> >>>>>>>>> I have not found how to do this correctly. >>>>>>>>> Any help greatly appreciated. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Jimmie >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Maybe a silly thing, but since %2C = , ... Did you tried already to >>>>>>>> make itself encode that? Like >>>>>>>> znClient >>>>>>>> addPath: '/v1/instruments'; >>>>>>>> queryAt: 'fields' putAll: 'displayName,instrument,pip'; >>>>>>>> get ; >>>>>>>> contents) >>>>>>>> >>>>>>>> I suspect it is using encoding internally, that is why % is also >>>>>>>> encoded if you try to put it. >>>>>>>> >>>>>>>> I hope that works >>>>>>>> >>>>>>> Not silly and no need to suspect, but absolutely correct ! >>>>>>> >>>>>>> Sven >>>>>>> >>>>>> My apologies for not having full disclosure. >>>>>> >>>>>> Pharo 4, new image, freshly installed Zinc stable version. >>>>>> Xubuntu 15.04 >>>>>> >>>>>> >>>>>> >>>>>> That is what I thought would happen and what I tried first. But it is not being encoded from what I can find. >>>>>> >>>>>> Inspect this in a workspace/playground. >>>>>> >>>>>> ZnClient new >>>>>> https; >>>>>> host: ' >>>>>> google.com >>>>>> '; >>>>>> addPath: '/commaTest'; >>>>>> queryAt: 'fields' put: 'displayName,instrument,pip'; >>>>>> yourself >>>>>> >>>>>> View the request / requestLine / uri. The commas are still present in the URI. >>>>>> So I tried encoding myself and get the other error. >>>>>> >>>>>> Of course Google won't understand this and in this snippet won't receive it. >>>>>> >>>>>> And please let me know if I am doing something wrong. >>>>>> >>>>>> Any help greatly appreciated. >>>>>> >>>>>> Jimmie >>>>>> >>>>>> >>>>>> >>>>>> >

June 11, 2015