Am 06.12.2011 um 16:23 schrieb Sven Van Caekenberghe:

Norbert,

On 05 Dec 2011, at 18:07, Norbert Hartl wrote:

...

As Zinc is now part of pharo may I assume that bug reports belong to the pharo tracker?

Ok.

Yes, please use the Pharo-Dev list.

...

I've discovered a problem in Zinc. If you execute this piece of code

| client entity | client := ZnNeoClient new. entity := client applicationFormUrlEncodedEntity. entity at: 'testkey' put: 'ö'. Transcript show: 'entity type: ', entity contentType printString; cr. Transcript show: 'encoded body: ', (String streamContents: [:stream| entity writeRepresentationOn: stream ])

You'll get (in pharo #13315)

entity type: application/x-www-form-urlencoded;charset=iso-8859-1 encoded body: testkey=%C3%B6

As you can see the content type contains latin-1 as encoding while the body is utf-8 encoded. The utf-8 encoding comes from

ZnUtils class>>writeQueryFields:on:

that uses encodeForHTTP which uses utf-8 by default.

Norbert

Hmm,

In my image, there is no charset attribute for that mime type, did you add it yourself ?

ZnApplicationFormUrlEncodedEntity new contentType printString

gives

'application/x-www-form-urlencoded'

I was wondering myself. I added a fixed encoding in my own code to test. Before I never saw the latin-1 appearing aynwhere. But I checked monticello and I did not alter zinc code. I'll check again.

On the other hand you are right: ZnUtils class>>writeQueryFields:on: does indeed always use the UTF8 based #encodeForHTTP of which there is also an encoding based variant that would be more correct I guess.

I assume this is not a theoretical exercise and that you have actual code failing because of this ?

Yes. I need to issue url safe encoded parameters. When it comes to character encoding of safe url encoded things it gets hairy pretty quick. While with GET requests this is heavy to solve at all a charset addition should help when doing posts. So at least for tesing I need to take control over the encoding. And I think there are a lot of latin-1 encoded url safe encoded URLs and strings out there. And the server I need to deal with does not accept the utf-8 encoded stuff.

Please file a bug report. You could also suggest a fix based on the lastet version if you feel like it. Or at least some unit test.

http://code.google.com/p/pharo/issues/detail?id=5063

If not, I will try to fix it, time permitting.

I hope I can spend some more time. But the usage of the encoding is burried deep in the process so I couldn't came up with an easy fix looking the first time.

Thanks for reporting this.

Sven

PS: ZnNeoClient has been renamed ZnClient since more than a month, you probably best use the latest version.

Yes, I know. As I wrote I tested this with #13315 and there is only ZnNeoClient. I updated pharo two times and rolled back. Then I remember you call it "the client has been renamed" but my code those not work with ZnClient. It looks as it has less in common with ZnNeoClient. So I postpone upgrading day after day :) Norbert

Am 06.12.2011 um 16:23 schrieb Sven Van Caekenberghe:

In my image, there is no charset attribute for that mime type, did you add it yourself ?

ZnApplicationFormUrlEncodedEntity new contentType printString

gives

'application/x-www-form-urlencoded'

It is because MimeTypes types are the same instance. If I do | client entity | client := ZnClient new. entity := client applicationFormUrlEncodedEntity. entity contentType charSet: 'iso-8859-1' then in ZnClient>> applicationFormUrlEncodedEntity you use ZnMimeType>>applicationFormUrlEncoded which gets the sole instance from a pre-initialized dictionaries MimeTypes. If I set the charSet once it will alter subsequent entities created this way. Norbert

Sven, I've uploaded the fix. Information can be found in the issue http://code.google.com/p/pharo/issues/detail?id=5063 Norbert Am 06.12.2011 um 18:03 schrieb Norbert Hartl:

Am 06.12.2011 um 16:23 schrieb Sven Van Caekenberghe:

...

Please file a bug report. You could also suggest a fix based on the lastet version if you feel like it. Or at least some unit test.

I have a fix for it. But I can't commit because squeaksource is down.

Basically I changed ZnUtils>>writeQueryFields:on: to ZnUtils>>writeQueryFields:withTextEncoding:on: which uses encodeForHTTPWithTextEncoding: instead of encodeForHTTP. In ZnApplicationFormUrlEncodedEntity it checks for the encoding and uses the appropriate method.

This is not nice but en par with the previous solution. I think the default behaviour (which encoding to use) needs to be configurable as well. The only thing that points towards a default is the w3c recommendation to use utf-8 for url safe encoding. But there is no strong reason to use any of those.