On 19.05.2010 02:17, jaayer wrote:
============ Forwarded message ============ From : jaayer<jaayer@zoho.com> To :<alexandre.bergel@inria.fr> Date : Tue, 18 May 2010 16:30:06 -0700 Subject : Re: Decoding bug with XMLParser ? ============ Forwarded message ============
---- On Tue, 18 May 2010 02:29:18 -0700 Alexandre Bergel<alexandre.bergel@inria.fr> wrote ----
To give a bit of context, the problem is:
-=-=-=-=-=-=-=-=-=-=-=-= exampleEncodedXML ^'<?xml version="1.0" encoding="UTF-8"?> <test-data>…</test-data> '
testDecodingCharacters | xmlDocument element | "XMLTokenizer testDecodingCharacters"
xmlDocument := XMLDOMParser parseDocumentFrom: self exampleEncodedXML readStream. element := xmlDocument firstTagNamed: #'test-data'.
self assert: element contentString first codePoint = 8230 -=-=-=-=-=-=-=-=-=-=-=-=
#testDecodingCharacters goes yellow
Thinking of it, it's not really an encoding problem, rather a bug in the entity->character conversion. I guess there should be a similar test where there is an actual ellipsis character in the xml, instead of the entity. Any idea how your test can goes green?
And now I realize our server will not be able to connect outside its DMZ, so I won't be able to use the fix :D DMZ ?
Cheers, Alexandre
Character references like the one above are handled using #nextCharReference. It does so by reading the number after the "&#" or "&x" prefix and then sending #value: to the class Unicode with that as the argument. If you evaluate the following code in a workspace with cmd-p: "(Unicode value: 8230) codePoint", you will see that the resulting code point is not what you would expect. For me it was "1069555750". The same behavior results when creating a Unicode character with #charFromUnicode:. Unless Unicode>>value: and Unicode>>charFromUnicode: are being used incorrectly, I am not sure that this is a bug, or least a bug in XML-Support.
(I am working on adding full DTD support with validation and refactoring and re-engineering the parser at the moment, which is why minor releases have slowed to a trickle. I will take a closer look at how character encoding is handled in the process.)
codePoint returns the raw value, which includes the leadingChar used to differentiate between different locale interpretations of the same character. In 1.0 this was 255 for WideCharacters, in 1.1 it has been changed to 0. ie, using codePoint in the test is erroneous, for a method which returns what you expect in both 1.0 and 1.1, use asUnicode. Cheers, Henry