[Pharo-project] Unicode/Latin1 handling in Pharo 1.3
Hi, I am trying to read a file from disk that is in latin1 encoding and then try to compare the string with a string provided as literal and it fails. My test case can be seen below. I am using Pharo 1.3 on a Linux machine with a UTF-8 locale. Is there something obvious that I am doing wrong? | stream text | stream := (FileStream fileNamed: 'pharo_example_latin1.txt') converter: ISO885915TextConverter new; yourself. text := stream contents. text = 'Teilrückzahlung'
Hi Holger, -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill On 08 May 2012, at 10:57, Holger Hans Peter Freyther wrote:
Hi,
I am trying to read a file from disk that is in latin1 encoding and then try to compare the string with a string provided as literal and it fails. My test case can be seen below. I am using Pharo 1.3 on a Linux machine with a UTF-8 locale. Is there something obvious that I am doing wrong?
| stream text | stream := (FileStream fileNamed: 'pharo_example_latin1.txt') converter: ISO885915TextConverter new; yourself. text := stream contents. text = 'Teilrückzahlung'
<pharo_example_latin1.txt>
This should work: | stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt') converter: Latin1TextConverter new; yourself. text := stream contents. text = 'Teilrückzahlung'. or this (latest Zn code): | stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt') binary; yourself. text := (ZnCharacterEncoder newForEncoding: 'iso-8859-15') decodeBytes: stream contents. text = 'Teilrückzahlung'. there seems to be an issue with ISO885915TextConverter, check the umlaut encoding: it adds something called the leadingChar which I don't think is needed (I don't know why it even exists). HTH, Sven
Am 08.05.2012 um 11:41 schrieb Sven Van Caekenberghe:
Hi Holger,
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 08 May 2012, at 10:57, Holger Hans Peter Freyther wrote:
Hi,
I am trying to read a file from disk that is in latin1 encoding and then try to compare the string with a string provided as literal and it fails. My test case can be seen below. I am using Pharo 1.3 on a Linux machine with a UTF-8 locale. Is there something obvious that I am doing wrong?
| stream text | stream := (FileStream fileNamed: 'pharo_example_latin1.txt') converter: ISO885915TextConverter new; yourself. text := stream contents. text = 'Teilrückzahlung'
<pharo_example_latin1.txt>
This should work:
| stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt') converter: Latin1TextConverter new; yourself. text := stream contents. text = 'Teilrückzahlung'.
or this (latest Zn code):
| stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt') binary; yourself. text := (ZnCharacterEncoder newForEncoding: 'iso-8859-15') decodeBytes: stream contents. text = 'Teilrückzahlung'.
there seems to be an issue with ISO885915TextConverter, check the umlaut encoding: it adds something called the leadingChar which I don't think is needed (I don't know why it even exists).
Yes, characters are mapped to characters with a leadingChar. Furthermore in Character the leadingChar value alters the output of asInteger, asciiValue, codePoint... because they all use the value of the character that includes leadingChar.But Latin1TextConverter doesn't do it so you get false for (ISO885915TextConverter new byteToUnicode: $ü) = (Latin1TextConverter new byteToUnicode: $ü) Because Character>>#= aCharacter "Primitive. Answer true if the receiver and the argument are the same object (have the same object pointer) and false otherwise. Optional. See Object documentation whatIsAPrimitive." ^ self == aCharacter or:[ aCharacter isCharacter and: [self asciiValue = aCharacter asciiValue]] is using asciiValue containing in one case 252 and in the other 71303420 (252 + leadingChar) So leadingChar can be considered harmful. It surely was useful in the 8bit days (that followed the 7bit days). But I personally cannot see why the encoding of character should persist if a character has been created. The approach with leadingChar is keeping the 128 >= asciiValue >= 255 and add the leadingChar for the encoding (comparable to code pages). Another approach would be to have a homogenous encoding inside the image. So a value of a character is always unicode (The 8bit trouble is solved because 8bit now is latin1). There would be only be unicode characters without encoding. And encoders that produce individual encodings. The "westeners first" approach then leads to a situation that everyone not living in the latin1 zone (or good ol' 7bit US) has to live with wide strings. This should be changed but I can see that's no easy task because the leadingChar is used throughout the system trying to help in collation things. It would be a bigger effort to clean this up. Norbert
leadingChar was here merely for handling Han unification http://en.wikipedia.org/wiki/Han_unification. As I understand it, the language variant is encoded in leadingChar, while the generic ideogram is encoded in the unicode value. We tried to clean a lot already and avoid using leadingChar, except maybe for east-asian languages, so in case of umlaut, I would classify this as a bug. Nicolas 2012/5/8 Norbert Hartl <norbert@hartl.name>:
Am 08.05.2012 um 11:41 schrieb Sven Van Caekenberghe:
Hi Holger,
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 08 May 2012, at 10:57, Holger Hans Peter Freyther wrote:
Hi,
I am trying to read a file from disk that is in latin1 encoding and then try to compare the string with a string provided as literal and it fails. My test case can be seen below. I am using Pharo 1.3 on a Linux machine with a UTF-8 locale. Is there something obvious that I am doing wrong?
| stream text | stream := (FileStream fileNamed: 'pharo_example_latin1.txt')            converter: ISO885915TextConverter new;            yourself. text := stream contents. text = 'Teilrückzahlung'
<pharo_example_latin1.txt>
This should work:
| stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt')            converter: Latin1TextConverter  new;            yourself. text := stream contents. text = 'Teilrückzahlung'.
or this (latest Zn code):
| stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt')            binary;            yourself. text := (ZnCharacterEncoder newForEncoding: 'iso-8859-15')        decodeBytes: stream contents. text = 'Teilrückzahlung'.
there seems to be an issue with ISO885915TextConverter, check the umlaut encoding: it adds something called the leadingChar which I don't think is needed (I don't know why it even exists).
Yes, characters are mapped to characters with a leadingChar. Furthermore in Character the leadingChar value alters the output of asInteger, asciiValue, codePoint... because they all use the value of the character that includes leadingChar.But Latin1TextConverter doesn't do it so you get false for
(ISO885915TextConverter new byteToUnicode: $ü) = (Latin1TextConverter new byteToUnicode: $ü)
Because
Character>>#= aCharacter     "Primitive. Answer true if the receiver and the argument are the same     object (have the same object pointer) and false otherwise. Optional. See     Object documentation whatIsAPrimitive."
    ^ self == aCharacter or:[         aCharacter isCharacter and: [self asciiValue = aCharacter asciiValue]]
is using asciiValue containing in one case 252 and in the other 71303420 (252 + leadingChar)
So leadingChar can be considered harmful. It surely was useful in the 8bit days (that followed the 7bit days). But I personally cannot see why the encoding of character should persist if a character has been created. The approach with leadingChar is keeping the 128 >= asciiValue >= 255 and add the leadingChar for the encoding (comparable to code pages). Another approach would be to have a homogenous encoding inside the image. So a value of a character is always unicode (The 8bit trouble is solved because 8bit now is latin1). There would be only be unicode characters without encoding. And encoders that produce individual encodings. The "westeners first" approach then leads to a situation that everyone not living in the latin1 zone (or good ol' 7bit US) has to live with wide strings.
This should be changed but I can see that's no easy task because the leadingChar is used throughout the system trying to help in collation things. It would be a bigger effort to clean this up.
Norbert
Am 08.05.2012 um 14:00 schrieb Nicolas Cellier:
leadingChar was here merely for handling Han unification http://en.wikipedia.org/wiki/Han_unification. As I understand it, the language variant is encoded in leadingChar, while the generic ideogram is encoded in the unicode value. We tried to clean a lot already and avoid using leadingChar, except maybe for east-asian languages, so in case of umlaut, I would classify this as a bug.
Ok, so there is a real tension to get rid of leadingChar. Would be good then to have Character>>#= using charCode instead of asciiValue? Norbert
2012/5/8 Norbert Hartl <norbert@hartl.name>:
Am 08.05.2012 um 11:41 schrieb Sven Van Caekenberghe:
Hi Holger,
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 08 May 2012, at 10:57, Holger Hans Peter Freyther wrote:
Hi,
I am trying to read a file from disk that is in latin1 encoding and then try to compare the string with a string provided as literal and it fails. My test case can be seen below. I am using Pharo 1.3 on a Linux machine with a UTF-8 locale. Is there something obvious that I am doing wrong?
| stream text | stream := (FileStream fileNamed: 'pharo_example_latin1.txt') converter: ISO885915TextConverter new; yourself. text := stream contents. text = 'Teilrückzahlung'
<pharo_example_latin1.txt>
This should work:
| stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt') converter: Latin1TextConverter new; yourself. text := stream contents. text = 'Teilrückzahlung'.
or this (latest Zn code):
| stream text | stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt') binary; yourself. text := (ZnCharacterEncoder newForEncoding: 'iso-8859-15') decodeBytes: stream contents. text = 'Teilrückzahlung'.
there seems to be an issue with ISO885915TextConverter, check the umlaut encoding: it adds something called the leadingChar which I don't think is needed (I don't know why it even exists).
Yes, characters are mapped to characters with a leadingChar. Furthermore in Character the leadingChar value alters the output of asInteger, asciiValue, codePoint... because they all use the value of the character that includes leadingChar.But Latin1TextConverter doesn't do it so you get false for
(ISO885915TextConverter new byteToUnicode: $ü) = (Latin1TextConverter new byteToUnicode: $ü)
Because
Character>>#= aCharacter "Primitive. Answer true if the receiver and the argument are the same object (have the same object pointer) and false otherwise. Optional. See Object documentation whatIsAPrimitive."
^ self == aCharacter or:[ aCharacter isCharacter and: [self asciiValue = aCharacter asciiValue]]
is using asciiValue containing in one case 252 and in the other 71303420 (252 + leadingChar)
So leadingChar can be considered harmful. It surely was useful in the 8bit days (that followed the 7bit days). But I personally cannot see why the encoding of character should persist if a character has been created. The approach with leadingChar is keeping the 128 >= asciiValue >= 255 and add the leadingChar for the encoding (comparable to code pages). Another approach would be to have a homogenous encoding inside the image. So a value of a character is always unicode (The 8bit trouble is solved because 8bit now is latin1). There would be only be unicode characters without encoding. And encoders that produce individual encodings. The "westeners first" approach then leads to a situation that everyone not living in the latin1 zone (or good ol' 7bit US) has to live with wide strings.
This should be changed but I can see that's no easy task because the leadingChar is used throughout the system trying to help in collation things. It would be a bigger effort to clean this up.
Norbert
On 08.05.2012 14:00, Nicolas Cellier wrote:
leadingChar was here merely for handling Han unification http://en.wikipedia.org/wiki/Han_unification. As I understand it, the language variant is encoded in leadingChar, while the generic ideogram is encoded in the unicode value. From what I can tell, it (based on the indexes) initially also allowed for non-unicode WideStrings (see Character asUnicode, and it's use of EncodedCharSet). That was probably a bad idea in the first place since there is no 16-bit string type in Pharo, I guess adding that would be the original thought behind it...
Later on it was changed to do Han unification, but: - The code for selecting/displaying font based on leadingChar is a mess, never tested, and no compatible font come installed in Pharo by default. - There is no code to, say, automatically add leadingChars based on current environment to character events, so even if it works, input might still display incorrectly. - Using it in Translation support requires storing in a special format which includes the leading characters. - The LanguageEnvironments that currently deal with this also have other roles, which is purely empirical and, in some cases, outdated (I'm looking at you, defaultSystemConverter)
We tried to clean a lot already and avoid using leadingChar, except maybe for east-asian languages, so in case of umlaut, I would classify this as a bug.
Nicolas Yeah, that's clearly something gone entirely wrong when the translation table was initialized. 17 has never been a valid leadingChar. FWIW, it was reinitialized and works as expected in 1.4.
TLDR of non-bug related stuff; If it were up to me, I'd say nuke leadingChar from orbit (which, as you say, is already done in most cases), then find some other way than what is currently done in LanguageEnvironment (if possible) to query the correct encoding of strings you pass to the system. Cheers, Henry
Am 08.05.2012 um 15:37 schrieb Henrik Sperre Johansen:
TLDR of non-bug related stuff; If it were up to me, I'd say nuke leadingChar from orbit (which, as you say, is already done in most cases), then find some other way than what is currently done in LanguageEnvironment (if possible) to query the correct encoding of strings you pass to the system.
+ any number you can imagine It is easy to state that it is broken now and leadingChar causes harm. So nuking it will probably just inverse the problem situation but then it can be cured somehow by fixing the bugs. Handling the characters right in encoding is only one side of the medal. Characters and strings are still need to be comparable. To get this right it would mean to have support ICU [1] in pharo which is a good idea anyway. Well, if it is doable in a pluggable fashion of course. Norbert [1] http://site.icu-project.org/
2012/5/8 Norbert Hartl <norbert@hartl.name>:
Am 08.05.2012 um 15:37 schrieb Henrik Sperre Johansen:
TLDR of non-bug related stuff; If it were up to me, I'd say nuke leadingChar from orbit (which, as you say, is already done in most cases), then find some other way than what is currently done in LanguageEnvironment (if possible) to query the correct encoding of strings you pass to the system.
One thing is sure, both clean-up and fast-up would result from leadingChar removal, and Pharo is the perfect place where this should happen. The main question left is what support shall we provide to East-asian-users... I don't know what works or what is broken already, and what are the required features exactly to support asian usage. What are Pharo plans? Nicolas
+ any number you can imagine
It is easy to state that it is broken now and leadingChar causes harm. So nuking it will probably just inverse the problem situation but then it can be cured somehow by fixing the bugs. Handling the characters right in encoding is only one side of the medal. Characters and strings are still need to be comparable. To get this right it would mean to have support ICU [1] in pharo which is a good idea anyway. Well, if it is doable in a pluggable fashion of course.
Norbert
salut nicolas
One thing is sure, both clean-up and fast-up would result from leadingChar removal, and Pharo is the perfect place where this should happen. The main question left is what support shall we provide to East-asian-users... I don't know what works or what is broken already, and what are the required features exactly to support asian usage.
Me too.
What are Pharo plans?
We do not really know because we are not expert. Now I agree with you that making the system simpler would be better. Then once the system is cleaner we can probably find a solution. To me right now it looks complex and without test cases. Stef
Nicolas
+ any number you can imagine
It is easy to state that it is broken now and leadingChar causes harm. So nuking it will probably just inverse the problem situation but then it can be cured somehow by fixing the bugs. Handling the characters right in encoding is only one side of the medal. Characters and strings are still need to be comparable. To get this right it would mean to have support ICU [1] in pharo which is a good idea anyway. Well, if it is doable in a pluggable fashion of course.
Norbert
participants (6)
-
Henrik Sperre Johansen -
Holger Hans Peter Freyther -
Nicolas Cellier -
Norbert Hartl -
Stéphane Ducasse -
Sven Van Caekenberghe