Re: [Pharo-project] Unicode/Latin1 handling in Pharo 1.3

May 8, 2012

      Hi Holger,

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

On 08 May 2012, at 10:57, Holger Hans Peter Freyther wrote:
...
Hi,
I am trying to read a file from disk that is in latin1 encoding and then try
to compare the string with a string provided as literal and it fails. My test
case can be seen below. I am using Pharo 1.3 on a Linux machine with a UTF-8
locale. Is there something obvious that I am doing wrong?
| stream text |
stream := (FileStream fileNamed: 'pharo_example_latin1.txt')
      	converter: ISO885915TextConverter new;
      	yourself.
text := stream contents.
text = 'TeilrÃ¼ckzahlung'
<pharo_example_latin1.txt>
This should work:

| stream text |
stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt')
			converter: Latin1TextConverter  new;
			yourself.
text := stream contents.
text = 'TeilrÃ¼ckzahlung'.

or this (latest Zn code):

| stream text |
stream := (FileStream fileNamed: '/Users/sven/Desktop/pharo_example_latin1.txt')
			binary;
			yourself.
text := (ZnCharacterEncoder newForEncoding: 'iso-8859-15')
		decodeBytes: stream contents.
text = 'TeilrÃ¼ckzahlung'.

there seems to be an issue with ISO885915TextConverter, check the umlaut encoding: it adds something called the leadingChar which I don't think is needed (I don't know why it even exists).

HTH,

Sven