Gmail Contact Export and #readStream
Two issues: 1. Exports for Outlook ============ Gmail seems to have inserted "160 asCharacter" in a few places in a gmail contact csv export, so for example this partial phone number: '(914)Â 469-' is written to file as: #[40 57 49 52 41 160 52 54 57 45] And then "aFile readStream contents" signals "ZnInvalidUTF8: Invalid utf8 input detected" when it encounters the 160 2. Google csv Exports ============ The file starts with the BOM #[255 254], which generates the same exception ============= Are these problems with gmail or Pharo? Thanks ----- Cheers, Sean -- View this message in context: http://forum.world.st/Gmail-Contact-Export-and-readStream-tp4759306.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
That is weird indeed, I would say they mixed up Latin1 and UTF8. BTW, if you do #[40 57 49 52 41 160 52 54 57 45] inspect. in Pharo 3, you'll see that it is actually a Latin1 encoded ByteArray, not a UTF8 one. In Latin1 (http://en.wikipedia.org/wiki/Latin1) 160 is NBSP (Non breaking space). In UTF8, this would be encoded differently ZnUTF8Encoder new encodeString: 160 asCharacter asString. #[194 160] But if the file starts with the Unicode BOM, that is really confusing (http://en.wikipedia.org/wiki/Unicode_Byte-Order_Mark#Usage), since in UTF8 it would be different, and from your byte sequence it can't be UTF16 either. On 16 May 2014, at 22:06, Sean P. DeNigris <sean@clipperadams.com> wrote:
Two issues:
1. Exports for Outlook ============ Gmail seems to have inserted "160 asCharacter" in a few places in a gmail contact csv export, so for example this partial phone number: '(914) 469-' is written to file as: #[40 57 49 52 41 160 52 54 57 45] And then "aFile readStream contents" signals "ZnInvalidUTF8: Invalid utf8 input detected" when it encounters the 160
2. Google csv Exports ============ The file starts with the BOM #[255 254], which generates the same exception
=============
Are these problems with gmail or Pharo? Thanks
----- Cheers, Sean -- View this message in context: http://forum.world.st/Gmail-Contact-Export-and-readStream-tp4759306.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
Sven Van Caekenberghe-2 wrote
That is weird indeed, I would say they mixed up Latin1 and UTF8
Yes! Changing to Latin1TextConverter did the trick. I assumed UTF8 because they did not specify and mention UTF8 elsewhere in gmail settings Sven Van Caekenberghe-2 wrote
But if the file starts with the Unicode BOM, that is really confusing... since in UTF8 it would be different, and from your byte sequence it can't be UTF16 either.
It was UTF16. The byte sequence was from a different output format. Changing to UTF16TextConverter successfully parsed the file. I wonder... can all this be handled automatically? When I open either file in OS X, it seems to just "do the right thing". I also wonder why gmail doesn't clearly display the encoding when you're exporting... ----- Cheers, Sean -- View this message in context: http://forum.world.st/Gmail-Contact-Export-and-readStream-tp4759306p4759320.... Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
On 17 May 2014, at 12:18 , Sean P. DeNigris <sean@clipperadams.com> wrote:
I wonder... can all this be handled automatically? When I open either file in OS X, it seems to just "do the right thing».
Sorta, though not 100% reliably. The problem is, if the automatic handling decides wrong, you end up with garbage content, rather than a nice, clean error. (Say, a file gets interpreted as ISO-8859-1 when itâs really ISO-8859-15) You *could* automatically detect BOM-marked files with no errors, but the frequency of such files compared to that of single-byte encoded files, or non-BOMâd UTF is rather low, so the payoff is rather small for the extra effort it takes trying to detect encoding for every stream opened. Which is why the standard choice fell on something that will signal an error if itâs the wrong choice, which means having to make an explicit decision which is the correct one on a case-by-case basis, as you experienced.
I also wonder why gmail doesn't clearly display the encoding when you're exportingâ¦
Yeah, itâd be nice if they were at least as usable as Notepadâs -> Save as⦠dialogue, huh? :) (Though, for some reason this seems hard to do in a usable yet flawless manner, that one allows you to select ASCII as encoding, and still save âæøåâ without an errorâ¦) Cheers, Henry
participants (3)
-
Henrik Johansen -
Sean P. DeNigris -
Sven Van Caekenberghe