On 23.05.2013 00:06, Nicolas Cellier wrote:
That sounds good. We could even try to fallback to UT-32 if we encounter zeros (but his should be very rare...).
For write, ZipArchive are un-aware of any encoding... They use latin1. In Squeak, I could place some squeakToUTF8 sends in MCMczWriter, and equivalent UTF8TextConverter in Pharo #serializeDefinitions:, maybe this is needed in some other serialize* (version, dependencies who knows...)
2013/5/22 Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>>
Am 22.05.2013 um 23:16 schrieb Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com <mailto:nicolas.cellier.aka.nice@gmail.com>>:
First thing would be to simplify #setConverterForCode and #selectTextConverterForCode. Do we still want to use a MacRomanTextConverter, seriously? I'm not even sure I've got that many files with that encoding on my Mac-OSX... Do we really need to put a ByteOrderMark for UTF-8, seriously? See http://en.wikipedia.org/wiki/Byte_order_mark, it's valueless, and not recommended. It were a Squeak way to specify that a Squeak source file would use UTF-8 rather than MacRoman, but now this should be obsolescent.
A BOM for utf-8 does not make sense. It could act as a switch between legacy encoding and utf-8. But it would also be a decision that will be regretted shortly after. Most files in monticello are 7bit so there wouldn't be a problem changing the default encoding. For every other file an exception will be thrown. So reading utf-8 and on exception reading the same thing in legacy might be a way to go.
A BOM does make sense in this case, explicitly to act as a legacy switch, just like how the current .st works. Notice any non-ascii files will (should) be written in utf8 after a switch, just as for regular .st. Do we need to use a MacRomanTextConverter? No, we don't, not for MC. For regular .st MacRoman makes sense, since that was the encoding using for ByteStrings before WideStrings were invented, and as such, any legacy non-ascii .st sources written would be in that encoding. For MC .st files, the equivalent fallback legacy encoding for maximum compatability would be Latin1. End result: Written by Readable in Old Readable in New Old ascii: yes yes Old non-ascii, <256 yes yes Old non-ascii, >256 no no New ascii: yes yes New non-ascii: no yes Find attached the minor changeset I wrote a long time ago doing this, IIRC, the main problems remaining were how to handle writing source to internal streams, for which you can't specify encoding; this is done in quite a few tests. Not sure if it's better to just disallow that case, or another approach should be taken. Cheers, Henry