On 23.05.2013 00:06, Nicolas Cellier wrote:
That sounds good. We could even try to fallback to UT-32 if we encounter zeros (but his should be very rare...).

For write, ZipArchive are un-aware of any encoding... They use latin1.
In Squeak, I could place some squeakToUTF8 sends in MCMczWriter, and equivalent UTF8TextConverter in Pharo #serializeDefinitions:, maybe this is needed in some other serialize* (version, dependencies who knows...)



2013/5/22 Norbert Hartl <norbert@hartl.name>


Am 22.05.2013 um 23:16 schrieb Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:

First thing would be to simplify #setConverterForCode and #selectTextConverterForCode.
Do we still want to use a MacRomanTextConverter, seriously? I'm not even sure I've got that many files with that encoding on my Mac-OSX...
Do we really need to put a ByteOrderMark for UTF-8, seriously? See http://en.wikipedia.org/wiki/Byte_order_mark, it's valueless, and not recommended. It were a Squeak way to specify that a Squeak source file would use UTF-8 rather than MacRoman, but now this should be obsolescent.

A BOM for utf-8 does not make sense. It could act as a switch between legacy encoding and utf-8. But it would also be a decision that will be regretted shortly after. Most files in monticello are 7bit so there wouldn't be a problem changing the default encoding. For every other file an exception will be thrown. So reading utf-8 and on exception reading the same thing in legacy might be a way to go.��
A BOM does make sense in this case, explicitly to act as a legacy switch, just like how the current .st works.
Notice any non-ascii files will (should) be written in utf8 after a switch, just as for regular .st.

Do we need to use a MacRomanTextConverter? No, we don't, not for MC.


For regular .st MacRoman makes sense, since that was the encoding using for ByteStrings before WideStrings were invented, and as such, any legacy non-ascii .st sources written would be in that encoding.
For MC .st files, the equivalent fallback legacy encoding for maximum compatability would be Latin1.

End result:
Written by �� ���� ������ ���� Readable in Old ���� Readable in New
Old ascii:������ ������ ���� ���� ������ ������ yes���� ������ ������ ������ ������ ������ �� yes
Old non-ascii, <256 �� ������ yes�� �� ���� ������ ������ ������ ������ �� yes
Old non-ascii, >256������ ���� no �� �������� ������ ������ ������ ������ �� no
New ascii:������ ������ ������ ������ �������� yes���� ���� ������ ������ ������ ������ �� yes
New non-ascii: ���� ������ ������ �� no������ ���� ������ ������ ������ ������ �� yes


Find attached the minor changeset I wrote a long time ago doing this, IIRC, the main problems remaining were how to handle writing source to internal streams, for which you can't specify encoding; this is done in quite a few tests.
Not sure if it's better to just disallow that case, or another approach should be taken.

Cheers,
Henry