I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support. This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry. I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data. Call to action: ========== If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them. If you would like to help out, please let me know. If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know. If you have comments or ideas on anything mentioned in this email In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html I have created a SqueakSource.com project called UnicodeSupport I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm. Current state of play =============== ByteString gets converted to WideString when need is automagically detected. Is there anything else that currently exists? Definition of Terms ============== A quick definition of terms before I go any further: Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic. Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints. (I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints). a composed character : a character made up of several codepoints Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form. The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not. You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination. Using compatibility characters can prevent proper sorting and equivalence testing. Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems. Currently my thinking is: a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator. Utf8Strings sometimes need to be sortable, and sometimes need to be compatible. So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one. And that there is a Utf8SortableString which uses *only* normal characters. We then need methods to convert between the two. aUtf8String asUtf8SortableString and aUtf8SortableString asUtf8String Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France) Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc along the lines of aListOfWords := SortedCollection sortBlock: deOrder If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially. To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint. aUtf8String cleanUtf8String With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.) (I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know) Cheers, Euan
Hi Euan I think itâs great that youâre trying this. I hope you know what youâre getting yourself into :) Iâm no Unicode expert but I want to add two points to your list (although youâve probably already thought of them): - Normalisation and conversion (http://unicode.org/faq/normalization.html). Unicode / ICU provide libraries (libuconv / libiconv) that handle this stuff. Specifically normalisation conversions arenât trivial and I think it wouldnât make much sense to reimplement those algorithms. I do think however, that having them available is important (where I work weâre currently writing a VM plugin for access to libiconv through primitives so that we can clean out combining characters through normalisation. And weâll obviously get nice sorting properties and speeds for free) - Sorting and comparison. Basically the same point as above. libuconv / libiconv provide algorithms for this. Do we need our own implementation? Cheers, Max
On 04 Dec 2015, at 12:42, EuanM <euanmee@gmail.com> wrote:
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 04 Dec 2015, at 17:00, Max Leske <maxleske@gmail.com> wrote:
Hi Euan
I think itâs great that youâre trying this. I hope you know what youâre getting yourself into :)
Iâm no Unicode expert but I want to add two points to your list (although youâve probably already thought of them): - Normalisation and conversion (http://unicode.org/faq/normalization.html). Unicode / ICU provide libraries (libuconv / libiconv) that handle this stuff. Specifically normalisation conversions arenât trivial and I think it wouldnât make much sense to reimplement those algorithms. I do think however, that having them available is important (where I work weâre currently writing a VM plugin for access to libiconv through primitives so that we can clean out combining characters through normalisation. And weâll obviously get nice sorting properties and speeds for free) - Sorting and comparison. Basically the same point as above. libuconv / libiconv provide algorithms for this. Do we need our own implementation?
These 2 are indeed missing and it would be good to add them. We already have UTF8/UTF16 encoding/decoding, even 2 implementations. See http://files.pharo.org/books/enterprisepharo/book/Zinc-Encoding-Meta/Zinc-En... for the modern version. But IMHO it would not be a good idea to try to implement functionality on in image strings with those representations, it would be too slow. But of course, if you want to try to implement something and show us, go for it.
Cheers, Max
On 04 Dec 2015, at 12:42, EuanM <euanmee@gmail.com> wrote:
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
Hi EuanM Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :) Stef What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
Sent from the road
On Dec 5, 2015, at 05:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time. Read http://utf8everywhere.org/ I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site. Sent from the road
On Dec 5, 2015, at 05:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
Hi todd thanks for the link. It looks really interesting. Stef Le 5/12/15 17:35, Todd Blanchard a écrit :
would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time.
Read http://utf8everywhere.org/
I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site.
Sent from the road
On Dec 5, 2015, at 05:08, stepharo <stepharo@free.fr <mailto:stepharo@free.fr>> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com <http://squeaksource.com> project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 05 Dec 2015, at 17:35, Todd Blanchard <tblanchard@mac.com> wrote:
would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time.
Read http://utf8everywhere.org/
I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site.
Well, I read the page/document/site as well. It was very interesting indeed, thanks for sharing it. In some sense it made me reconsider my aversion against in-image utf-8 encoding, maybe it could have some value. Absolute storage is more efficient, some processing might also be more efficient, i/o conversions to/from utf-8 become a no-op. What I found nice is the suggestion that most structured parsing (XML, JSON, CSV, STON, ...) could actually ignore the encoding for a large part and just assume its ASCII, which would/could be nice for performance. Also the fact that a lot of strings are (or should be) treated as opaque makes a lot of sense. What I did not like is that much of argumentation is based on issue in the Windows world, take all that away and the document shrinks in half. I would have liked a bit more fundamental CS arguments. Canonicalisation and sorting issues are hardly discussed. In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $é if you know that it is possible to represent $é as just $é and as $e + $´ ? Sven
Sent from the road
On Dec 5, 2015, at 05:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 06 Dec 2015, at 18:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 05 Dec 2015, at 17:35, Todd Blanchard <tblanchard@mac.com> wrote:
would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time.
Read http://utf8everywhere.org/
I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site.
Well, I read the page/document/site as well. It was very interesting indeed, thanks for sharing it.
In some sense it made me reconsider my aversion against in-image utf-8 encoding, maybe it could have some value. Absolute storage is more efficient, some processing might also be more efficient, i/o conversions to/from utf-8 become a no-op. What I found nice is the suggestion that most structured parsing (XML, JSON, CSV, STON, ...) could actually ignore the encoding for a large part and just assume its ASCII, which would/could be nice for performance. Also the fact that a lot of strings are (or should be) treated as opaque makes a lot of sense.
What I did not like is that much of argumentation is based on issue in the Windows world, take all that away and the document shrinks in half. I would have liked a bit more fundamental CS arguments.
Canonicalisation and sorting issues are hardly discussed.
In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $é if you know that it is possible to represent $é as just $é and as $e + $´ ?
Thatâs what normalization is for: http://unicode.org/faq/normalization.html. It will generate the same codepoint for two strings where one contains the combining character and the other is a âsingle characterâ.
Sven
Sent from the road
On Dec 5, 2015, at 05:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
"Canonicalisation and sorting issues are hardly discussed. In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $é if you know that it is possible to represent $é as just $é and as $e + $´ ?" This, for me, is one of the chief purposes of Unicode support. What you have it a convertor for "might contain compatibility codepoints" to "contains only composed sequences of codepoints, and no compatibility codepoints". As long as you're not using Strings where you should use Streams, it should be okay. And of course, for passing back to ISO Latin 1 or ASCII systems, you need to have a convertor to "contains only compatibility codepoints, and no composed sets of codepoints". As long as you can tell one type from the other, it's not a problem. Any string that mixes both can be converted in either direction by the same methods which I've just outlined. Once you have these, we can do this for all 1 byte characters. We can then expand this to have Classes and methods for character strings which contain the occasional character from other ISO character sets. Cheers, Euan On 6 December 2015 at 17:44, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 05 Dec 2015, at 17:35, Todd Blanchard <tblanchard@mac.com> wrote:
would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time.
Read http://utf8everywhere.org/
I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site.
Well, I read the page/document/site as well. It was very interesting indeed, thanks for sharing it.
In some sense it made me reconsider my aversion against in-image utf-8 encoding, maybe it could have some value. Absolute storage is more efficient, some processing might also be more efficient, i/o conversions to/from utf-8 become a no-op. What I found nice is the suggestion that most structured parsing (XML, JSON, CSV, STON, ...) could actually ignore the encoding for a large part and just assume its ASCII, which would/could be nice for performance. Also the fact that a lot of strings are (or should be) treated as opaque makes a lot of sense.
What I did not like is that much of argumentation is based on issue in the Windows world, take all that away and the document shrinks in half. I would have liked a bit more fundamental CS arguments.
Canonicalisation and sorting issues are hardly discussed.
In one place, the fact that a lot of special characters can have multiple representations is a big argument, while it is not mentioned how just treating things like a byte sequence would solve this (it doesn't AFAIU). Like how do you search for $e or $é if you know that it is possible to represent $é as just $é and as $e + $´ ?
Sven
Sent from the road
On Dec 5, 2015, at 05:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
Todd, As long as others are using it, it's useful to be able to send UTF16, and to successfully import it. I like systems that play well with others. :-) On 5 December 2015 at 16:35, Todd Blanchard <tblanchard@mac.com> wrote:
would suggest that the only worthwhile encoding is UTF8 - the rest are distractions except for being able to read and convert from other encodings to UTF8. UTF16 is a complete waste of time.
Read http://utf8everywhere.org/
I have extensive Unicode chops from around 1999 to 2004 and my experience leads me to strongly agree with the views on that site.
Sent from the road
On Dec 5, 2015, at 05:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our
Unicode support is. I am doing this to establish what still needs to
be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it
once, and then have the same codebase incorporated into the Smalltalks
that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for
strings which have ligatures and diacritic characters; and correct
round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action:
==========
If you have comments on these proposals - such as "but we already have
that facility" or "the reason we do not have these facilities is
because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or
would be willing to be, in the 'council of experts' for this project,
please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be:
http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist.
Except where they prevent us reaching the goals of:
- sortable UTF8 strings
- sortable UTF16 strings
- equivalence testing of 2 UTF8 strings
- equivalence testing of 2 UTF16 strings
- round-tripping UTF8 strings through Smalltalk
- roundtripping UTF16 strings through Smalltalk.
As I understand it, we have limited Unicode support atm.
Current state of play
===============
ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms
==============
A quick definition of terms before I go any further:
Standard terms from the Unicode standard
===============================
a compatibility character : an additional encoding of a *normal*
character, for compatibility and round-trip conversion purposes. For
instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms
============
a convenience codepoint : a single codepoint which represents an item
that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility
codepoint interchangably. The standard only refers to them as
compatibility characters. However, the standard is determined to
emphasise that characters are abstract and that codepoints are
concrete. So I think it is often more useful and productive to think
of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained
=====================
A convenience codepoint can therefore be thought of as a code point
used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in
one byte, sometimes not. Sometimes you can encode it in two bytes,
sometimes not.
You can therefore have a long stream of ASCII which is single-byte
Unicode. If there is an occasional Cyrillic or Greek character in the
stream, it would be represented either by a compatibility character or
by a multi-byte combination.
Using compatibility characters can prevent proper sorting and
equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility
and round-tripping probelms. Although avoiding them can *also* cause
compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class
an Ordered collection, with 1 byte characters as the modal element,
but short arrays of wider strings where necessary
a Utf16String class
an Ordered collection, with 2 byte characters as the modal element,
but short arrays of wider strings
beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints,
for round-tripping. And where there are multiple convenience
codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany
have different sort orders for the same diacritic-ed characters. Some
countries have one order in general usage, and another for specific
usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and
conversion methods
A list of sorted words would be a SortedCollection, and there could be
pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder,
seOrder, ukOrder, etc
along the lines of
aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String,
then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way
of cleaning up any convenience codepoints which were valid, but which
were for a character which has multiple equally-valid alternative
convenience codepoints, and for which the string currently had the
"wrong" convenience codepoint. (i.e for any character with valid
alternative convenience codepoints, we would choose one to be in the
well-formed Utf8String, and we would need a method for cleaning the
alternative convenience codepoints out of the string, and replacing
them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except
round-tripping(although I'm sure I have seen something recently about
4-byte strings that also have an additional bit. Which would make
some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious,
subtle, or somewhere in between, please let me know)
Cheers,
Euan
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and... My current thinking, and understanding. ============================== 0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes. 1) Smalltalk has long had multiple String classes. 2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex. 3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1 4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII 5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both. b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1 6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8. 7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8. 8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8 9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character. 10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints". 11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters. 12) Some concrete examples: A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex £ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla) 13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display. 14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently. 15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently. 16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order. 17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations. Some practical upshots ================== 1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid. 2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1. 3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1. 4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML). 5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints. Class + protocol proposals a Utf8CompatibilityString class. asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex. asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1 asAsciiString - ensures only characters 00hex - 7F hex are used. asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings. Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases. Custom sortBlocks will define the applicable sort order. We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks. This will work for all UTF8 strings of ISO-8859-1 and ASCII strings. If anyone has better names for the classes, please let me know. If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know. n.b. I have had absolutely no experience of Ropes. My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing. Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in. Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990. So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10). The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure) Cheers, Euan On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
I am sorry but one of your basic assumptions is completely wrong: 'Les élèves Français' encodeWith: #iso99591. => #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115] 'Les élèves Français' utf8Encoded. => #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115] ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !! Or shorter, $é is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8. So more than half the points you make, or the facts that you state, are thus plain wrong. The only thing that is correct is that the code points are equal, but that is not the same as the encoding !
From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String.
Sorry. PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world.
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
Verifying assumptions is the key reason why you should documents like this out for review. Sven - Cuis is encoded with ISO 8859-15 (aka ISO Latin 9) Sven, this is *NOT* as you state, ISO 99591, (and not as I stated, 8859-1). We caught the right specification bug for the wrong reason. Juan: "Cuis: Chose not to use Squeak approach. Chose to make the base image include and use only 1-byte strings. Chose to use ISO-8859-15" I have double-checked - each character encoded in ISO Latin 15 (ISO 8859-15) is exactly the character represented by the corresponding 1-byte codepoint in Unicode 0000 to 00FF, with the following exceptions: codepoint 20ac - Euro symbol character code a4 (replaces codepoint 00a4 generic currency symbol) codepoint 0160 Latin Upper Case S with Caron character code a6 (replaces codepoint 00A6 was | Unix pipe character) codepoint 0161 Latin Lower Case s with Caron character code a8 (replaces codepoint 00A8 was dierisis) codepoint 017d Latin Upper Case Z with Caron character code b4 (replaces codepoint 00b4 was Acute accent) codepoint 017e Latin Lower Case Z with Caron character code b8 (replaces codepoint 00b8 was cedilla) codepoint 0152 Upper Case OE ligature = Ethel character code bc (replaces codepoint 00bc was 1/4 symbol) codepoint 0153 Lower Case oe ligature = ethel character code bd (replaces codepoint 00bd was 1/2 symbol) codepoint 0178 Upper Case Y diaeresis character code be (replaces codepoint 00be was 3/4 symbol) Juan - I don't suppose we could persuade you to change to ISO Latin-1 from ISO Latin-9 ? It means we could run the same 1 byte string encoding across Cuis, Squeak, Pharo, and, as far as I can make out so far, Dolphin Smalltalk and Gnu Smalltalk. The downside would be that French Y diaeresis would lose the ability to use that character, along with users of oe, OE, and s, S, z, Z with caron. Along with the Euro. https://en.wikipedia.org/wiki/ISO/IEC_8859-15. I'm confident I understand the use of UTF-8 in principal. On 7 December 2015 at 08:27, Sven Van Caekenberghe <sven@stfx.eu> wrote:
I am sorry but one of your basic assumptions is completely wrong:
'Les élèves Français' encodeWith: #iso99591.
=> #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115]
'Les élèves Français' utf8Encoded.
=> #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115]
ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !!
Or shorter, $é is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8.
So more than half the points you make, or the facts that you state, are thus plain wrong.
The only thing that is correct is that the code points are equal, but that is not the same as the encoding !
From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String.
Sorry.
PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world.
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
And indeed, in principle. On 7 December 2015 at 10:51, EuanM <euanmee@gmail.com> wrote:
Verifying assumptions is the key reason why you should documents like this out for review.
Sven -
Cuis is encoded with ISO 8859-15 (aka ISO Latin 9)
Sven, this is *NOT* as you state, ISO 99591, (and not as I stated, 8859-1).
We caught the right specification bug for the wrong reason.
Juan: "Cuis: Chose not to use Squeak approach. Chose to make the base image include and use only 1-byte strings. Chose to use ISO-8859-15"
I have double-checked - each character encoded in ISO Latin 15 (ISO 8859-15) is exactly the character represented by the corresponding 1-byte codepoint in Unicode 0000 to 00FF,
with the following exceptions:
codepoint 20ac - Euro symbol character code a4 (replaces codepoint 00a4 generic currency symbol)
codepoint 0160 Latin Upper Case S with Caron character code a6 (replaces codepoint 00A6 was | Unix pipe character)
codepoint 0161 Latin Lower Case s with Caron character code a8 (replaces codepoint 00A8 was dierisis)
codepoint 017d Latin Upper Case Z with Caron character code b4 (replaces codepoint 00b4 was Acute accent)
codepoint 017e Latin Lower Case Z with Caron character code b8 (replaces codepoint 00b8 was cedilla)
codepoint 0152 Upper Case OE ligature = Ethel character code bc (replaces codepoint 00bc was 1/4 symbol)
codepoint 0153 Lower Case oe ligature = ethel character code bd (replaces codepoint 00bd was 1/2 symbol)
codepoint 0178 Upper Case Y diaeresis character code be (replaces codepoint 00be was 3/4 symbol)
Juan - I don't suppose we could persuade you to change to ISO Latin-1 from ISO Latin-9 ?
It means we could run the same 1 byte string encoding across Cuis, Squeak, Pharo, and, as far as I can make out so far, Dolphin Smalltalk and Gnu Smalltalk.
The downside would be that French Y diaeresis would lose the ability to use that character, along with users of oe, OE, and s, S, z, Z with caron. Along with the Euro.
https://en.wikipedia.org/wiki/ISO/IEC_8859-15.
I'm confident I understand the use of UTF-8 in principal.
On 7 December 2015 at 08:27, Sven Van Caekenberghe <sven@stfx.eu> wrote:
I am sorry but one of your basic assumptions is completely wrong:
'Les élèves Français' encodeWith: #iso99591.
=> #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115]
'Les élèves Français' utf8Encoded.
=> #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115]
ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !!
Or shorter, $é is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8.
So more than half the points you make, or the facts that you state, are thus plain wrong.
The only thing that is correct is that the code points are equal, but that is not the same as the encoding !
From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String.
Sorry.
PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world.
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 07 Dec 2015, at 11:51 , EuanM <euanmee@gmail.com> wrote:
And indeed, in principle.
On 7 December 2015 at 10:51, EuanM <euanmee@gmail.com> wrote:
Verifying assumptions is the key reason why you should documents like this out for review.
Sven -
I'm confident I understand the use of UTF-8 in principal.
I can only second Sven's sentiment that you need to better differentiate code points (an abstract numerical representation of a character, where a set of such mappings define a charset, such as Unicode), and character encoding forms. (which are how code points are represented in bytes by a defined process such as UTF-8, UTF-16 etc). I know you'll probably think I'm arguing semantics again, but these are *important* semantics ;) Cheers, Henry
Hi Henry, To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-) codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character e.g. The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041) The representation of a component of the composed (i.e. preferred) version of the abstract character â« #(0041 030a) â« (#00C5) represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a). â« (#212b) also represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a). With any luck, this satisfies both our semantic understandings of the concept of "codepoint" Would you agree with that? In Unicode, codepoints are *NOT* an abstract numerical representation of a text character. At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings. codepoints represent "*encoded characters*" and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard) Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class) i.e. in other words *Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases. The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*. The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint. In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint. Cheers, Euan On 7 December 2015 at 11:11, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On 07 Dec 2015, at 11:51 , EuanM <euanmee@gmail.com> wrote:
And indeed, in principle.
On 7 December 2015 at 10:51, EuanM <euanmee@gmail.com> wrote:
Verifying assumptions is the key reason why you should documents like this out for review.
Sven -
I'm confident I understand the use of UTF-8 in principal.
I can only second Sven's sentiment that you need to better differentiate code points (an abstract numerical representation of a character, where a set of such mappings define a charset, such as Unicode), and character encoding forms. (which are how code points are represented in bytes by a defined process such as UTF-8, UTF-16 etc).
I know you'll probably think I'm arguing semantics again, but these are *important* semantics ;)
Cheers, Henry
On 07 Dec 2015, at 1:05 , EuanM <euanmee@gmail.com> wrote:
Hi Henry,
To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-)
codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character
e.g.
The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041)
The representation of a component of the composed (i.e. preferred) version of the abstract character â« #(0041 030a)
â« (#00C5) represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
â« (#212b) also represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
With any luck, this satisfies both our semantic understandings of the concept of "codepoint"
Would you agree with that?
In Unicode, codepoints are *NOT* an abstract numerical representation of a text character.
At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings.
I agree, they are numerical representations of what Unicode refers to as characters.
codepoints represent "*encoded characters*"
No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding.
and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard)
Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class)
i.e. in other words
*Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases.
The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*.
The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint.
In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character
There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint.
Cheers, Euan
I agree you have a good grasp of the distinction between an abstract character (characters and character sequences which should be treated equivalent wrt, equality / sorting / display, etc.) and a character (which each have a code point assigned). That is besides the point both Sven and I tried to get through, which is the difference between a code point and the encoded form(s) of said code point. When you write: "and therefore encodable in UTF-8 as compatibility codepoint e9 hex and as the composed character #(0065 00b4) (all in hex) and as the same composed character as both #(feff 0065 00b4) and #(ffef 0065 00b4) when endianness markers are included" I's quite clear you confuse the two. 0xFEFF is the codepoint of the character used as bom. When you state that it can be written ffef (I assume you meant FFFE), you are again confusing the code point and its encoded value (an encoded value which only occurs in UTF16/32, no less). When this distinction is clear, it might be easier to see that value in that Strings are kept as Unicode code points arrays, and converted to encoded forms when entering/exiting the system. Cheers, Henry
On 07 Dec 2015, at 2:06 , Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
codepoints represent "*encoded characters*"
No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding.
You were right on this point, I see I remembered the terminology of this incorrectly. http://www.unicode.org/versions/Unicode8.0.0/ch02.pdf figure 2.8 does use "encoded characters" for the mapping of abstract characters to its equivalent codepoint (s/ sequences). What I thought it meant is better described as a codepoint's byte output using an "encoding scheme". An accurate description following that terminology, would be that Pharo/Squeak Strings keep data in UTF32 encoding form, where 1 codepoint = 1 code unit, dynamically switched between Latin1 (ByteStrings) and UTF32 (WideStrings) encoding schemes as needed. With the same terminology, the difference between a code point, a code unit, how an encoding scheme represents a codepoint as code units/bytes, are the concepts it is important to distinguish. Quite a mouthful though! Cheers Henry
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding." No. A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time). Codepoints represent a single encoding of a single concept. Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character. This is the key difference between Unicode and most character encodings. A codepoint does not always represent a whole character. On 7 December 2015 at 13:06, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On 07 Dec 2015, at 1:05 , EuanM <euanmee@gmail.com> wrote:
Hi Henry,
To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-)
codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character
e.g.
The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041)
The representation of a component of the composed (i.e. preferred) version of the abstract character â« #(0041 030a)
â« (#00C5) represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
â« (#212b) also represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
With any luck, this satisfies both our semantic understandings of the concept of "codepoint"
Would you agree with that?
In Unicode, codepoints are *NOT* an abstract numerical representation of a text character.
At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings.
I agree, they are numerical representations of what Unicode refers to as characters.
codepoints represent "*encoded characters*"
No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding.
and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard)
Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class)
i.e. in other words
*Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases.
The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*.
The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint.
In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character
There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint.
Cheers, Euan
I agree you have a good grasp of the distinction between an abstract character (characters and character sequences which should be treated equivalent wrt, equality / sorting / display, etc.) and a character (which each have a code point assigned). That is besides the point both Sven and I tried to get through, which is the difference between a code point and the encoded form(s) of said code point. When you write: "and therefore encodable in UTF-8 as compatibility codepoint e9 hex and as the composed character #(0065 00b4) (all in hex) and as the same composed character as both #(feff 0065 00b4) and #(ffef 0065 00b4) when endianness markers are included"
I's quite clear you confuse the two. 0xFEFF is the codepoint of the character used as bom. When you state that it can be written ffef (I assume you meant FFFE), you are again confusing the code point and its encoded value (an encoded value which only occurs in UTF16/32, no less).
When this distinction is clear, it might be easier to see that value in that Strings are kept as Unicode code points arrays, and converted to encoded forms when entering/exiting the system.
Cheers, Henry
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote. Can you explain a bit more about what you mean by abstract character and concept?
This is the key difference between Unicode and most character encodings.
A codepoint does not always represent a whole character.
On 7 December 2015 at 13:06, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On 07 Dec 2015, at 1:05 , EuanM <euanmee@gmail.com> wrote:
Hi Henry,
To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-)
codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character
e.g.
The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041)
The representation of a component of the composed (i.e. preferred) version of the abstract character â« #(0041 030a)
â« (#00C5) represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
â« (#212b) also represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
With any luck, this satisfies both our semantic understandings of the concept of "codepoint"
Would you agree with that?
In Unicode, codepoints are *NOT* an abstract numerical representation of a text character.
At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings.
I agree, they are numerical representations of what Unicode refers to as characters.
codepoints represent "*encoded characters*"
No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding.
and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard)
Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class)
i.e. in other words
*Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases.
The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*.
The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint.
In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character
There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint.
Cheers, Euan
I agree you have a good grasp of the distinction between an abstract character (characters and character sequences which should be treated equivalent wrt, equality / sorting / display, etc.) and a character (which each have a code point assigned). That is besides the point both Sven and I tried to get through, which is the difference between a code point and the encoded form(s) of said code point. When you write: "and therefore encodable in UTF-8 as compatibility codepoint e9 hex and as the composed character #(0065 00b4) (all in hex) and as the same composed character as both #(feff 0065 00b4) and #(ffef 0065 00b4) when endianness markers are included"
I's quite clear you confuse the two. 0xFEFF is the codepoint of the character used as bom. When you state that it can be written ffef (I assume you meant FFFE), you are again confusing the code point and its encoded value (an encoded value which only occurs in UTF16/32, no less).
When this distinction is clear, it might be easier to see that value in that Strings are kept as Unicode code points arrays, and converted to encoded forms when entering/exiting the system.
Cheers, Henry
On 09 Dec 2015, at 10:35, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
I am pretty sure that this whole discussion does more harm than good for most people's understanding of Unicode. It is best and (mostly) correct to think of a Unicode string as a sequence of Unicode characters, each defined/identified by a code point (out of 10.000s covering all languages). That is what we have today in Pharo (with the distinction between ByteString and WideString as mostly invisible implementation details). To encode Unicode for external representation as bytes, we use UTF-8 like the rest of the modern world. So far, so good. Why all the confusion ? Because the world is a complex place and the Unicode standard tries to cover all possible things. Citing all these exceptions and special cases will make people crazy and give up. I am sure that most stopped reading this thread. Why then is there confusion about the seemingly simple concept of a character ? Because Unicode allows different ways to say the same thing. The simplest example in a common language is (the French letter é) is LATIN SMALL LETTER E WITH ACUTE [U+00E9] which can also be written as LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT [U+0301] The former being a composed normal form, the latter a decomposed normal form. (And yes, it is even much more complicated than that, it goes on for 1000s of pages). In the above example, the concept of character/string is indeed fuzzy. HTH, Sven
"To encode Unicode for external representation as bytes, we use UTF-8 like the rest of the modern world. So far, so good. Why all the confusion ?" The confusion arises because simply providing *a* valid UTF-8 encoding of does not ensure sortability, nor equivalence testability. It might provide sortable strings. It might not. It might provide a string that can be compared to another string successfully. It might not. So being able to perform valid UTF-8 encoding is *necessary*, but *not sufficient*. i.e. the confusion arises because UTF-8 can provide for several competing, non-sortable encodings of even a single character. This means that *valid* UTF-8 cannot be relied upon to provide these facilities *unless* all the UTF-8 strings can be relied upon to have been encoded to UTF-8 by the same specification of process. i.e. *unless* it has gone through a process of being converted by *a specific* valid method of encoding to UTF-8. Understanding the concept of abstract character is, imo key to understanding the differences between the various valid UTF-8 forms of a given abstract character. Cheers, Euan On 9 December 2015 at 10:45, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 09 Dec 2015, at 10:35, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
I am pretty sure that this whole discussion does more harm than good for most people's understanding of Unicode.
It is best and (mostly) correct to think of a Unicode string as a sequence of Unicode characters, each defined/identified by a code point (out of 10.000s covering all languages). That is what we have today in Pharo (with the distinction between ByteString and WideString as mostly invisible implementation details).
To encode Unicode for external representation as bytes, we use UTF-8 like the rest of the modern world.
So far, so good.
Why all the confusion ? Because the world is a complex place and the Unicode standard tries to cover all possible things. Citing all these exceptions and special cases will make people crazy and give up. I am sure that most stopped reading this thread.
Why then is there confusion about the seemingly simple concept of a character ? Because Unicode allows different ways to say the same thing. The simplest example in a common language is (the French letter é) is
LATIN SMALL LETTER E WITH ACUTE [U+00E9]
which can also be written as
LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT [U+0301]
The former being a composed normal form, the latter a decomposed normal form. (And yes, it is even much more complicated than that, it goes on for 1000s of pages).
In the above example, the concept of character/string is indeed fuzzy.
HTH,
Sven
On 09 Dec 2015, at 14:16, EuanM <euanmee@gmail.com> wrote:
"To encode Unicode for external representation as bytes, we use UTF-8 like the rest of the modern world.
So far, so good.
Why all the confusion ?"
That was a rhetorical question. I know that we lack normalization, we don't need another encoding or representation. Sorting/collation can also be done regardless of encoding or representation. These are orthogonal concerns to the working situation that we have today.
The confusion arises because simply providing *a* valid UTF-8 encoding of does not ensure sortability, nor equivalence testability.
It might provide sortable strings. It might not.
It might provide a string that can be compared to another string successfully. It might not.
So being able to perform valid UTF-8 encoding is *necessary*, but *not sufficient*.
i.e. the confusion arises because UTF-8 can provide for several competing, non-sortable encodings of even a single character. This means that *valid* UTF-8 cannot be relied upon to provide these facilities *unless* all the UTF-8 strings can be relied upon to have been encoded to UTF-8 by the same specification of process. i.e. *unless* it has gone through a process of being converted by *a specific* valid method of encoding to UTF-8.
Understanding the concept of abstract character is, imo key to understanding the differences between the various valid UTF-8 forms of a given abstract character.
Cheers, Euan
On 9 December 2015 at 10:45, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 09 Dec 2015, at 10:35, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
I am pretty sure that this whole discussion does more harm than good for most people's understanding of Unicode.
It is best and (mostly) correct to think of a Unicode string as a sequence of Unicode characters, each defined/identified by a code point (out of 10.000s covering all languages). That is what we have today in Pharo (with the distinction between ByteString and WideString as mostly invisible implementation details).
To encode Unicode for external representation as bytes, we use UTF-8 like the rest of the modern world.
So far, so good.
Why all the confusion ? Because the world is a complex place and the Unicode standard tries to cover all possible things. Citing all these exceptions and special cases will make people crazy and give up. I am sure that most stopped reading this thread.
Why then is there confusion about the seemingly simple concept of a character ? Because Unicode allows different ways to say the same thing. The simplest example in a common language is (the French letter é) is
LATIN SMALL LETTER E WITH ACUTE [U+00E9]
which can also be written as
LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT [U+0301]
The former being a composed normal form, the latter a decomposed normal form. (And yes, it is even much more complicated than that, it goes on for 1000s of pages).
In the above example, the concept of character/string is indeed fuzzy.
HTH,
Sven
I am pretty sure that this whole discussion does more harm than good for most people's understanding of Unicode.
It is best and (mostly) correct to think of a Unicode string as a sequence of Unicode characters, each defined/identified by a code point (out of 10.000s covering all languages). That is what we have today in Pharo (with the distinction between ByteString and WideString as mostly invisible implementation details).
To encode Unicode for external representation as bytes, we use UTF-8 like the rest of the modern world.
So far, so good.
Why all the confusion ? Because the world is a complex place and the Unicode standard tries to cover all possible things. Citing all these exceptions and special cases will make people crazy and give up. I am sure that most stopped reading this thread.
like me ;) I will wait for a conclusion with code :) Stef
Why then is there confusion about the seemingly simple concept of a character ? Because Unicode allows different ways to say the same thing. The simplest example in a common language is (the French letter é) is
LATIN SMALL LETTER E WITH ACUTE [U+00E9]
which can also be written as
LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT [U+0301]
The former being a composed normal form, the latter a decomposed normal form. (And yes, it is even much more complicated than that, it goes on for 1000s of pages).
In the above example, the concept of character/string is indeed fuzzy.
HTH,
Sven
See example with ANGSTROM Abstract Characters (Unicode) http://wiki.squeak.org/squeak/6256 On 12/9/15, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
This is the key difference between Unicode and most character encodings.
A codepoint does not always represent a whole character.
On 7 December 2015 at 13:06, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On 07 Dec 2015, at 1:05 , EuanM <euanmee@gmail.com> wrote:
Hi Henry,
To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-)
codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character
e.g.
The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041)
The representation of a component of the composed (i.e. preferred) version of the abstract character â« #(0041 030a)
â« (#00C5) represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
â« (#212b) also represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
With any luck, this satisfies both our semantic understandings of the concept of "codepoint"
Would you agree with that?
In Unicode, codepoints are *NOT* an abstract numerical representation of a text character.
At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings.
I agree, they are numerical representations of what Unicode refers to as characters.
codepoints represent "*encoded characters*"
No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding.
and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard)
Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class)
i.e. in other words
*Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases.
The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*.
The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint.
In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character
There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint.
Cheers, Euan
I agree you have a good grasp of the distinction between an abstract character (characters and character sequences which should be treated equivalent wrt, equality / sorting / display, etc.) and a character (which each have a code point assigned). That is besides the point both Sven and I tried to get through, which is the difference between a code point and the encoded form(s) of said code point. When you write: "and therefore encodable in UTF-8 as compatibility codepoint e9 hex and as the composed character #(0065 00b4) (all in hex) and as the same composed character as both #(feff 0065 00b4) and #(ffef 0065 00b4) when endianness markers are included"
I's quite clear you confuse the two. 0xFEFF is the codepoint of the character used as bom. When you state that it can be written ffef (I assume you meant FFFE), you are again confusing the code point and its encoded value (an encoded value which only occurs in UTF16/32, no less).
When this distinction is clear, it might be easier to see that value in that Strings are kept as Unicode code points arrays, and converted to encoded forms when entering/exiting the system.
Cheers, Henry
"Well. I do not agree with this. I agree with the quote. Can you explain a bit more about what you mean by abstract character and concept?" Guillermo The problem with the quote is, that *while true*, it *does not disambiguate* between: either compatibility character and abstract character; or character as composable component of an abstract character and character as the entire embodiment of an abstract character; Abstract character is the key concept of Unicode. Differentiation between abstract character and codepoints is the key differentiator of the Unicode approach and most previous approaches to character encoding, e,g, ASCII, EBCDIC, ISO Latin 1, etc Please see my previous posts which use the example of Angstrom, Capital A with circle (or whatever the canonical name is) and the composed sequence of "Capital A" and "circle above a letter" for a fuller explanation of the concept of "abstract character". On 9 December 2015 at 09:35, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
This is the key difference between Unicode and most character encodings.
A codepoint does not always represent a whole character.
On 7 December 2015 at 13:06, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On 07 Dec 2015, at 1:05 , EuanM <euanmee@gmail.com> wrote:
Hi Henry,
To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-)
codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character
e.g.
The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041)
The representation of a component of the composed (i.e. preferred) version of the abstract character â« #(0041 030a)
â« (#00C5) represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
â« (#212b) also represents one valid compatibility form of the abstract character â« which is most properly represented by #(0041 030a).
With any luck, this satisfies both our semantic understandings of the concept of "codepoint"
Would you agree with that?
In Unicode, codepoints are *NOT* an abstract numerical representation of a text character.
At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings.
I agree, they are numerical representations of what Unicode refers to as characters.
codepoints represent "*encoded characters*"
No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding.
and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard)
Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class)
i.e. in other words
*Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases.
The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*.
The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint.
In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character
There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint.
Cheers, Euan
I agree you have a good grasp of the distinction between an abstract character (characters and character sequences which should be treated equivalent wrt, equality / sorting / display, etc.) and a character (which each have a code point assigned). That is besides the point both Sven and I tried to get through, which is the difference between a code point and the encoded form(s) of said code point. When you write: "and therefore encodable in UTF-8 as compatibility codepoint e9 hex and as the composed character #(0065 00b4) (all in hex) and as the same composed character as both #(feff 0065 00b4) and #(ffef 0065 00b4) when endianness markers are included"
I's quite clear you confuse the two. 0xFEFF is the codepoint of the character used as bom. When you state that it can be written ffef (I assume you meant FFFE), you are again confusing the code point and its encoded value (an encoded value which only occurs in UTF16/32, no less).
When this distinction is clear, it might be easier to see that value in that Strings are kept as Unicode code points arrays, and converted to encoded forms when entering/exiting the system.
Cheers, Henry
On Wed, Dec 9, 2015 at 5:35 PM, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
This seems to be what Swift is doing, where Strings are not composed not of codepoints but of graphemes.
"Every instance of Swiftâs Character type represents a single extended grapheme cluster. An extended grapheme cluster is a sequence** of one or more Unicode scalars that (when combined) produce a single human-readable character. [1]
** i.e. not an array
Hereâs an example. The letter é can be represented as the single Unicode scalar é (LATIN SMALL LETTER E WITH ACUTE, or U+00E9). However, the same letter can also be represented as a pair of scalarsâa standard letter e (LATIN SMALL LETTER E, or U+0065), followed by the COMBINING ACUTE ACCENT scalar (U+0301). TheCOMBINING ACUTE ACCENT scalar is graphically applied to the scalar that precedes it, turning an e into an éwhen it is rendered by a Unicode-aware text-rendering system. [1]
In both cases, the letter é is represented as a single Swift Character value that represents an extended grapheme cluster. In the first case, the cluster contains a single scalar; in the second case, it is a cluster of two scalars:" [1]
Swiftʼs string implemenation makes working with Unicode easier and significantly less error-prone. As a programmer, you still have to be aware of possible edge cases, but this probably cannot be avoided completely considering the characteristics of Unicode. [2]
Indeed I've tried searched for what problems it causes and get a null result. So I read *all*good* things about Swift's unicode implementation reducing common errors dealing with Unicode. Can anyone point to complaints about Swift's unicode implementation? Maybe this...
An argument could be made that the implementation of String as a sequence that requires iterating over characters from the beginning of the string for many operations poses a significant performance problem but I do not think so. My guess is that Appleʼs engineers have considered the implications of their implementation and apps that do not deal with enormous amounts of text will be fine. Moreover, the idea that you could get away with an implementation that supports random access of characters is an illusion given the complexity of Unicode. [2]
Considering our common pattern: Make it work, Make it right, Make it fast -- maybe Strings as arrays are a premature optimisation, that was the right choice in the past prior to Unicode, but considering Moore's Law versus programmer time, is not the best choice now. Should we at least start with a UnicodeString and UnicodeCharacter that operates like Swift, and over time *maybe* move the tools to use them. [1] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift... [2] http://oleb.net/blog/2014/07/swift-strings/ cheers -ben
This is the key difference between Unicode and most character encodings.
A codepoint does not always represent a whole character.
On 7 December 2015 at 13:06, Henrik Johansen
I agree with all of that, Ben. I'm currently fairly certain that fully-composed abstract characters is a term that is 1:1 mapped with the term "grapheme cluster" (i.e. one is an older Unicode description of a newer Unicode term). And once we create these, I think this sort of implementation is straightforward. For particular values of "straightforward", of course :-) i.e. the Swift approach is equivalent to the approach I originally proposed and asked for critiques of. One thing I don't understand.... why does the fact the composed abstract character (aka grapheme cluster) is a sequence mean that an array cannot be used to hold the sequence? If people then also want a compatibility-codepoints-only UTF-8 representation, it is simple to provide comparable (i.e equivalence-testable) versions of any UTF-8 string - because we are creating them from composed forms by a *single* defined method. For my part, the reason I think we ought to implement it *in* Smalltalk is ... this is the String class of the new age. I want Smalltalk to be handle Strings as native objects. On 10 December 2015 at 23:41, Ben Coman <btc@openinworld.com> wrote:
On Wed, Dec 9, 2015 at 5:35 PM, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
This seems to be what Swift is doing, where Strings are not composed not of codepoints but of graphemes.
"Every instance of Swiftâs Character type represents a single extended grapheme cluster. An extended grapheme cluster is a sequence** of one or more Unicode scalars that (when combined) produce a single human-readable character. [1]
** i.e. not an array
Hereâs an example. The letter é can be represented as the single Unicode scalar é (LATIN SMALL LETTER E WITH ACUTE, or U+00E9). However, the same letter can also be represented as a pair of scalarsâa standard letter e (LATIN SMALL LETTER E, or U+0065), followed by the COMBINING ACUTE ACCENT scalar (U+0301). TheCOMBINING ACUTE ACCENT scalar is graphically applied to the scalar that precedes it, turning an e into an éwhen it is rendered by a Unicode-aware text-rendering system. [1]
In both cases, the letter é is represented as a single Swift Character value that represents an extended grapheme cluster. In the first case, the cluster contains a single scalar; in the second case, it is a cluster of two scalars:" [1]
Swiftʼs string implemenation makes working with Unicode easier and significantly less error-prone. As a programmer, you still have to be aware of possible edge cases, but this probably cannot be avoided completely considering the characteristics of Unicode. [2]
Indeed I've tried searched for what problems it causes and get a null result. So I read *all*good* things about Swift's unicode implementation reducing common errors dealing with Unicode. Can anyone point to complaints about Swift's unicode implementation? Maybe this...
An argument could be made that the implementation of String as a sequence that requires iterating over characters from the beginning of the string for many operations poses a significant performance problem but I do not think so. My guess is that Appleʼs engineers have considered the implications of their implementation and apps that do not deal with enormous amounts of text will be fine. Moreover, the idea that you could get away with an implementation that supports random access of characters is an illusion given the complexity of Unicode. [2]
Considering our common pattern: Make it work, Make it right, Make it fast -- maybe Strings as arrays are a premature optimisation, that was the right choice in the past prior to Unicode, but considering Moore's Law versus programmer time, is not the best choice now. Should we at least start with a UnicodeString and UnicodeCharacter that operates like Swift, and over time *maybe* move the tools to use them.
[1] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift... [2] http://oleb.net/blog/2014/07/swift-strings/
cheers -ben
This is the key difference between Unicode and most character encodings.
A codepoint does not always represent a whole character.
On 7 December 2015 at 13:06, Henrik Johansen
Hi Euan,
On Dec 10, 2015, at 6:43 PM, EuanM <euanmee@gmail.com> wrote:
I agree with all of that, Ben.
I'm currently fairly certain that fully-composed abstract characters is a term that is 1:1 mapped with the term "grapheme cluster" (i.e. one is an older Unicode description of a newer Unicode term).
And once we create these, I think this sort of implementation is straightforward. For particular values of "straightforward", of course :-)
i.e. the Swift approach is equivalent to the approach I originally proposed and asked for critiques of.
One thing I don't understand.... why does the fact the composed abstract character (aka grapheme cluster) is a sequence mean that an array cannot be used to hold the sequence?
Of course an Array can be used, but one good reason to use bits organized as four-byte units is that the garbage collector spends no time scanning them, whereas as far as its concerned the Array representation is all objects and must be scanned. Another reason is that foreign code may find the bits representation compatible and so they can be passed through the FFI to other languages whereas the Array of tagged characters will always require conversion. Yet another reason is that in 64-bits the Array takes twice the space of the bits object.
If people then also want a compatibility-codepoints-only UTF-8 representation, it is simple to provide comparable (i.e equivalence-testable) versions of any UTF-8 string - because we are creating them from composed forms by a *single* defined method.
For my part, the reason I think we ought to implement it *in* Smalltalk is ... this is the String class of the new age. I want Smalltalk to be handle Strings as native objects.
There's little if any difference in convenience of use between an Array of characters and a bits array with the string at:/at:put: primitives since both require at:/at:put: to access, but the latter is (efficiently) type checked (by the VM), whereas there's nothing to prevent storing other than characters in the Areay unless one introduces the overhead of skier explicit type checks in Smalltalk, and the Areay starts life as a sequence of nils (invalid until every element is set to a character) whereas the bits representation begins fully initialized with 0 asCharacter. So there's nothing more "natively objecty" about the Array. Smalltalk objects hide their representation from clients and externally they behave the same, except for space and time. Given that this is a dynamically-typed language there's nothing to prevent one providing both implementations beyond maintenance cost and complexity/confusion. So at least it's easy to do performance comparisons between the two. But I still think the bits representation is superior if what you want is a sequence of Characters.
On 10 December 2015 at 23:41, Ben Coman <btc@openinworld.com> wrote: On Wed, Dec 9, 2015 at 5:35 PM, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
This seems to be what Swift is doing, where Strings are not composed not of codepoints but of graphemes.
"Every instance of Swiftâs Character type represents a single extended grapheme cluster. An extended grapheme cluster is a sequence** of one or more Unicode scalars that (when combined) produce a single human-readable character. [1]
** i.e. not an array
Hereâs an example. The letter é can be represented as the single Unicode scalar é (LATIN SMALL LETTER E WITH ACUTE, or U+00E9). However, the same letter can also be represented as a pair of scalarsâa standard letter e (LATIN SMALL LETTER E, or U+0065), followed by the COMBINING ACUTE ACCENT scalar (U+0301). TheCOMBINING ACUTE ACCENT scalar is graphically applied to the scalar that precedes it, turning an e into an éwhen it is rendered by a Unicode-aware text-rendering system. [1]
In both cases, the letter é is represented as a single Swift Character value that represents an extended grapheme cluster. In the first case, the cluster contains a single scalar; in the second case, it is a cluster of two scalars:" [1]
Swiftʼs string implemenation makes working with Unicode easier and significantly less error-prone. As a programmer, you still have to be aware of possible edge cases, but this probably cannot be avoided completely considering the characteristics of Unicode. [2]
Indeed I've tried searched for what problems it causes and get a null result. So I read *all*good* things about Swift's unicode implementation reducing common errors dealing with Unicode. Can anyone point to complaints about Swift's unicode implementation? Maybe this...
An argument could be made that the implementation of String as a sequence that requires iterating over characters from the beginning of the string for many operations poses a significant performance problem but I do not think so. My guess is that Appleʼs engineers have considered the implications of their implementation and apps that do not deal with enormous amounts of text will be fine. Moreover, the idea that you could get away with an implementation that supports random access of characters is an illusion given the complexity of Unicode. [2]
Considering our common pattern: Make it work, Make it right, Make it fast -- maybe Strings as arrays are a premature optimisation, that was the right choice in the past prior to Unicode, but considering Moore's Law versus programmer time, is not the best choice now. Should we at least start with a UnicodeString and UnicodeCharacter that operates like Swift, and over time *maybe* move the tools to use them.
[1] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift... [2] http://oleb.net/blog/2014/07/swift-strings/
cheers -ben
This is the key difference between Unicode and most character encodings.
A codepoint does not always represent a whole character.
On 7 December 2015 at 13:06, Henrik Johansen
_,,,^..^,,,_ (phone)
Eliot - thank you for explaining to me why my original idea was bad. :-) I always assumed it would be. Otherwise I'd've just built it the way I proposed. I'm thoroughly delighted to have more knowledgeable people contributing. On 11 December 2015 at 09:29, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Euan,
On Dec 10, 2015, at 6:43 PM, EuanM <euanmee@gmail.com> wrote:
I agree with all of that, Ben.
I'm currently fairly certain that fully-composed abstract characters is a term that is 1:1 mapped with the term "grapheme cluster" (i.e. one is an older Unicode description of a newer Unicode term).
And once we create these, I think this sort of implementation is straightforward. For particular values of "straightforward", of course :-)
i.e. the Swift approach is equivalent to the approach I originally proposed and asked for critiques of.
One thing I don't understand.... why does the fact the composed abstract character (aka grapheme cluster) is a sequence mean that an array cannot be used to hold the sequence?
Of course an Array can be used, but one good reason to use bits organized as four-byte units is that the garbage collector spends no time scanning them, whereas as far as its concerned the Array representation is all objects and must be scanned. Another reason is that foreign code may find the bits representation compatible and so they can be passed through the FFI to other languages whereas the Array of tagged characters will always require conversion. Yet another reason is that in 64-bits the Array takes twice the space of the bits object.
If people then also want a compatibility-codepoints-only UTF-8 representation, it is simple to provide comparable (i.e equivalence-testable) versions of any UTF-8 string - because we are creating them from composed forms by a *single* defined method.
For my part, the reason I think we ought to implement it *in* Smalltalk is ... this is the String class of the new age. I want Smalltalk to be handle Strings as native objects.
There's little if any difference in convenience of use between an Array of characters and a bits array with the string at:/at:put: primitives since both require at:/at:put: to access, but the latter is (efficiently) type checked (by the VM), whereas there's nothing to prevent storing other than characters in the Areay unless one introduces the overhead of skier explicit type checks in Smalltalk, and the Areay starts life as a sequence of nils (invalid until every element is set to a character) whereas the bits representation begins fully initialized with 0 asCharacter. So there's nothing more "natively objecty" about the Array. Smalltalk objects hide their representation from clients and externally they behave the same, except for space and time.
Given that this is a dynamically-typed language there's nothing to prevent one providing both implementations beyond maintenance cost and complexity/confusion. So at least it's easy to do performance comparisons between the two. But I still think the bits representation is superior if what you want is a sequence of Characters.
On 10 December 2015 at 23:41, Ben Coman <btc@openinworld.com> wrote: On Wed, Dec 9, 2015 at 5:35 PM, Guillermo Polito <guillermopolito@gmail.com> wrote:
On 8 dic 2015, at 10:07 p.m., EuanM <euanmee@gmail.com> wrote:
"No. a codepoint is the numerical value assigned to a character. An "encoded character" is the way a codepoint is represented in bytes using a given encoding."
No.
A codepoint may represent a component part of an abstract character, or may represent an abstract character, or it may do both (but not always at the same time).
Codepoints represent a single encoding of a single concept.
Sometimes that concept represents a whole abstract character. Sometimes it represent part of an abstract character.
Well. I do not agree with this. I agree with the quote.
Can you explain a bit more about what you mean by abstract character and concept?
This seems to be what Swift is doing, where Strings are not composed not of codepoints but of graphemes.
"Every instance of Swiftâs Character type represents a single extended grapheme cluster. An extended grapheme cluster is a sequence** of one or more Unicode scalars that (when combined) produce a single human-readable character. [1]
** i.e. not an array
Hereâs an example. The letter é can be represented as the single Unicode scalar é (LATIN SMALL LETTER E WITH ACUTE, or U+00E9). However, the same letter can also be represented as a pair of scalarsâa standard letter e (LATIN SMALL LETTER E, or U+0065), followed by the COMBINING ACUTE ACCENT scalar (U+0301). TheCOMBINING ACUTE ACCENT scalar is graphically applied to the scalar that precedes it, turning an e into an éwhen it is rendered by a Unicode-aware text-rendering system. [1]
In both cases, the letter é is represented as a single Swift Character value that represents an extended grapheme cluster. In the first case, the cluster contains a single scalar; in the second case, it is a cluster of two scalars:" [1]
Swiftʼs string implemenation makes working with Unicode easier and significantly less error-prone. As a programmer, you still have to be aware of possible edge cases, but this probably cannot be avoided completely considering the characteristics of Unicode. [2]
Indeed I've tried searched for what problems it causes and get a null result. So I read *all*good* things about Swift's unicode implementation reducing common errors dealing with Unicode. Can anyone point to complaints about Swift's unicode implementation? Maybe this...
An argument could be made that the implementation of String as a sequence that requires iterating over characters from the beginning of the string for many operations poses a significant performance problem but I do not think so. My guess is that Appleʼs engineers have considered the implications of their implementation and apps that do not deal with enormous amounts of text will be fine. Moreover, the idea that you could get away with an implementation that supports random access of characters is an illusion given the complexity of Unicode. [2]
Considering our common pattern: Make it work, Make it right, Make it fast -- maybe Strings as arrays are a premature optimisation, that was the right choice in the past prior to Unicode, but considering Moore's Law versus programmer time, is not the best choice now. Should we at least start with a UnicodeString and UnicodeCharacter that operates like Swift, and over time *maybe* move the tools to use them.
[1] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift... [2] http://oleb.net/blog/2014/07/swift-strings/
cheers -ben
This is the key difference between Unicode and most character encodings.
A codepoint does not always represent a whole character.
On 7 December 2015 at 13:06, Henrik Johansen
_,,,^..^,,,_ (phone)
On 07 Dec 2015, at 11:51, EuanM <euanmee@gmail.com> wrote:
Verifying assumptions is the key reason why you should documents like this out for review.
Fair enough, discussion can only help.
Sven -
Cuis is encoded with ISO 8859-15 (aka ISO Latin 9)
Sven, this is *NOT* as you state, ISO 99591, (and not as I stated, 8859-1).
Ah, that was a typo, I meant, of course (and sorry for the confusion): 'Les élèves Français' encodeWith: #iso88591. "#[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115]" 'Les élèves Français' utf8Encoded "#[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115]" Or shorter, $é is encoded in ISO-88591-1 as #[233], but as #[195 169] in UTF-8. That Cuis chose ISO-8859-15 makes no real difference. The thing is: you started talking about UTF-8 encoded strings in the image, and then the difference between code point and encoding is really important. Only in ASCII is the encoding identical, not for anything else.
We caught the right specification bug for the wrong reason.
Juan: "Cuis: Chose not to use Squeak approach. Chose to make the base image include and use only 1-byte strings. Chose to use ISO-8859-15"
I have double-checked - each character encoded in ISO Latin 15 (ISO 8859-15) is exactly the character represented by the corresponding 1-byte codepoint in Unicode 0000 to 00FF,
with the following exceptions:
codepoint 20ac - Euro symbol character code a4 (replaces codepoint 00a4 generic currency symbol)
codepoint 0160 Latin Upper Case S with Caron character code a6 (replaces codepoint 00A6 was | Unix pipe character)
codepoint 0161 Latin Lower Case s with Caron character code a8 (replaces codepoint 00A8 was dierisis)
codepoint 017d Latin Upper Case Z with Caron character code b4 (replaces codepoint 00b4 was Acute accent)
codepoint 017e Latin Lower Case Z with Caron character code b8 (replaces codepoint 00b8 was cedilla)
codepoint 0152 Upper Case OE ligature = Ethel character code bc (replaces codepoint 00bc was 1/4 symbol)
codepoint 0153 Lower Case oe ligature = ethel character code bd (replaces codepoint 00bd was 1/2 symbol)
codepoint 0178 Upper Case Y diaeresis character code be (replaces codepoint 00be was 3/4 symbol)
Juan - I don't suppose we could persuade you to change to ISO Latin-1 from ISO Latin-9 ?
It means we could run the same 1 byte string encoding across Cuis, Squeak, Pharo, and, as far as I can make out so far, Dolphin Smalltalk and Gnu Smalltalk.
The downside would be that French Y diaeresis would lose the ability to use that character, along with users of oe, OE, and s, S, z, Z with caron. Along with the Euro.
https://en.wikipedia.org/wiki/ISO/IEC_8859-15.
I'm confident I understand the use of UTF-8 in principal.
On 7 December 2015 at 08:27, Sven Van Caekenberghe <sven@stfx.eu> wrote:
I am sorry but one of your basic assumptions is completely wrong:
'Les élèves Français' encodeWith: #iso99591.
=> #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115]
'Les élèves Français' utf8Encoded.
=> #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115]
ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !!
Or shorter, $é is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8.
So more than half the points you make, or the facts that you state, are thus plain wrong.
The only thing that is correct is that the code points are equal, but that is not the same as the encoding !
From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String.
Sorry.
PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world.
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
Hi Sven, okay I'm plodding my through https://tools.ietf.org/html/rfc3629 and https://en.wikipedia.org/wiki/UTF-8#Examples to see what's what. On 7 December 2015 at 11:01, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 07 Dec 2015, at 11:51, EuanM <euanmee@gmail.com> wrote:
Verifying assumptions is the key reason why you should documents like this out for review.
Fair enough, discussion can only help.
Sven -
Cuis is encoded with ISO 8859-15 (aka ISO Latin 9)
Sven, this is *NOT* as you state, ISO 99591, (and not as I stated, 8859-1).
Ah, that was a typo, I meant, of course (and sorry for the confusion):
'Les élèves Français' encodeWith: #iso88591.
"#[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115]"
'Les élèves Français' utf8Encoded
"#[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115]"
Or shorter, $é is encoded in ISO-88591-1 as #[233], but as #[195 169] in UTF-8.
That Cuis chose ISO-8859-15 makes no real difference.
The thing is: you started talking about UTF-8 encoded strings in the image, and then the difference between code point and encoding is really important.
Only in ASCII is the encoding identical, not for anything else.
We caught the right specification bug for the wrong reason.
Juan: "Cuis: Chose not to use Squeak approach. Chose to make the base image include and use only 1-byte strings. Chose to use ISO-8859-15"
I have double-checked - each character encoded in ISO Latin 15 (ISO 8859-15) is exactly the character represented by the corresponding 1-byte codepoint in Unicode 0000 to 00FF,
with the following exceptions:
codepoint 20ac - Euro symbol character code a4 (replaces codepoint 00a4 generic currency symbol)
codepoint 0160 Latin Upper Case S with Caron character code a6 (replaces codepoint 00A6 was | Unix pipe character)
codepoint 0161 Latin Lower Case s with Caron character code a8 (replaces codepoint 00A8 was dierisis)
codepoint 017d Latin Upper Case Z with Caron character code b4 (replaces codepoint 00b4 was Acute accent)
codepoint 017e Latin Lower Case Z with Caron character code b8 (replaces codepoint 00b8 was cedilla)
codepoint 0152 Upper Case OE ligature = Ethel character code bc (replaces codepoint 00bc was 1/4 symbol)
codepoint 0153 Lower Case oe ligature = ethel character code bd (replaces codepoint 00bd was 1/2 symbol)
codepoint 0178 Upper Case Y diaeresis character code be (replaces codepoint 00be was 3/4 symbol)
Juan - I don't suppose we could persuade you to change to ISO Latin-1 from ISO Latin-9 ?
It means we could run the same 1 byte string encoding across Cuis, Squeak, Pharo, and, as far as I can make out so far, Dolphin Smalltalk and Gnu Smalltalk.
The downside would be that French Y diaeresis would lose the ability to use that character, along with users of oe, OE, and s, S, z, Z with caron. Along with the Euro.
https://en.wikipedia.org/wiki/ISO/IEC_8859-15.
I'm confident I understand the use of UTF-8 in principal.
On 7 December 2015 at 08:27, Sven Van Caekenberghe <sven@stfx.eu> wrote:
I am sorry but one of your basic assumptions is completely wrong:
'Les élèves Français' encodeWith: #iso99591.
=> #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115]
'Les élèves Français' utf8Encoded.
=> #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115]
ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !!
Or shorter, $é is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8.
So more than half the points you make, or the facts that you state, are thus plain wrong.
The only thing that is correct is that the code points are equal, but that is not the same as the encoding !
From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String.
Sorry.
PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world.
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
As for the issue of lower case e acute, it is compatibility codepoint 00e9 hex and therefore encodable in UTF-8 as compatibility codepoint e9 hex and as the composed character #(0065 00b4) (all in hex) and as the same composed character as both #(feff 0065 00b4) and #(ffef 0065 00b4) when endianness markers are included and as I understand it, should also be legitimate to encode it in UTF8 as a composed character #(65 b4) (all hex) etc On 7 December 2015 at 08:27, Sven Van Caekenberghe <sven@stfx.eu> wrote:
I am sorry but one of your basic assumptions is completely wrong:
'Les élèves Français' encodeWith: #iso99591.
=> #[76 101 115 32 233 108 232 118 101 115 32 70 114 97 110 231 97 105 115]
'Les élèves Français' utf8Encoded.
=> #[76 101 115 32 195 169 108 195 168 118 101 115 32 70 114 97 110 195 167 97 105 115]
ISO-9959-1 (~Latin1) is NOT AT ALL identical to UTF-8 in its upper, non-ACII part !!
Or shorter, $é is encoded in ISO-9959-1 as #[233], but as #[195 169] in UTF-8.
So more than half the points you make, or the facts that you state, are thus plain wrong.
The only thing that is correct is that the code points are equal, but that is not the same as the encoding !
From this I am inclined to conclude that you do not fundamentally understand how UTF-8 works, which does not strike me as good basis to design something called a UTF8String.
Sorry.
PS: Note also that Cuis' choice to use ISO-9959-1 only is pretty limiting in a Unicode world.
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
On 07 Dec 2015, at 04:21, EuanM <euanmee@gmail.com> wrote:
This a long email. A *lot* of it is encapsulated in the Venn diagram both: http://smalltalk.uk.to/unicode-utf8.html and my Smalltalk in Small Steps blog at: http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and...
My current thinking, and understanding. ==============================
0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte. b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.
1) Smalltalk has long had multiple String classes.
2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex is encoded as a UTF-8 codepoint of nn hex.
3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex. https://en.wikipedia.org/wiki/ISO/IEC_8859-1
4) All valid ASCII characters have a character code between 00 hex and 7E hex. https://en.wikipedia.org/wiki/ASCII
5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.
b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1
6) All character codes defined in ASCII (00 hex to 7E hex) are defined identically in Unicode UTF-8.
7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.
8) => some Unicode codepoints map to both ASCII and ISO-8859-1. all ASCII maps 1:1 to Unicode UTF-8 all ISO-8859-1 maps 1:1 to Unicode UTF-8
9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.
10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".
11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.
12) Some concrete examples:
A - aka Upper Case A In ASCII, in ISO 8859-1 ASCII A - 41 hex ISO-8859-1 A - 41 hex UTF-8 A - 41 hex
BEL (a bell sound, often invoked by a Ctrl-g keyboard chord) In ASCII, not in ISO 8859-1 ASCII : BEL - 07 hex ISO-8859-1 : 07 hex is not a valid character code UTF-8 : BEL - 07 hex
£ (GBP currency symbol) In ISO-8859-1, not in ASCII ASCII : A3 hex is not a valid ASCII code UTF-8: £ - A3 hex ISO-8859-1: £ - A3 hex
Upper Case C cedilla In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints ASCII : C7 hex is not a valid ASCII character code ISO-8859-1 : Upper Case C cedilla - C7 hex UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex Unicode preferred Upper Case C cedilla (composed set of codepoints) Upper case C 0043 hex (Upper case C) followed by cedilla 00B8 hex (cedilla)
13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.
14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.
15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character i.e. "C" and "C cedilla" will be treated very differently. "Ã" and "fs" will be treated very differently.
16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.
17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.
Some practical upshots ==================
1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.
2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.
3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.
4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).
5) To successfully *accept* all UTF-8 we much be able to do either: a) accept UTF-8 strings with composed characters b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.
Class + protocol proposals
a Utf8CompatibilityString class.
asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.
asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1
asAsciiString - ensures only characters 00hex - 7F hex are used.
asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints
a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.
Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.
Custom sortBlocks will define the applicable sort order.
We can create a collection... a Dictionary, thinking about it, of named, prefabricated sortBlocks.
This will work for all UTF8 strings of ISO-8859-1 and ASCII strings.
If anyone has better names for the classes, please let me know.
If anyone else wants to help - build these, - create SUnit tests for these - write documentation for these Please let me know.
n.b. I have had absolutely no experience of Ropes.
My own background with this stuff: In the early 90's as a Project Manager implementing office automation systems across a global company, with offices in the Americas, Western, Eastern and Central Europe, (including Slavic and Cyrillic users) nations, Japan and China. The mission-critical application was word-processing.
Our offices were spread around the globe, and we needed those offices to successfully exchange documents with their sister offices, and with the customers in each region the offices were in.
Unicode was then new, and our platform supplier was the NeXT Corporation, who had been founder members in of the Unicode Consortium in 1990.
So far: I've read the latest version of the Unicode Standard (v8.0). This is freely downloadable. I've purchased a paper copy of an earlier release. New releases typically consist additional codespaces (i.e. alphabets). So old copies are useful, as well as cheap. (Paper copies of version 4.0 are available second-hand for < $10 / â¬10).
The typical change with each release is the addition of further codespaces (i.e alphabets (more or less) ), so you don't lose a lot. (I'll be going through my V4.0 just to make sure)
Cheers, Euan
On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
EuanM wrote
... all ISO-8859-1 maps 1:1 to Unicode UTF-8 ...
I am late coming in to this conversation. If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor. Unicode defines the meaning of the code points. UTF-8 (and -16) define an interchange mechanism. In other words, when you write the code points to an external medium (socket, file, whatever), encode them via UTF-whatever. Read UTF-whatever from an external medium and re-instantiate the code points. (Personally, I see no use for UTF-16 as an interchange mechanism. Others may have justification for it. I don't.) Having characters be a consistent size in their object representation makes everything easier. #at:, #indexOf:, #includes: ... no one wants to be scanning through bytes representing variable sized characters. Model Unicode strings using classes such as e.g. Unicode7, Unicode16, and Unicode32, with automatic coercion to the larger character width. -- View this message in context: http://forum.world.st/Unicode-Support-tp4865139p4866610.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
"If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor." --Richard Sargent I agree. :-) Regarding UTF-16, I just want to be able to export to, and receive from, Windows (and any other platforms using UTF-16 as their native character representation). Windows will always be able to accept UTF-16. All Windows apps *might well* export UTF-16. There may be other platforms which use UTF-16 as their native format. I'd just like to be able to cope with those situations. Nothing more. All this is requires is a Utf16String class that has an asUtf8String method (and any other required conversion methods). And other string classes to have asUtf16String classes. Once we have the other classes and methods, this should be a trivial extensions. Export will just be transformations of existing formats of valid strings. Import just needs to transform to (one of) our preferred format(s), and have a validity check performed after the transform is complete. On 11 December 2015 at 15:37, Richard Sargent <richard.sargent@gemtalksystems.com> wrote:
EuanM wrote
... all ISO-8859-1 maps 1:1 to Unicode UTF-8 ...
I am late coming in to this conversation. If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor.
Unicode defines the meaning of the code points. UTF-8 (and -16) define an interchange mechanism.
In other words, when you write the code points to an external medium (socket, file, whatever), encode them via UTF-whatever. Read UTF-whatever from an external medium and re-instantiate the code points. (Personally, I see no use for UTF-16 as an interchange mechanism. Others may have justification for it. I don't.)
Having characters be a consistent size in their object representation makes everything easier. #at:, #indexOf:, #includes: ... no one wants to be scanning through bytes representing variable sized characters.
Model Unicode strings using classes such as e.g. Unicode7, Unicode16, and Unicode32, with automatic coercion to the larger character width.
-- View this message in context: http://forum.world.st/Unicode-Support-tp4865139p4866610.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
On Dec 11, 2015, at 12:19, EuanM <euanmee@gmail.com> wrote:
"If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor." --Richard Sargent
Well, yes. But I think you guys are making this way too hard. A unicode character is an abstract idea - for instance the letter 'a'. The letter 'a' has a code point - its the number 97. How the number 97 is represented in the computer is irrelevant. Now we get to transfer encodings. These are UTF8, UTF16, etc.... A transfer encoding specifies the binary representation of the sequence of code points. UTF8 is a variable length byte encoding. You read it one byte at a time, aggregating byte sequences to 'code points'. ByteArray would be an excellent choice as a superclass but it must be understood that #at: or #at:put refers to a byte, not a character. If you want characters, you have to start at the beginning and process it sequentially, like a stream (if working in the ASCII domain - you can generally 'cheat' this a bit). A C representation would be char utf8[] UTF16 is also a variable length encoding of two byte quantities - what C used to call a 'short int'. You process it in two byte chunks instead of one byte chunks. Like UTF8, you must read it sequentially to interpret the characters. #at and #at:put: would necessarily refer to byte pairs and not characters. A C representation would be short utf16[]; It would also to 50% space inefficient for ASCII - which is normally the bulk of your text. Realistically, you need exactly one in-memory format and stream readers/writers that can convert (these are typically table driven state machines). My choice would be UTF8 for the internal memory format and the ability to read and write from UTF8 to UTF16. But I stress again...strings don't really need indexability as much as you think and neither UTF8 nor UTF16 provide this property anyhow as they are variable length encodings. I don't see any sensible reason to have more than one in-memory binary format in the image. My $0.02c
I agree. :-)
Regarding UTF-16, I just want to be able to export to, and receive from, Windows (and any other platforms using UTF-16 as their native character representation).
Windows will always be able to accept UTF-16. All Windows apps *might well* export UTF-16. There may be other platforms which use UTF-16 as their native format. I'd just like to be able to cope with those situations. Nothing more.
All this is requires is a Utf16String class that has an asUtf8String method (and any other required conversion methods).
Hi Todd,
On Dec 11, 2015, at 12:57 PM, Todd Blanchard <tblanchard@mac.com> wrote:
On Dec 11, 2015, at 12:19, EuanM <euanmee@gmail.com> wrote:
"If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor." --Richard Sargent
Well, yes. But I think you guys are making this way too hard.
A unicode character is an abstract idea - for instance the letter 'a'. The letter 'a' has a code point - its the number 97. How the number 97 is represented in the computer is irrelevant.
Now we get to transfer encodings. These are UTF8, UTF16, etc.... A transfer encoding specifies the binary representation of the sequence of code points.
UTF8 is a variable length byte encoding. You read it one byte at a time, aggregating byte sequences to 'code points'. ByteArray would be an excellent choice as a superclass but it must be understood that #at: or #at:put refers to a byte, not a character. If you want characters, you have to start at the beginning and process it sequentially, like a stream (if working in the ASCII domain - you can generally 'cheat' this a bit). A C representation would be char utf8[]
UTF16 is also a variable length encoding of two byte quantities - what C used to call a 'short int'. You process it in two byte chunks instead of one byte chunks. Like UTF8, you must read it sequentially to interpret the characters. #at and #at:put: would necessarily refer to byte pairs and not characters. A C representation would be short utf16[]; It would also to 50% space inefficient for ASCII - which is normally the bulk of your text.
Realistically, you need exactly one in-memory format and stream readers/writers that can convert (these are typically table driven state machines). My choice would be UTF8 for the internal memory format and the ability to read and write from UTF8 to UTF16.
But I stress again...strings don't really need indexability as much as you think and neither UTF8 nor UTF16 provide this property anyhow as they are variable length encodings. I don't see any sensible reason to have more than one in-memory binary format in the image.
The only reasons are space and time. If a string only contains code points in the range 0-255 there's no point in squandering 4 bytes per code point (same goes for 0-65535). Further, if in some application interchange is more important than random access it may make sense in performance grounds to use utf-8 directly. Again, Smalltalk's dynamic typing makes it easy to have one's cake and eat it too.
My $0.02c
_,,,^..^,,,_ (phone)
I agree. :-)
Regarding UTF-16, I just want to be able to export to, and receive from, Windows (and any other platforms using UTF-16 as their native character representation).
Windows will always be able to accept UTF-16. All Windows apps *might well* export UTF-16. There may be other platforms which use UTF-16 as their native format. I'd just like to be able to cope with those situations. Nothing more.
All this is requires is a Utf16String class that has an asUtf8String method (and any other required conversion methods).
Elliot, what's your take on having heterogenous collections for the composed Unicode? i.e. collections with one element for each character, with some characters being themselves a collection of characters (Simple character like "a" is one char, and a character which is a collection of characters is the fully composed version of Ç (01d5), a U (0055) with a diaeresis - ¨ - ( 00a8 aka 0308 in combining form ) on top to form the compatibility character à (ooDC) which then gets a macron- Ì -( 0304) on top of that so a #(0061) Ç #(01d5) = #( 00dc 0304) = #( 0055 0308 0304) i.e a string which alternated those two characters 'aÇaÇaÇaÇ' would be represented by something equivalent to: #( 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) ) as opposed to a string of compatibility characters: #( 0061 01d5 0061 01d5 0061 01d5 0061 01d5) Does alternating the type used for characters in a string have a significant effect on speed? On 11 December 2015 at 23:08, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Todd,
On Dec 11, 2015, at 12:57 PM, Todd Blanchard <tblanchard@mac.com> wrote:
On Dec 11, 2015, at 12:19, EuanM <euanmee@gmail.com> wrote:
"If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor." --Richard Sargent
Well, yes. But I think you guys are making this way too hard.
A unicode character is an abstract idea - for instance the letter 'a'. The letter 'a' has a code point - its the number 97. How the number 97 is represented in the computer is irrelevant.
Now we get to transfer encodings. These are UTF8, UTF16, etc.... A transfer encoding specifies the binary representation of the sequence of code points.
UTF8 is a variable length byte encoding. You read it one byte at a time, aggregating byte sequences to 'code points'. ByteArray would be an excellent choice as a superclass but it must be understood that #at: or #at:put refers to a byte, not a character. If you want characters, you have to start at the beginning and process it sequentially, like a stream (if working in the ASCII domain - you can generally 'cheat' this a bit). A C representation would be char utf8[]
UTF16 is also a variable length encoding of two byte quantities - what C used to call a 'short int'. You process it in two byte chunks instead of one byte chunks. Like UTF8, you must read it sequentially to interpret the characters. #at and #at:put: would necessarily refer to byte pairs and not characters. A C representation would be short utf16[]; It would also to 50% space inefficient for ASCII - which is normally the bulk of your text.
Realistically, you need exactly one in-memory format and stream readers/writers that can convert (these are typically table driven state machines). My choice would be UTF8 for the internal memory format and the ability to read and write from UTF8 to UTF16.
But I stress again...strings don't really need indexability as much as you think and neither UTF8 nor UTF16 provide this property anyhow as they are variable length encodings. I don't see any sensible reason to have more than one in-memory binary format in the image.
The only reasons are space and time. If a string only contains code points in the range 0-255 there's no point in squandering 4 bytes per code point (same goes for 0-65535). Further, if in some application interchange is more important than random access it may make sense in performance grounds to use utf-8 directly.
Again, Smalltalk's dynamic typing makes it easy to have one's cake and eat it too.
My $0.02c
_,,,^..^,,,_ (phone)
I agree. :-)
Regarding UTF-16, I just want to be able to export to, and receive from, Windows (and any other platforms using UTF-16 as their native character representation).
Windows will always be able to accept UTF-16. All Windows apps *might well* export UTF-16. There may be other platforms which use UTF-16 as their native format. I'd just like to be able to cope with those situations. Nothing more.
All this is requires is a Utf16String class that has an asUtf8String method (and any other required conversion methods).
Hi Euan, On Fri, Dec 11, 2015 at 5:45 PM, EuanM <euanmee@gmail.com> wrote:
Elliot, what's your take on having heterogenous collections for the composed Unicode?
I'm not sure I'm understanding the question, but... I'm told by someone in the know that string concatenation is a big deal in certain applications, so providing tree-like representations for strings can be a win since concatenation is O(1) (allocate a new root and assign the two subtrees). It seems reasonable to have a rich library with several representations available with different trade-offs. But I'd let requirements drive design, not feature dreams.
i.e. collections with one element for each character, with some characters being themselves a collection of characters
(Simple character like "a" is one char, and a character which is a collection of characters is the fully composed version of Ç (01d5), a U (0055) with a diaeresis - ¨ - ( 00a8 aka 0308 in combining form ) on top to form the compatibility character à (ooDC) which then gets a macron- Ì -( 0304) on top of that
so a #(0061)
Ç #(01d5) = #( 00dc 0304) = #( 0055 0308 0304)
i.e a string which alternated those two characters
'aÇaÇaÇaÇ'
would be represented by something equivalent to:
#( 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) )
as opposed to a string of compatibility characters: #( 0061 01d5 0061 01d5 0061 01d5 0061 01d5)
Does alternating the type used for characters in a string have a significant effect on speed?
I honestly don't know. You've just gone well beyond my familiarity with the issues :-). I'm just a VM guy :-). But I will say that in cases like this, real applications and the profiler are your friends. Be guided by what you need now, not by what you think you'll need further down the road.
On 11 December 2015 at 23:08, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Todd,
On Dec 11, 2015, at 12:57 PM, Todd Blanchard <tblanchard@mac.com> wrote:
On Dec 11, 2015, at 12:19, EuanM <euanmee@gmail.com> wrote:
"If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor." --Richard Sargent
Well, yes. But I think you guys are making this way too hard.
A unicode character is an abstract idea - for instance the letter 'a'. The letter 'a' has a code point - its the number 97. How the number 97 is represented in the computer is irrelevant.
Now we get to transfer encodings. These are UTF8, UTF16, etc.... A transfer encoding specifies the binary representation of the sequence of code points.
UTF8 is a variable length byte encoding. You read it one byte at a time, aggregating byte sequences to 'code points'. ByteArray would be an excellent choice as a superclass but it must be understood that #at: or #at:put refers to a byte, not a character. If you want characters, you have to start at the beginning and process it sequentially, like a stream (if working in the ASCII domain - you can generally 'cheat' this a bit). A C representation would be char utf8[]
UTF16 is also a variable length encoding of two byte quantities - what C used to call a 'short int'. You process it in two byte chunks instead of one byte chunks. Like UTF8, you must read it sequentially to interpret the characters. #at and #at:put: would necessarily refer to byte pairs and not characters. A C representation would be short utf16[]; It would also to 50% space inefficient for ASCII - which is normally the bulk of your text.
Realistically, you need exactly one in-memory format and stream readers/writers that can convert (these are typically table driven state machines). My choice would be UTF8 for the internal memory format and the ability to read and write from UTF8 to UTF16.
But I stress again...strings don't really need indexability as much as you think and neither UTF8 nor UTF16 provide this property anyhow as they are variable length encodings. I don't see any sensible reason to have more than one in-memory binary format in the image.
The only reasons are space and time. If a string only contains code points in the range 0-255 there's no point in squandering 4 bytes per code point (same goes for 0-65535). Further, if in some application interchange is more important than random access it may make sense in performance grounds to use utf-8 directly.
Again, Smalltalk's dynamic typing makes it easy to have one's cake and eat it too.
My $0.02c
_,,,^..^,,,_ (phone)
I agree. :-)
Regarding UTF-16, I just want to be able to export to, and receive from, Windows (and any other platforms using UTF-16 as their native character representation).
Windows will always be able to accept UTF-16. All Windows apps *might well* export UTF-16. There may be other platforms which use UTF-16 as their native format. I'd just like to be able to cope with those situations. Nothing more.
All this is requires is a Utf16String class that has an asUtf8String method (and any other required conversion methods).
-- _,,,^..^,,,_ best, Eliot
Thanks for those pointers, Steph. I'll make sure they are on my reading list. (I have a limited weekly time-budget for Unicode work, but I expect this is a long-term project). I'll keep in touch with Steph, so any new facilities can be immediately useful to Pharo, and someone can guide them to a proper home in Pharo's Class hierarchy. For now, I've stuck stuff on my blog, http://smalltalkinsmallsteps.blogspot.co.uk/2015/12/utf-8-for-cuis-pharo-and... in an email here and at smalltalk.uk.to/unicode-utf.html On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
Steph - I'll dig out the Fr phone book ordering from wherever it was I read about it! I thought I ghad it to hand, but I haven;t found it tonight. It can't be far away. On 5 December 2015 at 13:08, stepharo <stepharo@free.fr> wrote:
Hi EuanM
Le 4/12/15 12:42, EuanM a écrit :
I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support.
this is great. Thanks for pushing this. I wrote and collected some roadmap (analyses on different topics) on the pharo github project feel free to add this one there.
This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry.
I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data.
Go! My suggestion is start small make steady progress write tests commit often :)
Stef
What is the french phoneBook ordering because this is the first time I hear about it.
Call to action: ==========
If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them.
If you would like to help out, please let me know.
If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know.
If you have comments or ideas on anything mentioned in this email
In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html
I have created a SqueakSource.com project called UnicodeSupport
I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm.
Current state of play =============== ByteString gets converted to WideString when need is automagically detected.
Is there anything else that currently exists?
Definition of Terms ============== A quick definition of terms before I go any further:
Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic.
Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints.
(I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints).
a composed character : a character made up of several codepoints
Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form.
The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not.
You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination.
Using compatibility characters can prevent proper sorting and equivalence testing.
Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems.
Currently my thinking is:
a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator.
Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one.
And that there is a Utf8SortableString which uses *only* normal characters.
We then need methods to convert between the two.
aUtf8String asUtf8SortableString
and
aUtf8SortableString asUtf8String
Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France)
Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods
A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc
along the lines of aListOfWords := SortedCollection sortBlock: deOrder
If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially.
To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint.
aUtf8String cleanUtf8String
With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.)
(I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know)
Cheers, Euan
participants (11)
-
Ben Coman -
Eliot Miranda -
EuanM -
Guillermo Polito -
H. Hirzel -
Henrik Johansen -
Max Leske -
Richard Sargent -
stepharo -
Sven Van Caekenberghe -
Todd Blanchard