Elliot, what's your take on having heterogenous collections for the
composed Unicode?

i.e. collections with one element for each character, with some
characters being themselves a collection of characters

(Simple character like "a" is one char, and a character which is a
collection of characters is the fully composed version of �� (01d5), a
U (0055) with a diaeresis - �� - ( 00a8�� aka 0308 in combining form )
on top to form the compatibility character �� (ooDC) which then gets a
macron-�� ���� -( 0304) on top of that

so
a�� #(0061)

���� #(01d5) = #( 00dc 0304) = #( 0055 0308 0304)

i.e a string which alternated those two characters

'a��a��a��a��'

would be represented by something equivalent to:

#( 0061 #( 0055 0308 0304)�� 0061 #( 0055 0308 0304)�� 0061 #( 0055 0308
0304)�� 0061 #( 0055 0308 0304) )

as opposed to a string of compatibility characters:
#( 0061 01d5�� 0061 01d5 0061 01d5 0061 01d5)

Does alternating the type used for characters in a string have a
significant effect on speed?

On 11 December 2015 at 23:08, Eliot Miranda <eliot.miranda@gmail.com> wrote:
> Hi Todd,
>
> On Dec 11, 2015, at 12:57 PM, Todd Blanchard <tblanchard@mac.com> wrote:
>
>
> On Dec 11, 2015, at 12:19, EuanM <euanmee@gmail.com> wrote:
>
> "If it hasn't already been said, please do not conflate Unicode and
> UTF-8. I think that would be a recipe for
> a high P.I.T.A. factor."�� --Richard Sargent
>
>
> Well, yes. But�� I think you guys are making this way too hard.
>
> A unicode character is an abstract idea - for instance the letter 'a'.
> The letter 'a' has a code point - its the number 97.�� How the number 97 is
> represented in the computer is irrelevant.
>
> Now we get to transfer encodings.�� These are UTF8, UTF16, etc....�� A
> transfer encoding specifies the binary representation of the sequence of
> code points.
>
> UTF8 is a variable length byte encoding.�� You read it one byte at a time,
> aggregating byte sequences to 'code points'.�� ByteArray would be an
> excellent choice as a superclass but it must be understood that #at: or
> #at:put refers to a byte, not a character.�� If you want characters, you have
> to start at the beginning and process it sequentially, like a stream (if
> working in the ASCII domain - you can generally 'cheat' this a bit).�� A C
> representation would be char utf8[]
>
> UTF16 is also a variable length encoding of two byte quantities - what C
> used to call a 'short int'.�� You process it in two byte chunks instead of
> one byte chunks.�� Like UTF8, you must read it sequentially to interpret the
> characters.�� #at and #at:put: would necessarily refer to byte pairs and not
> characters.�� A C representation would be short utf16[];�� It would also to
> 50% space inefficient for ASCII - which is normally the bulk of your text.
>
> Realistically, you need exactly one in-memory format and stream
> readers/writers that can convert (these are typically table driven state
> machines).�� My choice would be UTF8 for the internal memory format and the
> ability to read and write from UTF8 to UTF16.
>
> But I stress again...strings don't really need indexability as much as you
> think and neither UTF8 nor UTF16 provide this property anyhow as they are
> variable length encodings.�� I don't see any sensible reason to have more
> than one in-memory binary format in the image.
>
>
> The only reasons are space and time.�� If a string only contains code points
> in the range 0-255 there's no point in squandering 4 bytes per code point
> (same goes for 0-65535).�� Further, if in some application interchange is
> more important than random access it may make sense in performance grounds
> to use utf-8 directly.
>
> Again, Smalltalk's dynamic typing makes it easy to have one's cake and eat
> it too.
>
> My $0.02c
>
>
> _,,,^..^,,,_ (phone)
>
>
> I agree. :-)
>
> Regarding UTF-16, I just want to be able to export to, and receive
> from, Windows (and any other platforms using UTF-16 as their native
> character representation).
>
> Windows will always be able to accept UTF-16.�� All Windows apps *might
> well* export UTF-16.�� There may be other platforms which use UTF-16 as
> their native format.�� I'd just like to be able to cope with those
> situations.�� Nothing more.
>
> All this is requires is a Utf16String class that has an asUtf8String
> method (and any other required conversion methods).
>
>