Elliot, what's your take on having heterogenous collections for the composed Unicode? i.e. collections with one element for each character, with some characters being themselves a collection of characters (Simple character like "a" is one char, and a character which is a collection of characters is the fully composed version of Ç (01d5), a U (0055) with a diaeresis - ¨ - ( 00a8 aka 0308 in combining form ) on top to form the compatibility character à (ooDC) which then gets a macron- Ì -( 0304) on top of that so a #(0061) Ç #(01d5) = #( 00dc 0304) = #( 0055 0308 0304) i.e a string which alternated those two characters 'aÇaÇaÇaÇ' would be represented by something equivalent to: #( 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) 0061 #( 0055 0308 0304) ) as opposed to a string of compatibility characters: #( 0061 01d5 0061 01d5 0061 01d5 0061 01d5) Does alternating the type used for characters in a string have a significant effect on speed? On 11 December 2015 at 23:08, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Todd,
On Dec 11, 2015, at 12:57 PM, Todd Blanchard <tblanchard@mac.com> wrote:
On Dec 11, 2015, at 12:19, EuanM <euanmee@gmail.com> wrote:
"If it hasn't already been said, please do not conflate Unicode and UTF-8. I think that would be a recipe for a high P.I.T.A. factor." --Richard Sargent
Well, yes. But I think you guys are making this way too hard.
A unicode character is an abstract idea - for instance the letter 'a'. The letter 'a' has a code point - its the number 97. How the number 97 is represented in the computer is irrelevant.
Now we get to transfer encodings. These are UTF8, UTF16, etc.... A transfer encoding specifies the binary representation of the sequence of code points.
UTF8 is a variable length byte encoding. You read it one byte at a time, aggregating byte sequences to 'code points'. ByteArray would be an excellent choice as a superclass but it must be understood that #at: or #at:put refers to a byte, not a character. If you want characters, you have to start at the beginning and process it sequentially, like a stream (if working in the ASCII domain - you can generally 'cheat' this a bit). A C representation would be char utf8[]
UTF16 is also a variable length encoding of two byte quantities - what C used to call a 'short int'. You process it in two byte chunks instead of one byte chunks. Like UTF8, you must read it sequentially to interpret the characters. #at and #at:put: would necessarily refer to byte pairs and not characters. A C representation would be short utf16[]; It would also to 50% space inefficient for ASCII - which is normally the bulk of your text.
Realistically, you need exactly one in-memory format and stream readers/writers that can convert (these are typically table driven state machines). My choice would be UTF8 for the internal memory format and the ability to read and write from UTF8 to UTF16.
But I stress again...strings don't really need indexability as much as you think and neither UTF8 nor UTF16 provide this property anyhow as they are variable length encodings. I don't see any sensible reason to have more than one in-memory binary format in the image.
The only reasons are space and time. If a string only contains code points in the range 0-255 there's no point in squandering 4 bytes per code point (same goes for 0-65535). Further, if in some application interchange is more important than random access it may make sense in performance grounds to use utf-8 directly.
Again, Smalltalk's dynamic typing makes it easy to have one's cake and eat it too.
My $0.02c
_,,,^..^,,,_ (phone)
I agree. :-)
Regarding UTF-16, I just want to be able to export to, and receive from, Windows (and any other platforms using UTF-16 as their native character representation).
Windows will always be able to accept UTF-16. All Windows apps *might well* export UTF-16. There may be other platforms which use UTF-16 as their native format. I'd just like to be able to cope with those situations. Nothing more.
All this is requires is a Utf16String class that has an asUtf8String method (and any other required conversion methods).