Hi Henry, On 21 Oct 2013, at 11:18, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

On Oct 18, 2013, at 6:34 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

Hi,

So once again we have an issue with Character>>#leadingChar, see

https://pharo.fogbugz.com/f/cases/6368

Do we really need this ? Any Japanese, Chinese or Korean users willing to comment ?

Thx,

Sven

I'm not any of those, but my short answer would be no.

Thanks for the reply and explanation, it seems you confirm what I suspected.

As for the long answer: LeadingChar has too many responsibilities: - Character set of string - Font selection (see StrikeFontSet) - Han unification disambiguation (through the above font selection)

The conflation of these, and confusion of which leadingChar actually implies, easily leads to bugs, and has done so already. (see Character >> asUnicode as opposed to JapaneseEnvironment >> fromJISX0208String: for example). I would bet 100€ StrikeFontSet no longer works as intended either, that is, being able to display > latin1 glyphs using StrikeFonts.

Now, here's why I feel those areas are not worth keeping, especially in their current, buggy state: - Non-unicode character sets The main reasons for supporting this would be 1) Size reduction. All Widestrings are 32bits per character, so that's moot. 2) No need for converting codepoints when using Fonts stored with JISX0208 etc. codePoints . I've yet to see a free/truetype font using anything but Unicode, and since we'd be the creators of any theoretical StrikeFontSet covering other languages, we'd be able to avoid it anyways.

If, in the future, it'd be desirable to support encodings other than Unicode for internal strings, I feel separate subclasses are a cleaner solution.

- Font selection / Han unification disambiguation IMHO, obsoleted by the use of standard TrueType fonts. As long as one does not use StrikeFontSets to display a string, it currently has no benefits. Yes, one could potentially select different FreeTypeFonts based on it when a run is encountered as well, but the fonts themselves do not contain metadata pertaining to which variant of the glyphs they include, afaik (if they even support them; automatic fallback to another font when current font doesn't cover a glyph would be a higher area of priority) Even in that case, it could be a property of the current locale instead, while it means you can't display both korean/japanese text in the same image correctly, it'd be a (imho) acceptable tradeoff.

Although from what I understand about Han Unification and from the code, that last point would seem to be the case, but I am not 100% convinced. Most modern systems use Unicode everywhere, so they would all have that problem - a bit hard to believe. We need some feedback from native speakers as well. Sven

Cheers, Henry

On Oct 21, 2013, at 11:56 , Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

My guess is most if not all systems which do support that case, implement a higher level abstraction than "String" to take care of it.

… or not :) http://www.unicode.org/faq/vs.html http://www.unicode.org/reports/tr37/ Which means, you can solve the problems caused by Han Unification using standard Unicode. Seems like a lot of fun to implement support for :) Cheers, Henry

On Oct 21, 2013, at 2:13 , Igor Stasenko <siguctua@gmail.com> wrote:

On 21 October 2013 13:41, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

On Oct 21, 2013, at 11:56 , Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

My guess is most if not all systems which do support that case, implement a higher level abstraction than "String" to take care of it.

… or not :) http://www.unicode.org/faq/vs.html http://www.unicode.org/reports/tr37/

Which means, you can solve the problems caused by Han Unification using standard Unicode. Seems like a lot of fun to implement support for :)

Why we should care?

We define a string as a sequence of Characters. We say that every Character can be uniquely identified by its unicode value.

and we say nothing about things like locale, language etc.. because it is higher level concepts, e.g. things like mapping unicode value (or sequence of them) into sequence of glyphs to display on screen, using whatever font, is outside of scope of 'String' definition.

Cheers, Henry

The proposition was that leadingChar might be valuable on the grounds that Unicode doesn't allow you to differentiate Korean/Japanese characters in the same document. Variation sequences show Unicode has acquired a built-in mechanism for doing just that, so the proposition is false. The work that would be involved in implementing support for them in paths from User input -> String instance and String -> Glyph display is outside the definition of a String, sure, but work nonetheless. Maybe I should've put quotes around "fun". :P One would probably also need to deal with them if implementing Unicode functionality that arguably *is* within String scope btw, such as equality, collation and normalization. Just because Strings currently treat code point = character, doesn't make it 100% correct :) Cheers, Henry

Hi guys I would love to see some effort cleaning this part (removing the leading char) and using Unicode. It will simplify a lot from what I understand. Who would like to think about a roadmap and share some effort? Stef On Oct 21, 2013, at 11:37 AM, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

As an added bonus, asInteger / asUnicode / codePoint / charCode / asciiValue would all share the same definition; ^value :)

Cheers, Henry

P.S. codePoint is currently bugged, it should be ^self asUnicode I'd hardly say the leadingChar-tagged value in potentially different character sets it currently returns meets the ANSI definition of: "Return the encoding value of the receiver in the implementation defined execution character set."

On Oct 21, 2013, at 11:18 , Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On Oct 18, 2013, at 6:34 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

Hi,

So once again we have an issue with Character>>#leadingChar, see

https://pharo.fogbugz.com/f/cases/6368

Do we really need this ? Any Japanese, Chinese or Korean users willing to comment ?

Thx,

Sven

I'm not any of those, but my short answer would be no.

As for the long answer: LeadingChar has too many responsibilities: - Character set of string - Font selection (see StrikeFontSet) - Han unification disambiguation (through the above font selection)

The conflation of these, and confusion of which leadingChar actually implies, easily leads to bugs, and has done so already. (see Character >> asUnicode as opposed to JapaneseEnvironment >> fromJISX0208String: for example). I would bet 100€ StrikeFontSet no longer works as intended either, that is, being able to display > latin1 glyphs using StrikeFonts.

Now, here's why I feel those areas are not worth keeping, especially in their current, buggy state: - Non-unicode character sets The main reasons for supporting this would be 1) Size reduction. All Widestrings are 32bits per character, so that's moot. 2) No need for converting codepoints when using Fonts stored with JISX0208 etc. codePoints . I've yet to see a free/truetype font using anything but Unicode, and since we'd be the creators of any theoretical StrikeFontSet covering other languages, we'd be able to avoid it anyways.

If, in the future, it'd be desirable to support encodings other than Unicode for internal strings, I feel separate subclasses are a cleaner solution.

- Font selection / Han unification disambiguation IMHO, obsoleted by the use of standard TrueType fonts. As long as one does not use StrikeFontSets to display a string, it currently has no benefits. Yes, one could potentially select different FreeTypeFonts based on it when a run is encountered as well, but the fonts themselves do not contain metadata pertaining to which variant of the glyphs they include, afaik (if they even support them; automatic fallback to another font when current font doesn't cover a glyph would be a higher area of priority) Even in that case, it could be a property of the current locale instead, while it means you can't display both korean/japanese text in the same image correctly, it'd be a (imho) acceptable tradeoff.

Cheers, Henry

Phil, On 21 Oct 2013, at 21:09, "phil@highoctane.be" <phil@highoctane.be> wrote:

Well, for one, removing leadingChar will make the urlEncode/urlDecode work properly for the test, as now, the leadingChar thing prevents it.

Read my last comment in https://pharo.fogbugz.com/f/cases/6368/deprecate-unescapePercents-and-encode... I rewrote the test without the language environment stuff, it should pass.

As it is now, leadingChar generates a code point outside of the unicode range, making leadingChar a non supportable element.

Let's get that out.

Yes, longer term I think we should indeed remove it.

Phil

Sven

On 21 Oct 2013, at 21:33, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:

What would be the possible steps?

There are 23 implementors and 56 senders of #leadingChar ;-)

Stef

On Oct 21, 2013, at 9:26 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

Phil,

On 21 Oct 2013, at 21:09, "phil@highoctane.be" <phil@highoctane.be> wrote:

...

Well, for one, removing leadingChar will make the urlEncode/urlDecode work properly for the test, as now, the leadingChar thing prevents it.

Read my last comment in https://pharo.fogbugz.com/f/cases/6368/deprecate-unescapePercents-and-encode...

I rewrote the test without the language environment stuff, it should pass.

...

As it is now, leadingChar generates a code point outside of the unicode range, making leadingChar a non supportable element.

Let's get that out.

Yes, longer term I think we should indeed remove it.

...

Phil

Sven