November 2010 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] TextConverter handling of binary streams

by StÃ©phane Ducasse

fix fix fix :) Stef > I would even go further, the test UTF8TextConverterTest>>#testPutSingleCharacter is plain wrong, it seems to have been written as an afterthought: > > testPutSingleCharacter > | actual | > actual := ByteArray streamContents: [ :stream | > | converter | > converter := UTF8TextConverter new. > converter > nextPut: $a > toStream: stream. > converter > nextPut: (Unicode value: 16r20AC) > toStream: stream ]. > self assert: actual = #[97 0 0 32 172] > > The correct result is #[97 226 130 172] !! > > If you would give these bytes to any other system, they would never be able to decode them as UTF-8.

Nov. 30, 2010

Re: [Pharo-project] TextConverter handling of binary streams

by StÃ©phane Ducasse

On Nov 30, 2010, at 11:46 AM, Sven Van Caekenberghe wrote: > I understand very well that you are very busy, StÃ©phane, I am not asking for a solution. :) but I would love to have time :) > > It is exactly the same in Squeak. > > Like Philippe said, it looks wrong. so open a ticket > > On 29 Nov 2010, at 20:12, StÃ©phane Ducasse wrote: > >> sven >> >> I'm terribly and more than that busy until mid or more dec. >> Now did you check if the behavior is the same in squeak? >> S. >> >> On Nov 29, 2010, at 3:23 PM, Sven Van Caekenberghe wrote: >> >>> Hi, >>> >>> TextConverter and its subclasses seem to break the contract of #nextFromStream: and #nextPut:toStream: when the stream #isBinary. Consider the following two examples: >>> >>> ByteArray streamContents: [ :stream | | encoder | >>> encoder := UTF8TextConverter new. >>> 'Ã©lÃ¨ve en franÃ§ais' do: [ :each | encoder nextPut: each toStream: stream ] ]. >>> >>> #[233 108 232 118 101 32 101 110 32 102 114 97 110 231 97 105 115] >>> >>> (String streamContents: [ :stream | | encoder | >>> encoder := UTF8TextConverter new. >>> 'Ã©lÃ¨ve en franÃ§ais' do: [ :each | encoder nextPut: each toStream: stream ] ]) asByteArray. >>> >>> #[195 169 108 195 168 118 101 32 101 110 32 102 114 97 110 195 167 97 105 115] >>> >>> The first answer is incorrect, the second is correct (as far as I understand it). >>> >>> This is apparently on purpose, from the implementation of, for example, UTF8TextConverter>>#nextPut:toStream: >>> >>> nextPut: aCharacter toStream: aStream >>> | leadingChar nBytes mask shift ucs2code | >>> aStream isBinary ifTrue: [^aCharacter storeBinaryOn: aStream]. >>> leadingChar := aCharacter leadingChar. >>> (leadingChar = 0 and: [aCharacter asciiValue < 128]) ifTrue: [ >>> aStream basicNextPut: aCharacter. >>> ^ aStream. >>> ]. >>> >>> "leadingChar > 3 ifTrue: [^ aStream]." >>> >>> ucs2code := aCharacter asUnicode. >>> ucs2code ifNil: [^ aStream]. >>> >>> nBytes := ucs2code highBit + 3 // 5. >>> mask := #(128 192 224 240 248 252 254 255) at: nBytes. >>> shift := nBytes - 1 * -6. >>> aStream basicNextPut: (Character value: (ucs2code bitShift: shift) + mask). >>> 2 to: nBytes do: [:i | >>> shift := shift + 6. >>> aStream basicNextPut: (Character value: ((ucs2code bitShift: shift) bitAnd: 63) + 128). >>> ]. >>> >>> ^ aStream. >>> >>> I would say that the contract of #nextPut:toStream: is to take a Character object and write a binary representation using a specific encoding to a stream. However, when given a #isBinary stream, it does no longer do any encoding at all ! >>> >>> The same is true for the other converters as well as for #nextFromStream:. >>> >>> Does anyone know why that is the case ? >>> >>> And if it is by design, how should one do UTF8 encoding on a binary stream ?? >>> >>> Thx, >>> >>> Sven >>> >>> PS: If others also think this is strange, I could make an issue, I am just not sure this is a bug. >>> >>> >>> >> >> > >

Nov. 30, 2010

Re: [Pharo-project] [ANN] Hudson build server for Pharo

by Luc Fabresse

Thanks Marcus! Really great news! #Luc 2010/11/30 Tudor Girba <tudor.girba(a)gmail.com> > Hi, > > Here is the apache2 configuration we use for the > hudson.moosetechnology.org server. Perhaps it is useful: > > <VirtualHost *> > ServerName hudson.moosetechnology.org > > DocumentRoot /srv/hudson.moosetechnology.org/web > > <Location /> > Order deny,allow > Allow from all > </Location> > > RewriteEngine On > RewriteRule ^/(.*)$ http://localhost:XXXX/$1 [proxy,last] > ProxyPassReverse / http://localhost:XXXX/ > ProxyRequests Off > > <Proxy http://localhost:XXXX/*> > Order deny,allow > Allow from all > </Proxy> > </VirtualHost> > > Doru > > On 30 Nov 2010, at 09:37, Marcus Denker wrote: > > > > > On Nov 29, 2010, at 11:08 PM, Miguel CobÃ¡ wrote: > > > >> > >> One question, you have the pharo-project.org domain, why isn't this > >> hudson a subdomain of that domain. Hosted on inria, but accessible by > >> the official project domain? > >> > > One step at a time :-) > > > > > >> Anyway, very good news. I suppose that the download page of pharo will > >> point to the latest successful build link of the hudson server very > >> soon. > > > > > > Yes... that's the goal. The artefact that people download should be > generated completely > > automatically. (This way we can move much faster). > > > > Marcus > > > > > > -- > > Marcus Denker -- http://www.marcusdenker.de > > INRIA Lille -- Nord Europe. Team RMoD. > > > > > > -- > www.tudorgirba.com > > "What is more important: To be happy, or to make happy?" > > >

Nov. 30, 2010

Re: [Pharo-project] Issue 3291 in pharo: Missing text field with completion

by pharo＠googlecode.com

Comment #5 on issue 3291 by aplantec: Missing text field with completion http://code.google.com/p/pharo/issues/detail?id=3291 In PharoInbox SLICE-Issue-3291-Missing-text-field-with-completion

Nov. 30, 2010

Re: [Pharo-project] TextConverter handling of binary streams

by Sven Van Caekenberghe

Philippe, I would even go further, the test UTF8TextConverterTest>>#testPutSingleCharacter is plain wrong, it seems to have been written as an afterthought: testPutSingleCharacter | actual | actual := ByteArray streamContents: [ :stream | | converter | converter := UTF8TextConverter new. converter nextPut: $a toStream: stream. converter nextPut: (Unicode value: 16r20AC) toStream: stream ]. self assert: actual = #[97 0 0 32 172] The correct result is #[97 226 130 172] !! If you would give these bytes to any other system, they would never be able to decode them as UTF-8. Sven On 30 Nov 2010, at 08:46, Philippe Marschall wrote: > On 11/29/2010 03:23 PM, Sven Van Caekenberghe wrote: >> Hi, >> >> TextConverter and its subclasses seem to break the contract of #nextFromStream: and #nextPut:toStream: when the stream #isBinary. Consider the following two examples: >> >> ByteArray streamContents: [ :stream | | encoder | >> encoder := UTF8TextConverter new. >> 'Ã©lÃ¨ve en franÃ§ais' do: [ :each | encoder nextPut: each toStream: stream ] ]. >> >> #[233 108 232 118 101 32 101 110 32 102 114 97 110 231 97 105 115] >> >> (String streamContents: [ :stream | | encoder | >> encoder := UTF8TextConverter new. >> 'Ã©lÃ¨ve en franÃ§ais' do: [ :each | encoder nextPut: each toStream: stream ] ]) asByteArray. >> >> #[195 169 108 195 168 118 101 32 101 110 32 102 114 97 110 195 167 97 105 115] >> >> The first answer is incorrect, the second is correct (as far as I understand it). > > I would agree. > >> This is apparently on purpose, from the implementation of, for example, UTF8TextConverter>>#nextPut:toStream: >> >> nextPut: aCharacter toStream: aStream >> | leadingChar nBytes mask shift ucs2code | >> aStream isBinary ifTrue: [^aCharacter storeBinaryOn: aStream]. >> leadingChar := aCharacter leadingChar. >> (leadingChar = 0 and: [aCharacter asciiValue < 128]) ifTrue: [ >> aStream basicNextPut: aCharacter. >> ^ aStream. >> ]. >> >> "leadingChar > 3 ifTrue: [^ aStream]." >> >> ucs2code := aCharacter asUnicode. >> ucs2code ifNil: [^ aStream]. >> >> nBytes := ucs2code highBit + 3 // 5. >> mask := #(128 192 224 240 248 252 254 255) at: nBytes. >> shift := nBytes - 1 * -6. >> aStream basicNextPut: (Character value: (ucs2code bitShift: shift) + mask). >> 2 to: nBytes do: [:i | >> shift := shift + 6. >> aStream basicNextPut: (Character value: ((ucs2code bitShift: shift) bitAnd: 63) + 128). >> ]. >> >> ^ aStream. >> >> I would say that the contract of #nextPut:toStream: is to take a Character object and write a binary representation using a specific encoding to a stream. However, when given a #isBinary stream, it does no longer do any encoding at all ! > > There are many cases where Squeak/Pharo do silent coercion, where > obviously something went wrong (you pass a ByteArray where a String is > expected or vice versa) and the system tries to be clever about it and > do something else than raising an exception and forcing you to fix your > code. The result is often wrong because the system can not guess what > you wanted to do. > >> The same is true for the other converters as well as for #nextFromStream:. >> >> Does anyone know why that is the case ? >> >> And if it is by design, how should one do UTF8 encoding on a binary stream ?? >> >> Thx, >> >> Sven >> >> PS: If others also think this is strange, I could make an issue, I am just not sure this is a bug. > > I find that strange as well. > > Cheers > Philippe

Nov. 30, 2010

Re: [Pharo-project] TextConverter handling of binary streams

by Sven Van Caekenberghe

I understand very well that you are very busy, StÃ©phane, I am not asking for a solution. It is exactly the same in Squeak. Like Philippe said, it looks wrong. On 29 Nov 2010, at 20:12, StÃ©phane Ducasse wrote: > sven > > I'm terribly and more than that busy until mid or more dec. > Now did you check if the behavior is the same in squeak? > S. > > On Nov 29, 2010, at 3:23 PM, Sven Van Caekenberghe wrote: > >> Hi, >> >> TextConverter and its subclasses seem to break the contract of #nextFromStream: and #nextPut:toStream: when the stream #isBinary. Consider the following two examples: >> >> ByteArray streamContents: [ :stream | | encoder | >> encoder := UTF8TextConverter new. >> 'Ã©lÃ¨ve en franÃ§ais' do: [ :each | encoder nextPut: each toStream: stream ] ]. >> >> #[233 108 232 118 101 32 101 110 32 102 114 97 110 231 97 105 115] >> >> (String streamContents: [ :stream | | encoder | >> encoder := UTF8TextConverter new. >> 'Ã©lÃ¨ve en franÃ§ais' do: [ :each | encoder nextPut: each toStream: stream ] ]) asByteArray. >> >> #[195 169 108 195 168 118 101 32 101 110 32 102 114 97 110 195 167 97 105 115] >> >> The first answer is incorrect, the second is correct (as far as I understand it). >> >> This is apparently on purpose, from the implementation of, for example, UTF8TextConverter>>#nextPut:toStream: >> >> nextPut: aCharacter toStream: aStream >> | leadingChar nBytes mask shift ucs2code | >> aStream isBinary ifTrue: [^aCharacter storeBinaryOn: aStream]. >> leadingChar := aCharacter leadingChar. >> (leadingChar = 0 and: [aCharacter asciiValue < 128]) ifTrue: [ >> aStream basicNextPut: aCharacter. >> ^ aStream. >> ]. >> >> "leadingChar > 3 ifTrue: [^ aStream]." >> >> ucs2code := aCharacter asUnicode. >> ucs2code ifNil: [^ aStream]. >> >> nBytes := ucs2code highBit + 3 // 5. >> mask := #(128 192 224 240 248 252 254 255) at: nBytes. >> shift := nBytes - 1 * -6. >> aStream basicNextPut: (Character value: (ucs2code bitShift: shift) + mask). >> 2 to: nBytes do: [:i | >> shift := shift + 6. >> aStream basicNextPut: (Character value: ((ucs2code bitShift: shift) bitAnd: 63) + 128). >> ]. >> >> ^ aStream. >> >> I would say that the contract of #nextPut:toStream: is to take a Character object and write a binary representation using a specific encoding to a stream. However, when given a #isBinary stream, it does no longer do any encoding at all ! >> >> The same is true for the other converters as well as for #nextFromStream:. >> >> Does anyone know why that is the case ? >> >> And if it is by design, how should one do UTF8 encoding on a binary stream ?? >> >> Thx, >> >> Sven >> >> PS: If others also think this is strange, I could make an issue, I am just not sure this is a bug. >> >> >> > >

Nov. 30, 2010

Re: [Pharo-project] [update 1.2] #12264

by StÃ©phane Ducasse

this is cool! > 12264 > ----- > > Issue 3359: categorize some methods > Issue 3358: Another fix for SourceReference > Issue 3351: Comments in MIMEDocument erroneously refer to MIMEObject > Issue 3344: No brainer fix for hasBindingThatBeginWith: > > ... and hudson auto-build was triggered successfully. > > https://pharo-ic.lille.inria.fr/hudson/job/Pharo%20Core%201.2/ > > -- > Marcus Denker -- http://www.marcusdenker.de > INRIA Lille -- Nord Europe. Team RMoD. > >

Nov. 30, 2010

Re: [Pharo-project] NewInspector does not work well with delegated collections

by StÃ©phane Ducasse

what we learned in the story is that the newinspector is too intrusive and that we should make sure that we limit the api to be used when building introspective tools. This is also why we will remove it from the next pharo-dev Stef On Nov 29, 2010, at 9:38 PM, jaayer wrote: > Enclosed is a fileout of a sample collection named DelegatedSet that works by delegating to an instance of Set stored in an instance variable. It implements the basic protocol Collection requires, including #add:, #remove:ifAbsent: and #do: as well as some additional messages, all through delegation. Create an instance of it and then inspect that instance with NewInsepctor, and you will notice the "Elements" item does not work. Nothing happens when you click it. To inspect the elements, you need to click "set," the instance variable containing the Set, and then inspect that object further. Apparently, unless you override one or more messages under the extension category *Newinspector, NewInspector will not know how to inspect your collection properly. Worse, as soon as you do implement those methods, unless you change their category, they will now be considered part of the *Newinspector package rather than the package your collection belongs to and will not be saved along with the collection. > > I think at a minimum, any collection understanding #do: and #size should be inspectable by NewInspector.<DelegatedSet.st>

Nov. 30, 2010

Re: [Pharo-project] Issue 3344 in pharo: No brainer fix for hasBindingThatBeginWith:

by pharo＠googlecode.com

Updates: Status: Closed Comment #1 on issue 3344 by marcus.denker: No brainer fix for hasBindingThatBeginWith: http://code.google.com/p/pharo/issues/detail?id=3344 12264

Nov. 30, 2010

[Pharo-project] [update 1.2] #12264

by Marcus Denker

12264 ----- Issue 3359: categorize some methods Issue 3358: Another fix for SourceReference Issue 3351: Comments in MIMEDocument erroneously refer to MIMEObject Issue 3344: No brainer fix for hasBindingThatBeginWith: ... and hudson auto-build was triggered successfully. https://pharo-ic.lille.inria.fr/hudson/job/Pharo%20Core%201.2/ -- Marcus Denker -- http://www.marcusdenker.de INRIA Lille -- Nord Europe. Team RMoD.

Nov. 30, 2010