June 2012 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Fastest utf-8 encoder contest

by Henrik Sperre Johansen

On 13.06.2012 14:59, Igor Stasenko wrote: > On 13 June 2012 10:31, Philippe Marschall > <philippe.marschall(a)netcetera.ch> wrote: >> On 06/13/2012 04:44 AM, Igor Stasenko wrote: >>> Hi, hardcore hackers. >>> please take a look at the code and tell if it can be improved. >>> >>> The AsmJit snippet below transforms an unicode integer value >>> to 1..4-byte sequence of utf-8 >>> >>> then the outer piece of code (which is not yet written) will >>> accumulate the results of this snippet >>> to do a memory-aligned (4byte) writes.. >>> like that, if 4 unicode characters can be encoded into 4 utf-8 bytes >>> (which mostly the case for latin-1 char range), then there will be >>> 4 memory reads (to read four 32-bit unicode values) but only single >>> memory write (to write four 8-bit utf-8 encoded values). >>> >>> The idea is to make utf-8 encoding speed close to memory copying speed :) >> In Seaside we use an other trick that Andreas Raab come up with. The >> assumption is that most of the strings are ASCII [1]. We use a CharSet / >> bitmap to quickly scan the string for the index of the first non-ASCII >> character. If we find none we just answer the argument. No copying at all. >> > Well, in my case i will need copying because i need to null-terminate it, > to represent it as null-terminated string. > This is what cairo library expects as input for rendering text. > And this also means that i can use a single buffer for conversions to > avoid generating garbage, i.e. > i take input string, convert it to utf8 in private buffer, then pass > that buffer as input to external call, > on next call an input can be any other string, but output will be the > same private buffer. > I will be needing to allocate new buffer if incoming string does not > fits into it. So, is this a one-off for Cairo, or are you planning on introducing a <var: #myStringParameter encoding: #utfXX> pragma or something to NBFFI? Because, on Windows system calls either expect the local system code page, or utf16 (depending on which version of the API you use), while the in other libraries the expected encoding may be different from the one of the system libraries. Heck, the same lib may expect different encodings, case in point: The expected encoding of the Oracle client lib (and by extension DBXTalk*) depends on either an environment variable, what you told it to expect using an API function, or one of many other fallbacks I can't remember at the moment. Cheers, Henry *<rant> OpenDBX makes an explicity choice to simply pass the data you hand it over to whatever backend you have hooked up raw. Very nice of a library that does a good job of abstracting the actual API of communicating with a database to leave it up to you to decide which encoding to use (which depends on the abstracted away backend...) if you want your data stored correctly... There is a reason those benchmarks are fast! :) </rant>

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by petton.nicolas＠gmail.com

StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> writes: > If one group of guys would take the lead we could introduce XTream in > 2.0 and slowly start to integrate and deprecate slowly the old > streams. Who is interested in that? Me. Nico > > Stef > >>>>> Tx >>>>> >>>>> now there is a difference between adjustments and fully new API and I >>>>> would have preferred adjustments. >>>>> >>>>> Stef >>>> >>>> IMO, Xtreams design is simply superior. >>> >>> +100. >>> I gave up on trying to mold the current Streams into something a bit more >>> reasonable and still be backwards compatible quite some time ago. >>> >>> That other thread about Converters? >>> Totally related. For example, you can't do what Sven did in the Zn >>> converters without breaking backwards compatability. Nor could you do it >>> reasonably fast in the general case, since fast string -> bytearray >>> streaming works for some streams, but not for others. (which, I guess, is >>> one reason why the legacy converters do string -> string.) >>> >>> IMHO, Xtreams would be a perfect fit to go with the new FileSystem as well. >>> (using compositing on top of a "dumb" terminal) >>> >> >> Quite sure that it is highly ranked in Colin's plans. >> >>>> But i agree, we cannot throw existing streams out.. too much things >>>> tied to them, and rely >>>> on using stream protocols. :( >>> >>> Well, one*could* try to introduce something akin to the Zn HTTPClient >>> compatability protocol, strictly speaking what you can do with Xtreams is a >>> superset of what Streams currently do. >>> The problem is all the moronic quirks of the current Streams, and clients >>> potentially relying on that behaviour. >>> >>> Cheers, >>> Henry >>> >>> >>> >> > > -- Nicolas Petton http://nicolas-petton.fr

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by StÃ©phane Ducasse

I would also really like to have a chapter based on the documentation (if any). > If one group of guys would take the lead we could introduce XTream in 2.0 and slowly start to integrate and deprecate slowly the old streams. Who is interested in that?

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by StÃ©phane Ducasse

>> > > Indeed, with the new FS being almost ready in 2.0 the next big mess to > tackle are the streams. > > Personally I really liked the clean separation of the low level stream > features (bytes) and the high-level interfaces to it (converters, characters...) > > but the question remains on how to integrate them into a running system ;) > > after having done the FS take-over I see that it's rather easy to create > an image that works with the replaced classes. But creating a nice loadable > slice / changeset is a rather fragile and tedious task :P We can just have dedicated sprints. I think that tomorrow I would like to finish with you the FS part. Then I would not open the Xtreams work now but plan it. Stef

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by Camillo Bruni

>>> >> +100. >> I gave up on trying to mold the current Streams into something a bit more >> reasonable and still be backwards compatible quite some time ago. >> >> That other thread about Converters? >> Totally related. For example, you can't do what Sven did in the Zn >> converters without breaking backwards compatability. Nor could you do it >> reasonably fast in the general case, since fast string -> bytearray >> streaming works for some streams, but not for others. (which, I guess, is >> one reason why the legacy converters do string -> string.) >> >> IMHO, Xtreams would be a perfect fit to go with the new FileSystem as well. >> (using compositing on top of a "dumb" terminal) >> > > Quite sure that it is highly ranked in Colin's plans. Indeed, with the new FS being almost ready in 2.0 the next big mess to tackle are the streams. Personally I really liked the clean separation of the low level stream features (bytes) and the high-level interfaces to it (converters, characters...) but the question remains on how to integrate them into a running system ;) after having done the FS take-over I see that it's rather easy to create an image that works with the replaced classes. But creating a nice loadable slice / changeset is a rather fragile and tedious task :P

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by StÃ©phane Ducasse

If one group of guys would take the lead we could introduce XTream in 2.0 and slowly start to integrate and deprecate slowly the old streams. Who is interested in that? Stef >>>> Tx >>>> >>>> now there is a difference between adjustments and fully new API and I >>>> would have preferred adjustments. >>>> >>>> Stef >>> >>> IMO, Xtreams design is simply superior. >> >> +100. >> I gave up on trying to mold the current Streams into something a bit more >> reasonable and still be backwards compatible quite some time ago. >> >> That other thread about Converters? >> Totally related. For example, you can't do what Sven did in the Zn >> converters without breaking backwards compatability. Nor could you do it >> reasonably fast in the general case, since fast string -> bytearray >> streaming works for some streams, but not for others. (which, I guess, is >> one reason why the legacy converters do string -> string.) >> >> IMHO, Xtreams would be a perfect fit to go with the new FileSystem as well. >> (using compositing on top of a "dumb" terminal) >> > > Quite sure that it is highly ranked in Colin's plans. > >>> But i agree, we cannot throw existing streams out.. too much things >>> tied to them, and rely >>> on using stream protocols. :( >> >> Well, one*could* try to introduce something akin to the Zn HTTPClient >> compatability protocol, strictly speaking what you can do with Xtreams is a >> superset of what Streams currently do. >> The problem is all the moronic quirks of the current Streams, and clients >> potentially relying on that behaviour. >> >> Cheers, >> Henry >> >> >> >

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by Nicolas Cellier

2012/6/14 Henrik Sperre Johansen <henrik.s.johansen(a)veloxit.no>: > On 14.06.2012 08:43, Igor Stasenko wrote: >> >> On 14 June 2012 08:33, StÃ©phane Ducasse<stephane.ducasse(a)inria.fr> Â wrote: >>> >>> Tx >>> >>> now there is a difference between adjustments and fully new API and I >>> would have preferred adjustments. >>> >>> Stef >> >> IMO, Xtreams design is simply superior. > > +100. > I gave up on trying to mold the current Streams into something a bit more > reasonable and still be backwards compatible quite some time ago. > > That other thread about Converters? > Totally related. For example, you can't do what Sven did in the Zn > converters without breaking backwards compatability. Nor could you do it > reasonably fast in the general case, since fast string -> bytearray > streaming works for some streams, but not for others. (which, I guess, is > one reason why the legacy converters do string -> string.) > > IMHO, Xtreams would be a perfect fit to go with the new FileSystem as well. > (using compositing on top of a "dumb" terminal) > Quite sure that it is highly ranked in Colin's plans. >> But i agree, we cannot throw existing streams out.. too much things >> tied to them, and rely >> on using stream protocols. :( > > Well, one*could* try to introduce something akin to the Zn HTTPClient > compatability protocol, strictly speaking what you can do with Xtreams is a > superset of what Streams currently do. > The problem is all the moronic quirks of the current Streams, and clients > potentially relying on that behaviour. > > Cheers, > Henry > > >

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by Nicolas Cellier

Here is a bit less dumb evaluation of stream protocols, top 100 Stream-messages sent in Pharo2.0 but by Stream: selectors := (Stream withAllSubclasses inject: Set new into: [:allSelectors :aClass | allSelectors addAll: aClass selectors; yourself]) sorted. references := selectors collect: [:e | e -> ((SystemNavigation default allCallsOn: e) reject: [:aRef | (Smalltalk at: (aRef classSymbol readStream upTo: Character space) asSymbol) inheritsFrom: Stream]) size ]. (references sorted: [:a :b | a value > b value ]) first: 100. {#=->7340. #size->3817. #do:->2612. #name->1366. #isEmpty->1345. #nextPutAll:->991. #nextPut:->887. #value:->792. #contents->741. #last->679. #cr->666. #initialize->634. #on:->558. #with:->547. #readStream->497. #position->314. #close->275. #white->266. #space->256. #next->245. #black->244. #printOn:->226. #print:->215. #position:->184. #string->175. #lf->171. #atEnd->165. #hash->161. #reset->156. #tab->130. #text->123. #'<<'->120. #open->112. #match:->106. #show:->97. #crlf->93. #binary->91. #confirm:->84. #timeStamp->83. #next:->80. #element->74. #fullName->68. #upToEnd->66. #removeAll->65. #peek->64. #flush->64. #skip:->63. #findFirst:->61. #postCopy->55. #replace:with:->52. #nextChunkPut:->52. #braceArray->52. #isDirectory->49. #fork->49. #url->46. #upTo:->41. #isBinary->40. #nextLine->39. #header->38. #sendCommand:->34. #release->33. #request:->32. #nextChunk->30. #uri->29. #store:->29. #directory->28. #peekFor:->26. #setToEnd->24. #crtab:->24. #trailer->23. #nextChunkText->23. #skipSeparators->21. #asUrl->20. #initialize:->19. #upToAll:->18. #textStyle->18. #unregister->16. #readOnlyCopy->16. #localName->16. #register->15. #nextOrNil->15. #fileIn->15. #objectAt:->14. #crtab->14. #contentsOfEntireFile->14. #clear->14. #on:limit:->13. #nextChar->13. #findString:->13. #write:->12. #skipTo:->12. #converter->12. #string:->11. #nextWord->11. #next:putAll:startingAt:->11. #initializeOn:->11. #endEntry->11. #converter:->11. #basicNextPut:->11. #basicNext->11} - Some should be removed due to false polymorphism - Some should be isolated because specific (TranscriptStream, Generator, Transcripter ...) - Some should be removed thanks to a rewrite of bloated converters (basic*) But this give an idea of main protocol usage in image. As for 3rd party packages, this is another story... But Pharo is in a better position than Squeak for such hard decisions. Nicolas 2012/6/14 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: > I would address the problem with a compatibility layer like > Â Â 'foo' reading withLegacyProtocol > You have many options to add a protocol > - simply add the protocol at XTReadStream/XTWriteSTream roots > - just create another Xtreams wrapper, > - use a Trait > > I wish I had lightweight traits, I mean with traitified classes > created on the fly, with a simple naming like XTCollectionReadStream + > TLegacyStreamProtocol in case we have to debug it, because this way I > could also keep a clean Xtreams hierarchy in parallel with legacy at > very low maintenance costs. > > However, this is going to be some work for sure, here is a rough measurement: > > (Stream withAllSubclasses inject: Set new into: [:allSelectors :aClass > | allSelectors addAll: aClass selectors; yourself]) sorted size > -> 594 > > (XTReadStream withAllSubclasses , XTWriteStream withAllSubclasses > inject: Set new into: [:allSelectors :aClass | allSelectors addAll: > aClass selectors; yourself]) sorted size > -> 119 > > You certainly don't want to support 594 compatibility messages... > > Nicolas > > 2012/6/14 Igor Stasenko <siguctua(a)gmail.com>: >> On 14 June 2012 08:33, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> wrote: >>> Tx >>> >>> now there is a difference between adjustments and fully new API and I would have preferred adjustments. >>> >>> Stef >> >> IMO, Xtreams design is simply superior. >> But i agree, we cannot throw existing streams out.. too much things >> tied to them, and rely >> on using stream protocols. :( >> >> -- >> Best regards, >> Igor Stasenko. >>

June 14, 2012

Re: [Pharo-project] Process Tests and

by Stefan Marr

Hi: On 14 Jun 2012, at 11:34, Frank Shearar wrote: >>> How do you #join Processes? I can't see anything in Process' protocol >>> for doing so. so what am I missing? >> >> >> Nothing. Theres no support for join. You have to roll your own. > > OK, that's what I thought. Stefan's comment intrigued me. Yes, my fault. I added it to some random subclass somewhere, because I thought it was handy. It is indeed not in the standard images. > > It would be kind've handy to be able to say - even though we don't use > native threads - something like this: > > (0..10).map {|i| > Thread.new { sleep(1); i } > }.map {|t| t.value} > > only in a decent language: > > ((0 to: 10) > collect: [:i | [(Delay forSeconds: 1) wait. i] fork]) > collect: [:p | p value]. Yes, that was part of the discussion. For the options discussed of the return value of fork, we have three options: 1. self [Smalltalk-80 defines it this way, apparently] self is the block, it is the consistent return value for methods without real return value. -> I think, this is the least useful option 2. Process [current implementation] It's the object created as the side-effect of fork. Allows to query whether the fork already completed. Process could be extended to offer #join (at a cost of a semaphore, presumably). 3. Promise As the placeholder for the result value of the block computation. Benefit is that we get an immediate handle on the result, and the ability to wait for the completion of the process, even if we only care for its side-effects. There is however currently no Promise class in Pharo. Squeak got one, I think. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525

June 14, 2012

Re: [Pharo-project] WriteStream, nextPutAll:, OrderedCollections

by Frank Shearar

On 13 June 2012 23:11, Frank Shearar <frank.shearar(a)gmail.com> wrote: > On 13 June 2012 20:25, StÃ©phane Ducasse <Stephane.Ducasse(a)inria.fr> wrote: >> I would have really prefer that Xtreams provide a compatible API. >> Because changing API means also rewrite documentationâ¦. > > If I recall correctly, Michael & Martin said that it was impossible to > provide a compatible API because the ANSI Stream protocol was > fundamentally broken in a few key ways. I forget the details though, > sadly. > >> Nicolas else did you get Xtream working well in Pharo? > > It should work just fine. As I recall there wasn't much adjustment > necessary for Squeak (but of course Nicolas would know better than > anyone!). As it happens, the latest ConfigurationOfXtreams fails to load in Pharo 1.4. ConfigurationOfXtreams class >> #bootstrapPackage:from: tries to load ConfigurationOfMetacello-dkh.674 from SqueakSource. SS doesn't have that version, so #bootstrapPackage:from:'s caller, #ensureMetacello, should retry, and fetch the version from Metacello's canonical repository. In Squeak this works, but not in Pharo. I'm unsure why, unless it's just that Zinc+Monticello behaves differently in Pharo than in Squeak. Xtreams uses FFI and in Pharo-1.4 there's a single test failure, in FFIPluginTests >> #testBlitToAndFromExternalForm because Form class doesn't understand #makeStar. However, Xtreams' tests are green in Pharo 1.4! > frank > >> Because we will nearly killed all the old FileDirectory code. So stream would be good candidate to clean too. >> Stef >> >> >>> If it's about switching to a clean library maximizing the power of >>> composition use Xtreams. >>> If it's about maximizing compatibility, use Nile. The composition in >>> Nile is done via Traits, this is interesting but spectrum is quite >>> narrow. Nile also introduces lazy streams and filters, but this is >>> less systematic than Xtreams, because the main goal was to replace >>> legacy Stream, innovation being secondary. >>> >>> Of course, in the first case, you break a lot of clients because >>> design decision is to change the whole API and force a rewrite. >>> So you must have a long transition phase maintaining legacy Streams >>> >>> And in second case, you can eliminate the legacy streams more rapidly >>> and just replace them with Nile, but you still carry some cruft, >>> compatibility oblige... >>> >>> Not an easy choice, but you can guess my opinion ;) >>> >>> Nicolas >>> >>> 2012/6/13 Denis Kudriashov <dionisiydk(a)gmail.com>: >>>> And why no Xtreams? >>>> >>>> 2012/6/13 <petton.nicolas(a)gmail.com> >>>> >>>>> Guillermo Polito <guillermopolito(a)gmail.com> writes: >>>>> >>>>>> Ok, just was expecting (very very deeply) not having to modify too >>>>>> much in Glorp :). Anyway, I'll see what can I do in this front. >>>>>> >>>>>> Now Damien, if streaming is only intended for arrays and strings, I'd >>>>>> expect the expression 'OrderedCollection new writeStream' to raise an >>>>>> error... :/ >>>>> >>>>> Indeed. Also, just for my information, why aren't we using Nile? >>>>> >>>>> Nico >>>>> >>>>>> >>>>>> On Wed, Jun 13, 2012 at 5:29 PM, Levente Uzonyi <leves(a)elte.hu> wrote: >>>>>> >>>>>> Â Â On Wed, 13 Jun 2012, Guillermo Polito wrote: >>>>>> >>>>>> Â Â Â Â soo, what do I do? :D >>>>>> >>>>>> >>>>>> >>>>>> Â Â Streaming into an OrderedCollection is unnecessary, because >>>>>> Â Â OrderedCollection is already a stream-like object. If I were >>>>>> Â Â implementing Smalltalk now, I would probably add the stream >>>>>> Â Â protocol to it (#nextPut:, #nextPutAll:, etc). >>>>>> Â Â A simple fix to your problem is to replace the line >>>>>> >>>>>> Â Â Â Â Â grownCollection := collection class new: newSize. >>>>>> >>>>>> Â Â with >>>>>> >>>>>> Â Â Â Â Â grownCollection := collection class ofSize: newSize. >>>>>> >>>>>> Â Â in WriteStream >> #growTo: (note that I didn't check current Pharo >>>>>> Â Â code, so this might be different there). >>>>>> >>>>>> >>>>>> Â Â Levente >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Â Â Â Â On 6/13/12, Camillo Bruni <camillobruni(a)gmail.com> wrote: >>>>>> >>>>>> Â Â Â Â yup I know that :D >>>>>> Â Â Â Â Â Â And I provided a fix on e year ago, that got lost in a big >>>>>> Â Â Â Â Â Â refactoring... >>>>>> Â Â Â Â Â Â - I added an explicit #streamSpecies on the Collection >>>>>> Â Â Â Â Â Â classes. >>>>>> Â Â Â Â Â Â - By default it returns the same class >>>>>> Â Â Â Â Â Â - on Set / OrderedCollection / Symbol it returns the >>>>>> Â Â Â Â Â Â mutable types >>>>>> Â Â Â Â Â Â (LinkedList as well I think) >>>>>> Â Â Â Â Â Â - overwrote #streamContents: to convert from the >>>>>> Â Â Â Â Â Â #streamSpecies type back to >>>>>> Â Â Â Â Â Â the original class >>>>>> >>>>>> >>>>>> >>>>>> Â Â Â Â Â Â On 2012-06-13, at 14:56, Guillermo Polito wrote: >>>>>> >>>>>> Â Â Â Â Â Â Â Â Â Â Â Â Hi guys! >>>>>> >>>>>> Â Â Â Â Â Â Â Â I'm chasing a bug that appeared in glorp under pharo >>>>>> Â Â Â Â Â Â Â Â 1.4. Â Now, the >>>>>> Â Â Â Â Â Â Â Â bug is due to some behavior changed in >>>>>> Â Â Â Â Â Â Â Â OrderedCollection I think. Look >>>>>> Â Â Â Â Â Â Â Â at this piece of code: >>>>>> >>>>>> >>>>>> Â Â Â Â Â Â Â Â oc := OrderedCollection new. >>>>>> Â Â Â Â Â Â Â Â ws := oc writeStream. >>>>>> >>>>>> Â Â Â Â Â Â Â Â "this explodes" >>>>>> Â Â Â Â Â Â Â Â ws nextPutAll: (OrderedCollection with: 1 with: 2 >>>>>> Â Â Â Â Â Â Â Â with: 3). >>>>>> >>>>>> Â Â Â Â Â Â Â Â "this works" >>>>>> Â Â Â Â Â Â Â Â ws nextPutAll: {1.2.3} >>>>>> >>>>>> >>>>>> Â Â Â Â Â Â Â Â And I'm puzzled, why should one work and the other not >>>>>> Â Â Â Â Â Â Â Â from the pov of the >>>>>> Â Â Â Â Â Â Â Â user? >>>>>> Â Â Â Â Â Â Â Â And how should I replace that behavior if it's my bug? >>>>>> >>>>>> Â Â Â Â Â Â Â Â putting an asArray for each nextPutAll: does not look >>>>>> Â Â Â Â Â Â Â Â good for me... :S >>>>>> >>>>>> Â Â Â Â Â Â Â Â Tx, >>>>>> Â Â Â Â Â Â Â Â Guille >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Nicolas Petton >>>>> http://nicolas-petton.fr >>>>> >>>> >>> >> >>

June 14, 2012