November 2011 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] WideString performance

by Nicolas Cellier

Also note that the more mainXtream Xtreams at http://www.squeaksource.com/Xtreams/ does not perform that bad... This ones gets a score of 3.0 seconds with a regular MultiByteFileStream nextRow | row char line nextValue noValue | line := stream nextLine reading. row := OrderedCollection new. nextValue := (WideString new: 32) writing. noValue := true. [[[(char := line get) isSeparator] whileTrue. noValue := false. char = $" ifTrue: [[nextValue write: (line ending: $") rest. (char := line get) = $"] whileTrue: [nextValue put: $"]. [char = separator] whileFalse: [char := line get]]. [char = separator] whileFalse: [nextValue put: char. char := line get]. row add: nextValue conclusion. nextValue := (WideString new: 32) writing] repeat] on: Incomplete do: [:exc | ]. noValue ifFalse: [row add: nextValue conclusion]. ^row with a XTFileReadStream, just rewrite this: MessageTally spyOn: [| stream separator | separator := $;. stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') reading encoding: #utf8) "rest reading". (stream ending: Character cr) slicing collect: [:line | | row char nextValue noValue | row := OrderedCollection new. nextValue := (WideString new: 32) writing. noValue := true. [[[(char := line get) isSeparator] whileTrue. noValue := false. char = $" ifTrue: [[nextValue write: (line ending: $") rest. (char := line get) = $"] whileTrue: [nextValue put: $"]. [char = separator] whileFalse: [char := line get]]. [char = separator] whileFalse: [nextValue put: char. char := line get]. row add: nextValue conclusion. nextValue := (WideString new: 32) writing] repeat] on: Incomplete do: [:exc | ]. noValue ifFalse: [row add: nextValue conclusion]. row]] For some reason, this was super slow (>20s), though my previous attempt without slicing was around 3.0s... (so you noticed the commented "rest reading" whose purpose is to load the whole file in memory) The major advantage is that you can just replace #collect: with #collecting: and get a stream of rows. Lazy is cool. Nicolas 2011/11/18 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: > 2011/11/17 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: >> 2011/11/17 Dennis Schetinin <chaetal(a)gmail.com>: >>> So, the answer for the question "Is there a simple way to improve the >>> performance?" would be "No", right? >>> >> >> No, the bottle neck is CSVParser as Henrik and Levente already noticed. >> >> My machine executes your code in 21.7s >> >> So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ >> and inlined row decoding in a single method: >> >> CSVParser>>nextRow >> Â Â Â Â | row char line nextValue | >> Â Â Â Â line := stream nextLine readXtream. >> Â Â Â Â row := OrderedCollection new. >> Â Â Â Â nextValue := (WideString new: 32) writeXtream. >> Â Â Â Â line endOfStreamAction: [^row add: (nextValue contents); yourself]. >> Â Â Â Â [[(char := line next) isSeparator] whileTrue. >> Â Â Â Â char = $" ifTrue: >> Â Â Â Â Â Â Â Â [[nextValue nextPutAll: (stream upTo: $"). > > Err, it's line upTo: $" > >> Â Â Â Â Â Â Â Â line peekFor: $"] whileTrue: >> Â Â Â Â Â Â Â Â Â Â Â Â [nextValue nextPut: $"]. >> Â Â Â Â Â Â Â Â [(char := line next) = separator] whileFalse]. >> Â Â Â Â [char = separator] >> Â Â Â Â Â Â Â Â whileFalse: >> Â Â Â Â Â Â Â Â Â Â Â Â [nextValue nextPut: char. >> Â Â Â Â Â Â Â Â Â Â Â Â char := line next]. >> Â Â Â Â row add: nextValue contents. >> Â Â Â Â nextValue reset] repeat >> >> This reduces execution time to 2.4 seconds >> >> I also tried to use SqueaXTream for the file itself (if you don't mind >> the ugly invocation); >> But it does not gain much because you file is essentially made of Wide chars... >> It would help only if you'd have more ASCII fields... >> >> MessageTally spyOn: [| stream | >> Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / >> 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. >> Â Â Â Â [(CSVParser on: stream) >> Â Â Â Â Â Â Â Â useDelimiter: $;; >> Â Â Â Â Â Â Â Â rows ] ensure: [stream close]]. >> >> This reduces execution time to 1.9 seconds though >> >> So I wouldn't say there's not plenty of room for performance ;) >> >> Nicolas >> >>> 2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> >>>> >>>> I think that there are several issues: >>>> >>>> 1) Character with codePoint < 256 are unique, but Wider are not. >>>> The characters are not stored in the WideString (which instead stores >>>> the code points), but any enumeration using at: rather than wordAt: >>>> will create an overhead. I presume that it would be better to have a >>>> #codePointAt: defined as #byteAt: in ByteString and #wordAt: in >>>> WideString, so we could still define generic methods in String. >>>> 2) A lot of ByteString methods are optimized with ad hoc optional >>>> primitives, but very few WideString methods are. >>>> 3) Some of these optimized operations work with 256 element masks and >>>> cannot be generalized to WideString that easily >>>> 4) Some Stream operations that copy one Collection to the stream >>>> buffer or vice versa are also not optimized for mixed byte/word >>>> collections >>>> 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters >>>> also are (because they can use more primitives in this case) >>>> 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe >>>> questionable from quality POV, but were not replaced from efficiency >>>> POV. >>>> >>>> I would say that a new VM with immediate Character values might help, >>>> but that won't be enough. >>>> >>>> Nicolas >>>> >>>> 2011/11/16 Sven Van Caekenberghe <sven(a)beta9.be>: >>>> > Dennis, >>>> > >>>> > On 16 Nov 2011, at 19:30, Dennis Schetinin wrote: >>>> > >>>> >> Hi! >>>> >> >>>> >> Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I >>>> >> found it really slow. It takes about 5 seconds to read a 53Kb file (136 >>>> >> lines) and several minutes to parse a 3.4Mb file (about 8K lines). >>>> >> >>>> >> Profiler shows that nearly all the time is eaten by WideString >> >>>> >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, >>>> >> this means that <primitive: 66> there does not work in this caseâ¦ So, to be >>>> >> short: is there a simple way to improve the performance? TIA >>>> >> >>>> >> -- >>>> >> Dennis Schetinin >>>> > >>>> > I am interested in this problem. >>>> > >>>> > Would it be possible to provide a test file (not that I can read >>>> > Russian) and your test code ? >>>> > >>>> > I would like to try this myself. >>>> > >>>> > Sven >>>> > >>>> >>> >>> >>> >>> -- >>> Dennis Schetinin >>> >> >

Nov. 17, 2011

Re: [Pharo-project] WideString performance

by Nicolas Cellier

2011/11/17 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: > 2011/11/17 Dennis Schetinin <chaetal(a)gmail.com>: >> So, the answer for the question "Is there a simple way to improve the >> performance?" would be "No", right? >> > > No, the bottle neck is CSVParser as Henrik and Levente already noticed. > > My machine executes your code in 21.7s > > So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ > and inlined row decoding in a single method: > > CSVParser>>nextRow > Â Â Â Â | row char line nextValue | > Â Â Â Â line := stream nextLine readXtream. > Â Â Â Â row := OrderedCollection new. > Â Â Â Â nextValue := (WideString new: 32) writeXtream. > Â Â Â Â line endOfStreamAction: [^row add: (nextValue contents); yourself]. > Â Â Â Â [[(char := line next) isSeparator] whileTrue. > Â Â Â Â char = $" ifTrue: > Â Â Â Â Â Â Â Â [[nextValue nextPutAll: (stream upTo: $"). Err, it's line upTo: $" > Â Â Â Â Â Â Â Â line peekFor: $"] whileTrue: > Â Â Â Â Â Â Â Â Â Â Â Â [nextValue nextPut: $"]. > Â Â Â Â Â Â Â Â [(char := line next) = separator] whileFalse]. > Â Â Â Â [char = separator] > Â Â Â Â Â Â Â Â whileFalse: > Â Â Â Â Â Â Â Â Â Â Â Â [nextValue nextPut: char. > Â Â Â Â Â Â Â Â Â Â Â Â char := line next]. > Â Â Â Â row add: nextValue contents. > Â Â Â Â nextValue reset] repeat > > This reduces execution time to 2.4 seconds > > I also tried to use SqueaXTream for the file itself (if you don't mind > the ugly invocation); > But it does not gain much because you file is essentially made of Wide chars... > It would help only if you'd have more ASCII fields... > > MessageTally spyOn: [| stream | > Â Â Â Â stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / > 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. > Â Â Â Â [(CSVParser on: stream) > Â Â Â Â Â Â Â Â useDelimiter: $;; > Â Â Â Â Â Â Â Â rows ] ensure: [stream close]]. > > This reduces execution time to 1.9 seconds though > > So I wouldn't say there's not plenty of room for performance ;) > > Nicolas > >> 2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> >>> >>> I think that there are several issues: >>> >>> 1) Character with codePoint < 256 are unique, but Wider are not. >>> The characters are not stored in the WideString (which instead stores >>> the code points), but any enumeration using at: rather than wordAt: >>> will create an overhead. I presume that it would be better to have a >>> #codePointAt: defined as #byteAt: in ByteString and #wordAt: in >>> WideString, so we could still define generic methods in String. >>> 2) A lot of ByteString methods are optimized with ad hoc optional >>> primitives, but very few WideString methods are. >>> 3) Some of these optimized operations work with 256 element masks and >>> cannot be generalized to WideString that easily >>> 4) Some Stream operations that copy one Collection to the stream >>> buffer or vice versa are also not optimized for mixed byte/word >>> collections >>> 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters >>> also are (because they can use more primitives in this case) >>> 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe >>> questionable from quality POV, but were not replaced from efficiency >>> POV. >>> >>> I would say that a new VM with immediate Character values might help, >>> but that won't be enough. >>> >>> Nicolas >>> >>> 2011/11/16 Sven Van Caekenberghe <sven(a)beta9.be>: >>> > Dennis, >>> > >>> > On 16 Nov 2011, at 19:30, Dennis Schetinin wrote: >>> > >>> >> Hi! >>> >> >>> >> Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I >>> >> found it really slow. It takes about 5 seconds to read a 53Kb file (136 >>> >> lines) and several minutes to parse a 3.4Mb file (about 8K lines). >>> >> >>> >> Profiler shows that nearly all the time is eaten by WideString >> >>> >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, >>> >> this means that <primitive: 66> there does not work in this caseâ¦ So, to be >>> >> short: is there a simple way to improve the performance? TIA >>> >> >>> >> -- >>> >> Dennis Schetinin >>> > >>> > I am interested in this problem. >>> > >>> > Would it be possible to provide a test file (not that I can read >>> > Russian) and your test code ? >>> > >>> > I would like to try this myself. >>> > >>> > Sven >>> > >>> >> >> >> >> -- >> Dennis Schetinin >> >

Nov. 17, 2011

Re: [Pharo-project] OpenGL in Pharo?

by Pavel Krivanek

I would like to ask on the future of FFI technologies in Pharo. What will be the prefered project? Alien, NativeBoost or both? That means what projects will have support in the "official" VM for all main platforms? Some week ago I played with FFI calls on Linux and the only project that worked well was NativeBoost. The Alien failed to locate the libraries (it was 32bit VM on 64bit system, possible source of complacations). Cheers -- Pavel 17.11.2011 v 18:51, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: > Hi michele > > Do not expect something like Jun because this is 10 or 15 years. Now I saw today a cairo graphics displayed on the machine of Javier :). > This is an example for the Alien chapter of the new pharo book (quite draft). > > I hope that I will get somebody will work on a minimal code city for moose based on a opengl solution. > Now if people have more cycles it will go faster. I hope igor will release soon the opengl binding for mac. > > Stef > <alien.pdf> > > > On Nov 17, 2011, at 8:52 AM, Michele Lanza wrote: > >> Dear all, >> I wanted to ask whether there are some news I missed with respect to having a decent OpenGL framework in Pharo. I checked the following page: >> >> http://book.pharo-project.org/book/LanguageAndLibraries/3DGraphicsAndOpenGL/ >> >> ..but it sounds all a bit outdated and not something I can rely on in the long run. Ideally there should be something like the Jun framework (available for VisualWorks), but when I met the Jun maker at ICSE and I asked him whether there were any plans of porting Jun to Pharo he told me that it was not in their plans. Long talk, short sense: Is anyone working on OpenGL for Pharo? >> >> Cheers >> >> Michele >

Nov. 17, 2011

Re: [Pharo-project] WideString performance

by Nicolas Cellier

2011/11/17 Dennis Schetinin <chaetal(a)gmail.com>: > So, the answer for the question "Is there a simple way to improve the > performance?" would be "No", right? > No, the bottle neck is CSVParser as Henrik and Levente already noticed. My machine executes your code in 21.7s So I decided to play with SqueaXTream http://www.squeaksource.com/XTream/ and inlined row decoding in a single method: CSVParser>>nextRow | row char line nextValue | line := stream nextLine readXtream. row := OrderedCollection new. nextValue := (WideString new: 32) writeXtream. line endOfStreamAction: [^row add: (nextValue contents); yourself]. [[(char := line next) isSeparator] whileTrue. char = $" ifTrue: [[nextValue nextPutAll: (stream upTo: $"). line peekFor: $"] whileTrue: [nextValue nextPut: $"]. [(char := line next) = separator] whileFalse]. [char = separator] whileFalse: [nextValue nextPut: char. char := line next]. row add: nextValue contents. nextValue reset] repeat This reduces execution time to 2.4 seconds I also tried to use SqueaXTream for the file itself (if you don't mind the ugly invocation); But it does not gain much because you file is essentially made of Wide chars... It would help only if you'd have more ASCII fields... MessageTally spyOn: [| stream | stream := (((FileDirectory on: '/Users/nicolas/Downloads/') / 'Data.csv') readXtream binary buffered ~> UTF8Decoder) buffered. [(CSVParser on: stream) useDelimiter: $;; rows ] ensure: [stream close]]. This reduces execution time to 1.9 seconds though So I wouldn't say there's not plenty of room for performance ;) Nicolas > 2011/11/16 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> >> >> I think that there are several issues: >> >> 1) Character with codePoint < 256 are unique, but Wider are not. >> The characters are not stored in the WideString (which instead stores >> the code points), but any enumeration using at: rather than wordAt: >> will create an overhead. I presume that it would be better to have a >> #codePointAt: defined as #byteAt: in ByteString and #wordAt: in >> WideString, so we could still define generic methods in String. >> 2) A lot of ByteString methods are optimized with ad hoc optional >> primitives, but very few WideString methods are. >> 3) Some of these optimized operations work with 256 element masks and >> cannot be generalized to WideString that easily >> 4) Some Stream operations that copy one Collection to the stream >> buffer or vice versa are also not optimized for mixed byte/word >> collections >> 5) UTF8 is optimized for ASCII anyway... And Squeak/Pharo converters >> also are (because they can use more primitives in this case) >> 6) Pharo ripped #squeakToUtf8 and #utf8ToSqueak which were maybe >> questionable from quality POV, but were not replaced from efficiency >> POV. >> >> I would say that a new VM with immediate Character values might help, >> but that won't be enough. >> >> Nicolas >> >> 2011/11/16 Sven Van Caekenberghe <sven(a)beta9.be>: >> > Dennis, >> > >> > On 16 Nov 2011, at 19:30, Dennis Schetinin wrote: >> > >> >> Hi! >> >> >> >> Parsing a UTF-8 CSV file with Russian symbols (using CSVParser), I >> >> found it really slow. It takes about 5 seconds to read a 53Kb file (136 >> >> lines) and several minutes to parse a 3.4Mb file (about 8K lines). >> >> >> >> Profiler shows that nearly all the time is eaten by WideString >> >> >> at:put: which is invoked from WriteStream>>nextPut:. If I'm not mistaken, >> >> this means that <primitive: 66> there does not work in this caseâ¦ So, to be >> >> short: is there a simple way to improve the performance? TIA >> >> >> >> -- >> >> Dennis Schetinin >> > >> > I am interested in this problem. >> > >> > Would it be possible to provide a test file (not that I can read >> > Russian) and your test code ? >> > >> > I would like to try this myself. >> > >> > Sven >> > >> > > > > -- > Dennis Schetinin >

Nov. 17, 2011

Re: [Pharo-project] how to get a string read stream from filesystem

by Norbert Hartl

Am 17.11.2011 um 22:01 schrieb Tudor Girba: > Hi, > > I am using Filesystem, and I would like to get a read stream that provides a string instead of bytes. > > If I do "aReference readStream", I get a FSReadStream which is a byte stream. > > To get to the string, I now do the ugly: "aReference readStream contents asString" > > Is there a cleverer way? > I don't know FileSystem. But in any case a bytestream is correct. In order to get a string you need to decode it. Only for ascii this is null codec. For any other encoded file you won't get what you want. So, using a textconverter with file encoding might get you what you want. Norbert

Nov. 17, 2011

Re: [Pharo-project] OpenGL in Pharo?

by Levente Uzonyi

On Thu, 17 Nov 2011, Javier PimÃ¡s wrote: > We are advancing on making OpenGL work with NativeBoost right now. If you > have to write an app > that uses OpenGL now, I would strongly recommend you to use NBOpenGL. It's I see the hype about NBOpenGL, but I don't see how is it better than the FFI based OpenGL implementation. Can someone shed some light on it? Levente > already working > on Windows, and we are fixing the missing stuff to make it work on the > other platforms. > > As a side effect, we are developing a base FFI library covering the > windowing systems. > I started today wrapping XLib and it's already working! here is a > screenshot. Of course, it > lacks asynchronous calls, but that's only for now. > > We are also writing a chapter in pharo by example to make it easier for > everybody. > > Cheers, > Javier. > > > On Thu, Nov 17, 2011 at 11:32 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > >> On 17 November 2011 11:22, Alexandre Bergel <alexandre.bergel(a)me.com> >> wrote: >>> Igor is working on it i've heard. I am also waiting for it >>> >> >> NativeBoost bindings for OpenGL working fine under Windows. >> On Macs and Linux there still missing platform-specific glue code to >> initialize context. >> I started Mac implementation recently, but i didn't finished it. >> Since there are increasing interest, i will try to finish it in nearest >> days. >> >> >>> Alexandre >>> >>> >>> On 17 Nov 2011, at 04:52, Michele Lanza <michele.lanza(a)usi.ch> wrote: >>> >>>> Dear all, >>>> I wanted to ask whether there are some news I missed with respect to >> having a decent OpenGL framework in Pharo. I checked the following page: >>>> >>>> >> http://book.pharo-project.org/book/LanguageAndLibraries/3DGraphicsAndOpenGL/ >>>> >>>> ..but it sounds all a bit outdated and not something I can rely on in >> the long run. Ideally there should be something like the Jun framework >> (available for VisualWorks), but when I met the Jun maker at ICSE and I >> asked him whether there were any plans of porting Jun to Pharo he told me >> that it was not in their plans. Long talk, short sense: Is anyone working >> on OpenGL for Pharo? >>>> >>>> Cheers >>>> >>>> Michele >>> >>> >> >> >> >> -- >> Best regards, >> Igor Stasenko. >> >> > > > -- > Lic. Javier PimÃ¡s > Ciudad de Buenos Aires >

Nov. 17, 2011

Re: [Pharo-project] * Smalltalk User Group

by Johannes Rasche

Hi Norbert possibly the people of HPI are the only ones in the area of Berlin up to now I'll contact them Johannes Am 17.11.11 13:21, schrieb Norbert Hartl: > Ha ha, > > I was asking that myself earlier in this century. AFAIK there are not > that much smalltalkers in Berlin. But Potsdam and the HPI are close. I > wanted to create a Berlin user group but now I don't live there > anymore (and I helped creating one in cologne instead). But I'm > inclined to attend anyway even now I'm closer to Lille than Berlin :) > > good luck, > > Norbert > Am 17.11.2011 um 12:53 schrieb StÃ©phane Ducasse: >> >> Begin forwarded message: >> >>> *From: *Johannes Rasche <johannesrasche(a)mac.com >>> <mailto:johannesrasche@mac.com>> >>> *Subject: *** Smalltalk User Group* >>> *Date: *November 17, 2011 12:27:15 PM GMT+01:00 >>> *To: *pharo-project-owner(a)lists.gforge.inria.fr >>> <mailto:pharo-project-owner@lists.gforge.inria.fr> >>> >>> as Sven just announced the Belgian Smalltalk User Group (BSUG) meeting: >>> >>> are there Pharo people in Berlin (I didn't realize it up to now)? >>> >>> It would be great to have a local community with the opportunity for >>> face-to-face exchange >>> and potential cooperation >>> >>> Johannes >>> >>> -- >>> Johannes Rasche /Hagelberger Str. 56 10965 Berlin mail >>> johannesrasche(a)mac.com mobile +49 1577 175 04 75/ >> > -- Johannes Rasche /Hagelberger Str. 56 10965 Berlin mail johannesrasche(a)mac.com mobile +49 1577 175 04 75/

Nov. 17, 2011

Re: [Pharo-project] MVP: quite interesting.

by StÃ©phane Ducasse

http://aviadezra.blogspot.com/2007/07/twisting-mvp-triad-say-hello-to-mvpc.… sounds also interesting. On Nov 17, 2011, at 10:49 PM, StÃ©phane Ducasse wrote: > http://www.mimuw.edu.pl/~sl/teaching/00_01/Delfin_EC/Overviews/ModelViewPre… > > Stef

Nov. 17, 2011

[Pharo-project] MVP: quite interesting.

by StÃ©phane Ducasse

http://www.mimuw.edu.pl/~sl/teaching/00_01/Delfin_EC/Overviews/ModelViewPre… Stef

Nov. 17, 2011

Re: [Pharo-project] Unsubscribing for Announcements

by Tudor Girba

On 17 Nov 2011, at 05:46, Igor Stasenko wrote: > On 17 November 2011 05:17, Sean P. DeNigris <sean(a)clipperadams.com> wrote: >> >> Tudor Girba wrote: >>> >>> The WeakAnnouncement are already on the agenda for 1.3. >>> >> >> Bump... >> What is the status of this? Also, what are people doing in the mean time? I >> don't see a way to safely use Announcements without automatic unsubscribing. >> I have not found a hook in Morphic that's guaranteed to be called. >> > > annoucer weak on: Announcement send: #foo to: self. > > when self will die, so its subscription. This works since quite a while. I use it in Glamour extensively. Cheers, Doru >> Sean >> >> -- >> View this message in context: http://forum.world.st/Unsubscribing-for-Announcements-tp3220751p4078804.html >> Sent from the Pharo Smalltalk mailing list archive at Nabble.com. >> >> > > > > -- > Best regards, > Igor Stasenko. > -- www.tudorgirba.com "Problem solving should be focused on describing the problem in a way that makes the solution obvious."

Nov. 17, 2011