November 2014 - Pharo-users

pharo consultants
by Norbert Hartl Nov. 16, 2014

Nov. 16, 2014

I just had a look at http://consultants.pharo.org/ <http://consultants.pharo.org/> (that is linked from pharo.org <http://pharo.org/>) and it seems pretty outdated. The page footer tells Page last generated on 2013-10-12T21:46:35.576104+02:00 by Pharo3.0 of 18 March 2013 update 30262 and the images linked on the page do not exist anymore. I forgot how people can add themselves. I remember a smalltalk snippet that one can send around to update the list. So if you think you know pharo quite ok please consider to add yourself on the site. I'm sure the proper information for adding/updating your information will follow shortly in a reply to this mail. Norbert

1 0

Re: [Pharo-users] Has anyone tried compiling the Pharo VM into JS using Emscripten?
by Andy Burnett Nov. 15, 2014

Nov. 15, 2014

Alain wrote <<< Hi Andy, If I understand your question, you want to remake Dan Ingall's lively kernel ? >>> Hi Alain, Although I am really impressed with Dan's work, Emscripten seems to be very different. In theory - and my knowledge in this is very limited - it might allow the VM C files to be transpiled into a very tight subset of JS. On the face of it, this is a crazy idea, but they have achieved amazing performance with things like the QT library. This made me wonder whether a Js version of pharo is possible. Just Idle conjecturing in. Friday night :-) > On 14 Nov 2014, at 18:09, pharo-users-request(a)lists.pharo.org wrote: > > Send Pharo-users mailing list submissions to > pharo-users(a)lists.pharo.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.pharo.org/mailman/listinfo/pharo-users_lists.pharo.org > or, via email, send a message with subject or body 'help' to > pharo-users-request(a)lists.pharo.org > > You can reach the person managing the list at > pharo-users-owner(a)lists.pharo.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Pharo-users digest..." > > > Today's Topics: > > 1. Re: running out of memory while processing a 220MB csv file > with NeoCSVReader - tips? (Paul DeBruicker) > 2. Re: running out of memory while processing a 220MB csv file > with NeoCSVReader - tips? (Sven Van Caekenberghe) > 3. Re: Has anyone tried compiling the Pharo VM into JS using > Emscripten? (Alain Rastoul) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 14 Nov 2014 14:14:51 -0800 (PST) > From: Paul DeBruicker <pdebruic(a)gmail.com> > To: pharo-users(a)lists.pharo.org > Subject: Re: [Pharo-users] running out of memory while processing a > 220MB csv file with NeoCSVReader - tips? > Message-ID: <1416003291281-4790341.post(a)n4.nabble.com> > Content-Type: text/plain; charset=us-ascii > > Hi Sven > > Yes, like I said earlier, after your first email, that I think its not a > problem with NeoCSV as with what I'm doing and an out of memory condition. > > Have you ever seen a stack after sending kill -SIGUSR1 that looks like this: > > output file stack is full. > output file stack is full. > output file stack is full. > output file stack is full. > output file stack is full. > .... > > > What does that mean? > > Answers to your questions below. > > Thanks again for helping me out > > > > Sven Van Caekenberghe-2 wrote >> OK then, you *can* read/process 300MB .csv files ;-) >> >> What does your CSV file look like, can you show a couple of lines ? >> >> here are 2 lines + a header: >> >> "provnum","Provname","address","city","state","zip","survey_date_output","SurveyType","defpref","tag","tag_desc","scope","defstat","statdate","cycle","standard","complaint","filedate" >> "015009","BURNS NURSING HOME, INC.","701 MONROE STREET >> NW","RUSSELLVILLE","AL","35653","2013-09-05","Health","F","0314","Give >> residents proper treatment to prevent new bed (pressure) sores or heal >> existing bed sores.","D","Deficient, Provider has date of >> correction","2013-10-10",1,"Y","N","2014-01-01" >> "015009","BURNS NURSING HOME, INC.","701 MONROE STREET >> NW","RUSSELLVILLE","AL","35653","2013-09-05","Health","F","0315","Ensure >> that each resident who enters the nursing home without a catheter is not >> given a catheter, unless medically necessary, and that incontinent >> patients receive proper services to prevent urinary tract infections and >> restore normal bladder functions.","D","Deficient, Provider has date of >> correction","2013-10-10",1,"Y","N","2014-01-01" >> >> >> You are using a custom record class of your own, what does that look like >> or do ? >> >> A custom record class. This is all publicly available data but I'm >> keeping track of the performance of US based health care providers during >> their annual inspections. So the records are notes of a deficiency during >> the inspection and I'm keeping those notes in a collection in an instance >> of the health care provider's class. The custom record class just >> converts the CSV record to objects (Integers, Strings, DateAndTime) and >> then gets stuffed in the health care provider's deficiency history >> OrderedCollection (which has about 100 items). Again I don't think its >> what I'm doing as much as the image isn't growing when it needs to. >> >> >> >> >> Maybe you can try using Array again ? >> >> I've attempted to do it where I parse and convert the entire CSV into >> domain objects then add them to the image and the parsing works fine, but >> the system runs out of resources during the update phase. >> >> >> What percentage of records read do you keep ? In my example it was very >> small. Have you tried calculating your memory usage ? >> >> >> I'm keeping some data from every record, but it doesn't load more than >> 500MB of the data before falling over. I am not attempting to load the >> 9GB of CSV files into one image. For 95% of the records in the CSV file >> 20 of the 22 columns of the data is the same from file to file, just a >> 'published date' and a 'time to expiration' date changes. Each file >> covers a month, with about 500k deficiencies. Each month some >> deficiencies are added to the file and some are resolved. So the total >> number of deficiencies in the image is about 500k. Of those records that >> don't expire in a given month I'm adding the published date to a >> collection of published dates for the record and also adding the "time to >> expiration" to a collection of those to record what was made public and >> letting the rest of the data get GC'd. I don't only load those two >> records because the other fields of the record in the CSV could change. >> >> I have not calculated the memory usage for the collection because I >> thought it would have no problem fitting in the 2GB of RAM I have on this >> machine. >> >> >> >>> On 14 Nov 2014, at 22:34, Paul DeBruicker < > >> pdebruic@ > >> > wrote: >>> >>> Yes. With the image & vm I'm having trouble with I get an array with >>> 9,942 >>> elements in it. So its works as you'd expect. >>> >>> While processing the CSV file the image stays at about 60MB in RAM. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Sven Van Caekenberghe-2 wrote >>>> Can you successfully run my example code ? >>>> >>>>> On 14 Nov 2014, at 22:03, Paul DeBruicker < >>> >>>> pdebruic@ >>> >>>> > wrote: >>>>> >>>>> Hi Sven, >>>>> >>>>> Thanks for taking a look and testing the NeoCSVReader portion for me. >>>>> You're right of course that there's something I'm doing that's slow. >>>>> But. >>>>> There is something I can't figure out yet. >>>>> >>>>> To provide a little more detail: >>>>> >>>>> When the 'csv reading' process completes successfully profiling shows >>>>> that >>>>> most of the time is spent in NeoCSVReader>>#peekChar and using >>>>> NeoCSVReader>>##addField: to convert a string to a DateAndTime. >>>>> Dropping >>>>> the DateAndTime conversion speeds things up but doesn't stop it from >>>>> running >>>>> out of memory. >>>>> >>>>> I start the image with >>>>> >>>>> ./pharo-ui --memory 1000m myimage.image >>>>> >>>>> Splitting the CSV file helps: >>>>> ~1.5MB 5,000 lines = 1.2 seconds. >>>>> ~15MB 50,000 lines = 8 seconds. >>>>> ~30MB 100,000 lines = 16 seconds. >>>>> ~60MB 200,000 lines = 45 seconds. >>>>> >>>>> >>>>> It seems that when the CSV file crosses ~70MB in size things start >>>>> going >>>>> haywire with performance, and leads to the out of memory condition. >>>>> The >>>>> processing never ends. Sending "kill -SIGUSR1" prints a stack >>>>> primarily >>>>> composed of: >>>>> >>>>> 0xbffc5d08 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n) >>>>> OutOfMemory class >>>>> 0xbffc5d20 M OutOfMemory class(Behavior)>basicNew 0x1f7ac060: a(n) >>>>> OutOfMemory class >>>>> 0xbffc5d38 M OutOfMemory class(Behavior)>new 0x1f7ac060: a(n) >>>>> OutOfMemory >>>>> class >>>>> 0xbffc5d50 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n) >>>>> OutOfMemory class >>>>> 0xbffc5d68 M OutOfMemory class(Behavior)>basicNew 0x1f7ac060: a(n) >>>>> OutOfMemory class >>>>> 0xbffc5d80 M OutOfMemory class(Behavior)>new 0x1f7ac060: a(n) >>>>> OutOfMemory >>>>> class >>>>> 0xbffc5d98 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n) >>>>> OutOfMemory class >>>>> >>>>> So it seems like its trying to signal that its out of memory after its >>>>> out >>>>> of memory which triggers another OutOfMemory error. So that's why >>>>> progress >>>>> stops. >>>>> >>>>> >>>>> ** Aside - OutOfMemory should probably be refactored to be able to >>>>> signal >>>>> itself without taking up more memory, triggering itself infinitely. >>>>> Maybe >>>>> it & its signalling morph infrastructure would be good as a singleton >>>>> ** >>>>> >>>>> >>>>> >>>>> I'm confused about why it runs out of memory. According to htop the >>>>> image >>>>> only takes up about 520-540 MB of RAM when it reaches the 'OutOfMemory' >>>>> condition. This Macbook Air laptop has 4GB, and has plenty of room for >>>>> the >>>>> image to grow. Also I've specified a 1,000MB image size when starting. >>>>> So >>>>> it should have plenty of room. Is there something I should check or a >>>>> flag >>>>> somewhere that prevents it from growing on a Mac? This is the latest >>>>> Pharo30 VM. >>>>> >>>>> >>>>> Thanks for helping me get to the bottom of this >>>>> >>>>> Paul >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Sven Van Caekenberghe-2 wrote >>>>>> Hi Paul, >>>>>> >>>>>> I think you must be doing something wrong with your class, the #do: is >>>>>> implemented as streaming over the record one by one, never holding >>>>>> more >>>>>> than one in memory. >>>>>> >>>>>> This is what I tried: >>>>>> >>>>>> 'paul.csv' asFileReference writeStreamDo: [ :file| >>>>>> ZnBufferedWriteStream on: file do: [ :out | >>>>>> (NeoCSVWriter on: out) in: [ :writer | >>>>>> writer writeHeader: { #Number. #Color. #Integer. #Boolean}. >>>>>> 1 to: 1e7 do: [ :each | >>>>>> writer nextPut: { each. #(Red Green Blue) atRandom. 1e6 >>>>>> atRandom. >>>>>> #(true false) atRandom } ] ] ] ]. >>>>>> >>>>>> This results in a 300Mb file: >>>>>> >>>>>> $ ls -lah paul.csv >>>>>> -rw-r--r--@ 1 sven staff 327M Nov 14 20:45 paul.csv >>>>>> $ wc paul.csv >>>>>> 10000001 10000001 342781577 paul.csv >>>>>> >>>>>> This is a selective read and collect (loads about 10K records): >>>>>> >>>>>> Array streamContents: [ :out | >>>>>> 'paul.csv' asFileReference readStreamDo: [ :in | >>>>>> (NeoCSVReader on: (ZnBufferedReadStream on: in)) in: [ :reader | >>>>>> reader skipHeader; addIntegerField; addSymbolField; >>>>>> addIntegerField; >>>>>> addFieldConverter: [ :x | x = #true ]. >>>>>> reader do: [ :each | each third < 1000 ifTrue: [ out nextPut: each >>>>>> ] >>>>>> ] ] ] ]. >>>>>> >>>>>> This worked fine on my MacBook Air, no memory problems. It takes a >>>>>> while >>>>>> to parse that much data, of course. >>>>>> >>>>>> Sven >>>>>> >>>>>>> On 14 Nov 2014, at 19:08, Paul DeBruicker < >>>>> >>>>>> pdebruic@ >>>>> >>>>>> > wrote: >>>>>>> >>>>>>> Hi - >>>>>>> >>>>>>> I'm processing a 9 GBs of CSV files (the biggest file is 220MB or >>>>>>> so). >>>>>>> I'm not sure if its because of the size of the files or the code I've >>>>>>> written to keep track of the domain objects I'm interested in, but >>>>>>> I'm >>>>>>> getting out of memory errors & crashes in Pharo 3 on Mac with the >>>>>>> latest >>>>>>> VM. I haven't checked other vms. >>>>>>> >>>>>>> I'm going to profile my own code and attempt to split the files >>>>>>> manually >>>>>>> for now to see what else it could be. >>>>>>> >>>>>>> >>>>>>> Right now I'm doing something similar to >>>>>>> >>>>>>> |file reader| >>>>>>> file:= '/path/to/file/myfile.csv' asFileReference readStream. >>>>>>> reader: NeoCSVReader on: file >>>>>>> >>>>>>> reader >>>>>>> recordClass: MyClass; >>>>>>> skipHeader; >>>>>>> addField: #myField:; >>>>>>> .... >>>>>>> >>>>>>> >>>>>>> reader do:[:eachRecord | self seeIfRecordIsInterestingAndIfSoKeepIt: >>>>>>> eachRecord]. >>>>>>> file close. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Is there a facility in NeoCSVReader to read a file in batches (e.g. >>>>>>> 1000 >>>>>>> lines at a time) or an easy way to do that ? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Paul >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi… >>>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi… >>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > > > > > -- > View this message in context: http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi… > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > > > ------------------------------ > > Message: 2 > Date: Sat, 15 Nov 2014 00:07:35 +0100 > From: Sven Van Caekenberghe <sven(a)stfx.eu> > To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org> > Subject: Re: [Pharo-users] running out of memory while processing a > 220MB csv file with NeoCSVReader - tips? > Message-ID: <4196B7D2-63C3-4E17-89DF-A0C7ACA919EF(a)stfx.eu> > Content-Type: text/plain; charset=us-ascii > > >> On 14 Nov 2014, at 23:14, Paul DeBruicker <pdebruic(a)gmail.com> wrote: >> >> Hi Sven >> >> Yes, like I said earlier, after your first email, that I think its not a >> problem with NeoCSV as with what I'm doing and an out of memory condition. >> >> Have you ever seen a stack after sending kill -SIGUSR1 that looks like this: >> >> output file stack is full. >> output file stack is full. >> output file stack is full. >> output file stack is full. >> output file stack is full. >> .... >> >> >> What does that mean? > > I don't know, but I think that you are really out of memory. > BTW, I think that setting no flags is better, memory will expand maximally then. > I think the useful maximum is closer to 1GB than 2GB. > >> Answers to your questions below. > > It is difficult to follow what you are doing exactly, but I think that you underestimate how much memory a parsed, structured/nested object uses. Taking the second line of your example, the 20+ fields, with 3 DateAndTimes, easily cost between 512 and 1024 bytes per record. That would limit you to between 1M and 2M records. > > I tried this: > > Array streamContents: [ :data | > 5e2 timesRepeat: [ > data nextPut: (Array streamContents: [ :out | > 20 timesRepeat: [ out nextPut: Character alphabet ]. > 3 timesRepeat: [ out nextPut: DateAndTime now ] ]) ] ]. > > it worked to 5e5, but not for 5e6 - I didn't try numbers in between as it takes very long. > > Good luck, if you can solve this, please tell us how you did it. > >> Thanks again for helping me out >> >> >> >> Sven Van Caekenberghe-2 wrote >>> OK then, you *can* read/process 300MB .csv files ;-) >>> >>> What does your CSV file look like, can you show a couple of lines ? >>> >>> here are 2 lines + a header: >>> >>> "provnum","Provname","address","city","state","zip","survey_date_output","SurveyType","defpref","tag","tag_desc","scope","defstat","statdate","cycle","standard","complaint","filedate" >>> "015009","BURNS NURSING HOME, INC.","701 MONROE STREET >>> NW","RUSSELLVILLE","AL","35653","2013-09-05","Health","F","0314","Give >>> residents proper treatment to prevent new bed (pressure) sores or heal >>> existing bed sores.","D","Deficient, Provider has date of >>> correction","2013-10-10",1,"Y","N","2014-01-01" >>> "015009","BURNS NURSING HOME, INC.","701 MONROE STREET >>> NW","RUSSELLVILLE","AL","35653","2013-09-05","Health","F","0315","Ensure >>> that each resident who enters the nursing home without a catheter is not >>> given a catheter, unless medically necessary, and that incontinent >>> patients receive proper services to prevent urinary tract infections and >>> restore normal bladder functions.","D","Deficient, Provider has date of >>> correction","2013-10-10",1,"Y","N","2014-01-01" >>> >>> >>> You are using a custom record class of your own, what does that look like >>> or do ? >>> >>> A custom record class. This is all publicly available data but I'm >>> keeping track of the performance of US based health care providers during >>> their annual inspections. So the records are notes of a deficiency during >>> the inspection and I'm keeping those notes in a collection in an instance >>> of the health care provider's class. The custom record class just >>> converts the CSV record to objects (Integers, Strings, DateAndTime) and >>> then gets stuffed in the health care provider's deficiency history >>> OrderedCollection (which has about 100 items). Again I don't think its >>> what I'm doing as much as the image isn't growing when it needs to. >>> >>> >>> >>> >>> Maybe you can try using Array again ? >>> >>> I've attempted to do it where I parse and convert the entire CSV into >>> domain objects then add them to the image and the parsing works fine, but >>> the system runs out of resources during the update phase. >>> >>> >>> What percentage of records read do you keep ? In my example it was very >>> small. Have you tried calculating your memory usage ? >>> >>> >>> I'm keeping some data from every record, but it doesn't load more than >>> 500MB of the data before falling over. I am not attempting to load the >>> 9GB of CSV files into one image. For 95% of the records in the CSV file >>> 20 of the 22 columns of the data is the same from file to file, just a >>> 'published date' and a 'time to expiration' date changes. Each file >>> covers a month, with about 500k deficiencies. Each month some >>> deficiencies are added to the file and some are resolved. So the total >>> number of deficiencies in the image is about 500k. Of those records that >>> don't expire in a given month I'm adding the published date to a >>> collection of published dates for the record and also adding the "time to >>> expiration" to a collection of those to record what was made public and >>> letting the rest of the data get GC'd. I don't only load those two >>> records because the other fields of the record in the CSV could change. >>> >>> I have not calculated the memory usage for the collection because I >>> thought it would have no problem fitting in the 2GB of RAM I have on this >>> machine. >>> >>> >>> >>>> On 14 Nov 2014, at 22:34, Paul DeBruicker < >> >>> pdebruic@ >> >>> > wrote: >>>> >>>> Yes. With the image & vm I'm having trouble with I get an array with >>>> 9,942 >>>> elements in it. So its works as you'd expect. >>>> >>>> While processing the CSV file the image stays at about 60MB in RAM. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Sven Van Caekenberghe-2 wrote >>>>> Can you successfully run my example code ? >>>>> >>>>>> On 14 Nov 2014, at 22:03, Paul DeBruicker < >>>> >>>>> pdebruic@ >>>> >>>>> > wrote: >>>>>> >>>>>> Hi Sven, >>>>>> >>>>>> Thanks for taking a look and testing the NeoCSVReader portion for me. >>>>>> You're right of course that there's something I'm doing that's slow. >>>>>> But. >>>>>> There is something I can't figure out yet. >>>>>> >>>>>> To provide a little more detail: >>>>>> >>>>>> When the 'csv reading' process completes successfully profiling shows >>>>>> that >>>>>> most of the time is spent in NeoCSVReader>>#peekChar and using >>>>>> NeoCSVReader>>##addField: to convert a string to a DateAndTime. >>>>>> Dropping >>>>>> the DateAndTime conversion speeds things up but doesn't stop it from >>>>>> running >>>>>> out of memory. >>>>>> >>>>>> I start the image with >>>>>> >>>>>> ./pharo-ui --memory 1000m myimage.image >>>>>> >>>>>> Splitting the CSV file helps: >>>>>> ~1.5MB 5,000 lines = 1.2 seconds. >>>>>> ~15MB 50,000 lines = 8 seconds. >>>>>> ~30MB 100,000 lines = 16 seconds. >>>>>> ~60MB 200,000 lines = 45 seconds. >>>>>> >>>>>> >>>>>> It seems that when the CSV file crosses ~70MB in size things start >>>>>> going >>>>>> haywire with performance, and leads to the out of memory condition. >>>>>> The >>>>>> processing never ends. Sending "kill -SIGUSR1" prints a stack >>>>>> primarily >>>>>> composed of: >>>>>> >>>>>> 0xbffc5d08 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n) >>>>>> OutOfMemory class >>>>>> 0xbffc5d20 M OutOfMemory class(Behavior)>basicNew 0x1f7ac060: a(n) >>>>>> OutOfMemory class >>>>>> 0xbffc5d38 M OutOfMemory class(Behavior)>new 0x1f7ac060: a(n) >>>>>> OutOfMemory >>>>>> class >>>>>> 0xbffc5d50 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n) >>>>>> OutOfMemory class >>>>>> 0xbffc5d68 M OutOfMemory class(Behavior)>basicNew 0x1f7ac060: a(n) >>>>>> OutOfMemory class >>>>>> 0xbffc5d80 M OutOfMemory class(Behavior)>new 0x1f7ac060: a(n) >>>>>> OutOfMemory >>>>>> class >>>>>> 0xbffc5d98 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n) >>>>>> OutOfMemory class >>>>>> >>>>>> So it seems like its trying to signal that its out of memory after its >>>>>> out >>>>>> of memory which triggers another OutOfMemory error. So that's why >>>>>> progress >>>>>> stops. >>>>>> >>>>>> >>>>>> ** Aside - OutOfMemory should probably be refactored to be able to >>>>>> signal >>>>>> itself without taking up more memory, triggering itself infinitely. >>>>>> Maybe >>>>>> it & its signalling morph infrastructure would be good as a singleton >>>>>> ** >>>>>> >>>>>> >>>>>> >>>>>> I'm confused about why it runs out of memory. According to htop the >>>>>> image >>>>>> only takes up about 520-540 MB of RAM when it reaches the 'OutOfMemory' >>>>>> condition. This Macbook Air laptop has 4GB, and has plenty of room for >>>>>> the >>>>>> image to grow. Also I've specified a 1,000MB image size when starting. >>>>>> So >>>>>> it should have plenty of room. Is there something I should check or a >>>>>> flag >>>>>> somewhere that prevents it from growing on a Mac? This is the latest >>>>>> Pharo30 VM. >>>>>> >>>>>> >>>>>> Thanks for helping me get to the bottom of this >>>>>> >>>>>> Paul >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Sven Van Caekenberghe-2 wrote >>>>>>> Hi Paul, >>>>>>> >>>>>>> I think you must be doing something wrong with your class, the #do: is >>>>>>> implemented as streaming over the record one by one, never holding >>>>>>> more >>>>>>> than one in memory. >>>>>>> >>>>>>> This is what I tried: >>>>>>> >>>>>>> 'paul.csv' asFileReference writeStreamDo: [ :file| >>>>>>> ZnBufferedWriteStream on: file do: [ :out | >>>>>>> (NeoCSVWriter on: out) in: [ :writer | >>>>>>> writer writeHeader: { #Number. #Color. #Integer. #Boolean}. >>>>>>> 1 to: 1e7 do: [ :each | >>>>>>> writer nextPut: { each. #(Red Green Blue) atRandom. 1e6 >>>>>>> atRandom. >>>>>>> #(true false) atRandom } ] ] ] ]. >>>>>>> >>>>>>> This results in a 300Mb file: >>>>>>> >>>>>>> $ ls -lah paul.csv >>>>>>> -rw-r--r--@ 1 sven staff 327M Nov 14 20:45 paul.csv >>>>>>> $ wc paul.csv >>>>>>> 10000001 10000001 342781577 paul.csv >>>>>>> >>>>>>> This is a selective read and collect (loads about 10K records): >>>>>>> >>>>>>> Array streamContents: [ :out | >>>>>>> 'paul.csv' asFileReference readStreamDo: [ :in | >>>>>>> (NeoCSVReader on: (ZnBufferedReadStream on: in)) in: [ :reader | >>>>>>> reader skipHeader; addIntegerField; addSymbolField; >>>>>>> addIntegerField; >>>>>>> addFieldConverter: [ :x | x = #true ]. >>>>>>> reader do: [ :each | each third < 1000 ifTrue: [ out nextPut: each >>>>>>> ] >>>>>>> ] ] ] ]. >>>>>>> >>>>>>> This worked fine on my MacBook Air, no memory problems. It takes a >>>>>>> while >>>>>>> to parse that much data, of course. >>>>>>> >>>>>>> Sven >>>>>>> >>>>>>>> On 14 Nov 2014, at 19:08, Paul DeBruicker < >>>>>> >>>>>>> pdebruic@ >>>>>> >>>>>>> > wrote: >>>>>>>> >>>>>>>> Hi - >>>>>>>> >>>>>>>> I'm processing a 9 GBs of CSV files (the biggest file is 220MB or >>>>>>>> so). >>>>>>>> I'm not sure if its because of the size of the files or the code I've >>>>>>>> written to keep track of the domain objects I'm interested in, but >>>>>>>> I'm >>>>>>>> getting out of memory errors & crashes in Pharo 3 on Mac with the >>>>>>>> latest >>>>>>>> VM. I haven't checked other vms. >>>>>>>> >>>>>>>> I'm going to profile my own code and attempt to split the files >>>>>>>> manually >>>>>>>> for now to see what else it could be. >>>>>>>> >>>>>>>> >>>>>>>> Right now I'm doing something similar to >>>>>>>> >>>>>>>> |file reader| >>>>>>>> file:= '/path/to/file/myfile.csv' asFileReference readStream. >>>>>>>> reader: NeoCSVReader on: file >>>>>>>> >>>>>>>> reader >>>>>>>> recordClass: MyClass; >>>>>>>> skipHeader; >>>>>>>> addField: #myField:; >>>>>>>> .... >>>>>>>> >>>>>>>> >>>>>>>> reader do:[:eachRecord | self seeIfRecordIsInterestingAndIfSoKeepIt: >>>>>>>> eachRecord]. >>>>>>>> file close. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Is there a facility in NeoCSVReader to read a file in batches (e.g. >>>>>>>> 1000 >>>>>>>> lines at a time) or an easy way to do that ? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Paul >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi… >>>>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi… >>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >> >> >> >> >> >> -- >> View this message in context: http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi… >> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > > > > ------------------------------ > > Message: 3 > Date: Sat, 15 Nov 2014 00:09:48 +0100 > From: Alain Rastoul <alf.mmm.cat(a)gmail.com> > To: pharo-users(a)lists.pharo.org > Subject: Re: [Pharo-users] Has anyone tried compiling the Pharo VM > into JS using Emscripten? > Message-ID: <m4623q$btk$1(a)ger.gmane.org> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi Andy, > > If I understand your question, you want to remake Dan Ingall's lively > kernel ? > :) > http://lively-web.org/welcome.html > > Cheers, > Alain > > > Le 14/11/2014 22:31, Andy Burnett a ?crit : >> I just saw this implementation of SQLite as a JS system, via Emscripten, >> and I was curious whether something similar would be even vaguely >> possible for the VM. >> >> Cheers >> Andy >> ? > > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Pharo-users mailing list > Pharo-users(a)lists.pharo.org > http://lists.pharo.org/mailman/listinfo/pharo-users_lists.pharo.org > > > ------------------------------ > > End of Pharo-users Digest, Vol 19, Issue 53 > *******************************************

3 2

Has anyone tried compiling the Pharo VM into JS using Emscripten?
by Andy Burnett Nov. 15, 2014

Nov. 15, 2014

I just saw this implementation of SQLite as a JS system, via Emscripten, and I was curious whether something similar would be even vaguely possible for the VM. Cheers Andy á§

4 4

Problems with Roassal2 & Pharo 3.0
by volkert＠nivoba.de Nov. 15, 2014

Nov. 15, 2014

Started today with a fresh download of Pharo 3.0 and installed Roassal2 from the Configbrowser. Looks like Roassal now depends on the GTToolkit and other things, which are not correctly handled in the Config. What is missing here? What packages should be loaded? It is a bit disappointing how unstable/fragile the things are .... i see that you want make progress, but it is really hard to follow the development when not a "hardcore&fulltime" User .... Volkert

2 3

Ploting all history of the commits in a Monticello repository
by Offray Vladimir Luna CÃ¡rdenas Nov. 14, 2014

Nov. 14, 2014

Hi, I don't know if the terminology is the proper one. Anyway I'm trying to plot a "time line" of the history of a project which is stored on [1] and [2] (after rebranding) and also of the commits to documentation on [3] using Pharo/Roassal. [1] http://smalltalkhub.com/#!/~Offray/Ubakye/ [2] http://smalltalkhub.com/#!/~Offray/Grafoscopio/ [3] http://mutabit.com/deltas/repos.fossil/grafoscopio/timeline?n=200 Given I don't know any API of the sites, I thought it would be easier by scraping the data on the web pages and the using it into Roassal, but after testing some solutions on python, I came back to pharo to see if I can made everything on it, including the data collection. For [1] and [2] I think that is possible to send a message to Monticello to show all commits of a specific package (like it actually does from the UI) and export them with the information about date of commit for all commits. For [3] I'm thinking in using some combination of Zinc with Soup[4] despite of the lack of tutorials. [4] http://www.squeaksource.com/@Y-xGMSBYZmiWQToh/GJDsBRqL So here are my questions: - There is any way to say to Monticello to export all commits dates and times for a specific package? - There is any advised scraping strategy for getting the same information from the web page at [3]? The plot I would like to do is similar to the one of the ebola example at [5], but using high as number of commits and x axis as dates. [5] https://dl.dropboxusercontent.com/u/31543901/AgileVisualization/QuickStart/… Any help is welcomed, as always. Cheers, Offray

2 3

Uninstalling NonInteractiveTranscript (or reinstalling default Transcript)
by Esteban A. Maringolo Nov. 14, 2014

Nov. 14, 2014

How do I revert the installation of the following?: NonInteractiveTranscript stdout install. Is the following statement enough? ThreadSafeTranscript install Regards!

2 1

Metacello configuration based on subdirectory of Git repo
by Rafael Luque Nov. 13, 2014

Nov. 13, 2014

Hi all, I'm starting with Pharo and playing with the ways to integrate it with Git. I've read the "Git and Pharo" chapter on "Enterprise Pharo" book by Thierry Goubier and loaded the GitFileTree package. I'm working on a project with several microservices, one of them will be based on Pharo and the other ones in other technologies. The project repository is based in our own Git server and each microservice maps with one subdirectory in the repo. My question is if it is possible to define a Metacello configuration to load my packages from an specific subdirectory of a Git repository. I have tried with the following baseline method, but it fails because does not understand repoPath: baseline01: spec <version: '0.1-baseline'> spec for: #common do: [ spec blessing: #baseline. spec repository: 'git@my-git-server:my-project'. spec repoPath: 'my-pharo-subdirectory'. spec package: 'MyPackage'. . . . ] Thank you in advance.

3 6

Error when compiling a string with a quote
by Hilaire Nov. 13, 2014

Nov. 13, 2014

Hello, I get an error (see screenshot) when I compile at runtime a method where a quote is found in its body. I expect this compiled method: description ^ 'n''a pas sa force' The user inputs part of this method as: n'a pas sa force In its string representation, it is displayed as: n''a pas sa force with the quote correctly escaped. But compiling fail. I use this code to compile at runtime: stream := WriteStream on: String new. stream << 'description' << Character cr << Character tab << $^ <<$' << dialog description string << $'. scriptClass class compile: stream contents classified: 'public'. -- Dr. Geo - http://drgeo.eu iStoa - http://istoa.drgeo.eu

2 4

[Article] Lambda Calculus in Pharo
by Sven Van Caekenberghe Nov. 13, 2014

Nov. 13, 2014

Hi, I was hacking a bit and one thing let to another and I ended up writing and publishing another short article: Lambda Calculus in Pharo Yes, the Y Combinator is useful in normal programs https://medium.com/@svenvc/lambda-calculus-in-pharo-a4a571869594 Enjoy, Sven

1 0

Pharo how to get carect symbol
by Rafael Luque Nov. 13, 2014

Nov. 13, 2014

Hi all, I don't know how to get the caret (^) symbol inside Pharo in my Linux laptop. The usual combination of keys that work for me in other applications like browser, emacs or terminal does not work inside Pharo. This is the VM version output: M: NBCoInterpreter NativeBoost-CogPlugin-GuillermoPolito.19 uuid: acc98e51-2fba-4841-a965-2975997bba66 May 15 2014 NBCogit NativeBoost-CogPlugin-GuillermoPolito.19 uuid: acc98e51-2fba-4841-a965-2975997bba66 May 15 2014 https://github.com/pharo-project/pharo-vm.git Commit: ed4a4f59208968a21d82fd2406f75c2c4de558b2 Date: 2014-05-15 18:23:04 +0200 By: Esteban Lorenzano <estebanlm(a)gmail.com> Jenkins build #14826 I'm trying with the following images: * 4.0 #40360 * 3.0 #30860 Thank you.

3 4

Glamour stream presentations
by Evan Donahue Nov. 12, 2014

Nov. 12, 2014

Hello, I was just wondering if it was possible to use list/table presentations (or an analogous presentation) in Glamour to display streams that load on demand as you page through them (rather than loading all entries into memory before displaying the first page). If not, does anyone have any tips on the best way to go about creating one? Thanks, Evan

1 0

Anyone having a Rserve client?
by phil＠highoctane.be Nov. 12, 2014

Nov. 12, 2014

I am doing some R work and wanted to integrate with Pharo. Is there any support for Rserve around? http://www.rforge.net/Rserve/dev.html At this point I am doing OProcess style stuff with littler but Rserve would be cooler. There are binding for Ruby and Python. e.g. http://pythonhosted.org//pyRserve/ https://github.com/ralhei/pyRserve Stats are becoming bigger and bigger and the leader of the pack is R. DHB is fine and I do use some of it but let's face it, it is nowhere near the power of R... Phil

4 11

Memoization Question
by Torsten Bergmann Nov. 12, 2014

Nov. 12, 2014

Werner Kassens wrote: >Hi, >now this a nice idea, especially the memoizedUsing:cache idea. i would >really appreciate it, if that would be MIT licenced. >werner It already is, thanks to the permission of author John Cromartie who answered us today. Both methods are in a slice already: Details in https://pharo.fogbugz.com/f/cases/14458 Thx T.

4 4

String representation of Class
by Hilaire Nov. 12, 2014

Nov. 12, 2014

Hi, Dr. Geo script are becoming first class citizen -- before it was just a method hooked to an existing DrGeoUserScripts class ; then, it was easy to export the script method as a string to include it in the XML sketch file. Now to save a Dr. Geo sketch using a script, the whole script class needs to be embedded in the sketch XML representation. Is it possible to export a whole class as a string, then import it back from a string representation to a class? Thanks Hilaire -- Dr. Geo - http://drgeo.eu iStoa - http://istoa.drgeo.eu

4 8

Re: [Pharo-users] Is there a way to store workspace 'gists' in Pharo?
by Andy Burnett Nov. 11, 2014

Nov. 11, 2014

Stef said: << I usually create a method and a class. because script do not compose well. >> á§ > > Thanks, that sounds interesting. And, just for clarity, do you mean that you create a single class e.g. MyGists and then create methods for each of your useful snippets? Or, is there a benefit in creating a class per gist? Cheers

2 1

Re: [Pharo-users] Memoization Question
by Torsten Bergmann Nov. 11, 2014

Nov. 11, 2014

Phil wrote: >thanks for the pointer. >Now, how do I use that? First: create a method BlockClosure>>#memoized in your Pharo image (see https://gist.github.com/jcromartie/2705526) memoized "returns memoized version of an unary function" | cache | cache := Dictionary new. ^ [ :x | cache at: x ifAbsentPut: [ self value: x ] ]! So the idea of #memoized is to return a new "outer" block who evaluates the original block only when the computation of the inner block was not yet done/the result of the inner evaluation is not yet cached. As you see from the comment it works for "unary functions" which would mean it works for "one argument blocks". Simple example: |square memo | square:= [:x | Transcript show: 'Processing'. x * x]. memo := square memoized. memo value: 2. memo value: 3. memo value: 2. The Transcript only shows "Processing" two times instead of three times, so only for the first two #value: calls we have to calculate x * x. For the last evaluation on the memo block (with argument 2 again) the result is already known from the cache, so the square block is not evaluated again. Instead the cached result could be returned which could be faster than doing the computation again. x*x is only an example - this "memoization" can usually speed up when a block is costly (performancewise) and is often called with the same arguments. A precondition (for this caching) is that the block evalutes the same way/returns the same result for the same input argument. You also try the two prepared workspaces in Pharo: Non-memoized version: http://ws.stfx.eu/5ZE87XSFL9IH Memoized Version: http://ws.stfx.eu/QLQLSLT7PZ4C Hope this makes it more clear. This simple example is a nice performance trick and shows again the power of Smalltalk and block closures. Dont know about the license - but we could ask author John Cromartie for MIT license to include this simple method into the standard image. I will try to contact him. Bye T.

3 2

Is there a way to store workspace 'gists' in Pharo?
by Andy Burnett Nov. 11, 2014

Nov. 11, 2014

>From time to time I write a workspace script that I would like to keep. Obviously, I could save them to a file, but I was wondering if there was a way of saving them in the image, and browsing them in some friendly way? Cheers Andy á§

5 4

Memoization Question
by Torsten Bergmann Nov. 10, 2014

Nov. 10, 2014

1 0

Re: [Pharo-users] Ploting all history of the commits in a Monticello repository
by Stephan Eggermont Nov. 10, 2014

Nov. 10, 2014

You might get some inspiration by taking a look at http://smalltalkhub.com/#!/~StephanEggermont/DeprecationFinder Stephan

1 0

Re: [Pharo-users] Is there a way to store workspace 'gists' in Pharo?
by Andy Burnett Nov. 10, 2014

Nov. 10, 2014

á§ Doru said <<< If you work with Pharo 4, they should already be saved in a play-cache folder. >> Great, I shall go and experiment. Cheers Andy

1 0

AST #copy behavior
by Mark Rizun Nov. 10, 2014

Nov. 10, 2014

Hi guys! I'm using AST really often, and recently stumbled on problem with copying an AST. It doesn't copy stuff related to computing sourceInterval of AST, and I need it to copy:) I know there are #deepCopy and #veryDeepCopy, but they are too heavy methods for AST, especially in my case, as I need to copy AST all the time. Earlier, I discussed this problem with Marcus Denker. He suggested to fix #copy of AST (to be more accurate #postCopy). If I want to copy stuff related to sourceInterval, I would have to implement #postCopy for each RBNode... differently. RBProgramNode>>#postCopy super postCopy. properties := properties copy So for each RBNode... it will be like: RBNode...>>#postCopy super postCopy. "here some different code each time" I'm just interested: What do you think about it? Thanks, Mark -- View this message in context: http://forum.world.st/AST-copy-behavior-tp4789447.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

2 2

Webdoc: status?
by phil＠highoctane.be Nov. 10, 2014

Nov. 10, 2014

I'd like to use WebDoc for generating my project code doc; I see that https://ci.inria.fr/pharo-contribution/job/WebDoc/ is pretty much red for a long time. Started by timer Building remotely on pharo-contribution-win7.ci.inria.fr <https://ci.inria.fr/pharo-contribution/computer/pharo-contribution-win7.ci.…> (win7 32 win) in workspace C:\builds\workspace\WebDoc Deleting project workspace... done Triggering 30,development,vm <https://ci.inria.fr/pharo-contribution/job/WebDoc/PHARO=30,VERSION=developm…>30,development,vm <https://ci.inria.fr/pharo-contribution/job/WebDoc/PHARO=30,VERSION=developm…> completed with result FAILURE Deleting matrix project workspace... Skipped based on build state FAILURE Finished: FAILURE But the green one isn't more indicative of success... What's the status? Is it used by someone? I tried the config and it crashed; github-camillobruniPPMarkdown184.... does not exit. :-( The catalog says: https://ci.inria.fr/pharo-contribution/job/PharoProjectCatalog/HTML_Report/ So, - I cloned Esteban's github repo (git(a)github.com=estebanlm/pharo-webdoc.git - git submodule init update --init Loaded by hand the packages in the submodule (petitmarkdown) Loaded the WebDoc package (the SimpleWebDoc doesn't load, even if I have Seaside in my image). In a workspace: (WebDocExporter packages: RPackageOrganizer default packages) exportTo: './doc' asFileReference. Which started to generate html files in doc/ nicely but choked while exporting FFITestLibrary Ideas why it fails? Known issue? TIA Phil

1 0

[Survey] Reasoning about objects
by Andrei Chis Nov. 10, 2014

Nov. 10, 2014

Hi all, In order to improve object inspectors we are doing a survey. We want to know whether several features are important to you or not when trying to understand objects. This is a very short survey and should't take you more than 10 minutes to complete. Please take this time to participate. Here is the link to the survey: http://goo.gl/forms/xKPxqLxJJv Cheers, Andrei

1 0

next sttep after the example tutorial
by Roelof Wobben Nov. 8, 2014

Nov. 8, 2014

Hello, Is there a book / tutorial/ challenges that I can use to become a expert in Pharo ? Roelof

7 16

About PBE status was Re: How not to serve an AngularJs application from Pharo
by kilon alios Nov. 8, 2014

Nov. 8, 2014

I converted the first 5 chapters to Pharo 3 , 1 chapter has been ported by Stef and another from Damien, so we are more or less at 50% but I gave up as I was burned out and fed up documenting stuff :D. But (Arnold accent) I will be back ! My conversion was not flawless since this was my first project using Pillar. I think however that is ok quality. By the way I am aware of the work on the documentation for all books generally because I follow my github feed. On Fri, Nov 7, 2014 at 3:11 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Kilon, > > You did start with converting PBE, right ? What is the current status ? I > kind of liked the git book. And it is badly needed, indeed. > > Sven > > > On 07 Nov 2014, at 13:56, kilon alios <kilon.alios(a)gmail.com> wrote: > > > > Stef I did not meant it as offence, of course you do great amount of > work for documentation. But I cannot ignore the fact that PBE is still > stuck to 1.4, that there is no documentation for Pharo Morphic and most > libraries if not all are partly documented."lacks documentation" does not > mean that there is no documentation. I meant exactly what I typed. > > > > Stef you guys do an amazing job and I am proud to be a member of the > Pharo community and make my own small contributions to make Pharo more > popular and better. > > > > On Fri, Nov 7, 2014 at 2:27 PM, stepharo <stepharo(a)free.fr> wrote: > > > > On 4/11/14 13:53, kilon alios wrote: > >> I have welcomed Amber with open arms, but I have failed miserably to > wrap my head around it. > >> > >> Is ironic if you think about, Pharo really lacks documentation even in > basic areas, > > Are you joking? I spend my time writing books and chapters and their > quality is quite good. > >> libraries are nowhere near as powerful as those offered by javascript > yet I have found Pharo way easier to use. Web development looks to me like > a big pile of unnecessary mess. Actually Amber is the best part of it :D > >> > >> So the problem for me is not so much the System Browser , but the steep > learning curve . Or maybe its just the fact that I dont like many of the > web frameworkds, including html / js / css etc. > >> > >> I wish I could make something in pharo and click a button to deploy it > as a website but alas thats just a dream. > > > > I would like to have a solution for that without having a smalltalk > compiler in JS as amber is doing. > >> I think however that for the amount of developers Amber has been making > some amazing progress. > >> > >> > >> > >> On Tue, Nov 4, 2014 at 4:55 AM, Pierce Ng <pierce(a)samadhiweb.com> > wrote: > >> On Mon, Oct 27, 2014 at 10:21:18PM +0200, kilon alios wrote: > >> > Even though I always have been a supporter of Amber the IDE has still > a > >> > very long way to go. > >> > >> I've been meaning to build a code browser for Amber (and GST) in Pharo > with > >> Omnibrowser using the Shampoo protocol[1], once I get some copious > spare time. > >> > >> [1] http://dmitrymatveev.co.uk/shampoo > >> > >> Pierce > >> > >> > > > > > > >

4 4