July 2017 - Pharo-users - lists.pharo.org

Object count/memory usage in Pharo6 WAS Re: Critical issues for Dr. Geo on P6

by Holger Freyther

> On 22. Jul 2017, at 14:00, Hilaire <hilaire(a)drgeo.eu> wrote: > > Hi Holger, > > With Pharo3 I was proceeding like this, by uninstallation of packages. However there is the promise of building from a minimal image, but it is not documented AFAIK. > > By curiosity, what is the size of your resulting image. I see 31mb image size for Pharo3 and 49mb for Pharo6. The changes file is about the same. From the top command on a test system. "pharo" is a Pharo6 VM and image and pharo-vm is the pharo3 system. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4206 root 20 0 1056684 74036 1600 S 5.7 1.2 1582:15 pharo-vm 10459 root 20 0 104040 90348 2124 S 5.0 1.5 369:00.29 pharo I looked a bit at memory usage and see if some random changes make a difference * Shrink ZnMimeType ExtensionMaps to a minimum when not handling extensions? * ASTCache reset to get rid of the spotter cache. Not sure if the SessionManager runs ASTCache>>#shutDown properly. It looks like it should but it doesn't? * ByteString compaction? I did strs := ByteString allInstances. str -> strs asSet. The numbers on a not fully clean image are 99874 vs. 42139. I wonder in which hell I end if I use >>#becomeForward: to compact it (I tried this once for GNU Smalltalk and I am sure others tried it before). I think anyone using == on String is using the wrong class? Any good arguments against doing it? I am moving my Pharo6 image into production right now and then will look into setting up CI for Pharo7 with a minimal image. cheers holger

July 26, 2017

Re: [Pharo-users] Critical issues for Dr. Geo on P6

by Holger Freyther

> On 22. Jul 2017, at 12:07, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: > > Hi holger Hey Stef! > "the growth from Pharo3->Pharo6" us too :) > this is why we invested in the bootstrap and this is why we will > remove (and we started) packages and classes. > And this is also why we will continue to repackage the system. yes this is very fascinating for Pharo7. It would be nice if we could track size (image+changes but also ram usage, object count) over time to see our progress. I recently added ordinary metric upload to my "bob-bench.org" test and metric tracking. ;) holger

July 26, 2017

Zinc: Are start and stop meant to be idempotent?

by Herby VojÄÃk

Hi, Looking at ZnServer >> start and ZnSingleThreadedServer >> stop (as ZnServer >> stop is #subclassResponsibility), I had the impression they are meant to be idempotent. The start comment is: "Start me. I will start listening on my port for incoming HTTP connections. If I am running, I will first stop and thus effectively restart" The stop comment is: "Stop me. I will stop listening on my port for incoming HTTP connections. Does nothing when I am not running" So, it seems like both start and stop try to be safe to be sent twice in a row. I did a few Transcript prints: --- 1. just stop: is ok. ZnManagingMultiThreadedServer allInstances do: #stop. ZnServer managedServers do: #unregister. Smalltalk garbageCollect. (ZnServer on: 4998) stop. (ZnServer on: 4997) stop; stop. Transcript show: ZnServer managedServers; cr; show: ZnManagingMultiThreadedServer allInstances; cr. ==> an IdentitySet() an Array(a ZnManagingMultiThreadedServer(stopped 4998) a ZnManagingMultiThreadedServer(stopped 4997)) --- 2. start; stop: is ok. ZnManagingMultiThreadedServer allInstances do: #stop. ZnServer managedServers do: #unregister. Smalltalk garbageCollect. (ZnServer on: 4998) start; stop. (ZnServer on: 4997) start; stop; stop. Transcript show: ZnServer managedServers; cr; show: ZnManagingMultiThreadedServer allInstances; cr. ==> an IdentitySet() an Array(a ZnManagingMultiThreadedServer(stopped 4998) a ZnManagingMultiThreadedServer(stopped 4997)) --- 3. register; start; stop: is ok. ZnManagingMultiThreadedServer allInstances do: #stop. ZnServer managedServers do: #unregister. Smalltalk garbageCollect. (ZnServer on: 4998) register; start; stop. (ZnServer on: 4997) register; start; stop; stop. Transcript show: ZnServer managedServers; cr; show: ZnManagingMultiThreadedServer allInstances; cr. ==> an IdentitySet() an Array(a ZnManagingMultiThreadedServer(stopped 4998) a ZnManagingMultiThreadedServer(stopped 4997)) --- 4. start: is ok. ZnManagingMultiThreadedServer allInstances do: #stop. ZnServer managedServers do: #unregister. Smalltalk garbageCollect. (ZnServer on: 4998) start. (ZnServer on: 4997) start; start. Transcript show: ZnServer managedServers; cr; show: ZnManagingMultiThreadedServer allInstances; cr. ==> an IdentitySet() an Array(a ZnManagingMultiThreadedServer(running 4998) a ZnManagingMultiThreadedServer(running 4997)) 5. register; start: is not ok. ZnManagingMultiThreadedServer allInstances do: #stop. ZnServer managedServers do: #unregister. Smalltalk garbageCollect. (ZnServer on: 4998) register; start. (ZnServer on: 4997) register; start; start. Transcript show: ZnServer managedServers; cr; show: ZnManagingMultiThreadedServer allInstances; cr. ==> an IdentitySet(a ZnManagingMultiThreadedServer(running 4998)) an Array(a ZnManagingMultiThreadedServer(running 4998) a ZnManagingMultiThreadedServer(running 4997)) The last case is not idempotent for #start. With one #start, the server is registered, with two #start's, it is not registered. Shouldn't the last case be idempotent as well? If it is, it is the bug that should be put into issue tracker? Herby

July 26, 2017

Re: [Pharo-users] NeoCSV on Irregular Files

by Esteban A. Maringolo

2017-07-26 13:04 GMT-03:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > I agree. > > If the file is non-homegeneous it is not longer CSV by definition. > > Holding on to the original stream and creating new readers for each section is one option, an other one could be to add a #reset method. > > The big question is how to known when one section begins/ends. In my experience I looked for certain delimiters, like a header row with the field names. Oil & Gas telemetry instruments generate outputs like that, like a concatenation of several CSVs into one, maybe even with a non-csv like header of 10 rows of data. What I had to do to deal with that was either: a) Reading it line by line, buffering the hole "segment" until EOF or the next delimiter is found, or... b) Pre-scanning the whole file, and marking start and end positions of each segment, generating a new readStream with the contents and passed it the CSV parser (which doesn't care nor know about segments). > NeoCSVReader holds a one char buffer, so you could peek for something, just maybe. Then you could discover the section switches while parsing (a bit like #atEnd is used from #upToEnd, add a #atSectionEnd). But it all depends on your specific format. It's harder to do if it is char based, instead of "line" based. Or at least harder to code. Regards! Esteban A. Maringolo

July 26, 2017

Re: [Pharo-users] NeoCSV on Irregular Files

by Sven Van Caekenberghe

I agree. If the file is non-homegeneous it is not longer CSV by definition. Holding on to the original stream and creating new readers for each section is one option, an other one could be to add a #reset method. The big question is how to known when one section begins/ends. NeoCSVReader holds a one char buffer, so you could peek for something, just maybe. Then you could discover the section switches while parsing (a bit like #atEnd is used from #upToEnd, add a #atSectionEnd). But it all depends on your specific format. > On 26 Jul 2017, at 17:45, Esteban A. Maringolo <emaringolo(a)gmail.com> wrote: > > There is no way to perform this with NeoJSON or any other CSV > framework I'm aware of. > > I had to deal with that kind of "format" (which is likely an export > format), and the way to deal with it was to process each "segment" > using a different instance of the CSV reader, the segments where > scanned in the stream using the delimiting heuristics of your choice > (headers, blank lines, etc.), and then each segment was extracted and > passed as argument to the reader of that segment. > > The drawback was that if the file was big there was no way to have "a > stream over a stream" (like a window function), so passing the segment > to the reader implied copying its string contents within the segment > delimiters. > > It's something I already put some thought into, but never had the will > to code and share publicly. > > Regards, > > Esteban A. Maringolo > > > 2017-07-26 12:02 GMT-03:00 Sean P. DeNigris <sean(a)clipperadams.com>: >> I have a CSV file that has several subsections, each with its own format. >> What I'd like to do is parse one, reset the NeoCSVReader, set it up for the >> next section, and continue. I didn't see an API for this. Is it possible? >> Thanks. >> >> >> >> ----- >> Cheers, >> Sean >> -- >> View this message in context: http://forum.world.st/NeoCSV-on-Irregular-Files-tp4956850.html >> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >> >

July 26, 2017

Re: [Pharo-users] NeoCSV on Irregular Files

by Esteban A. Maringolo

There is no way to perform this with NeoJSON or any other CSV framework I'm aware of. I had to deal with that kind of "format" (which is likely an export format), and the way to deal with it was to process each "segment" using a different instance of the CSV reader, the segments where scanned in the stream using the delimiting heuristics of your choice (headers, blank lines, etc.), and then each segment was extracted and passed as argument to the reader of that segment. The drawback was that if the file was big there was no way to have "a stream over a stream" (like a window function), so passing the segment to the reader implied copying its string contents within the segment delimiters. It's something I already put some thought into, but never had the will to code and share publicly. Regards, Esteban A. Maringolo 2017-07-26 12:02 GMT-03:00 Sean P. DeNigris <sean(a)clipperadams.com>: > I have a CSV file that has several subsections, each with its own format. > What I'd like to do is parse one, reset the NeoCSVReader, set it up for the > next section, and continue. I didn't see an API for this. Is it possible? > Thanks. > > > > ----- > Cheers, > Sean > -- > View this message in context: http://forum.world.st/NeoCSV-on-Irregular-Files-tp4956850.html > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >

July 26, 2017

NeoCSV on Irregular Files

by Sean P. DeNigris

I have a CSV file that has several subsections, each with its own format. What I'd like to do is parse one, reset the NeoCSVReader, set it up for the next section, and continue. I didn't see an API for this. Is it possible? Thanks. ----- Cheers, Sean -- View this message in context: http://forum.world.st/NeoCSV-on-Irregular-Files-tp4956850.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

July 26, 2017

Re: [Pharo-users] Issue with the --headless option on Windows 10

by Stephane Ducasse

Thanks you for checking that. All the startup logic got rewritten in Pharo 6 to be able to have a more modular system and there are probably things to align. Stef On Wed, Jul 26, 2017 at 3:01 PM, Herby VojÄÃk <herby(a)mailbox.sk> wrote: > Herby VojÄÃk wrote: >> >> Nicolai Hess wrote: >>> >>> >>> >>> 2017-07-26 13:08 GMT+02:00 Herby VojÄÃk <herby(a)mailbox.sk >>> <mailto:herby@mailbox.sk>>: >>> >>> Nicolai Hess wrote: >>> >>> >>> >>> 2017-07-26 12:09 GMT+02:00 <herby(a)mailbox.sk >>> <mailto:herby@mailbox.sk> <mailto:herby@mailbox.sk >>> <mailto:herby@mailbox.sk>>>: >>> >>> >>> >>> On July 26, 2017 11:53:41 AM GMT+02:00, Sven Van Caekenberghe >>> <sven(a)stfx.eu <mailto:sven@stfx.eu> <mailto:sven@stfx.eu >>> <mailto:sven@stfx.eu>>> wrote: >>> > >>> > > On 25 Jul 2017, at 19:46, Cyril Ferlicot D. >>> ><cyril.ferlicot(a)gmail.com <mailto:cyril.ferlicot@gmail.com> >>> <mailto:cyril.ferlicot@gmail.com >>> <mailto:cyril.ferlicot@gmail.com>>> wrote: >>> > > >>> > > Le 25/07/2017 Ã 19:37, Sven Van Caekenberghe a Ã©crit : >>> > >> Hi, >>> > >> >>> > >> User Herby VojÄÃk seems to have an issue with the >>> --headless option >>> >on Windows 10 >>> > >> >>> > >> >>> >>> > >https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w… >>> >>> >>> <https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w…> >>> >>> >>> <https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w… >>> >>> >>> <https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w…>> >>> >>> > >> >>> > >> I cannot reproduce his problem. >>> > >> >>> > >> Maybe there are Windows users out there who can have a look >>> and help >>> >him ? >>> > >>> >Anyone ? >>> > >>> >>> >>> I can reproduce the strange output: >>> >>> pharo --headless Pharo.image st sample.st <http://sample.st> >>> <http://sample.st> >>> >>> with sample.st <http://sample.st> <http://sample.st> contains >>> >>> (ZnServer startOn: 8888) register. >>> Smalltalk snapshot: true andQuit: true. >>> >>> Now starting the pharo image in gui mode >>> >>> pharo --headless Pharo.image >>> >>> >>> This should probably be without --headless to be less strange >>> (running in gui mode does not need --headless, if --headless starts >>> gui without cli args in Win, that is probably another issue that >>> should be raised for win vm) >>> >>> >>> Oh, yes of course, I meant running >>> pharo Pharo.image >>> >>> >>> I can access localhost:8888 >>> >>> but >>> ZnServer managedServers evaluates to "an IdentitySet()" >>> >>> running >>> >>> pharo --headless Pharo.image eval "ZnServer managedServers" shows >>> an IdentitySet(a ZnManagingMultiThreadedServer(stopped 8888)) >>> >>> >>> Wow, it did not come to my mind to check that. That means it is >>> indeed the headless to gui transition that is problematic as >>> headless to headless is fine. >>> >>> running eval without headless option: >>> >>> pharo Pharo.image eval "ZnServer managedServers" >>> the set is again empty: >>> an IdentitySet() >>> >>> >>> >>> Thanks, Herby >>> >>> >>> It lools like ZnServer resets the ManagedServers variable, if the image >>> starts without the headless option. >> >> >> The problem is not image started without headless. It works fine if it >> started without headless, just the sample.st must also be run without >> headless. Only problem is sample.st run with headless and subsequently >> image run without headless. >> >> It unregisters it as a result of [ :each | each start ] during startup >> phase in which the server(s) is(are) already in running state, as shown >> in my last pharodebug.log posted here. >> >> Maybe it launches startup phase twice, first doing it correctly, second >> time failing it because start message is in fact >> startWhenStoppedButRestartAndUnregisterWhenRunnning. > > > It indeed looks like startup is happening twice (or Zinc installs two > startup hooks). I changed ZnSingleThreadedServer >> start to include > > Transcript cr; show: '--- ZnServer >> start ---'; show: DateAndTime now; > show: ' ---'; cr; print: self; cr. > > as the first statement, added 5 second pause between the --headless script > and gui image start and got this in the transcript window: > > > --- ZnServer >> start ---2017-07-26T14:40:57.478631+02:00 --- > a ZnManagingMultiThreadedServer(stopped 4998) > > --- ZnServer >> start ---2017-07-26T14:40:57.705631+02:00 --- > a ZnManagingMultiThreadedServer(running 4998) > > Now for the scarier part: when I ran the script in gui, and subsequently > loaded the image, the transcript shows: > > > Loaded -> Mocketry-Domain-Tests-cypress.1 --- > github://dionisiydk/Mocketry:v4.0.x [c21364a:v4.0.x] --- cache > ...finished baseline > --- ZnServer >> start ---2017-07-26T14:51:12.643052+02:00 --- > a ZnManagingMultiThreadedServer(stopped 4998) > > --- ZnServer >> start ---2017-07-26T14:51:18.359584+02:00 --- > a ZnManagingMultiThreadedServer(stopped 4998) > > It seems only one start was sent in second image, because there is 5 second > delay. > > But it is at the same time scary, as in brings more questions than answers. > >> >> Herby >> > >

July 26, 2017

Re: [Pharo-users] Issue with the --headless option on Windows 10

by Herby VojÄÃk

Herby VojÄÃk wrote: > Nicolai Hess wrote: >> >> >> 2017-07-26 13:08 GMT+02:00 Herby VojÄÃk <herby(a)mailbox.sk >> <mailto:herby@mailbox.sk>>: >> >> Nicolai Hess wrote: >> >> >> >> 2017-07-26 12:09 GMT+02:00 <herby(a)mailbox.sk >> <mailto:herby@mailbox.sk> <mailto:herby@mailbox.sk >> <mailto:herby@mailbox.sk>>>: >> >> >> >> On July 26, 2017 11:53:41 AM GMT+02:00, Sven Van Caekenberghe >> <sven(a)stfx.eu <mailto:sven@stfx.eu> <mailto:sven@stfx.eu >> <mailto:sven@stfx.eu>>> wrote: >> > >> > > On 25 Jul 2017, at 19:46, Cyril Ferlicot D. >> ><cyril.ferlicot(a)gmail.com <mailto:cyril.ferlicot@gmail.com> >> <mailto:cyril.ferlicot@gmail.com >> <mailto:cyril.ferlicot@gmail.com>>> wrote: >> > > >> > > Le 25/07/2017 Ã 19:37, Sven Van Caekenberghe a Ã©crit : >> > >> Hi, >> > >> >> > >> User Herby VojÄÃk seems to have an issue with the >> --headless option >> >on Windows 10 >> > >> >> > >> >> >https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w… >> >> <https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w…> >> >> <https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w… >> >> <https://pharo.fogbugz.com/f/cases/20271/ZnServer-managedServers-disappear-w…>> >> >> > >> >> > >> I cannot reproduce his problem. >> > >> >> > >> Maybe there are Windows users out there who can have a look >> and help >> >him ? >> > >> >Anyone ? >> > >> >> >> I can reproduce the strange output: >> >> pharo --headless Pharo.image st sample.st <http://sample.st> >> <http://sample.st> >> >> with sample.st <http://sample.st> <http://sample.st> contains >> >> (ZnServer startOn: 8888) register. >> Smalltalk snapshot: true andQuit: true. >> >> Now starting the pharo image in gui mode >> >> pharo --headless Pharo.image >> >> >> This should probably be without --headless to be less strange >> (running in gui mode does not need --headless, if --headless starts >> gui without cli args in Win, that is probably another issue that >> should be raised for win vm) >> >> >> Oh, yes of course, I meant running >> pharo Pharo.image >> >> >> I can access localhost:8888 >> >> but >> ZnServer managedServers evaluates to "an IdentitySet()" >> >> running >> >> pharo --headless Pharo.image eval "ZnServer managedServers" shows >> an IdentitySet(a ZnManagingMultiThreadedServer(stopped 8888)) >> >> >> Wow, it did not come to my mind to check that. That means it is >> indeed the headless to gui transition that is problematic as >> headless to headless is fine. >> >> running eval without headless option: >> >> pharo Pharo.image eval "ZnServer managedServers" >> the set is again empty: >> an IdentitySet() >> >> >> >> Thanks, Herby >> >> >> It lools like ZnServer resets the ManagedServers variable, if the image >> starts without the headless option. > > The problem is not image started without headless. It works fine if it > started without headless, just the sample.st must also be run without > headless. Only problem is sample.st run with headless and subsequently > image run without headless. > > It unregisters it as a result of [ :each | each start ] during startup > phase in which the server(s) is(are) already in running state, as shown > in my last pharodebug.log posted here. > > Maybe it launches startup phase twice, first doing it correctly, second > time failing it because start message is in fact > startWhenStoppedButRestartAndUnregisterWhenRunnning. It indeed looks like startup is happening twice (or Zinc installs two startup hooks). I changed ZnSingleThreadedServer >> start to include Transcript cr; show: '--- ZnServer >> start ---'; show: DateAndTime now; show: ' ---'; cr; print: self; cr. as the first statement, added 5 second pause between the --headless script and gui image start and got this in the transcript window: --- ZnServer >> start ---2017-07-26T14:40:57.478631+02:00 --- a ZnManagingMultiThreadedServer(stopped 4998) --- ZnServer >> start ---2017-07-26T14:40:57.705631+02:00 --- a ZnManagingMultiThreadedServer(running 4998) Now for the scarier part: when I ran the script in gui, and subsequently loaded the image, the transcript shows: Loaded -> Mocketry-Domain-Tests-cypress.1 --- github://dionisiydk/Mocketry:v4.0.x [c21364a:v4.0.x] --- cache ...finished baseline --- ZnServer >> start ---2017-07-26T14:51:12.643052+02:00 --- a ZnManagingMultiThreadedServer(stopped 4998) --- ZnServer >> start ---2017-07-26T14:51:18.359584+02:00 --- a ZnManagingMultiThreadedServer(stopped 4998) It seems only one start was sent in second image, because there is 5 second delay. But it is at the same time scary, as in brings more questions than answers. > > Herby >

July 26, 2017

Re: [Pharo-users] Performance of zero conf install since 6.1 seems very slow?

by Tim Mackinnon

Yes I see the same - it was seconds and now minutes. Maybe it is worth having a mirror for CI (or does it get edge cached anyway- probably not from what we are seeing, but possibly if could be?) Tim Sent from my iPhone > On 26 Jul 2017, at 09:17, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > it may been related to recent INRIA infrastructure changes. > there is nothing 6.1 itself can do with that (6.1 is just a bunch of zips) > > Esteban > >> On 26 Jul 2017, at 09:44, Guillermo Polito <guillermopolito(a)gmail.com> wrote: >> >> We were also seeing it yesterday. In travis, 10m to download an image (and thus timeout). Before it took seconds. >> >> On Wed, Jul 26, 2017 at 9:41 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: >>> Hi. >>> >>> I don't know answer. But what time it takes before? >>> >>> 2017-07-26 9:30 GMT+02:00 Tim Mackinnon <tim(a)testit.works>: >>>> Hi - has anyone else noticed that since 6.1 - using zero conf for installs is very very slow (it takes minutes to download pharo64.zip when using >>>> >>>> curl get.pharo.org/64/ | bash >>>> >>>> This is both on my local computer but also on the gitlab ci server? Is it possibly something isnât configured right? >>>> >>>> Tim >>> >> >> >> >> -- >> >> Guille Polito >> >> Research Engineer >> French National Center for Scientific Research - http://www.cnrs.fr >> >> >> Web: http://guillep.github.io >> Phone: +33 06 52 70 66 13 >

July 26, 2017