July 2013 - Pharo-users - lists.pharo.org

Re: [Pharo-users] Pharo performance

by Mariano Martinez Peck

On Wed, Jul 31, 2013 at 6:32 PM, Chris <cpmbailey(a)btinternet.com> wrote: > It definitely seems to be because most things in the V2 driver are coming > in as strings and then being converted. Running the same query on the three > systems gives the following for a 65000 double array > > PGV2 770m/s > PGV3 200m/s > Psycopg2 130m/s > > Now I just need to find out how to get PGV3 as the backend for Glorp! > > PLease please please use the Glorp version you get from the DBX suite. You need to create a subclass of DatabaseDriver. You can take a look to NativePostgresDriver and create a similar one for V3. I would also like to have Glorp + PostgresV3 running. Also...we should somehow use NativePostgresPoolingDriver and NativePostgresConnectionPool for V3... we need some refactor here... We should join forces!! > Cheers > Chris > > > Hi Yanni, >> >> On 30 Jul 2013, at 05:17, Yanni Chiu <yanni(a)rogers.com> wrote: >> >> On 29/07/13 7:08 PM, Sven Van Caekenberghe wrote: >>> >>>> The explanation for the slowdown must be in the PgV2 driver. >>>> >>> The PgV2 protocol is described at: >>> http://www.postgresql.org/**docs/7.1/static/protocol-** >>> message-formats.html<http://www.postgresql.org/docs/7.1/static/protocol-message-formats.html> >>> >>> Have a glance at the "AsciiRow" and "BinaryRow" message formats. The >>> driver reads the data off the socket, parsing the the data, as described as >>> described by the message format. With the V2 protocol design, you have to >>> read the result row, one field at a time. >>> >>> IIUC, in the newer V3? protocol, the AsciiRow/BinaryRow message is >>> replaced by a DataRow message. The DataRow message has the message size >>> included, which could allow the driver to read the entire set of fields for >>> one data row, using a single socket read (or a few buffer sized reads). >>> >>> I recall seeing an experimental V3 protocol implementation, a few years >>> back - sorry, no links handy. It would be nice to see some benchmarks. >>> >>> Hope that helps. >>> >> Thanks for the response. >> >> I believe the V3 project is here http://www.squeaksource.com/** >> PostgresV3.html <http://www.squeaksource.com/PostgresV3.html>. >> >> Now, I probably spoke too fast and should have taken Mariano's advice to >> never speculate and first measure. Here is my quick test that, for me, >> shows that PostgresV2 seems more than fast enough (something that I had >> experienced before without doing benchmarks). >> >> [ self execute: 'select longitude,latitude from log_data limit 10000;' ] >> timeToRun. >> >> => 76 ms >> >> [ self execute: 'select longitude,latitude from log_data limit 100000;' ] >> timeToRun. >> >> => 765 ms >> >> This is querying for 2 floats from a huge table, over the network. Pretty >> fast ;-) >> >> So, back to Chris: what exactly are you doing that is (so) slow ? >> >> Anyway, thanks Yanni for all your work on the existing driver ! >> >> Sven >> >> -- >> Sven Van Caekenberghe >> Proudly supporting Pharo >> http://pharo.org >> http://association.pharo.org >> http://consortium.pharo.org >> >> >> >> >> >> >> > > -- Mariano http://marianopeck.wordpress.com

July 31, 2013

Re: [Pharo-users] Pharo performance

by Chris

It definitely seems to be because most things in the V2 driver are coming in as strings and then being converted. Running the same query on the three systems gives the following for a 65000 double array PGV2 770m/s PGV3 200m/s Psycopg2 130m/s Now I just need to find out how to get PGV3 as the backend for Glorp! Cheers Chris > Hi Yanni, > > On 30 Jul 2013, at 05:17, Yanni Chiu <yanni(a)rogers.com> wrote: > >> On 29/07/13 7:08 PM, Sven Van Caekenberghe wrote: >>> The explanation for the slowdown must be in the PgV2 driver. >> The PgV2 protocol is described at: >> http://www.postgresql.org/docs/7.1/static/protocol-message-formats.html >> >> Have a glance at the "AsciiRow" and "BinaryRow" message formats. The driver reads the data off the socket, parsing the the data, as described as described by the message format. With the V2 protocol design, you have to read the result row, one field at a time. >> >> IIUC, in the newer V3? protocol, the AsciiRow/BinaryRow message is replaced by a DataRow message. The DataRow message has the message size included, which could allow the driver to read the entire set of fields for one data row, using a single socket read (or a few buffer sized reads). >> >> I recall seeing an experimental V3 protocol implementation, a few years back - sorry, no links handy. It would be nice to see some benchmarks. >> >> Hope that helps. > Thanks for the response. > > I believe the V3 project is here http://www.squeaksource.com/PostgresV3.html. > > Now, I probably spoke too fast and should have taken Mariano's advice to never speculate and first measure. Here is my quick test that, for me, shows that PostgresV2 seems more than fast enough (something that I had experienced before without doing benchmarks). > > [ self execute: 'select longitude,latitude from log_data limit 10000;' ] timeToRun. > > => 76 ms > > [ self execute: 'select longitude,latitude from log_data limit 100000;' ] timeToRun. > > => 765 ms > > This is querying for 2 floats from a huge table, over the network. Pretty fast ;-) > > So, back to Chris: what exactly are you doing that is (so) slow ? > > Anyway, thanks Yanni for all your work on the existing driver ! > > Sven > > -- > Sven Van Caekenberghe > Proudly supporting Pharo > http://pharo.org > http://association.pharo.org > http://consortium.pharo.org > > > > > >

July 31, 2013

Re: [Pharo-users] RFB on 2.0 + linux - semaphore primitive failed

by Igor Stasenko

On 31 July 2013 22:17, Paul DeBruicker <pdebruic(a)gmail.com> wrote: > I don't know. I don't get the errors that used to occur with the max > semaphores being exceeded. I thought Igor Stasenko fixed that issue in > the VM so the array that holds the semaphores grows without blocking. > Or something. > You can have as big as you want array with semaphores. There is no limits and maxExternalSemaphoresSilently is just does nothing and ignored. > > > What I do know is that if I 'abandon' that warning and save & quit and > then reopen the image then the UI is locked and I have to use Seaside's > WAScreenshot tool to suspend and resume the UI to unfreeze it. > > weird. i would start from figuring out, why primitive failed..and for which semaphore. -- Best regards, Igor Stasenko.

July 31, 2013

Re: [Pharo-users] RFB on 2.0 + linux - semaphore primitive failed

by Paul DeBruicker

I don't know. I don't get the errors that used to occur with the max semaphores being exceeded. I thought Igor Stasenko fixed that issue in the VM so the array that holds the semaphores grows without blocking. Or something. What I do know is that if I 'abandon' that warning and save & quit and then reopen the image then the UI is locked and I have to use Seaside's WAScreenshot tool to suspend and resume the UI to unfreeze it. On 07/31/2013 11:57 AM, Mariano Martinez Peck wrote: > Maybe this is related to #maxExternalSemaphores: ? > > Did you try sending a large number? For my seaside images where I have > RFB I do: > > Smalltalk vm maxExternalSemaphoresSilently: 65535. > > Cheers, > > > On Wed, Jul 31, 2013 at 3:45 PM, Paul DeBruicker <pdebruic(a)gmail.com > <mailto:pdebruic@gmail.com>> wrote: > > Hi - > > In Pharo2 do I need to stop and start the RFB server? I 've > been > getting a > > primitive #signal in Semaphore failed > > error after the image starts. I'm using this version of RFB > > RFB-MarianoMartinezPeck.41 > > Thanks > > Paul > > > > > -- > Mariano > http://marianopeck.wordpress.com

July 31, 2013

Re: [Pharo-users] [Seaside] [ANN] Pharo 2 Summer released

by Tudor Girba

Thank you! Doru On Wed, Jul 31, 2013 at 5:41 PM, Esteban Lorenzano <estebanlm(a)gmail.com>wrote: > The Pharo Team is happy to announce the Pharo 2 Summer release, > the regular half of the year bugfix edition. > > For this version, we incorporated really few changes since 2.0 was a > really successful release. Anyway, we managed to sneak in a couple of minor > additions: > > - Integrated ProfStef > - Updated Configurations Browser > > Also, we integrated 42 bugfixes, 20 of them backported from 3. 0 alpha. > > As always, Summer edition can be downloaded at: > > http://www.pharo-project.org/pharo-download > > We want to thank to all members of the community for their help and > support and specially to all who submitted the reports, fixed the bugs and > tested them. > > The Pharo Team > > > List complete of bugfixes and backports: > =============================== > > Fixed: > -------- > > 7345 DNU with Glamour: PaginatedMorphTreeMorph>>setSelectedMorph: > https://code.google.com/p/pharo/issues/detail?id=7345 > > 10014 World menu > System > Switch User serves no purpose > > https://pharo.fogbugz.com/f/cases/10014/World-menu-System-Switch-User-serve… > 10004 SmalltalkImage current reportCPUandRAM > DNU > > https://pharo.fogbugz.com/f/cases/10004/SmalltalkImage-current-reportCPUand… > > 10017 ProtocolCLient do not propagate the erros from the server > https://pharo.fogbugz.com/f/cases/10017 > > 7627 FileReference>>deleteAll follows symlinks > https://pharo.fogbugz.com/f/cases/7627 > > 10027 Failing Test: FileReferenceTest>>#testDeleteAl > https://pharo.fogbugz.com/f/cases/10027 > > 10123 Bug in warning in CriticBrowser > https://pharo.fogbugz.com/f/cases/10123 > > 10196 Change shortcuts default settings of 2.0 to the traditional ones > https://pharo.fogbugz.com/f/cases/10196 > 10195 Replace ugly Keymappings API on:do: > https://pharo.fogbugz.com/f/cases/10195 > 10200 Date instances dayOfWeek wrong > https://pharo.fogbugz.com/f/cases/10200 > 10109 MNU : SmalltalkEditor>>browseItHere: > https://pharo.fogbugz.com/f/cases/10109 > 10394 Duplicated instance #model in NewList (already defined in > MorphicModel) > https://pharo.fogbugz.com/f/cases/10394 > 10336 Port 2.0: Document difference between TextModel and > TextInputFieldModel > https://pharo.fogbugz.com/f/cases/10336 > 10653 Some Jenkins images show a stdout is closed error > https://pharo.fogbugz.com/f/cases/10653 > > 10653 Some Jenkins images show a stdout is closed error > https://pharo.fogbugz.com/f/cases/10653 > 10415 Integrate 2.0: 10411 flush after nextChunkPut: > https://pharo.fogbugz.com/f/cases/10415 > > 10888 Backport 2.0: Compound Extension Methods Unpackaged > > https://pharo.fogbugz.com/f/cases/10888/Backport-2-0-Compound-Extension-Met… > > 10730 Fix for stack serialization with Fuel > > https://pharo.fogbugz.com/f/cases/10730/Fix-for-stack-serialization-with-Fu… > > 10946 HistoryCollection package is empty > https://pharo.fogbugz.com/f/cases/10946 > > 11101 fileOutMethod:on: uncategorized in 2.0 > https://pharo.fogbugz.com/f/cases/11101 > 11179 Config browser buttons broken due to missing method > https://pharo.fogbugz.com/f/cases/11179 > > 10860 option click on list in TestRunner and others: freeze system > https://pharo.fogbugz.com/f/cases/10860 > > Backported: > ---------------- > 10194 Backport to 2.0: special objects array recreation crash > https://pharo.fogbugz.com/f/cases/10194 > 10185 Backport 2.0: 10173 ClassBuilder cannot safely modify a class why > using it > https://pharo.fogbugz.com/f/cases/10185 > 10174 Backport 2.0 Issue 10162: multiple selection of methods in method > pane ... > https://pharo.fogbugz.com/f/cases/10174 > > 10249 Backport 2.0: Red Light when all test are green > https://pharo.fogbugz.com/f/cases/10249 > 10230 backport 2.0: 10229 Fix sender of #copyWithTempsFromMethodNode: > https://pharo.fogbugz.com/f/cases/10230 > 10179 backport 2.0: 10038 Wrong deprecated message > https://pharo.fogbugz.com/f/cases/10179 > > 10260 backport 2.0: 10258 DivideByZero in SystemProgressMorph > https://pharo.fogbugz.com/f/cases/10260 > 10406 backport 2.0: 5819 Implement a way to get the user directories based > on the current OS > https://pharo.fogbugz.com/f/cases/10406 > 10396 backport 2.0 10395 DNU SmallInteger>>#add: when loading metacello > configuration > https://pharo.fogbugz.com/f/cases/10396 > 10387 backport 2.0: 10102 SocketStream>>next: should not signal > ConncetionClosed > https://pharo.fogbugz.com/f/cases/10387 > 10311 Backport 2.0: 10228 Fix tab key for autocompletion > https://pharo.fogbugz.com/f/cases/10311 > > 10537 Backport 2.0: StartupPreferences interferes with zeroconf pharo > binary > https://pharo.fogbugz.com/f/cases/10537 > > 10938 backport 2.0 : an MCOrganizationDefinition should only remove empty > categories during unload > https://pharo.fogbugz.com/f/cases/10938 > 10909 Backport Pharo2.0: Prepare isHeadless for new VMs with double dash > -- arguments > https://pharo.fogbugz.com/f/cases/10909 > > 10994 Backport 2.0: Delays do not work properly directly on image startup > > https://pharo.fogbugz.com/f/cases/10994/Backport-2-0-Delays-do-not-work-pro… > > 10931 Backport 2.0: Filing out extension package in Nautilus produces > empty file > https://pharo.fogbugz.com/f/cases/10931 > 10978 Backport 20: ZipArchive utf8 handling > https://pharo.fogbugz.com/f/cases/10978 > 11125 backport 2.0: 10859 Monticello freezing the image > https://pharo.fogbugz.com/f/cases/11125 > 10569 Backport 2.0: Better Interrruptionche > https://pharo.fogbugz.com/f/cases/10569 > 10940 Backport 2.0: 10925 Use caching in Gofer > https://pharo.fogbugz.com/f/cases/10940 > > 11252 backport 2.0: have SliceMaker use fogbugz > https://pharo.fogbugz.com/f/cases/11252 > > _______________________________________________ > seaside mailing list > seaside(a)lists.squeakfoundation.org > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside > > -- www.tudorgirba.com "Every thing has its own flow"

July 31, 2013

Re: [Pharo-users] RFB on 2.0 + linux - semaphore primitive failed

by Mariano Martinez Peck

Maybe this is related to #maxExternalSemaphores: ? Did you try sending a large number? For my seaside images where I have RFB I do: Smalltalk vm maxExternalSemaphoresSilently: 65535. Cheers, On Wed, Jul 31, 2013 at 3:45 PM, Paul DeBruicker <pdebruic(a)gmail.com> wrote: > Hi - > > In Pharo2 do I need to stop and start the RFB server? I 've been > getting a > > primitive #signal in Semaphore failed > > error after the image starts. I'm using this version of RFB > > RFB-MarianoMartinezPeck.41 > > Thanks > > Paul > > -- Mariano http://marianopeck.wordpress.com

July 31, 2013

Re: [Pharo-users] Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

by Mariano Martinez Peck

hehehe I did know about the executors, but I didn't know about the WeakRegistry, I always thought it was only the #default instance :) BTW, I have just send an email about the 200MB image... Cheers, On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck < marianopeck(a)gmail.com> wrote: > Guys, I have images also with seaside, magritte, glorp, postgresV2, etc > and it is also around 200MB. > I will try to do some research today and let you know. > > Cheers, > > > On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker <marcus.denker(a)inria.fr>wrote: > >> >> On Jul 30, 2013, at 1:49 PM, "phil(a)highoctane.be" <phil(a)highoctane.be> >> wrote: >> >> > the changes file contained passwords and I replaced the text. So >> offsets may be wrong due to that. >> > >> Yes, the first thing I wanted to do is to recompile everything. Does not >> work. >> >> > Memorymonitor is not doing fanct stuff. It just counts instances. >> > >> Yes, but maybe it holds on to these instances? >> >> Marcus >> >> >> > > > -- > Mariano > http://marianopeck.wordpress.com > -- Mariano http://marianopeck.wordpress.com

July 31, 2013

Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

by Mariano Martinez Peck

OK, this was was my experiment.... Image fresh with all my app and dependencies loades: 30MB After using it for some days/weeks: 160MB. SpaceTally new printSpaceAnalysis showed: Class code space # instances inst space percent inst average size ByteString 2785 413144 116244078 69.90 281.36 Array 3712 181772 8466668 5.10 46.58 ByteArray 8574 1319 8186802 4.90 6206.82 Bitmap 3653 303 6656340 4.00 21968.12 CompiledMethod 22467 90554 5685374 3.40 62.78 After executing ImageCleaner cleanUpForRelease: 36MB Then...I searched which part of #cleanUpForRelease: was making the difference, and it was: Smalltalk cleanUp: true except: #() confirming: false. So now it was time to know WHICH class did the diference, so I modified #cleanUp: aggressive except: exclusions confirming: aBool in these lines: "Run the cleanup code" classes do:[:aClass| Transcript show: 'Image size before cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. aClass cleanUp: aggressive. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. Transcript show: 'Image size after cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString. ] displayingProgress: [:aClass| 'Cleaning up in ', aClass name]. I then opened a Transcript, and evaluated Smalltalk cleanUp: true except: #() confirming: false. I went to prepare Mate, and when I come back, the result was, of course: "Image size after cleaning MCFileBasedRepository : 39744008" That clean up ends up doing: flushAllCaches self allSubInstancesDo: [:ea | ea flushCache] So it sends #flushCache to all instances of MCHttpRepository and MCFileBasedRepository. Now what I wanted to see if it there was a particular repo that could take most of the space (like package-cache). And indeed, it was...I modified #flushCaches to: flushAllCaches | file | file := 'repos.txt' asFileReference writeStream text. self allSubInstancesDo: [:each | file nextPutAll: 'Image size before cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString; cr. each flushCache. 3 timesRepeat: [Smalltalk garbageCollect]. Smalltalk snapshot: true andQuit: false. file nextPutAll: 'Image size after cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString;cr. ]. file flush; close. And then I looked in the 'repos.txt' file. My package cache repo cleaned 60 MB. Glorp cleaned 35MB. Seaside30 cleaned 10MB. So...cleaning cache of just 3 repos frees approx 100MB. The question is....can we flush the cache safely? If they are called "cache", then I guess yes, we can. Thoughts? Thanks, On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck < marianopeck(a)gmail.com> wrote: > Guys, I have images also with seaside, magritte, glorp, postgresV2, etc > and it is also around 200MB. > I will try to do some research today and let you know. > > Cheers, > > > On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker <marcus.denker(a)inria.fr>wrote: > >> >> On Jul 30, 2013, at 1:49 PM, "phil(a)highoctane.be" <phil(a)highoctane.be> >> wrote: >> >> > the changes file contained passwords and I replaced the text. So >> offsets may be wrong due to that. >> > >> Yes, the first thing I wanted to do is to recompile everything. Does not >> work. >> >> > Memorymonitor is not doing fanct stuff. It just counts instances. >> > >> Yes, but maybe it holds on to these instances? >> >> Marcus >> >> >> > > > -- > Mariano > http://marianopeck.wordpress.com > -- Mariano http://marianopeck.wordpress.com

July 31, 2013

RFB on 2.0 + linux - semaphore primitive failed

by Paul DeBruicker

Hi - In Pharo2 do I need to stop and start the RFB server? I 've been getting a primitive #signal in Semaphore failed error after the image starts. I'm using this version of RFB RFB-MarianoMartinezPeck.41 Thanks Paul

July 31, 2013

Re: [Pharo-users] Git filetree workflow

by joerick

In the Gofer #push case, it's quite strange... some classes end up the wrong package, and some don't get output at all. It seems to work using Gofer #push when each is in its own repository folder, and it also works with them all in the same folder when using Monticello Browser's 'save' button. Perhaps I should try moving to 2.0. Last time we tried there was a package that hadn't been ported yet, I can't recall what it was... fuel perhaps? -- View this message in context: http://forum.world.st/Git-filetree-workflow-tp4701465p4701734.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

July 31, 2013