May 2013 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Reflectivity beta

by Camille Teruel

On 31 mai 2013, at 11:16, Pavel Krivanek wrote: > Really cool! Has anyone already tried to build coverage tests based on it? Not yet, that's brand new stuff so nobody did anything with it. > -- Pavel > > On Thu, May 30, 2013 at 6:15 PM, Camille Teruel <camille.teruel(a)gmail.com> wrote: > Hello everyone, > > Reflectivity beta is out ! > > Reflectivity is a tool that permit to annotate AST nodes with "metalinks". > You can put metalinks at different "positions": > before: the metalink is executed before the node > instead: the metalink is executed instead the node > after: the metalink is executed after the node > onError: the metalink is executed only if the execution of the node raises an error > onSuccess: the metalink is executed only if the execution of the node raises no error > > When you put metalinks on some node of a method's AST, a wrapper is installed in place of the method. When executed, this wrapper compiles an expanded version of the AST that takes metalinks into account and install the resulting compiled method. > > Examples: > > increaseAllNumbersIn: aCompiledMethod > "A method that increases all numbers in aCompiledMethod" > aCompiledMethod reflectiveAST > forAllNodes: [ :node | node isLiteral and: [ node value isNumber ] ] > putInstead: [ :node | RFMetalink fromExpression: (node value + 1) asString ]. > > removeMetalinksIn: aCompiledMethod > aCompiledMethod reflectiveAST removeAllMetalinks > > In nautilus, you have a menu entry called 'Edit metalinks' that permit to edit the metalinks of the node corresponding to the selected piece of code. > As an example use case, another entry called 'Put breakpoint' adds a metalink that corresponds to 'Halt now' before the selected node. > > Remember that it is a beta, so you might find errors and things are likely to change. > > You can load Reflectivity with: > > Gofer it > smalltalkhubUser: 'RMoD' project: 'Reflectivity'; > configuration; > loadDevelopment > > Or you can download it from RMoD's CI: > https://ci.inria.fr/rmod/job/Reflectivity/ > > Camille >

May 31, 2013

Re: [Pharo-dev] Keeping Pharo CPU usage low: how?

by Norbert Hartl

Am 31.05.2013 um 11:38 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: > > On 31 May 2013, at 11:26, Norbert Hartl <norbert(a)hartl.name> wrote: > >> >> Am 27.05.2013 um 10:17 schrieb phil(a)highoctane.be: >> >>> I was looking at the output of "top" for while and Pharo was trusting the top spot indeed. >>> >>> Well, I'll look at all that this evening and give back the impressions. >>> >>> I want to run several instances of Pharo on the box and it will for sure add up. Say, 20 x 2% gives an awful lot. >>> >> What is the use of having 20 idle images on one server? :) I think you can't easily say that it always adds on top. If your image is running at 100% the 2% off even if fully countable wouldn't be that much. But nevertheless it is a valid point and I cannot see pharo becomes more modern by stop polling in the mid-term future. >> Anyway I played a little bit with the tweaks and there are things that make me wonder. I raised the limit for server mode but didn't get much benefit. Having a cycle pause of 100ms in serverMode and stopping the ui process still gave me a 1% CPU usage which is really a lot. Does anyone know in-depth what an image is doing? How to investigate? Maybe there are a lot of native polling things involved, too? >> I need to dig deeper. > > Killing as much non-needed process is certainly step one. A clean image has these processes: > > 1. Delay scheduling process > 2. Input events fetching process > 3. The low space watcher > 4. The weak array finalization process > 5. Morphic UI process > 6. The idle process > > So you killed 5. How did you do that exactly ? I didn't kill it. I just did MorphicUIManager default uiProcess suspend > Is #serverMode still relevant when 5 is gone ? I was asking myself the same question. > Would it be possible to kill 2 ? Is it needed for a headless server ? That could be one of the more busy processes. So if we stop 5. there is no point in running 2., right? Or asking the other way round: What is included in input events fetching? keyboard and mouse or something else, too? > As long as file and socket IO keeps working, we're good. > Yes and it is quite interesting to see how much polling is going on there. I think those two are the ones that would benefit the most from having an event based native adaption. Norbert > The rest is probably all needed. > > Are there any (slightly busy) processes inside the VM that we can't see at the Smalltalk level ? > >> Norbert >> >>> Phil >>> >>> >>> On Mon, May 27, 2013 at 10:07 AM, Norbert Hartl <norbert(a)hartl.name> wrote: >>> >>> Am 27.05.2013 um 09:49 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: >>> >>>> BTW, load & cpu usage numbers can be very hard to interpret. >>> >>> really? CPU usage is pretty easy and you can see this in most OSses directly. 100% CPU usage is full :) Load isn't that much harder. It's just the load of the scheduler queue. So if some needs a rule of thumb >>> >>> safe CPU load = number of cores/processes * 0.7 >>> theoretical optimal CPU load = number of core/processes >>> >>> Everything above should really be investigated because you entered the zone where additional CPU load can drive your machine unresponsive. >>> You have less CPU usage but high load? Investigate I/O usage! >>> >>> I know you know that, Sven. I just wanted to mention it because this comes up from time to time. Compared to memory consumption, CPU, scheduler and I/O are easier targets IMHO. >>> >>> For Phil the point isn't the health of the system. It is EC2. You have to pay for the CPU usage on EC2 so I think he had the impression there is room for improvement. And indeed there is. But if you calculate it through it isn't really a factor. >>> >>> Norbert >>> >>> >> > >

May 31, 2013

Re: [Pharo-dev] Keeping Pharo CPU usage low: how?

by phil＠highoctane.be

I don't know how to do this for Linux based VMs (which is what I'll need in the end) but can do some experiments with Apple Xcode Instruments. That's something I've wanted to do for a long time as this gives deep insight on how a program is behaving. http://en.wikipedia.org/wiki/Instruments_%28application%29 http://developer.apple.com/library/mac/#documentation/DeveloperTools/Concep… Thanks for looking into this. Maybe could we do a chapter on these matters in one of the Pharo books. The point of having idle VMs is that when one machine is hit by a peak of usage, the other instances can swallow the load. Now, I am not talking about a single EC2 VM, but a group of them, each having a set of VMs. FWIW, AIMP3 playing MP3s all the time is 9%-11% on an old PIV 3Ghz, while Skype takes 1% idling Phil On Fri, May 31, 2013 at 11:26 AM, Norbert Hartl <norbert(a)hartl.name> wrote: > > Am 27.05.2013 um 10:17 schrieb phil(a)highoctane.be: > > I was looking at the output of "top" for while and Pharo was trusting the > top spot indeed. > > Well, I'll look at all that this evening and give back the impressions. > > I want to run several instances of Pharo on the box and it will for sure > add up. Say, 20 x 2% gives an awful lot. > > What is the use of having 20 idle images on one server? :) I think you > can't easily say that it always adds on top. If your image is running at > 100% the 2% off even if fully countable wouldn't be that much. But > nevertheless it is a valid point and I cannot see pharo becomes more modern > by stop polling in the mid-term future. > Anyway I played a little bit with the tweaks and there are things that > make me wonder. I raised the limit for server mode but didn't get much > benefit. Having a cycle pause of 100ms in serverMode and stopping the ui > process still gave me a 1% CPU usage which is really a lot. Does anyone > know in-depth what an image is doing? How to investigate? Maybe there are a > lot of native polling things involved, too? > I need to dig deeper. > > Norbert > > Phil > > > On Mon, May 27, 2013 at 10:07 AM, Norbert Hartl <norbert(a)hartl.name>wrote: > >> >> Am 27.05.2013 um 09:49 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: >> >> BTW, load & cpu usage numbers can be very hard to interpret. >> >> >> really? CPU usage is pretty easy and you can see this in most OSses >> directly. 100% CPU usage is full :) Load isn't that much harder. It's just >> the load of the scheduler queue. So if some needs a rule of thumb >> >> safe CPU load = number of cores/processes * 0.7 >> theoretical optimal CPU load = number of core/processes >> >> Everything above should really be investigated because you entered the >> zone where additional CPU load can drive your machine unresponsive. >> You have less CPU usage but high load? Investigate I/O usage! >> >> I know you know that, Sven. I just wanted to mention it because this >> comes up from time to time. Compared to memory consumption, CPU, scheduler >> and I/O are easier targets IMHO. >> >> For Phil the point isn't the health of the system. It is EC2. You have to >> pay for the CPU usage on EC2 so I think he had the impression there is room >> for improvement. And indeed there is. But if you calculate it through it >> isn't really a factor. >> >> Norbert >> >> > >

May 31, 2013

Re: [Pharo-dev] Keeping Pharo CPU usage low: how?

by Sven Van Caekenberghe

On 31 May 2013, at 11:26, Norbert Hartl <norbert(a)hartl.name> wrote: > > Am 27.05.2013 um 10:17 schrieb phil(a)highoctane.be: > >> I was looking at the output of "top" for while and Pharo was trusting the top spot indeed. >> >> Well, I'll look at all that this evening and give back the impressions. >> >> I want to run several instances of Pharo on the box and it will for sure add up. Say, 20 x 2% gives an awful lot. >> > What is the use of having 20 idle images on one server? :) I think you can't easily say that it always adds on top. If your image is running at 100% the 2% off even if fully countable wouldn't be that much. But nevertheless it is a valid point and I cannot see pharo becomes more modern by stop polling in the mid-term future. > Anyway I played a little bit with the tweaks and there are things that make me wonder. I raised the limit for server mode but didn't get much benefit. Having a cycle pause of 100ms in serverMode and stopping the ui process still gave me a 1% CPU usage which is really a lot. Does anyone know in-depth what an image is doing? How to investigate? Maybe there are a lot of native polling things involved, too? > I need to dig deeper. Killing as much non-needed process is certainly step one. A clean image has these processes: 1. Delay scheduling process 2. Input events fetching process 3. The low space watcher 4. The weak array finalization process 5. Morphic UI process 6. The idle process So you killed 5. How did you do that exactly ? Is #serverMode still relevant when 5 is gone ? Would it be possible to kill 2 ? Is it needed for a headless server ? As long as file and socket IO keeps working, we're good. The rest is probably all needed. Are there any (slightly busy) processes inside the VM that we can't see at the Smalltalk level ? > Norbert > >> Phil >> >> >> On Mon, May 27, 2013 at 10:07 AM, Norbert Hartl <norbert(a)hartl.name> wrote: >> >> Am 27.05.2013 um 09:49 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: >> >>> BTW, load & cpu usage numbers can be very hard to interpret. >> >> really? CPU usage is pretty easy and you can see this in most OSses directly. 100% CPU usage is full :) Load isn't that much harder. It's just the load of the scheduler queue. So if some needs a rule of thumb >> >> safe CPU load = number of cores/processes * 0.7 >> theoretical optimal CPU load = number of core/processes >> >> Everything above should really be investigated because you entered the zone where additional CPU load can drive your machine unresponsive. >> You have less CPU usage but high load? Investigate I/O usage! >> >> I know you know that, Sven. I just wanted to mention it because this comes up from time to time. Compared to memory consumption, CPU, scheduler and I/O are easier targets IMHO. >> >> For Phil the point isn't the health of the system. It is EC2. You have to pay for the CPU usage on EC2 so I think he had the impression there is room for improvement. And indeed there is. But if you calculate it through it isn't really a factor. >> >> Norbert >> >> >

May 31, 2013

Re: [Pharo-dev] Keeping Pharo CPU usage low: how?

by Norbert Hartl

Am 27.05.2013 um 10:17 schrieb phil(a)highoctane.be: > I was looking at the output of "top" for while and Pharo was trusting the top spot indeed. > > Well, I'll look at all that this evening and give back the impressions. > > I want to run several instances of Pharo on the box and it will for sure add up. Say, 20 x 2% gives an awful lot. > What is the use of having 20 idle images on one server? :) I think you can't easily say that it always adds on top. If your image is running at 100% the 2% off even if fully countable wouldn't be that much. But nevertheless it is a valid point and I cannot see pharo becomes more modern by stop polling in the mid-term future. Anyway I played a little bit with the tweaks and there are things that make me wonder. I raised the limit for server mode but didn't get much benefit. Having a cycle pause of 100ms in serverMode and stopping the ui process still gave me a 1% CPU usage which is really a lot. Does anyone know in-depth what an image is doing? How to investigate? Maybe there are a lot of native polling things involved, too? I need to dig deeper. Norbert > Phil > > > On Mon, May 27, 2013 at 10:07 AM, Norbert Hartl <norbert(a)hartl.name> wrote: > > Am 27.05.2013 um 09:49 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: > >> BTW, load & cpu usage numbers can be very hard to interpret. > > really? CPU usage is pretty easy and you can see this in most OSses directly. 100% CPU usage is full :) Load isn't that much harder. It's just the load of the scheduler queue. So if some needs a rule of thumb > > safe CPU load = number of cores/processes * 0.7 > theoretical optimal CPU load = number of core/processes > > Everything above should really be investigated because you entered the zone where additional CPU load can drive your machine unresponsive. > You have less CPU usage but high load? Investigate I/O usage! > > I know you know that, Sven. I just wanted to mention it because this comes up from time to time. Compared to memory consumption, CPU, scheduler and I/O are easier targets IMHO. > > For Phil the point isn't the health of the system. It is EC2. You have to pay for the CPU usage on EC2 so I think he had the impression there is room for improvement. And indeed there is. But if you calculate it through it isn't really a factor. > > Norbert > >

May 31, 2013

Re: [Pharo-dev] Reflectivity beta

by Pavel Krivanek

Really cool! Has anyone already tried to build coverage tests based on it? -- Pavel On Thu, May 30, 2013 at 6:15 PM, Camille Teruel <camille.teruel(a)gmail.com>wrote: > Hello everyone, > > Reflectivity beta is out ! > > Reflectivity is a tool that permit to annotate AST nodes with "metalinks". > You can put metalinks at different "positions": > > - before: the metalink is executed before the node > - instead: the metalink is executed instead the node > - after: the metalink is executed after the node > - onError: the metalink is executed only if the execution of the node > raises an error > - onSuccess: the metalink is executed only if the execution of the > node raises no error > > > When you put metalinks on some node of a method's AST, a wrapper is > installed in place of the method. When executed, this wrapper compiles an > expanded version of the AST that takes metalinks into account and install > the resulting compiled method. > > Examples: > > increaseAllNumbersIn: aCompiledMethod > "A method that increases all numbers in aCompiledMethod" > aCompiledMethod reflectiveAST > forAllNodes: [ :node | node isLiteral and: [ node value isNumber ] ] > putInstead: [ :node | RFMetalink fromExpression: (node value + 1) asString > ]. > > removeMetalinksIn: aCompiledMethod > aCompiledMethod reflectiveAST removeAllMetalinks > > In nautilus, you have a menu entry called 'Edit metalinks' that permit to > edit the metalinks of the node corresponding to the selected piece of code. > As an example use case, another entry called 'Put breakpoint' adds a > metalink that corresponds to 'Halt now' before the selected node. > > Remember that it is a beta, so you might find errors and things are likely > to change. > > You can load Reflectivity with: > > Gofer it > smalltalkhubUser: 'RMoD' project: 'Reflectivity'; > configuration; > loadDevelopment > > Or you can download it from RMoD's CI: > https://ci.inria.fr/rmod/job/Reflectivity/ > > Camille >

May 31, 2013

Re: [Pharo-dev] Amber's new stepping debugger

by StÃ©phane Ducasse

cool! Stef On May 30, 2013, at 11:26 PM, Nicolas Petton <petton.nicolas(a)gmail.com> wrote: > Hi guys! > > Today I was able to debug Amber code with a stepping debugger for the first time! > The debugger is based on an AST interpreter with a small stack implementation. > > Here's a short video of the debugger in action: > https://www.dropbox.com/s/43mli1wwbdwj5kx/amber-debugging.mov > > While it's an early version, it is already usable :) > > cheers, > nico

May 31, 2013

Handling error with PetitParser

by StÃ©phane Ducasse

We would like to see if we can use PetitParser to build syntax hilighting and for this we need to see if there is a way to specify error handling. Does any of you know? So far we tried to use the RBSCanner but it does not really work for what we need Stef

May 31, 2013

Re: [Pharo-dev] Reflectivity beta

by StÃ©phane Ducasse

> I would love to have an editor where you manipulates ast nodes directly. Me too. And we got killed by the state of the Scanner. Old infrastructure always bite us hard. We ask gisela to check with PetitParser and sync with Camillo/Damien/"Lukas"/Guillaume for the introduction of error handling in PetitParser. Stef

May 31, 2013

Re: [Pharo-dev] No Internationalization (nor Multilingual) Project in bug tracker

by StÃ©phane Ducasse

On May 30, 2013, at 11:37 PM, Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> wrote: > OK, i have opened https://pharo.fogbugz.com/default.asp?10811 (pre-requisite) and https://pharo.fogbugz.com/default.asp?10801 > I think that was the hardest part of my work > The rest is just Smalltalk coding ;) (you'll find two SLICES in inbox) > > The most boring stuff is yet to come: final harvesting process... > Nicolas we are working on parallelirizing it. If you want we can show you how you can commit directly to pharo. We could do a pair programming integration session. Stef

May 31, 2013