December 2016 - Pharo-users

Re: [Pharo-users] real world pharo web application set ups

by phil＠highoctane.be

I can try that: I've some 48-core 500+ GB RAM boxes around at the moment. If someone can make me a test script, can run it. Phil On Wed, Dec 14, 2016 at 8:54 PM, Stephan Eggermont <stephan(a)stack.nl> wrote: > On 14/12/16 19:41, Dimitris Chloupis wrote: > >> That 5% idle CPU consumption is not the only problem, creating a new >> process will add additional overheads anyway so it does not make sense >> to have more pharo processes than CPU cores. >> > > You can probably run a few thousand images on a single high-end x86 server > (2*22 Core Xeon, 1TB ram). I just started 21 images on my dual-core i5 MBP. > top tells me my system is 75-80% idle, pharo processes taking 2-3%CPU and > 1.6GB MEM (33MB-146MB) > > Stephan > > > > >

Dec. 14, 2016

Re: [Pharo-users] real world pharo web application set ups

by Vitor Medina Cruz

Pharo don't have non-blocking I/O? On Wed, Dec 14, 2016 at 6:59 PM, Ramon Leon <ramon.leon(a)allresnet.com> wrote: > On 12/14/2016 12:09 PM, Esteban A. Maringolo wrote: > >> Can you extend on suspending the UI process? I never did that. >> > > I feed my images a start script on the command line > > pharo-vm-nox \ > -vm-sound-null -vm-display-null \ > /var/pharo/app.image \ > /var/pharo/startScript > > startScript containing one line (among others) like so... > > Project uiProcess suspend. > > I'm on an older Pharo, but I presume the newer ones are the same or > similar. No sense in wasting CPU on a UI in a headless image > > Won't the idle use add up? >> > > Sure eventually, but you don't run more than a 2 or so per core so that'll > never be a problem. You shouldn't be running 5 images on a single core, > let alone more. > > In my case I served up to 20 concurrent users (out of ~100 total) with >> only 5 images. Plus another two images for the REST API. In a dual >> core server. >> > > That's barely a server, most laptops these days have more cores. Rent a > virtual server with a dozen or more cores, then you can run a few images > per core without the idle mattering at all and run 2 dozen images in total > per 12 core server. > > Scale by adding cores and ram allowing you to run more images per box; or > scale by running more boxes, ultimately, you need to spread out the load > across many many cores. > > -- > Ramon Leon > > >

Dec. 14, 2016

Re: [Pharo-users] real world pharo web application set ups

by Ramon Leon

On 12/14/2016 12:09 PM, Esteban A. Maringolo wrote: > Can you extend on suspending the UI process? I never did that. I feed my images a start script on the command line pharo-vm-nox \ -vm-sound-null -vm-display-null \ /var/pharo/app.image \ /var/pharo/startScript startScript containing one line (among others) like so... Project uiProcess suspend. I'm on an older Pharo, but I presume the newer ones are the same or similar. No sense in wasting CPU on a UI in a headless image > Won't the idle use add up? Sure eventually, but you don't run more than a 2 or so per core so that'll never be a problem. You shouldn't be running 5 images on a single core, let alone more. > In my case I served up to 20 concurrent users (out of ~100 total) with > only 5 images. Plus another two images for the REST API. In a dual > core server. That's barely a server, most laptops these days have more cores. Rent a virtual server with a dozen or more cores, then you can run a few images per core without the idle mattering at all and run 2 dozen images in total per 12 core server. Scale by adding cores and ram allowing you to run more images per box; or scale by running more boxes, ultimately, you need to spread out the load across many many cores. -- Ramon Leon

Dec. 14, 2016

Re: [Pharo-users] real world pharo web application set ups

by Esteban A. Maringolo

2016-12-14 16:31 GMT-03:00 Ramon Leon <ramon.leon(a)allresnet.com>: > On 12/14/2016 10:27 AM, Esteban A. Maringolo wrote: >> >> Memory is cheap, and the only limit now is CPU, the current Pharo >> image has an idle CPU consumption ~5%, so you can't have many >> concurrent images running in the same machine. > > > Sure you can, a multi-core server isn't going to get maxed out by idle CPU > consumption as each image will only do that on a single core; you can also > reduce consumption on headless images by suspending the UI process which > isn't needed getting you down to perhaps 2-3%. Can you extend on suspending the UI process? I never did that. >> whilst 10 idle Pharo images will consume 50%. > Only on a single CPU machine which isn't what you're going to have on a > server. Running two images per core on a 12 core virtual server and you'll > be sitting more like 5-6% idle which is totally fine. Spread that across 10 > or 15 virtual servers and you can handle 1000 concurrent users just fine. Won't the idle use add up? In my case I served up to 20 concurrent users (out of ~100 total) with only 5 images. Plus another two images for the REST API. In a dual core server. Regards. Esteban A. Maringolo

Dec. 14, 2016

Re: [Pharo-users] real world pharo web application set ups

by Stephan Eggermont

On 14/12/16 19:41, Dimitris Chloupis wrote: > That 5% idle CPU consumption is not the only problem, creating a new > process will add additional overheads anyway so it does not make sense > to have more pharo processes than CPU cores. You can probably run a few thousand images on a single high-end x86 server (2*22 Core Xeon, 1TB ram). I just started 21 images on my dual-core i5 MBP. top tells me my system is 75-80% idle, pharo processes taking 2-3%CPU and 1.6GB MEM (33MB-146MB) Stephan

Dec. 14, 2016

Re: [Pharo-users] real world pharo web application set ups

by Ramon Leon

On 12/14/2016 10:27 AM, Esteban A. Maringolo wrote: > Memory is cheap, and the only limit now is CPU, the current Pharo > image has an idle CPU consumption ~5%, so you can't have many > concurrent images running in the same machine. Sure you can, a multi-core server isn't going to get maxed out by idle CPU consumption as each image will only do that on a single core; you can also reduce consumption on headless images by suspending the UI process which isn't needed getting you down to perhaps 2-3%. > whilst 10 idle Pharo images will consume 50%. Only on a single CPU machine which isn't what you're going to have on a server. Running two images per core on a 12 core virtual server and you'll be sitting more like 5-6% idle which is totally fine. Spread that across 10 or 15 virtual servers and you can handle 1000 concurrent users just fine. -- Ramon Leon

Dec. 14, 2016

Re: [Pharo-users] real world pharo web application set ups

by Dimitris Chloupis

"A Pharo forked process is still running under the vm single threaded process... so it's more a matter of in-image blocking rather than external access." There is parallelism which means code that execute at the same time instruction by instruction There is concurency which means code that executes a bit of it from different parts giving the illusion that codes run at the same time OS threads do parallelism VM threads , aka green threads, do concurrency VM technically speaking uses several OS threads but the byte code is indeed executed on the main thread only , thus VM alllows only green threads for pharo code. I am not saying you do not know this stuff, I am just mention it here so everyone is on the same page. You cannot have more parallelism than cores, actually that may not be true because I think each core allows for two parallel threads because of hyperthreading but more or less thats the idea. Just for the record there are cases when multiple OS threads running on multiple cores can be actually slower than one thread running on one core. This the nightmare that is called parallelism and why coders generally avoid it. The usually reason why multiple OS threads may be slower it has to do with CPU cache and how it exchanges data between threads and CPU cores. I had the pleasure of getting a taste of this nightmare when I was learning OpenMP which is the most popular library for parallel execution for C/C++. Of course there are also the usual problems with synchronisation etc. Technically speaking you can do OS threads from Pharo with my CPP library. Thats possible because CPP works by having two isolated process (actually you can have an infinite amount of processes if your hardware can handle them ) , in this case one C++ executable and one for Pharo that share an area of memory that both can directly access to. You could implement callbacks to, you could do anything and you would not have to worry about joining threads or providing compatibility with Pharo VM because you deal with shared data and not code execution. Because CPP uses shared memory which in turn uses OS kernel native memory access and management functions, you get top speed. So you can have one process using OpenMP taking full advantage of the hardware acceleration (triggering multiple OS threads) of your CPU cores and one for Pharo that will drive the OpenMP process. There is still the issue of data synchronisation, who locks the data and who accesses it at what time, but those are usual issues for any kind of thread orientated coding, you wont be doing anything diffirent. "However any CPU intensive request will block the process, and that will happen with looping server like Pharo or an evented VM like NodeJS." Again you can avoid that with my CPP library. You do not need to block , you could let the external process do its thing and just block it when you decide to fetch data after it finishes or on specific intervals. There are so many ways you can do this. You can also have like the VM one central OS thread that is able to be blocked by Pharo and multiple secondary ones that Pharo will have no control over either than the level of control that the main thread provides. In any case, most servers run Apache which should deal with many of those issues, there are wrappers for many languages , Apaches is wrriten in C/C++ so its should be able to tap into this technology to avoid solving problems that have been already solved. So its a matter of writing wrappers for Pharo too. On Wed, Dec 14, 2016 at 8:59 PM Esteban A. Maringolo <emaringolo(a)gmail.com> wrote: > 2016-12-14 15:41 GMT-03:00 Dimitris Chloupis <kilon.alios(a)gmail.com>: > > That 5% idle CPU consumption is not the only problem, creating a new > process > > will add additional overheads anyway so it does not make sense to have > more > > pharo processes than CPU cores. > > Not exactly, thinking it that way would disable you from running Pharo > in a machine with other daemon services running (if daemons > cores). > But those other daemons at idle consume less than 1% of the total CPU, > whilst 10 idle Pharo images will consume 50%. > > With peak demand a single image can take 100% of the CPU core. > > > Concurrency can be handled by pharo forks. > > A Pharo forked process is still running under the vm single threaded > process... so it's more a matter of in-image blocking rather than > external access. > > > I am not web dev even by a remote imagination but my recent experience > with > > UFFI is that Pharo can go to insane speeds when it relies on C libraries > , > > especially performance orientated ones. Of course learning C and playing > > with C is a pain by itself but necessary evil if top performance is the > > goal. > > This is true. And non-blocking callbacks really alleviate the weight a > single threaded vm can support. > > However any CPU intensive request will block the process, and that > will happen with looping server like Pharo or an evented VM like > NodeJS. > > Unless you're targeting to an initial really high concurrent users > count, then you can scale up gradually. > > Esteban A. Maringolo > >

Dec. 14, 2016

GitFileTree in Pharo 5

by sergio ruiz

Hey, all. I am trying to install GitFileTree from the Catalog Browser. I get a warning that itâs not tested with Pharo 5. I go ahead anyway, and it just hangs. Has anyone else installed it on macOS? Is there a different package I should be using? Thanks! ---- peace, sergio photographer, journalist, visionary Public Key:Â http://bit.ly/29z9fG0 #BitMessageÂ BM-NBaswViL21xqgg9STRJjaJaUoyiNe2dV http://www.Village-Buzz.com http://www.ThoseOptimizeGuys.com http://www.coffee-black.com http://www.painlessfrugality.com http://www.twitter.com/sergio_101 http://www.facebook.com/sergio101

Dec. 14, 2016

Re: [Pharo-users] real world pharo web application set ups

by Esteban A. Maringolo

2016-12-14 15:41 GMT-03:00 Dimitris Chloupis <kilon.alios(a)gmail.com>: > That 5% idle CPU consumption is not the only problem, creating a new process > will add additional overheads anyway so it does not make sense to have more > pharo processes than CPU cores. Not exactly, thinking it that way would disable you from running Pharo in a machine with other daemon services running (if daemons > cores). But those other daemons at idle consume less than 1% of the total CPU, whilst 10 idle Pharo images will consume 50%. With peak demand a single image can take 100% of the CPU core. > Concurrency can be handled by pharo forks. A Pharo forked process is still running under the vm single threaded process... so it's more a matter of in-image blocking rather than external access. > I am not web dev even by a remote imagination but my recent experience with > UFFI is that Pharo can go to insane speeds when it relies on C libraries , > especially performance orientated ones. Of course learning C and playing > with C is a pain by itself but necessary evil if top performance is the > goal. This is true. And non-blocking callbacks really alleviate the weight a single threaded vm can support. However any CPU intensive request will block the process, and that will happen with looping server like Pharo or an evented VM like NodeJS. Unless you're targeting to an initial really high concurrent users count, then you can scale up gradually. Esteban A. Maringolo

Dec. 14, 2016

Re: [Pharo-users] What can we look forward to in 2017?

by Dimitris Chloupis

> > > Because reinventing the wheel is a good way to learn about wheels... > cheers -ben > > Absolutely , actually I find it funny when I hear or read "do not reinvent the wheel" . How many different wheels do we not already have out there, just a huge number and each serves a specific purpose very well. Out of the so many objects one could pick , wheel is the worst option because of the versatility and efficient of its variations in its designs. As you said reinvention allow you to basically reverse engineer and really learn your subject matter but that is half of the story, the other half is that you apply this knowledge to make a customised version which will be an actual improvement for the specific role you want it. This is why in most case scenarios a customised solution will work much better than a generic solution.

Dec. 14, 2016