- Pharo-dev - lists.pharo.org

Re: [Pharo-project] Nearly limitless Image: revisited.
by Janko MivÅ¡ek Feb. 29, 2012

Feb. 29, 2012

Good compromise is a step by step approach, something like: 1.step: image based persistence up to 1GB with hourly snapshot 2.step: parts migrated to Fuel and file based persistence 3.step: Gemstone, with "images" running in parallel (well, any DB with images in parallel, Gemstone is certainly the easiest to scale from image based persistence) 1GB limit is here just for simplicity, you can probably go further with 64bit images. Advantages: - very very simple start - freedom of pure OO modeling, - good enough for probably 90% of all projects - fastest way from your dreams to reality - speed of development - no impedance mismatch, no ORM nightmare - you won't believe how much data you can put in 1GB image - speed because of always in-memory data processing - you can always scale further if you make your design from the start with above steps in mind - reliability good enough, on reliable hardware probably even better than more complex solutions. Main reason: simplicity. Disadvantages: - you easy forgot to include later scalability requirements in upfront design - such scaling is easy only to OO database while migrating to NoSQL (not to mention SQL) database later is very hard if not possible - up to about 1GB only, because of GC problems as Igor described - active users limit (number of requests/s) - single point of failure - corrupted image will loose all data (but good backup approach helps) - undetected image corruption fear (after many otherwise successful snapshots, causing non-startable image) - lengthy snapshots of bigger images (can be improved with two step snapshots, first in memory, then on disk) - loss of data between snapshots in case of power or machine failure (but this is very rare those days) Bet regards Janko S, Igor Stasenko piÅ¡e: > Incidentally, we had a little chat with Marcus yesterday about that. > > No, i don't think it is feasible to use single image to store everything. > It is convenient, cheap and of course it is way better than dealing > with communicating with external DB/servers whatever. > > But there's one thing you should know already: the days of vertical > growth is over. > > Running a service (under VM or not) on a single machine is asking for troubles: > - limits on load > - susceptible to power outage and other reliability problems > etc > > Also, think that the amount of data you need to process correlates > with CPU horsepower available. > Which means that yes, you can run a huge image with 64Gb data in it.. > but that means that responsiveness > of your service will quite often fall beyond any usability limits. > > If we look in terms of VM and pick only one thing - garbage collection, > you will see that there is certain limits beyond which a performance > will drop too much, so you naturally > will start thinking about ways to split data to separate chunks and > run them on different machines/VMs. > > It is because GC's mark algorithm is O(n) bound, when n is total > number of references between objects, > and GC's scavenge algorithm is at best O(n) bound where n is total > number of objects in object memory, > and at worst is where n is total memory used by objects. > No matter how you turn it, i just wanted to indicate that time to run > GC is in linear dependency from the amount of data. > > Yes, we might invest a lot of effort in making GC more clever, more > complex and more robust.. but no matter what you do, > you cannot change the above facts. It means, that any improvements > will be about diminishing returns, but won't change the picture > radically. > > That means that sooner or later you will have to deal with it: a > problem of splitting data on multiple independent chunks, > and making your service to run on multiple machines , in order to use > more CPU power, more memory and be more reliable etc. > At this point, your main dilemma is to invent a fast and robust > interfaces to communicate between images or between image(s)/ database > etc. > > We should concentrate on things which dealing with inter-image > communication and image-database communication, > because it is the only way to ensure that we will answer upcoming > future problems. Relying on using a single huge image is way to > nowhere. > -- Janko MivÅ¡ek Aida/Web Smalltalk Web Application Server http://www.aidaweb.si

2 1

[Pharo-project] New CogVM binaries as per VMMaker.oscog-eem.150/r2537
by Eliot Miranda Feb. 29, 2012

Feb. 29, 2012

http://www.mirandabanda.org/files/Cog/VM/VM.r2537/ CogVM binaries as per VMMaker.oscog-eem.150/r2537 Disable filling of the weakRoots table during fullGC. Report remappable oop and weak root table overflows as errors. Make primitiveObjectAtPut fail if used to store other than a SmallInteger into the method header. Better error message for inability to thread. Load function pointer early in ThreadedFFIPlugin's invocation checking sequence. This must come early for compatibility with the old FFIPlugin. Image-level code may assume the function pointer is loaded eagerly. Add 1007,1008 & 1009 attribute info to version info on linux, & 1009 on Windows. Add -version option to Mac VM (with similar info to linux). Include vm-display-fbdev on linux. Exclude fbdev in the newspeak build. Simplify ticker for older linuxes. -- best, Eliot

1 0

Re: [Pharo-project] Nearly limitless Image: revisited.
by Marcus Denker Feb. 29, 2012

Feb. 29, 2012

On Feb 29, 2012, at 1:35 PM, Sven Van Caekenberghe wrote: > > On 29 Feb 2012, at 12:58, Janko MivÅ¡ek wrote: > >> For Pharo there seems everyone is avoiding this idea, mostly because of >> image corruption fear. Well, we need to improve the robustness of Pharo >> image and VM then this fear will vanish. It is obviously doable, if VW >> is reliable enough, why not once Pharo? > > I am trying to be polite: remarks like this are unfair and not based on the experiences of most of the people in the Pharo community or of the companies using Pharo. It is perfectly possible to have stable long running servers based on Pharo. Sneaking in these phrases in simply spreading FUD. It's not fair to expect, for nothing, the same as you get from a very expensive product. Especially if even just entering a bug report is seemingly already too much to ask (!), while for VisualWorks, you have to pay real money. "It is obviously doable" has a differnt meaning with some Millions of EUR per year that one can spend for engineering than without. Without, it's *far* from obvious! Marcus -- Marcus Denker -- http://marcusdenker.de

3 2

Re: [Pharo-project] Nearly limitless Image: revisited.
by Marcus Denker Feb. 29, 2012

Feb. 29, 2012

On Feb 29, 2012, at 12:59 PM, Janko MivÅ¡ek wrote: > Hi Ted, > > I also support the image as a sole persistence idea and I'm actually > using it for years on VisualWorks. > > For Pharo there seems everyone is avoiding this idea, mostly because of > image corruption fear. Well, we need to improve the robustness of Pharo > image and VM then this fear will vanish. It is obviously doable, if VW > is reliable enough, why not once Pharo? > Someone needs to start fixing and improving. Why not, for example, you? Here is a link: http://code.google.com/p/pharo/issues/list It's much better than talking. Marcus -- Marcus Denker -- http://marcusdenker.de

4 3

[Pharo-project] MC and shared pools
by Igor Stasenko Feb. 29, 2012

Feb. 29, 2012

I just stumbled upon this.. I was assuming that it is perfectly fine to: - define a shared pool in package - make a class which using given pool in same package but when MC loads it , it looks like it doesn't care about loading order and if it loads the class which using pool first, it says that pool not exists and asks me to create it. if you create it, then when it loads the pool , it barks that given class already in system... i know that solution is to have 2 packages, 1 with pool, and another with classes which using them, but somehow i was assumed that it should work fine in single package.. So, here 2 questions: - does anybody knows if this worked ever? - do we need that? -- Best regards, Igor Stasenko.

1 0

[Pharo-project] [update 1.4] #14369
by Marcus Denker Feb. 29, 2012

Feb. 29, 2012

14369 ----- Issue 5331: use the glamour style for pretty print http://code.google.com/p/pharo/issues/detail?id=5331 Issue 5396: Deprecate TTMorph classes http://code.google.com/p/pharo/issues/detail?id=5396 -- Marcus Denker -- http://marcusdenker.de

1 0

[Pharo-project] "Do not feed the trolls"
by Dale Henrichs Feb. 29, 2012

Feb. 29, 2012

>From Wikipedia: In Internet slang, a troll is someone who posts inflammatory,[2] extraneous, or off-topic messages in an online community, such as an online discussion forum, chat room, or blog, with the primary intent of provoking readers into an emotional response[3] or of otherwise disrupting normal on-topic discussion.[4] The noun troll may refer to the provocative message itself, as in: "That was an excellent troll you posted".

29 78

[Pharo-project] VNC client
by Norbert Hartl Feb. 29, 2012

Feb. 29, 2012

I wonder what vnc client do you use if you use pharo + RFB? I had a running via macports. Now that I'm about to switch to homebrew I have problems finding a working one. Be it in brew or app store, etc. thanks, Norbert

5 10

Re: [Pharo-project] Nearly limitless Image: revisited.
by Janko MivÅ¡ek Feb. 29, 2012

Feb. 29, 2012

Hi Ted, I also support the image as a sole persistence idea and I'm actually using it for years on VisualWorks. For Pharo there seems everyone is avoiding this idea, mostly because of image corruption fear. Well, we need to improve the robustness of Pharo image and VM then this fear will vanish. It is obviously doable, if VW is reliable enough, why not once Pharo? Best regards Janko S, ted f.a. van gaalen piÅ¡e: > "I first used image persistence but the image grow too large.." > thanks for reminding, Norbert. > > > Hi all, > > perhaps you'll find me an idiot, I don't care. > (i know that I am: (class SelfReflection :o) ) > > Nevertheless, some brainstorming here. > > I've brought this up before, but it seems nobody is interested in this > topic.. why? > > it's about image size. > > Currently, Pharo 1.3-13315 at my 4 Gigabyte Debian amd64 Linux machine > reports > > memory 75,392,160 bytes > old 61,370,144 bytes (81.4%) > young 2,169,692 bytes (2.9000000000000004%) > used 63,539,836 bytes (84.30000000000001%) > free 11,852,324 bytes (15.700000000000001%) > > (nothing peculiar added, just the Seaside package and a few classes of > my own. ) > > So, if i understand this well I have only 11,8 megabytes to play with? > > To begin with: why is only 75 megabyte available in Pharo (or Squeak as > well) > when there is 4 gigabyte physical memory available on my machine?? > > Anyway, IMHO this very limited storage violates the Smalltalk-all-in-one > image ideal, > because with this limited storage one cannot avoid the use of external > data stores > like databases (of course, exception: there always will be import an > export of objects/data) > > External data storage is complex, unreliable and should be something of > the past. > I'd suggest: don't waste time with this. rather improve the Smalltalk > internal data handling. > > I wish to have a very large image so e.q. I can have an entire > company's administration > (client and accounting data etc.) alive in it. > I want to use Collections and descendants, not external databases. > So, everything within the image. > > The beautiful (although opinions may vary) principle > of a Smalltalk environment is Image Persistence, isn't it? > > IMHO this requires: > - virtually unlimited image memory size. > -no external databases like e.g. Mongo or DB2 whatever. > -no persistent external storage at all. > > -You should see the (currently still) faster database performance as a > challenge > to improve the Collection and other related data handling classes > > -You would have to completely restructure the architecture. > or just extend the address width: 64 bits? > > what about an image of virtually unlimited size > gigabytes so > > (for now, if it goes beyond the physical memory size (say 32 GB?) > one could use a VM that works with virtual storage based VM > Virtual storage was first in use successfully with IBM mainframes > in the 1970s (and still is) ) > > on the other hand, I estimate holographic? physical memory > going in Terabytes is about to be available in 5-10 years from now. > blindingly fast, compared with today's standards. > > So, one has about 5 years to come up with such a system :o) > if one does, Smalltalk is ahead of everything then.. > > Don't laugh. One didn't even dream of 4 gigabytes of memory > during the Commodore 64 era. 1982-1994. > > In short: what would have to be changed to enable this? > > Some thoughts and speculations? > Anyone for tennis? > > Thanks > Ted > > > > > > -- Janko MivÅ¡ek Aida/Web Smalltalk Web Application Server http://www.aidaweb.si

2 1

[Pharo-project] Who is using Fuel?
by Mariano Martinez Peck Feb. 29, 2012

Feb. 29, 2012

Of course, not the fuel you use to go to work every day ;) I am talking about Fuel serializer. In particular, I wonder if our page deserves an update: rmod.lille.inria.fr/web/pier/software/Fuel/Software-using-Fuel Let me know if there is something I should add. Thanks, -- Mariano http://marianopeck.wordpress.com

4 19