Nearly limitless Image: revisited.
"I first used image persistence but the image grow too large.." thanks for reminding, Norbert. Hi all, perhaps you'll find me an idiot, I don't care. (i know that I am: (class SelfReflection :o) ) Nevertheless, some brainstorming here. I've brought this up before, but it seems nobody is interested in this topic.. why? it's about image size. Currently, Pharo 1.3-13315 at my 4 Gigabyte Debian amd64 Linux machine reports memory 75,392,160 bytes old 61,370,144 bytes (81.4%) young 2,169,692 bytes (2.9000000000000004%) used 63,539,836 bytes (84.30000000000001%) free 11,852,324 bytes (15.700000000000001%) (nothing peculiar added, just the Seaside package and a few classes of my own. ) So, if i understand this well I have only 11,8 megabytes to play with? To begin with: why is only 75 megabyte available in Pharo (or Squeak as well) when there is 4 gigabyte physical memory available on my machine?? Anyway, IMHO this very limited storage violates the Smalltalk-all-in-one image ideal, because with this limited storage one cannot avoid the use of external data stores like databases (of course, exception: there always will be import an export of objects/data) External data storage is complex, unreliable and should be something of the past. I'd suggest: don't waste time with this. rather improve the Smalltalk internal data handling. I wish to have a very large image so e.q. I can have an entire company's administration (client and accounting data etc.) alive in it. I want to use Collections and descendants, not external databases. So, everything within the image. The beautiful (although opinions may vary) principle of a Smalltalk environment is Image Persistence, isn't it? IMHO this requires: - virtually unlimited image memory size. -no external databases like e.g. Mongo or DB2 whatever. -no persistent external storage at all. -You should see the (currently still) faster database performance as a challenge to improve the Collection and other related data handling classes -You would have to completely restructure the architecture. or just extend the address width: 64 bits? what about an image of virtually unlimited size > gigabytes so (for now, if it goes beyond the physical memory size (say 32 GB?) one could use a VM that works with virtual storage based VM Virtual storage was first in use successfully with IBM mainframes in the 1970s (and still is) ) on the other hand, I estimate holographic? physical memory going in Terabytes is about to be available in 5-10 years from now. blindingly fast, compared with today's standards. So, one has about 5 years to come up with such a system :o) if one does, Smalltalk is ahead of everything then.. Don't laugh. One didn't even dream of 4 gigabytes of memory during the Commodore 64 era. 1982-1994. In short: what would have to be changed to enable this? Some thoughts and speculations? Anyone for tennis? Thanks Ted -- Ted F.A. van Gaalen Danketsweiler 608 D-88263 Horgenzell Germany T: +49 750 491 48 38 M: +49 151 587 862 47
On 28 Feb 2012, at 23:25, ted f.a. van gaalen wrote:
it's about image size.
Currently, Pharo 1.3-13315 at my 4 Gigabyte Debian amd64 Linux machine reports
memory 75,392,160 bytes old 61,370,144 bytes (81.4%) young 2,169,692 bytes (2.9000000000000004%) used 63,539,836 bytes (84.30000000000001%) free 11,852,324 bytes (15.700000000000001%)
(nothing peculiar added, just the Seaside package and a few classes of my own. )
So, if i understand this well I have only 11,8 megabytes to play with?
To begin with: why is only 75 megabyte available in Pharo (or Squeak as well) when there is 4 gigabyte physical memory available on my machine??
You are interpreting this wrongly: total memory allocation of the heap did not yet grow beyond 75Mb because you still have 11Mb free, so there was not need for it. See this thread: http://forum.world.st/Big-Image-Tests-td4188045.html#a4188548 Although you cannot allocate much more than 1Gb, that is already more than enough for a lot of purposes. Of course, 64-bit VMs / Smalltalk images would be helpful in growing even larger, there seems to be some progress there. Sven
Am 28.02.2012 um 23:25 schrieb ted f.a. van gaalen:
In short: what would have to be changed to enable this?
You can read the LOOM paper [1] or you can have a look at GemStone [2]
Some thoughts and speculations?
Yes. Predictions about how long it takes for something technical to become true are always too short termed. :) I think that the gap between volatile memory and persistent memory is only a necessity nothing natural. The gap will be closed somewhere in the future. Having a LOOM architecture helps mitigate the effect of the gap if not make it unnoticable. From then on you just need to buy the newest things like SSDs to make it happen sooner.
Anyone for tennis?
Sure. Didn't play for a long time and over 500Km is bit far off, don't you think? :) Norbert [1] http://dl.acm.org/citation.cfm?id=94112 [2] http://www.gemstone.com/products/gemstone
On 2/28/12 5:44 PM, Norbert Hartl wrote:
You can read the LOOM paper [1] or you can have a look at GemStone [2]
I was interested in this as well. After some googling for the LOOM paper, I discovered it's in the " Smalltalk-80 Bits of History, Words of Advice " (green book). Pages 251-271. http://stephane.ducasse.free.fr/FreeBooks/BitsOfHistory/BitsOfHistory.pdf
Hi Ted, I also support the image as a sole persistence idea and I'm actually using it for years on VisualWorks. For Pharo there seems everyone is avoiding this idea, mostly because of image corruption fear. Well, we need to improve the robustness of Pharo image and VM then this fear will vanish. It is obviously doable, if VW is reliable enough, why not once Pharo? Best regards Janko S, ted f.a. van gaalen piše:
"I first used image persistence but the image grow too large.." thanks for reminding, Norbert.
Hi all,
perhaps you'll find me an idiot, I don't care. (i know that I am: (class SelfReflection :o) )
Nevertheless, some brainstorming here.
I've brought this up before, but it seems nobody is interested in this topic.. why?
it's about image size.
Currently, Pharo 1.3-13315 at my 4 Gigabyte Debian amd64 Linux machine reports
memory 75,392,160 bytes old 61,370,144 bytes (81.4%) young 2,169,692 bytes (2.9000000000000004%) used 63,539,836 bytes (84.30000000000001%) free 11,852,324 bytes (15.700000000000001%)
(nothing peculiar added, just the Seaside package and a few classes of my own. )
So, if i understand this well I have only 11,8 megabytes to play with?
To begin with: why is only 75 megabyte available in Pharo (or Squeak as well) when there is 4 gigabyte physical memory available on my machine??
Anyway, IMHO this very limited storage violates the Smalltalk-all-in-one image ideal, because with this limited storage one cannot avoid the use of external data stores like databases (of course, exception: there always will be import an export of objects/data)
External data storage is complex, unreliable and should be something of the past. I'd suggest: don't waste time with this. rather improve the Smalltalk internal data handling.
I wish to have a very large image so e.q. I can have an entire company's administration (client and accounting data etc.) alive in it. I want to use Collections and descendants, not external databases. So, everything within the image.
The beautiful (although opinions may vary) principle of a Smalltalk environment is Image Persistence, isn't it?
IMHO this requires: - virtually unlimited image memory size. -no external databases like e.g. Mongo or DB2 whatever. -no persistent external storage at all.
-You should see the (currently still) faster database performance as a challenge to improve the Collection and other related data handling classes
-You would have to completely restructure the architecture. or just extend the address width: 64 bits?
what about an image of virtually unlimited size > gigabytes so
(for now, if it goes beyond the physical memory size (say 32 GB?) one could use a VM that works with virtual storage based VM Virtual storage was first in use successfully with IBM mainframes in the 1970s (and still is) )
on the other hand, I estimate holographic? physical memory going in Terabytes is about to be available in 5-10 years from now. blindingly fast, compared with today's standards.
So, one has about 5 years to come up with such a system :o) if one does, Smalltalk is ahead of everything then..
Don't laugh. One didn't even dream of 4 gigabytes of memory during the Commodore 64 era. 1982-1994.
In short: what would have to be changed to enable this?
Some thoughts and speculations? Anyone for tennis?
Thanks Ted
-- Janko Mivšek Aida/Web Smalltalk Web Application Server http://www.aidaweb.si
participants (5)
-
David Graham -
Janko Mivšek -
Norbert Hartl -
Sven Van Caekenberghe -
ted f.a. van gaalen