Ok, I'll try that in the coming days on my desktop box. I've got 32GB on that box, so it will be interesting. What would be a good way to stress test the GC? Phil On Wed, Nov 9, 2016 at 3:59 PM, Thierry Goubier <thierry.goubier@gmail.com> wrote:

2016-11-09 15:23 GMT+01:00 philippe.back@highoctane.be < philippe.back@gmail.com>:

...

Anyone having tested such 64 bit goodness on a Linux box?

Only as a professional demo for code developped up to the very last moment on Pharo 32 bits. So I didn't try to explose the available RAM on my laptop (yet).

Thierry

...

Phil

2016-11-09 23:20 GMT+01:00 Sven Van Caekenberghe <sven@stfx.eu>:

I ran a similar test [1] on a 16GB Ubuntu box:

# uname -a Linux ubuntu-m-16gb-nyc3-01 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/issue Ubuntu 16.04.1 LTS

# history | grep wget 5 wget http://files.pharo.org/get-files/60/pharo-64.zip 6 wget http://files.pharo.org/sources/PharoV50.sources.zip 7 wget https://bintray.com/estebanlm/pharo-vm/download_file?file_ path=pharo-linux-x86_64threaded.0a4ccdf.zip

# ./bin/pharo -vm-display-null pharo-64.image st --quit memtest.st >& out.log &

Which gave the following (abbreviated) output:

uptime 0h0m0s memory 70,918,144 bytes old 61,966,112 bytes (87.4%) young 2,781,608 bytes (3.9000000000000004%)

I see yet another bad usage of round:/roundTo: --------------^ Ah, but it was another thread ;)

used 46,301,016 bytes (65.3%) free 18,446,704 bytes (26.0%) GCs 1 (207ms between GCs) full 0 totalling 0ms (0.0% uptime) incr 1 totalling 0ms (0.0% uptime), avg 0.0ms tenures 0

Iteration 1 ...

Iteration 8 ...

uptime 0h12m44s memory 5,238,300,672 bytes old 5,229,348,640 bytes (99.80000000000001%) young 1,054,128 bytes (0.0%) used 5,207,806,800 bytes (99.4%) free 22,595,968 bytes (0.4%) GCs 23,509 (33ms between GCs) full 19 totalling 147,969ms (19.400000000000002% uptime), avg 7787.8ms incr 23490 totalling 143,396ms (18.8% uptime), avg 6.1000000000000005ms tenures 107,267,532 (avg 0 GCs/tenure) Since last view 23,508 (33ms between GCs) uptime 764.4000000000001s full 19 totalling 147,969ms (19.400000000000002% uptime), avg 7787.8ms incr 23489 totalling 143,396ms (18.8% uptime), avg 6.1000000000000005ms tenures 107,267,532 (avg 0 GCs/tenure)

# free -m total used free shared buff/cache available Mem: 16047 4632 10647 3 768 11129 Swap: 0 0 0

So, yes, it works on Linux too.

Sven

[1]

| meta hold |

NonInteractiveTranscript stdout install.

Transcript crShow: SmalltalkImage current vm statisticsReport; cr.

hold := (1 to: 8) collect: [ :i |

Transcript crShow: 'Iteration ', i asString.

ASTCache reset. meta := Dictionary new. Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | (classMeta := Dictionary new) at: #name put: each name asSymbol; at: #comment put: each comment; at: #definition put: each definition; at: #object put: each. methods := Dictionary new. classMeta at: #methods put: methods. each methodsDo: [ :method | | methodMeta | (methodMeta := Dictionary new) at: #name put: method selector; at: #source put: method sourceCode; at: #ast put: method ast; at: #args put: method argumentNames asArray; at: #formatted put: method ast formattedCode; at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); at: #object put: method. methods at: method selector put: methodMeta ]. meta at: each name asSymbol put: classMeta ]. meta.

].

3 timesRepeat: [ Smalltalk garbageCollect ].

Transcript crShow: SmalltalkImage current vm statisticsReport; cr.

On 9 Nov 2016, at 17:18, phil@highoctane.be wrote:

Ok, I'll try that in the coming days on my desktop box. I've got 32GB on that box, so it will be interesting.

What would be a good way to stress test the GC?

Phil

On Wed, Nov 9, 2016 at 3:59 PM, Thierry Goubier <thierry.goubier@gmail.com

...

wrote:

2016-11-09 15:23 GMT+01:00 philippe.back@highoctane.be <philippe.back@ gmail.com>: Anyone having tested such 64 bit goodness on a Linux box?

Only as a professional demo for code developped up to the very last moment on Pharo 32 bits. So I didn't try to explose the available RAM on my laptop (yet).

Thierry

Phil

On 10 Nov 2016, at 10:10, Denis Kudriashov <dionisiydk@gmail.com> wrote:

2016-11-09 23:30 GMT+01:00 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>: uptime 0h0m0s memory 70,918,144 bytes old 61,966,112 bytes (87.4%) young 2,781,608 bytes (3.9000000000000004%) I see yet another bad usage of round:/roundTo: --------------^

It just printed float :). I think anybody round values in this statistics.

Nothing should be rounded. Just compute the percentage and then use #printShowingDecimalPlaces: or #printOn:showingDecimalPlaces:

Am 10.11.2016 um 07:27 schrieb Tudor Girba <tudor@tudorgirba.com>:

Hi Igor,

Please refrain from speaking down on people. +1

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

+10 Norbert

Cheers, Tudor

...

On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua@gmail.com> wrote:

Nice progress, indeed. Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)

On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven@stfx.eu> wrote: OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

<Screen Shot 2016-11-09 at 11.25.28.png>

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ Image here: http://files.pharo.org/get-files/60/pharo-64.zip

[2]

| meta | ASTCache reset. meta := Dictionary new. Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | (classMeta := Dictionary new) at: #name put: each name asSymbol; at: #comment put: each comment; at: #definition put: each definition; at: #object put: each. methods := Dictionary new. classMeta at: #methods put: methods. each methodsDo: [ :method | | methodMeta | (methodMeta := Dictionary new) at: #name put: method selector; at: #source put: method sourceCode; at: #ast put: method ast; at: #args put: method argumentNames asArray; at: #formatted put: method ast formattedCode; at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); at: #object put: method. methods at: method selector put: methodMeta ]. meta at: each name asSymbol put: classMeta ]. meta.

-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com

"We can create beautiful models in a vacuum. But, to get them effective we have to deal with the inconvenience of reality."

Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image. Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies. FWIW, I have 2PB of data. Not going to fit in RAM. But also would takes ages to load anyway, so we work on pieces. FWIW2: one Hadoop cluster I am managing now will grow by, I think, some 100s of servers in the coming months. 64bit Pharo is something I can deploy on the boxes, which are 128GB-256GB things with like 32-48 cores. As I mentioned before, if you want to run something on it with Pharo, be my guest. Phil On Thu, Nov 10, 2016 at 9:12 AM, Igor Stasenko <siguctua@gmail.com> wrote:

On 10 November 2016 at 07:27, Tudor Girba <tudor@tudorgirba.com> wrote:

...

Hi Igor,

Please refrain from speaking down on people.

Hi, Doru! I just wanted to hear you :)

...

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it in image (in most optimal cases - only the portion that user sees/operates with). As i said to you before, you will hit this wall inevitably, no matter how much memory is available. So, what stops you from digging in that direction? Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to 100 MB or less. I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks. So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about doing what they need. We have Fuel for how many years now? So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again. Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution.

...

Cheers, Tudor

...

On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua@gmail.com> wrote:

Nice progress, indeed. Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)

On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven@stfx.eu> wrote: OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

<Screen Shot 2016-11-09 at 11.25.28.png>

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ Image here: http://files.pharo.org/get-files/60/pharo-64.zip

[2]

| meta | ASTCache reset. meta := Dictionary new. Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | (classMeta := Dictionary new) at: #name put: each name asSymbol; at: #comment put: each comment; at: #definition put: each definition; at: #object put: each. methods := Dictionary new. classMeta at: #methods put: methods. each methodsDo: [ :method | | methodMeta | (methodMeta := Dictionary new) at: #name put: method selector; at: #source put: method sourceCode; at: #ast put: method ast; at: #args put: method argumentNames asArray; at: #formatted put: method ast formattedCode; at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); at: #object put: method. methods at: method selector put: methodMeta ]. meta at: each name asSymbol put: classMeta ]. meta.

-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com

"We can create beautiful models in a vacuum. But, to get them effective we have to deal with the inconvenience of reality."

-- Best regards, Igor Stasenko.

On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

2016-11-10 9:49 GMT+01:00 phil@highoctane.be <phil@highoctane.be>:

...

Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.

Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies.

Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero. Basically that is what happens with Spark. http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo https://0x0fff.com/spark-misconceptions/ and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs. Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't). Phil

On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda@gmail.com> wrote:

Hi Phil,

On Thu, Nov 10, 2016 at 2:19 AM, phil@highoctane.be <phil@highoctane.be> wrote:

...

On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

...

2016-11-10 9:49 GMT+01:00 phil@highoctane.be <phil@highoctane.be>:

...

Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.

Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies.

Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero.

Basically that is what happens with Spark.

http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo https://0x0fff.com/spark-misconceptions/

While global GC may not be useful for big-data scavenging probably will be for any non-trivial query. But I think I see a misconception here. The large RAM on a multiword machine would be divided up between the cores. It makes no sense to run a single Smalltalk across lots of cores (we're a long way from having a thread-safe class library). It makes much more sense to have one Smalltalk per core. So that brings the heap sizes down and makes GC less scary.

yep, that approach what we're tried in HydraVM

...

and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs.

Sure. We have all the facilities we need to do this. We can add and remove code at runtime so we can keep live instances running, and send the code to them along with the data we want them to crunch.

...

Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't).

Or have policies which one can switch. There are quite a few levers into the GC from the image and one can easily switch off global GC with the right levers. One doesn't need a VM that doesn't contain a GC. One needs an image that is using the right policy.

or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that.

Phil

...

_,,,^..^,,,_ best, Eliot

-- Best regards, Igor Stasenko.

On 22 Nov 2016, at 19:16, phil@highoctane.be wrote:

On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua@gmail.com> wrote:

On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda@gmail.com> wrote: Hi Phil,

On Thu, Nov 10, 2016 at 2:19 AM, phil@highoctane.be <phil@highoctane.be> wrote:

On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

2016-11-10 9:49 GMT+01:00 phil@highoctane.be <phil@highoctane.be>: Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.

Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies.

Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero.

Basically that is what happens with Spark.

http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo https://0x0fff.com/spark-misconceptions/

While global GC may not be useful for big-data scavenging probably will be for any non-trivial query. But I think I see a misconception here. The large RAM on a multiword machine would be divided up between the cores. It makes no sense to run a single Smalltalk across lots of cores (we're a long way from having a thread-safe class library). It makes much more sense to have one Smalltalk per core. So that brings the heap sizes down and makes GC less scary.

yep, that approach what we're tried in HydraVM

and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs.

Sure. We have all the facilities we need to do this. We can add and remove code at runtime so we can keep live instances running, and send the code to them along with the data we want them to crunch.

Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't).

Or have policies which one can switch. There are quite a few levers into the GC from the image and one can easily switch off global GC with the right levers. One doesn't need a VM that doesn't contain a GC. One needs an image that is using the right policy.

or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that.

Could be very useful for all kinds of large binary data, like videos and sounds that we can load once and keep in the heap space.

How hard would it be to get something like that?

Large binary data poses no problem (as long as it's not a copying GC). Since a binary blob contains no subpointers, no work needs to be done. A 1M or 1G ByteArray is the same amount of GC work.

Phil

Phil

_,,,^..^,,,_ best, Eliot

-- Best regards, Igor Stasenko.

On Wed, Nov 23, 2016 at 12:53 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:

On Tue, Nov 22, 2016 at 10:26 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

...

On 22 Nov 2016, at 19:16, phil@highoctane.be wrote:

On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua@gmail.com> wrote:

On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda@gmail.com> wrote: Hi Phil,

On Thu, Nov 10, 2016 at 2:19 AM, phil@highoctane.be <phil@highoctane.be> wrote:

On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov < dionisiydk@gmail.com> wrote:

2016-11-10 9:49 GMT+01:00 phil@highoctane.be <phil@highoctane.be>: Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.

Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies.

Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero.

Basically that is what happens with Spark.

http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo https://0x0fff.com/spark-misconceptions/

While global GC may not be useful for big-data scavenging probably will be for any non-trivial query. But I think I see a misconception here. The large RAM on a multiword machine would be divided up between the cores. It makes no sense to run a single Smalltalk across lots of cores (we're a long way from having a thread-safe class library). It makes much more sense to have one Smalltalk per core. So that brings the heap sizes down and makes GC less scary.

yep, that approach what we're tried in HydraVM

and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs.

Sure. We have all the facilities we need to do this. We can add and remove code at runtime so we can keep live instances running, and send the code to them along with the data we want them to crunch.

Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't).

Or have policies which one can switch. There are quite a few levers into the GC from the image and one can easily switch off global GC with the right levers. One doesn't need a VM that doesn't contain a GC. One needs an image that is using the right policy.

or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that.

Could be very useful for all kinds of large binary data, like videos and sounds that we can load once and keep in the heap space.

How hard would it be to get something like that?

Large binary data poses no problem (as long as it's not a copying GC). Since a binary blob contains no subpointers, no work needs to be done. A 1M or 1G ByteArray is the same amount of GC work.

+1

Amen to that. But a dataset made of a gazillion of composites is not the same, right? Phil

_,,,^..^,,,_ best, Eliot

On Wed, Nov 23, 2016 at 10:51 AM, Igor Stasenko <siguctua@gmail.com> wrote:

On 23 November 2016 at 10:50, phil@highoctane.be <phil@highoctane.be> wrote:

...

On Wed, Nov 23, 2016 at 12:53 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:

...

On Tue, Nov 22, 2016 at 10:26 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

...

On 22 Nov 2016, at 19:16, phil@highoctane.be wrote:

On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua@gmail.com> wrote:

On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda@gmail.com> wrote: Hi Phil,

On Thu, Nov 10, 2016 at 2:19 AM, phil@highoctane.be < phil@highoctane.be> wrote:

On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov < dionisiydk@gmail.com> wrote:

2016-11-10 9:49 GMT+01:00 phil@highoctane.be <phil@highoctane.be>: Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.

Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies.

Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero.

Basically that is what happens with Spark.

http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo https://0x0fff.com/spark-misconceptions/

While global GC may not be useful for big-data scavenging probably will be for any non-trivial query. But I think I see a misconception here. The large RAM on a multiword machine would be divided up between the cores. It makes no sense to run a single Smalltalk across lots of cores (we're a long way from having a thread-safe class library). It makes much more sense to have one Smalltalk per core. So that brings the heap sizes down and makes GC less scary.

yep, that approach what we're tried in HydraVM

and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs.

Sure. We have all the facilities we need to do this. We can add and remove code at runtime so we can keep live instances running, and send the code to them along with the data we want them to crunch.

Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't).

Or have policies which one can switch. There are quite a few levers into the GC from the image and one can easily switch off global GC with the right levers. One doesn't need a VM that doesn't contain a GC. One needs an image that is using the right policy.

or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that.

Could be very useful for all kinds of large binary data, like videos and sounds that we can load once and keep in the heap space.

How hard would it be to get something like that?

Large binary data poses no problem (as long as it's not a copying GC). Since a binary blob contains no subpointers, no work needs to be done. A 1M or 1G ByteArray is the same amount of GC work.

+1

Amen to that. But a dataset made of a gazillion of composites is not the same, right?

yep, as soon as you have references in your data, you add more work for GC

That's what I tought. I have seen Craig Latta marking some objects with special flags in the object headers. Could there be some generic mechanism there now that we have 64-bit, super large headers? Like setting/resetting a kind of bitmask to let some spaces be GC'd or left alone? Things that we could manage image side? (damn, I need more money in the bank to let me work on these things for a long stretch, it is so frustrating </end of rant>). Phil

...

Phil

...

_,,,^..^,,,_ best, Eliot

-- Best regards, Igor Stasenko.

Le 23 nov. 2016 12:07, "Igor Stasenko" <siguctua@gmail.com> a écrit :

On 23 November 2016 at 12:41, phil@highoctane.be <phil@highoctane.be>

wrote:

...

On Wed, Nov 23, 2016 at 10:51 AM, Igor Stasenko <siguctua@gmail.com>

wrote:

...

...

On 23 November 2016 at 10:50, phil@highoctane.be <phil@highoctane.be>

wrote:

...

...

On Wed, Nov 23, 2016 at 12:53 AM, Eliot Miranda <

eliot.miranda@gmail.com> wrote:

...

...

On Tue, Nov 22, 2016 at 10:26 AM, Sven Van Caekenberghe <sven@stfx.eu>

wrote:

...

...

...

On 22 Nov 2016, at 19:16, phil@highoctane.be wrote:

On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua@gmail.com>

wrote:

...

...

On 15 November 2016 at 02:18, Eliot Miranda <

eliot.miranda@gmail.com> wrote:

...

Hi Phil,

On Thu, Nov 10, 2016 at 2:19 AM, phil@highoctane.be < phil@highoctane.be> wrote:

On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov < dionisiydk@gmail.com> wrote:

2016-11-10 9:49 GMT+01:00 phil@highoctane.be <phil@highoctane.be>: Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.

Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies.

Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero.

Basically that is what happens with Spark.

http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo

...

https://0x0fff.com/spark-misconceptions/

While global GC may not be useful for big-data scavenging probably will be for any non-trivial query. But I think I see a misconception here. The large RAM on a multiword machine would be divided up between the cores. It makes no sense to run a single Smalltalk across lots of cores (we're a long way from having a thread-safe class library). It makes much more sense to have one Smalltalk per core. So that brings the heap sizes down and makes GC less scary.

yep, that approach what we're tried in HydraVM

and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs.

Sure. We have all the facilities we need to do this. We can add and remove code at runtime so we can keep live instances running, and send the code to them along with the data we want them to crunch.

Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't).

Or have policies which one can switch. There are quite a few levers into the GC from the image and one can easily switch off global GC with the right levers. One doesn't need a VM that doesn't contain a GC. One needs an image that is using the right policy.

or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that.

Could be very useful for all kinds of large binary data, like videos and sounds that we can load once and keep in the heap space.

How hard would it be to get something like that?

Large binary data poses no problem (as long as it's not a copying GC). Since a binary blob contains no subpointers, no work needs to be done. A 1M or 1G ByteArray is the same amount of GC work.

+1

Amen to that. But a dataset made of a gazillion of composites is not the same, right?

yep, as soon as you have references in your data, you add more work for GC

That's what I tought. I have seen Craig Latta marking some objects with special flags in the object headers. Could there be some generic mechanism there now that we have 64-bit, super large headers? Like setting/resetting a kind of bitmask to let some spaces be GC'd or left alone? Things that we could manage image side?

well, adding bit(s) is just a simplest part of story. the main one is implement GC discipline to not walk over marked object(s), but as well, is by having a mechanism to ensure that marked object(s) form a closed subgraph (i.e. there's no references coming outside of it) scanning+marking a graph is usually a simple matter, you just need to provide a root(s). I had experiments with it in HydraVM, with a process we called mytosis - but it has slightly different purpose: - i implemented two primitives, the one that scans graph and reports if it fully isolated and another one is to basically clone the graph into separate memory region to start it as an image in own thread etc. But in our scenario, i imagine, that you cannot fully avoid external references - the most obvious one is instance->class references. In that case, we need some kind of mechanism to ensure that class objects that referenced by object(s) in desired data set are kept in a system as long as our blob is unchanges. That could be solved by simply declaring a 'fixed' set of external references per a subgraph which live as a normal object(s) in a system, with only exception, like i mentioned, that it need to ensured it won't be GCed, or even better won't be moved as long as our isolated graph is in use. Then the only what is left is to set the whole graph into read-only mode and you're ready to go.. And then, as you can imagine, having such mechanism opens even more interesting opportunities, like offloading graph on disk and/or (re)loading it on demand etc. Which is closely related to my flame-topic in this thread :) But the point is, that identifying subgraph(s) and designating it cannot be automated - this will always be a responsibility of user(s), because only user knows best, what he wants to be used as a static data and what are not etc etc.

Yes, I understand the implications and the root object thing. I also read about Mariano's work on Marea which could do the disk piece. Maybe a package manifest can help for doing the specification fo what should stay put. Is there any way we could get a grant or something for such a project? It is really important to have such features to avoid massive GC pauses. My use case is to load the data sets from here. https://www.google.be/url?sa=t&source=web&rct=j&url=http://proba-v.vgt.vito.... Phil

...

(damn, I need more money in the bank to let me work on these things for

a long stretch, it is so frustrating </end of rant>).

...

Phil

...

...

Phil

...

_,,,^..^,,,_ best, Eliot

-- Best regards, Igor Stasenko.

-- Best regards, Igor Stasenko.

Thanks Thierry. Please also see that with new satellites, the resolution is ever increasing (e.g. Sentinel http://m.esa.int/Our_Activities/Observing_the_Earth/Copernicus/Overview4) I understand the tile thing and indeed a lot of the algos work on tiles, but there are other ways to do this and especially with real time geo queries on custom defined polygons, you go only so far with tiles. A reason why we are using GeoTrellis backed by Accumulo in order to pump data very fast in random order. We are adding 30+ servers to the cluster at the moment just to deal with the sizes as there is a project mapping energy landscape https://vito.be/en/land-use/land-use/energy-landscapes. This thing is throwing YARN containers and uses CPU like, intensively. It is not uncommon for me to see their workload eating everything for a serious amount of CPU seconds. It would be silly not to plug Pharo into all of this infrastructure I think. Especially given the PhD/Postdoc/brainiacs per square meter there. If you have seen the Lost TV show, well, it kind of feels working there at that place. Especially given that is is kind of hidden in the woods. Maybe you could have interesting interactions with them. These guys also have their own nuclear reactor and geothermal drilling. Phil On Wed, Nov 23, 2016 at 1:30 PM, Thierry Goubier <thierry.goubier@gmail.com> wrote:

Hi Phil,

2016-11-23 12:17 GMT+01:00 philippe.back@highoctane.be < philippe.back@gmail.com>:

...

[ ...]

It is really important to have such features to avoid massive GC pauses.

My use case is to load the data sets from here. https://www.google.be/url?sa=t&source=web&rct=j&url=http://p roba-v.vgt.vito.be/sites/default/files/Product_User_Manual. pdf&ved=0ahUKEwjwlOG-4L7QAhWBniwKHZVmDZcQFggpMAI&usg=AFQjCNG RME9ZyHWQ8yCPgAQBDi1PUmzhbQ&sig2=eyaT4DlWCTjqUdQGBhFY0w

I've used that type of data before, a long time ago.

I consider that tiled / on-demand block loading is the way to go for those. Work with the header as long as possible, stream tiles if you need to work on the full data set. There is a good chance that:

1- You're memory bound for anything you compute with them 2- I/O times dominates, or become low enough to don't care (very fast SSDs) 3- It's very rare that you need full random access on the complete array 4- GC doesn't matter

Stream computing is your solution! This is how the raster GIS are implemented.

What is hard for me is manipulating a very large graph, or a sparse very large structure, like a huge Famix model or a FPGA layout model with a full design layed out on top. There, you're randomly accessing the whole of the structure (or at least you see no obvious partition) and the structure is too large for the memory or the GC.

This is why I had a long time ago this idea of a in-memory working-set / on-disk full structure with automatic determination of what the working set is.

For pointers, have a look at the Graph500 and HPCG benchmarks, especially the efficiency (ratio to peak) of HPCG runs, to see how difficult these cases are.

Regards,

Thierry

On Wed, Nov 23, 2016 at 4:16 PM, Thierry Goubier <thierry.goubier@gmail.com> wrote:

2016-11-23 15:46 GMT+01:00 phil@highoctane.be <phil@highoctane.be>:

...

Thanks Thierry.

Please also see that with new satellites, the resolution is ever increasing (e.g. Sentinel http://m.esa.int/Our_ Activities/Observing_the_Earth/Copernicus/Overview4)

It has allways been so. Anytime you reach a reasonable size, they send a new satellite with higher res / larger images :)

...

I understand the tile thing and indeed a lot of the algos work on tiles, but there are other ways to do this and especially with real time geo queries on custom defined polygons, you go only so far with tiles. A reason why we are using GeoTrellis backed by Accumulo in order to pump data very fast in random order.

But that mean you're dealing with preprocessed / graph georeferenced data (aka openstreetmap type of data). If you're dealing with raster, your polygons are approximated by a set of tiles (with a nice tile size well suited to your network / disk array).

I had reasonable success a long time ago (1991, I think), for Ifremer, with an unbalanced, sort of quadtree based decomposition for highly irregular curves on the seabed. Tree node size / tile size was computed to be exactly equal to the disk block size on a very slow medium. That sort of work is in the line of a geographic index for a database: optimise query accesses to geo-referenced objects... what is hard, and probably what you are doing, is combining geographic queries with graph queries (give me all houses in Belgium within a ten minutes bus + walk trip to a primary school)(*)

(*) One can work that out on a raster for speed. This is what GRASS does for example.

(**) I asked a student to accelerate some raster processing on a very small FPGA a long time ago. Once he had understood he could pipeline the design to increase the frequency, he then discovered that the FPGA would happily grok data faster than the computer bus could provide it :) leaving no bandwith for the data to be written back to memory.

Yes, but network can be pretty fast with bonded Ethernet interfaces these days.

...

We are adding 30+ servers to the cluster at the moment just to deal with the sizes as there is a project mapping energy landscape https://vito.be/en/land-use/land-use/energy-landscapes. This thing is throwing YARN containers and uses CPU like, intensively. It is not uncommon for me to see their workload eating everything for a serious amount of CPU seconds.

Only a few seconds ?

CPU-seconds, that the cluster usage unit for CPU. http://serverfault.com/questions/138703/a-definition-for-a-cpu-second So, says couple millions of them on a 640 core setup. CPU power is the limiting factor in these workloads it seems.

...

It would be silly not to plug Pharo into all of this infrastructure I think.

I've had quite bad results with Pharo on compute intensive code recently, so I'd plan carefully how I use it. On that sort of hardware, in the projects I'm working on, 1000x faster than Pharo on a single node is about an expected target.

Sure, but lower level C/C++ things are run from Python or Java, so Pharo will not do worse. The good bit about Pharo is that one can ship a preloaded image and that is easier than sending gigabyte (!) sized uberjars around, that Java will unzip before running, also true with Python myriad of dependencies. An image file appears super small then.

...

Especially given the PhD/Postdoc/brainiacs per square meter there. If you have seen the Lost TV show, well, it kind of feels working there at that place. Especially given that is is kind of hidden in the woods.

Maybe you could have interesting interactions with them. These guys also have their own nuclear reactor and geothermal drilling.

I'd be interested, because we're working a bit on high performance parallel runtimes and compilation for those. If one day you happen to be ready to talk about it in our place? South of Paris, not too hard to reach by public transport :)

Sure, that would be awesome. But Q1Y17 then because my schedule is pretty

packed at the moment. I can show you the thing over the web from my side, so you can see where are in terms of systems. I guess you are much more advanced but one of the goals of the project here is to be pretty approachable and gather a community that will cross pollinate algos and datasets for network effects. Phil

Thierry

...

Phil

On Wed, Nov 23, 2016 at 1:30 PM, Thierry Goubier < thierry.goubier@gmail.com> wrote:

...

Hi Phil,

2016-11-23 12:17 GMT+01:00 philippe.back@highoctane.be < philippe.back@gmail.com>:

...

[ ...]

It is really important to have such features to avoid massive GC pauses.

My use case is to load the data sets from here. https://www.google.be/url?sa=t&source=web&rct=j&url=http://p roba-v.vgt.vito.be/sites/default/files/Product_User_Manual.p df&ved=0ahUKEwjwlOG-4L7QAhWBniwKHZVmDZcQFggpMAI&usg=AFQjCNGR ME9ZyHWQ8yCPgAQBDi1PUmzhbQ&sig2=eyaT4DlWCTjqUdQGBhFY0w

I've used that type of data before, a long time ago.

I consider that tiled / on-demand block loading is the way to go for those. Work with the header as long as possible, stream tiles if you need to work on the full data set. There is a good chance that:

1- You're memory bound for anything you compute with them 2- I/O times dominates, or become low enough to don't care (very fast SSDs) 3- It's very rare that you need full random access on the complete array 4- GC doesn't matter

Stream computing is your solution! This is how the raster GIS are implemented.

What is hard for me is manipulating a very large graph, or a sparse very large structure, like a huge Famix model or a FPGA layout model with a full design layed out on top. There, you're randomly accessing the whole of the structure (or at least you see no obvious partition) and the structure is too large for the memory or the GC.

This is why I had a long time ago this idea of a in-memory working-set / on-disk full structure with automatic determination of what the working set is.

For pointers, have a look at the Graph500 and HPCG benchmarks, especially the efficiency (ratio to peak) of HPCG runs, to see how difficult these cases are.

Regards,

Thierry

On Thu, Nov 10, 2016 at 1:31 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

2016-11-10 9:49 GMT+01:00 phil@highoctane.be <phil@highoctane.be>:

...

Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image.

Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies.

Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

Which is why we plan on implementing an incremental global GC that will not stop the world, but will divide global GC up into many small steps, each of which will be shorter than 10 milliseconds, and so not be noticeable. _,,,^..^,,,_ best, Eliot

Hi Igor, I am happy to see you getting active again. The next step is to commit code at the rate you reply emails. I’d be even happier :). To address your point, of course it certainly would be great to have more people work on automated support for swapping data in and out of the image. That was the original idea behind the Fuel work. I have seen a couple of cases on the mailing lists where people are actually using Fuel for caching purposes. I have done this a couple of times, too. But, at this point these are dedicated solutions and would be interesting to see it expand further. However, your assumption is that the best design is one that deals with small chunks of data at a time. This made a lot of sense when memory was expensive and small. But, these days the cost is going down very rapidly, and sizes of 128+ GB of RAM is nowadays quite cheap, and there are strong signs of super large non-volatile memories become increasingly accessible. The software design should take advantage of what hardware offers, so it is not unreasonable to want to have a GC that can deal with large size. We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados. Cheers, Doru

On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua@gmail.com> wrote:

On 10 November 2016 at 07:27, Tudor Girba <tudor@tudorgirba.com> wrote: Hi Igor,

Please refrain from speaking down on people.

Hi, Doru! I just wanted to hear you :)

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it in image (in most optimal cases - only the portion that user sees/operates with). As i said to you before, you will hit this wall inevitably, no matter how much memory is available. So, what stops you from digging in that direction? Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to 100 MB or less. I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks. So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about doing what they need. We have Fuel for how many years now? So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again. Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution.

Cheers, Tudor

...

On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua@gmail.com> wrote:

Nice progress, indeed. Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)

On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven@stfx.eu> wrote: OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

<Screen Shot 2016-11-09 at 11.25.28.png>

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ Image here: http://files.pharo.org/get-files/60/pharo-64.zip

[2]

| meta | ASTCache reset. meta := Dictionary new. Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | (classMeta := Dictionary new) at: #name put: each name asSymbol; at: #comment put: each comment; at: #definition put: each definition; at: #object put: each. methods := Dictionary new. classMeta at: #methods put: methods. each methodsDo: [ :method | | methodMeta | (methodMeta := Dictionary new) at: #name put: method selector; at: #source put: method sourceCode; at: #ast put: method ast; at: #args put: method argumentNames asArray; at: #formatted put: method ast formattedCode; at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); at: #object put: method. methods at: method selector put: methodMeta ]. meta at: each name asSymbol put: classMeta ]. meta.

-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com

"We can create beautiful models in a vacuum. But, to get them effective we have to deal with the inconvenience of reality."

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com "Not knowing how to do something is not an argument for how it cannot be done."

Am 10.11.2016 um 10:42 schrieb Tudor Girba <tudor@tudorgirba.com>:

Hi Igor,

I am happy to see you getting active again. The next step is to commit code at the rate you reply emails. I’d be even happier :).

+1

To address your point, of course it certainly would be great to have more people work on automated support for swapping data in and out of the image. That was the original idea behind the Fuel work. I have seen a couple of cases on the mailing lists where people are actually using Fuel for caching purposes. I have done this a couple of times, too. But, at this point these are dedicated solutions and would be interesting to see it expand further.

And still it would be to general. The only thing you can say is that swapping in/out will make it slower. So you usually don't want that it is swapping. It is comparable with swap space in OSes. In many use case scenarios having swap at all is an architectural design failure. So before having a problem that resources get sparse there are good points not to care too much. And if you want to do it there is no general solution to it. How do you swap out a partial graph with fuel? How can you load back a small part graph of the graph you swapped out? Do we need to reify object references into objects in order to make that smart? It is understandable from a developers perspective. You have a real problem you should solve but then you make up all sorts of technical problems that you think you need to solve instead of the original problem. That is one prominent way how projects fail.

However, your assumption is that the best design is one that deals with small chunks of data at a time. This made a lot of sense when memory was expensive and small. But, these days the cost is going down very rapidly, and sizes of 128+ GB of RAM is nowadays quite cheap, and there are strong signs of super large non-volatile memories become increasingly accessible. The software design should take advantage of what hardware offers, so it is not unreasonable to want to have a GC that can deal with large size.

Be it small chunks of data or not. A statement that general is most likely to be wrong. So the best way might be to ignore it. Indeed you are right that hardware got cheap. Even more important is the fact that hardware is almost always cheaper than personal costs. Solving all those technical problems instead of real ones and not trying to act in an economical way ruins a lot of companies out there. You can ignore economical facts (are any other) but that doesn't make you really smart! my 2 cents, Norbert

We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados.

Cheers, Doru

...

On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua@gmail.com> wrote:

On 10 November 2016 at 07:27, Tudor Girba <tudor@tudorgirba.com> wrote: Hi Igor,

Please refrain from speaking down on people.

Hi, Doru! I just wanted to hear you :)

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it in image (in most optimal cases - only the portion that user sees/operates with). As i said to you before, you will hit this wall inevitably, no matter how much memory is available. So, what stops you from digging in that direction? Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to 100 MB or less. I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks. So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about doing what they need. We have Fuel for how many years now? So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again. Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution.

Cheers, Tudor

...

On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua@gmail.com> wrote:

Nice progress, indeed. Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)

On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven@stfx.eu> wrote: OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

<Screen Shot 2016-11-09 at 11.25.28.png>

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ Image here: http://files.pharo.org/get-files/60/pharo-64.zip

[2]

| meta | ASTCache reset. meta := Dictionary new. Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | (classMeta := Dictionary new) at: #name put: each name asSymbol; at: #comment put: each comment; at: #definition put: each definition; at: #object put: each. methods := Dictionary new. classMeta at: #methods put: methods. each methodsDo: [ :method | | methodMeta | (methodMeta := Dictionary new) at: #name put: method selector; at: #source put: method sourceCode; at: #ast put: method ast; at: #args put: method argumentNames asArray; at: #formatted put: method ast formattedCode; at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); at: #object put: method. methods at: method selector put: methodMeta ]. meta at: each name asSymbol put: classMeta ]. meta.

-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com

"We can create beautiful models in a vacuum. But, to get them effective we have to deal with the inconvenience of reality."

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com

"Not knowing how to do something is not an argument for how it cannot be done."

Am 10.11.2016 um 12:27 schrieb Thierry Goubier <thierry.goubier@gmail.com>:

2016-11-10 12:18 GMT+01:00 Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>>: [ ...]

Be it small chunks of data or not. A statement that general is most likely to be wrong. So the best way might be to ignore it. Indeed you are right that hardware got cheap. Even more important is the fact that hardware is almost always cheaper than personal costs. Solving all those technical problems instead of real ones and not trying to act in an economical way ruins a lot of companies out there. You can ignore economical facts (are any other) but that doesn't make you really smart!

I disagree with that. In some areas (HPC, Exascale, HPDA), whatever the physical limit is, we will reach and go larger than that.

To what you disagree? I didn't say you never need it. In your case you have concrete examples where it is necessary. In a lot of other cases it is counter productive. Isn't that an agreement that you cannot say it in a way too general way?

Now, about that memory aspect, there is an entire field dedicated to algorithmic solutions that never require the entire data set in memory. You just have to look and implement the right underlying abstractions to allow those algorithms to be implemented and run efficiently.

And that is good. I think you got me wrong. I find it important to be able to handle partial graphs in memory. But should everyone doing some statistical research be one implementing that? Something like this I took from the complaint making that the most important part and it is not. Norbert

(my best example for that: satelite imagery viewers... have allways been able to handle images larger than the computer RAM size. Just need a buffered streaming interface to the file).

Thierry

my 2 cents,

Norbert

...

We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados.

Cheers, Doru

...

On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua@gmail.com <mailto:siguctua@gmail.com>> wrote:

On 10 November 2016 at 07:27, Tudor Girba <tudor@tudorgirba.com <mailto:tudor@tudorgirba.com>> wrote: Hi Igor,

Please refrain from speaking down on people.

Hi, Doru! I just wanted to hear you :)

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it in image (in most optimal cases - only the portion that user sees/operates with). As i said to you before, you will hit this wall inevitably, no matter how much memory is available. So, what stops you from digging in that direction? Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to 100 MB or less. I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks. So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about doing what they need. We have Fuel for how many years now? So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again. Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution.

Cheers, Tudor

...

On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua@gmail.com <mailto:siguctua@gmail.com>> wrote:

Nice progress, indeed. Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)

On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven@stfx.eu <mailto:sven@stfx.eu>> wrote: OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

<Screen Shot 2016-11-09 at 11.25.28.png>

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ <http://bintray.com/estebanlm/pharo-vm/build#files/> Image here: http://files.pharo.org/get-files/60/pharo-64.zip <http://files.pharo.org/get-files/60/pharo-64.zip>

[2]

| meta | ASTCache reset. meta := Dictionary new. Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | (classMeta := Dictionary new) at: #name put: each name asSymbol; at: #comment put: each comment; at: #definition put: each definition; at: #object put: each. methods := Dictionary new. classMeta at: #methods put: methods. each methodsDo: [ :method | | methodMeta | (methodMeta := Dictionary new) at: #name put: method selector; at: #source put: method sourceCode; at: #ast put: method ast; at: #args put: method argumentNames asArray; at: #formatted put: method ast formattedCode; at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); at: #object put: method. methods at: method selector put: methodMeta ]. meta at: each name asSymbol put: classMeta ]. meta.

-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org <http://pharo.org/> http://association.pharo.org <http://association.pharo.org/> http://consortium.pharo.org <http://consortium.pharo.org/>

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com <http://www.tudorgirba.com/> www.feenk.com <http://www.feenk.com/>

"We can create beautiful models in a vacuum. But, to get them effective we have to deal with the inconvenience of reality."

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com <http://www.tudorgirba.com/> www.feenk.com <http://www.feenk.com/>

"Not knowing how to do something is not an argument for how it cannot be done."

On 10 November 2016 at 11:42, Tudor Girba <tudor@tudorgirba.com> wrote:

Hi Igor,

I am happy to see you getting active again. The next step is to commit code at the rate you reply emails. I’d be even happier :).

To address your point, of course it certainly would be great to have more people work on automated support for swapping data in and out of the image. That was the original idea behind the Fuel work. I have seen a couple of cases on the mailing lists where people are actually using Fuel for caching purposes. I have done this a couple of times, too. But, at this point these are dedicated solutions and would be interesting to see it expand further.

However, your assumption is that the best design is one that deals with small chunks of data at a time. This made a lot of sense when memory was expensive and small. But, these days the cost is going down very rapidly, and sizes of 128+ GB of RAM is nowadays quite cheap, and there are strong signs of super large non-volatile memories become increasingly accessible. The software design should take advantage of what hardware offers, so it is not unreasonable to want to have a GC that can deal with large size.

The speed of GC will always be in linear dependency from the size of governed memory. Yes, yes.. super fast and super clever, made by some wizard.. but still same dependency. So, it will be always in your interest to keep memory footprint as small as possible. PERIOD.

We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados.

What you saying is just: okay, we have a problem here, we hit a wall.. But we don't look for solutions! Instead let us sit and wait till someone else will be so generous to help with it. WOW, what a brilliant strategy!! So, you putting fate of your project(s) into hands of 3-rd party, which a) maybe , only maybe will work to solve your problem in next 10 years b) may decide it not worth effort right now(never) and focus on something else, because they have own priorities after all Are you serious? "Our furniture don't fits in modern truck(s), so let us wait will industry invent bigger trucks, build larger roads and then we will move" Hilarious! In that case, the problem that you arising is not that mission-critical to you, and thus making constant noise about your problem(s) is just what it is: a noise. Which returns us to my original mail with offensive tone. Cheers,

Doru

-- www.tudorgirba.com www.feenk.com

"Not knowing how to do something is not an argument for how it cannot be done."

-- Best regards, Igor Stasenko.

On 10 Nov 2016, at 17:35, Aliaksei Syrel <alex.syrel@gmail.com> wrote:

...

The speed of GC will always be in linear dependency from the size of governed memory.

Asymptotic complexity of GC is O(N), where N is heap size - amount of objects, not memory size.

Even that is not necessarily true, Generational Garbage collection and other tricks can avoid a full heap GC for a long time, even (or especially) under memory allocation stress. Apart from that, of course we have to write the most resource efficient code that we can !

I agree, however, that it's not good to create a lot of short living objects. That is why there are many practices how to overcome this problem. For example Object Pool can be nice example.

Nevertheless I can imagine many usecasses when breaking 4GB limit is useful. For example double buffering during rendering process. 1 pixel takes 32bit of memory => 8k image (near future displays) would take 126Mb of memory. Double buffering would be useful for Roassal (huge zoomed out visualization).

Storing 126Mb array object takes a lot of memory but does not influence on GC performance since it is just one object on the heap.

Cheers Alex

On Nov 10, 2016 5:02 PM, "Igor Stasenko" <siguctua@gmail.com> wrote:

On 10 November 2016 at 11:42, Tudor Girba <tudor@tudorgirba.com> wrote: Hi Igor,

I am happy to see you getting active again. The next step is to commit code at the rate you reply emails. I’d be even happier :).

To address your point, of course it certainly would be great to have more people work on automated support for swapping data in and out of the image. That was the original idea behind the Fuel work. I have seen a couple of cases on the mailing lists where people are actually using Fuel for caching purposes. I have done this a couple of times, too. But, at this point these are dedicated solutions and would be interesting to see it expand further.

However, your assumption is that the best design is one that deals with small chunks of data at a time. This made a lot of sense when memory was expensive and small. But, these days the cost is going down very rapidly, and sizes of 128+ GB of RAM is nowadays quite cheap, and there are strong signs of super large non-volatile memories become increasingly accessible. The software design should take advantage of what hardware offers, so it is not unreasonable to want to have a GC that can deal with large size.

The speed of GC will always be in linear dependency from the size of governed memory. Yes, yes.. super fast and super clever, made by some wizard.. but still same dependency. So, it will be always in your interest to keep memory footprint as small as possible. PERIOD.

We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados.

What you saying is just: okay, we have a problem here, we hit a wall.. But we don't look for solutions! Instead let us sit and wait till someone else will be so generous to help with it. WOW, what a brilliant strategy!! So, you putting fate of your project(s) into hands of 3-rd party, which a) maybe , only maybe will work to solve your problem in next 10 years b) may decide it not worth effort right now(never) and focus on something else, because they have own priorities after all

Are you serious? "Our furniture don't fits in modern truck(s), so let us wait will industry invent bigger trucks, build larger roads and then we will move" Hilarious!

In that case, the problem that you arising is not that mission-critical to you, and thus making constant noise about your problem(s) is just what it is: a noise. Which returns us to my original mail with offensive tone.

Cheers, Doru

-- www.tudorgirba.com www.feenk.com

"Not knowing how to do something is not an argument for how it cannot be done."

-- Best regards, Igor Stasenko.

On 10 November 2016 at 18:41, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 10 Nov 2016, at 17:35, Aliaksei Syrel <alex.syrel@gmail.com> wrote:

...

The speed of GC will always be in linear dependency from the size of governed memory.

Asymptotic complexity of GC is O(N), where N is heap size - amount of objects, not memory size.

Even that is not necessarily true, Generational Garbage collection and other tricks can avoid a full heap GC for a long time, even (or especially) under memory allocation stress.

That's why it asymptotic.. Still more objects => more memory.. O(N) => O(K).. so my statement holds true. And all of the tricks is a puny attempts to workaround that , all those generational, multi-generational, permanent space etc etc. It does helps, of course, but not solves the problem. Since you can always invent a real-world scenario where you can put it on knees and so, that it from 'asymptotic' becomes quite 'symptotic'.. so, all those elaborations does not dismiss my argument, especially when we're talking about large data. When it comes about BIG data - manual data/resource management is the way to go. The rest is handwaving and self-delugion :)

Apart from that, of course we have to write the most resource efficient code that we can !

...

I agree, however, that it's not good to create a lot of short living objects. That is why there are many practices how to overcome this problem. For example Object Pool can be nice example.

Nevertheless I can imagine many usecasses when breaking 4GB limit is useful. For example double buffering during rendering process. 1 pixel takes 32bit of memory => 8k image (near future displays) would take 126Mb of memory. Double buffering would be useful for Roassal (huge zoomed out visualization).

Storing 126Mb array object takes a lot of memory but does not influence on GC performance since it is just one object on the heap.

Cheers Alex

On Nov 10, 2016 5:02 PM, "Igor Stasenko" <siguctua@gmail.com> wrote:

On 10 November 2016 at 11:42, Tudor Girba <tudor@tudorgirba.com> wrote: Hi Igor,

I am happy to see you getting active again. The next step is to commit code at the rate you reply emails. I’d be even happier :).

To address your point, of course it certainly would be great to have more people work on automated support for swapping data in and out of the image. That was the original idea behind the Fuel work. I have seen a couple of cases on the mailing lists where people are actually using Fuel for caching purposes. I have done this a couple of times, too. But, at this point these are dedicated solutions and would be interesting to see it expand further.

However, your assumption is that the best design is one that deals with small chunks of data at a time. This made a lot of sense when memory was expensive and small. But, these days the cost is going down very rapidly, and sizes of 128+ GB of RAM is nowadays quite cheap, and there are strong signs of super large non-volatile memories become increasingly accessible. The software design should take advantage of what hardware offers, so it is not unreasonable to want to have a GC that can deal with large size.

The speed of GC will always be in linear dependency from the size of governed memory. Yes, yes.. super fast and super clever, made by some wizard.. but still same dependency. So, it will be always in your interest to keep memory footprint as small as possible. PERIOD.

We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados.

What you saying is just: okay, we have a problem here, we hit a wall.. But we don't look for solutions! Instead let us sit and wait till someone else will be so generous to help with it. WOW, what a brilliant strategy!! So, you putting fate of your project(s) into hands of 3-rd party, which a) maybe , only maybe will work to solve your problem in next 10 years b) may decide it not worth effort right now(never) and focus on something else, because they have own priorities after all

Are you serious? "Our furniture don't fits in modern truck(s), so let us wait will industry invent bigger trucks, build larger roads and then we will move" Hilarious!

In that case, the problem that you arising is not that mission-critical to you, and thus making constant noise about your problem(s) is just what it is: a noise. Which returns us to my original mail with offensive tone.

Cheers, Doru

-- www.tudorgirba.com www.feenk.com

"Not knowing how to do something is not an argument for how it cannot be done."

-- Best regards, Igor Stasenko.

-- Best regards, Igor Stasenko.

On 10 November 2016 at 18:57, Tudor Girba <tudor@tudorgirba.com> wrote:

Hi Igor,

I see you are still having fun :). I am not sure what you are arguing about, but it does not seem to be much related to what I said.

It is not fun, seeing that after years since we discussed this problem, and i shared my view on it, nothing changed. I really wish that problem be lifted from your eyesight. But your rhetoric tells me that you prefer to sit and wait instead of solving it. But feel free to tell me, if i am wrong.

And again, I would be very happy to work with you on something concrete. Just let me know if this is of interest and perhaps we can channel the energy on solutions rather than on discussions like this.

Why bother? Lets wait till be will have desktops with 1TB of RAM :) Ohh. sorry. Yeah, unfortunately i don't have much free time right now to dedicate to Pharo. But who knows, it may change. As you can see i keep coming, because smalltalk is not something you can forget after you learned it :)

Please, don't take my tone too close. Its my frustration takes offensive forms. My frustration, that i assuming that you can help youself, because your problem is not that hard to solve. But instead, you prefer to rely on somebody else's effort(s). Arrhgghhh!! :) Cheers,

Doru

...

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com

"From an abstract enough point of view, any two things are similar."

-- Best regards, Igor Stasenko.

Hi, There is never any point in talking down on people. It never leads to anything except perhaps stifling action and participation. We want to foster an environment in which people should not be afraid to be a novice at something and ask for help. Cheers, Doru

On Nov 10, 2016, at 9:45 AM, phil@highoctane.be wrote:

Tudor,

Igor still has a point here. I was talking yesterday with a data science guy and he was indeed more interested in lamenting than working out solutions for his problems.

Which weren't that hard to begin with as all it took is an hour of work to get his results. But I think he felt better complaining and self aggrandizing than actually making things work and move on to the next challenge.

Example of his "issues":

Him:"I have a looooot of data" Me: "Like what, more or less than 1TB?" Him: "Less than that" Me: "kay, can you give me a sample set of this hard disk?" Him: "Yeah, but no, well, I need to get it first" Me: "Let's sit tomorrow over lunch so that we can ingest it all and work it out" Him: "Let me come back to you..."

I think he was more interested in uttering things like "Spark 2.0" "Lots of data" "Star schema" (and saying it loud so that people could hear it) than solving anything real.

Overgeneralizing yes, speaking down, heh, not so much. There are indeed super smart/efficient/effective people in data science. But there is also a crowd that is quite, how to say... more interested in the Egyptian-style grand priest status than in the actual problems.

Phil

On Thu, Nov 10, 2016 at 7:27 AM, Tudor Girba <tudor@tudorgirba.com> wrote: Hi Igor,

Please refrain from speaking down on people.

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

Cheers, Tudor

...

On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua@gmail.com> wrote:

Nice progress, indeed. Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)

On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven@stfx.eu> wrote: OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

<Screen Shot 2016-11-09 at 11.25.28.png>

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ Image here: http://files.pharo.org/get-files/60/pharo-64.zip

[2]

| meta | ASTCache reset. meta := Dictionary new. Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | (classMeta := Dictionary new) at: #name put: each name asSymbol; at: #comment put: each comment; at: #definition put: each definition; at: #object put: each. methods := Dictionary new. classMeta at: #methods put: methods. each methodsDo: [ :method | | methodMeta | (methodMeta := Dictionary new) at: #name put: method selector; at: #source put: method sourceCode; at: #ast put: method ast; at: #args put: method argumentNames asArray; at: #formatted put: method ast formattedCode; at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); at: #object put: method. methods at: method selector put: methodMeta ]. meta at: each name asSymbol put: classMeta ]. meta.

-- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org

-- Best regards, Igor Stasenko.

-- www.tudorgirba.com www.feenk.com

"We can create beautiful models in a vacuum. But, to get them effective we have to deal with the inconvenience of reality."

-- www.tudorgirba.com www.feenk.com "One cannot do more than one can do."