- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by phil＠highoctane.be

On Wed, Nov 23, 2016 at 12:53 AM, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > > > On Tue, Nov 22, 2016 at 10:26 AM, Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > >> >> > On 22 Nov 2016, at 19:16, phil(a)highoctane.be wrote: >> > >> > >> > >> > On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua(a)gmail.com> >> wrote: >> > >> > >> > On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda(a)gmail.com> >> wrote: >> > Hi Phil, >> > >> > On Thu, Nov 10, 2016 at 2:19 AM, phil(a)highoctane.be <phil(a)highoctane.be> >> wrote: >> > >> > >> > On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov < >> dionisiydk(a)gmail.com> wrote: >> > >> > 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: >> > Ah, but then it may be more interesting to have a data image (maybe a >> lot of these) and a front end image. >> > >> > Isn't Seamless something that could help us here? No need to bring the >> data back, just manipulate it through proxies. >> > >> > Problem that server image will anyway perform GC. And it will be slow >> if server image is big which will stop all world. >> > >> > What if we asked it to not do any GC at all? Like if we have tons of >> RAM, why bother? Especially if what it is used to is to keep datasets: load >> them, save image to disk. When needed trash the loaded stuff and reload >> from zero. >> > >> > Basically that is what happens with Spark. >> > >> > http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo >> > https://0x0fff.com/spark-misconceptions/ >> > >> > While global GC may not be useful for big-data scavenging probably will >> be for any non-trivial query. But I think I see a misconception here. The >> large RAM on a multiword machine would be divided up between the cores. It >> makes no sense to run a single Smalltalk across lots of cores (we're a long >> way from having a thread-safe class library). It makes much more sense to >> have one Smalltalk per core. So that brings the heap sizes down and makes >> GC less scary. >> > >> > yep, that approach what we're tried in HydraVM >> > >> > >> > and Tachyon/Alluxio is kind of solving this kind of issue (may be nice >> to have that interacting with Pharo image). http://www.alluxio.org/ This >> thing basically keeps stuff in memory in case one needs to reuse the data >> between workload runs. >> > >> > Sure. We have all the facilities we need to do this. We can add and >> remove code at runtime so we can keep live instances running, and send the >> code to them along with the data we want them to crunch. >> > >> > >> > Or have an object memory for work and one for datasets (first one gets >> GC'd, the other one isn't). >> > >> > Or have policies which one can switch. There are quite a few levers >> into the GC from the image and one can easily switch off global GC with the >> right levers. One doesn't need a VM that doesn't contain a GC. One needs >> an image that is using the right policy. >> > >> > or just mark whole data (sub)graphs with some bit, telling GC to skip >> over this so it won't attempt to scan it treating them as always alive.. >> > this is where we getting back to my idea of heap spaces, where you can >> toss a subgraph into a special heap space that has such policy, that it is >> never scanned/GCed automatically and can be triggered only manually or >> something like that. >> > >> > Could be very useful for all kinds of large binary data, like videos >> and sounds that we can load once and keep in the heap space. >> > >> > How hard would it be to get something like that? >> >> Large binary data poses no problem (as long as it's not a copying GC). >> Since a binary blob contains no subpointers, no work needs to be done. A 1M >> or 1G ByteArray is the same amount of GC work. >> > > +1 > Amen to that. But a dataset made of a gazillion of composites is not the same, right? Phil > > _,,,^..^,,,_ > best, Eliot >

Nov. 23, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Eliot Miranda

On Tue, Nov 22, 2016 at 10:26 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > On 22 Nov 2016, at 19:16, phil(a)highoctane.be wrote: > > > > > > > > On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua(a)gmail.com> > wrote: > > > > > > On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda(a)gmail.com> > wrote: > > Hi Phil, > > > > On Thu, Nov 10, 2016 at 2:19 AM, phil(a)highoctane.be <phil(a)highoctane.be> > wrote: > > > > > > On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk(a)gmail.com> > wrote: > > > > 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: > > Ah, but then it may be more interesting to have a data image (maybe a > lot of these) and a front end image. > > > > Isn't Seamless something that could help us here? No need to bring the > data back, just manipulate it through proxies. > > > > Problem that server image will anyway perform GC. And it will be slow if > server image is big which will stop all world. > > > > What if we asked it to not do any GC at all? Like if we have tons of > RAM, why bother? Especially if what it is used to is to keep datasets: load > them, save image to disk. When needed trash the loaded stuff and reload > from zero. > > > > Basically that is what happens with Spark. > > > > http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo > > https://0x0fff.com/spark-misconceptions/ > > > > While global GC may not be useful for big-data scavenging probably will > be for any non-trivial query. But I think I see a misconception here. The > large RAM on a multiword machine would be divided up between the cores. It > makes no sense to run a single Smalltalk across lots of cores (we're a long > way from having a thread-safe class library). It makes much more sense to > have one Smalltalk per core. So that brings the heap sizes down and makes > GC less scary. > > > > yep, that approach what we're tried in HydraVM > > > > > > and Tachyon/Alluxio is kind of solving this kind of issue (may be nice > to have that interacting with Pharo image). http://www.alluxio.org/ This > thing basically keeps stuff in memory in case one needs to reuse the data > between workload runs. > > > > Sure. We have all the facilities we need to do this. We can add and > remove code at runtime so we can keep live instances running, and send the > code to them along with the data we want them to crunch. > > > > > > Or have an object memory for work and one for datasets (first one gets > GC'd, the other one isn't). > > > > Or have policies which one can switch. There are quite a few levers > into the GC from the image and one can easily switch off global GC with the > right levers. One doesn't need a VM that doesn't contain a GC. One needs > an image that is using the right policy. > > > > or just mark whole data (sub)graphs with some bit, telling GC to skip > over this so it won't attempt to scan it treating them as always alive.. > > this is where we getting back to my idea of heap spaces, where you can > toss a subgraph into a special heap space that has such policy, that it is > never scanned/GCed automatically and can be triggered only manually or > something like that. > > > > Could be very useful for all kinds of large binary data, like videos and > sounds that we can load once and keep in the heap space. > > > > How hard would it be to get something like that? > > Large binary data poses no problem (as long as it's not a copying GC). > Since a binary blob contains no subpointers, no work needs to be done. A 1M > or 1G ByteArray is the same amount of GC work. > +1 _,,,^..^,,,_ best, Eliot

Nov. 22, 2016

Re: [Pharo-dev] test please ignore

by Dimitris Chloupis

Ignoring failed : Please try again On Tue, 22 Nov 2016 at 23:23, Ron Teitelbaum <ron(a)usmedrec.com> wrote: > Successfully ignored. > > Ron > > -----Original Message----- > From: Pharo-dev [mailto:pharo-dev-bounces@lists.pharo.org] On Behalf Of > stepharo > Sent: Tuesday, November 22, 2016 4:20 PM > To: pharo-dev(a)lists.pharo.org > Subject: [Pharo-dev] test please ignore > > > > -- > Using Opera's mail client: http://www.opera.com/mail/ > > > >

Nov. 22, 2016

Re: [Pharo-dev] test please ignore

by Ron Teitelbaum

Successfully ignored. Ron -----Original Message----- From: Pharo-dev [mailto:pharo-dev-bounces@lists.pharo.org] On Behalf Of stepharo Sent: Tuesday, November 22, 2016 4:20 PM To: pharo-dev(a)lists.pharo.org Subject: [Pharo-dev] test please ignore -- Using Opera's mail client: http://www.opera.com/mail/

Nov. 22, 2016

test please ignore

by stepharo

-- Using Opera's mail client: http://www.opera.com/mail/

Nov. 22, 2016

This is a test

by stepharo

-- Using Opera's mail client: http://www.opera.com/mail/

Nov. 22, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Sven Van Caekenberghe

> On 22 Nov 2016, at 19:16, phil(a)highoctane.be wrote: > > > > On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > Hi Phil, > > On Thu, Nov 10, 2016 at 2:19 AM, phil(a)highoctane.be <phil(a)highoctane.be> wrote: > > > On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: > Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image. > > Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies. > > Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world. > > What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero. > > Basically that is what happens with Spark. > > http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo > https://0x0fff.com/spark-misconceptions/ > > While global GC may not be useful for big-data scavenging probably will be for any non-trivial query. But I think I see a misconception here. The large RAM on a multiword machine would be divided up between the cores. It makes no sense to run a single Smalltalk across lots of cores (we're a long way from having a thread-safe class library). It makes much more sense to have one Smalltalk per core. So that brings the heap sizes down and makes GC less scary. > > yep, that approach what we're tried in HydraVM > > > and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs. > > Sure. We have all the facilities we need to do this. We can add and remove code at runtime so we can keep live instances running, and send the code to them along with the data we want them to crunch. > > > Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't). > > Or have policies which one can switch. There are quite a few levers into the GC from the image and one can easily switch off global GC with the right levers. One doesn't need a VM that doesn't contain a GC. One needs an image that is using the right policy. > > or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. > this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that. > > Could be very useful for all kinds of large binary data, like videos and sounds that we can load once and keep in the heap space. > > How hard would it be to get something like that? Large binary data poses no problem (as long as it's not a copying GC). Since a binary blob contains no subpointers, no work needs to be done. A 1M or 1G ByteArray is the same amount of GC work. > Phil > > > > Phil > > _,,,^..^,,,_ > best, Eliot > > > > -- > Best regards, > Igor Stasenko.

Nov. 22, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by phil＠highoctane.be

On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda(a)gmail.com> > wrote: > >> Hi Phil, >> >> On Thu, Nov 10, 2016 at 2:19 AM, phil(a)highoctane.be <phil(a)highoctane.be> >> wrote: >> >>> >>> >>> On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk(a)gmail.com >>> > wrote: >>> >>>> >>>> 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: >>>> >>>>> Ah, but then it may be more interesting to have a data image (maybe a >>>>> lot of these) and a front end image. >>>>> >>>>> Isn't Seamless something that could help us here? No need to bring the >>>>> data back, just manipulate it through proxies. >>>>> >>>> >>>> Problem that server image will anyway perform GC. And it will be slow >>>> if server image is big which will stop all world. >>>> >>> >>> What if we asked it to not do any GC at all? Like if we have tons of >>> RAM, why bother? Especially if what it is used to is to keep datasets: load >>> them, save image to disk. When needed trash the loaded stuff and reload >>> from zero. >>> >>> Basically that is what happens with Spark. >>> >>> http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo >>> https://0x0fff.com/spark-misconceptions/ >>> >> >> While global GC may not be useful for big-data scavenging probably will >> be for any non-trivial query. But I think I see a misconception here. The >> large RAM on a multiword machine would be divided up between the cores. It >> makes no sense to run a single Smalltalk across lots of cores (we're a long >> way from having a thread-safe class library). It makes much more sense to >> have one Smalltalk per core. So that brings the heap sizes down and makes >> GC less scary. >> > > yep, that approach what we're tried in HydraVM > > >> >> >>> and Tachyon/Alluxio is kind of solving this kind of issue (may be nice >>> to have that interacting with Pharo image). http://www.alluxio.org/ >>> This thing basically keeps stuff in memory in case one needs to reuse the >>> data between workload runs. >>> >> >> Sure. We have all the facilities we need to do this. We can add and >> remove code at runtime so we can keep live instances running, and send the >> code to them along with the data we want them to crunch. >> >> >>> >>> Or have an object memory for work and one for datasets (first one gets >>> GC'd, the other one isn't). >>> >> >> Or have policies which one can switch. There are quite a few levers into >> the GC from the image and one can easily switch off global GC with the >> right levers. One doesn't need a VM that doesn't contain a GC. One needs >> an image that is using the right policy. >> > > or just mark whole data (sub)graphs with some bit, telling GC to skip over > this so it won't attempt to scan it treating them as always alive.. > this is where we getting back to my idea of heap spaces, where you can > toss a subgraph into a special heap space that has such policy, that it is > never scanned/GCed automatically and can be triggered only manually or > something like that. > Could be very useful for all kinds of large binary data, like videos and sounds that we can load once and keep in the heap space. How hard would it be to get something like that? Phil > > >> >> Phil >>> >> >> _,,,^..^,,,_ >> best, Eliot >> > > > > -- > Best regards, > Igor Stasenko. >

Nov. 22, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Igor Stasenko

On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > Hi Phil, > > On Thu, Nov 10, 2016 at 2:19 AM, phil(a)highoctane.be <phil(a)highoctane.be> > wrote: > >> >> >> On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk(a)gmail.com> >> wrote: >> >>> >>> 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: >>> >>>> Ah, but then it may be more interesting to have a data image (maybe a >>>> lot of these) and a front end image. >>>> >>>> Isn't Seamless something that could help us here? No need to bring the >>>> data back, just manipulate it through proxies. >>>> >>> >>> Problem that server image will anyway perform GC. And it will be slow if >>> server image is big which will stop all world. >>> >> >> What if we asked it to not do any GC at all? Like if we have tons of RAM, >> why bother? Especially if what it is used to is to keep datasets: load >> them, save image to disk. When needed trash the loaded stuff and reload >> from zero. >> >> Basically that is what happens with Spark. >> >> http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo >> https://0x0fff.com/spark-misconceptions/ >> > > While global GC may not be useful for big-data scavenging probably will be > for any non-trivial query. But I think I see a misconception here. The > large RAM on a multiword machine would be divided up between the cores. It > makes no sense to run a single Smalltalk across lots of cores (we're a long > way from having a thread-safe class library). It makes much more sense to > have one Smalltalk per core. So that brings the heap sizes down and makes > GC less scary. > yep, that approach what we're tried in HydraVM > > >> and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to >> have that interacting with Pharo image). http://www.alluxio.org/ This >> thing basically keeps stuff in memory in case one needs to reuse the data >> between workload runs. >> > > Sure. We have all the facilities we need to do this. We can add and > remove code at runtime so we can keep live instances running, and send the > code to them along with the data we want them to crunch. > > >> >> Or have an object memory for work and one for datasets (first one gets >> GC'd, the other one isn't). >> > > Or have policies which one can switch. There are quite a few levers into > the GC from the image and one can easily switch off global GC with the > right levers. One doesn't need a VM that doesn't contain a GC. One needs > an image that is using the right policy. > or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that. > > Phil >> > > _,,,^..^,,,_ > best, Eliot > -- Best regards, Igor Stasenko.

Nov. 22, 2016

Re: [Pharo-dev] Running on Ubuntu?

by Igor Stasenko

On 22 November 2016 at 16:50, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Igor, > > For future reference, > > > On 22 Nov 2016, at 11:17, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > (this is what I use for headless) > > > > sudo dpkg --add-architecture i386 > > sudo apt-get update > > sudo apt-get install libc6:i386 > > sudo apt-get install libssl1.0.0:i386 > > sudo apt-get install libfreetype6:i386 > > On a fresh Ubuntu 16.04.1 LTS 64-bit I did the above > > > (you might need more for full UI) > > And then just one other install > > sudo apt-get install libgl1-mesa-glx:i386 > > which installed lots of dependencies. > > After that I was able to run the download > > get.pharo.org/60+vm > > in UI mode (using pharo-ui). > > Sven > > ohh.. wait 6.0 is on the sight.. and i was mangling with 5.0.. yeah, that explains why it needs 32-bit libs and all that mess :) -- Best regards, Igor Stasenko.

Nov. 22, 2016