November 2016 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by phil＠highoctane.be

Thanks Thierry. Please also see that with new satellites, the resolution is ever increasing (e.g. Sentinel http://m.esa.int/Our_Activities/Observing_the_Earth/Copernicus/Overview4) I understand the tile thing and indeed a lot of the algos work on tiles, but there are other ways to do this and especially with real time geo queries on custom defined polygons, you go only so far with tiles. A reason why we are using GeoTrellis backed by Accumulo in order to pump data very fast in random order. We are adding 30+ servers to the cluster at the moment just to deal with the sizes as there is a project mapping energy landscape https://vito.be/en/land-use/land-use/energy-landscapes. This thing is throwing YARN containers and uses CPU like, intensively. It is not uncommon for me to see their workload eating everything for a serious amount of CPU seconds. It would be silly not to plug Pharo into all of this infrastructure I think. Especially given the PhD/Postdoc/brainiacs per square meter there. If you have seen the Lost TV show, well, it kind of feels working there at that place. Especially given that is is kind of hidden in the woods. Maybe you could have interesting interactions with them. These guys also have their own nuclear reactor and geothermal drilling. Phil On Wed, Nov 23, 2016 at 1:30 PM, Thierry Goubier <thierry.goubier(a)gmail.com> wrote: > Hi Phil, > > 2016-11-23 12:17 GMT+01:00 philippe.back(a)highoctane.be < > philippe.back(a)gmail.com>: > >> [ ...] >> >> It is really important to have such features to avoid massive GC pauses. >> >> My use case is to load the data sets from here. >> https://www.google.be/url?sa=t&source=web&rct=j&url=http://p >> roba-v.vgt.vito.be/sites/default/files/Product_User_Manual. >> pdf&ved=0ahUKEwjwlOG-4L7QAhWBniwKHZVmDZcQFggpMAI&usg=AFQjCNG >> RME9ZyHWQ8yCPgAQBDi1PUmzhbQ&sig2=eyaT4DlWCTjqUdQGBhFY0w >> > I've used that type of data before, a long time ago. > > I consider that tiled / on-demand block loading is the way to go for > those. Work with the header as long as possible, stream tiles if you need > to work on the full data set. There is a good chance that: > > 1- You're memory bound for anything you compute with them > 2- I/O times dominates, or become low enough to don't care (very fast SSDs) > 3- It's very rare that you need full random access on the complete array > 4- GC doesn't matter > > Stream computing is your solution! This is how the raster GIS are > implemented. > > What is hard for me is manipulating a very large graph, or a sparse very > large structure, like a huge Famix model or a FPGA layout model with a full > design layed out on top. There, you're randomly accessing the whole of the > structure (or at least you see no obvious partition) and the structure is > too large for the memory or the GC. > > This is why I had a long time ago this idea of a in-memory working-set / > on-disk full structure with automatic determination of what the working set > is. > > For pointers, have a look at the Graph500 and HPCG benchmarks, especially > the efficiency (ratio to peak) of HPCG runs, to see how difficult these > cases are. > > Regards, > > Thierry >

Nov. 23, 2016

Re: [Pharo-dev] IDNA / punycode for Zinc?

by Sven Van Caekenberghe

> On 23 Nov 2016, at 15:36, Max Leske <maxleske(a)gmail.com> wrote: > > Great! > > Thereâs a punycode implementation on smalltalkhub (http://smalltalkhub.com/#!/~dTriangle/Punycode) but it needs some polishing. Wow, that looks good, it even has ZnUrl integration, so we're done ;-) There are no tests though. How come we never heard of this ? Last commit was in 2013, hopefully the author is still around. > Should I open an issue on FogBugz so we donât forget? Yes, OK. > Max > >> On 23 Nov 2016, at 15:00, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> Max, >> >>> On 23 Nov 2016, at 14:34, Max Leske <maxleske(a)gmail.com> wrote: >>> >>> Hi (Sven), >>> >>> Zinc canât currently handle unicode domain names (e.g. http://Ã¼ni.ch) Are there any plans to implement punycode / IDNA conversion for Zinc? Or is there an explicit reason not to support it? I see that #parseHostPort: expects the host portion to be percent escaped, what is the use case for this? I have never seen a percent escaped host portion. Usually the host portion is either pure ASCII, unicode or punycode (in my experience at least). >>> >>> Just curious, as I just added IDNA conversion to one of our applications (I just let python perform the conversion: https://docs.python.org/2/library/codecs.html#module-encodings.idna) >>> >>> Cheers, >>> Max >> >> Yes, that would be nice to have. >> >> Just for future reference, we are talking about the following (IDN(A)): >> >> https://en.wikipedia.org/wiki/Internationalized_domain_name >> https://en.wikipedia.org/wiki/Punycode >> https://tools.ietf.org/html/rfc3490 >> https://www.charset.org/punycode >> >> Normal DNS hostnames are ASCII only (or used to be like that anyway), that is why it is (currently) implemented like that. >> >> Sven >> >> >

Nov. 23, 2016

Re: [Pharo-dev] IDNA / punycode for Zinc?

by Max Leske

Great! Thereâs a punycode implementation on smalltalkhub (http://smalltalkhub.com/#!/~dTriangle/Punycode <http://smalltalkhub.com/#!/~dTriangle/Punycode>) but it needs some polishing. Should I open an issue on FogBugz so we donât forget? Max > On 23 Nov 2016, at 15:00, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Max, > >> On 23 Nov 2016, at 14:34, Max Leske <maxleske(a)gmail.com> wrote: >> >> Hi (Sven), >> >> Zinc canât currently handle unicode domain names (e.g. http://Ã¼ni.ch) Are there any plans to implement punycode / IDNA conversion for Zinc? Or is there an explicit reason not to support it? I see that #parseHostPort: expects the host portion to be percent escaped, what is the use case for this? I have never seen a percent escaped host portion. Usually the host portion is either pure ASCII, unicode or punycode (in my experience at least). >> >> Just curious, as I just added IDNA conversion to one of our applications (I just let python perform the conversion: https://docs.python.org/2/library/codecs.html#module-encodings.idna) >> >> Cheers, >> Max > > Yes, that would be nice to have. > > Just for future reference, we are talking about the following (IDN(A)): > > https://en.wikipedia.org/wiki/Internationalized_domain_name > https://en.wikipedia.org/wiki/Punycode > https://tools.ietf.org/html/rfc3490 > https://www.charset.org/punycode > > Normal DNS hostnames are ASCII only (or used to be like that anyway), that is why it is (currently) implemented like that. > > Sven > >

Nov. 23, 2016

Re: [Pharo-dev] IDNA / punycode for Zinc?

by Sven Van Caekenberghe

Max, > On 23 Nov 2016, at 14:34, Max Leske <maxleske(a)gmail.com> wrote: > > Hi (Sven), > > Zinc canât currently handle unicode domain names (e.g. http://Ã¼ni.ch) Are there any plans to implement punycode / IDNA conversion for Zinc? Or is there an explicit reason not to support it? I see that #parseHostPort: expects the host portion to be percent escaped, what is the use case for this? I have never seen a percent escaped host portion. Usually the host portion is either pure ASCII, unicode or punycode (in my experience at least). > > Just curious, as I just added IDNA conversion to one of our applications (I just let python perform the conversion: https://docs.python.org/2/library/codecs.html#module-encodings.idna) > > Cheers, > Max Yes, that would be nice to have. Just for future reference, we are talking about the following (IDN(A)): https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Punycode https://tools.ietf.org/html/rfc3490 https://www.charset.org/punycode Normal DNS hostnames are ASCII only (or used to be like that anyway), that is why it is (currently) implemented like that. Sven

Nov. 23, 2016

Re: [Pharo-dev] About ~= and ~~

by Aliaksei Syrel

Hi It is been a while... So, do we want to replace ~~ with a primitive? :) Cheers Alex -- View this message in context: http://forum.world.st/About-and-tp3898409p4924391.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.

Nov. 23, 2016

IDNA / punycode for Zinc?

by Max Leske

Hi (Sven), Zinc canât currently handle unicode domain names (e.g. http://Ã¼ni.ch <http://xn--ni-wka.ch/>). Are there any plans to implement punycode / IDNA conversion for Zinc? Or is there an explicit reason not to support it? I see that #parseHostPort: expects the host portion to be percent escaped, what is the use case for this? I have never seen a percent escaped host portion. Usually the host portion is either pure ASCII, unicode or punycode (in my experience at least). Just curious, as I just added IDNA conversion to one of our applications (I just let python perform the conversion: https://docs.python.org/2/library/codecs.html#module-encodings.idna <https://docs.python.org/2/library/codecs.html#module-encodings.idna>). Cheers, Max

Nov. 23, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Thierry Goubier

Hi Phil, 2016-11-23 12:17 GMT+01:00 philippe.back(a)highoctane.be < philippe.back(a)gmail.com>: > [ ...] > > It is really important to have such features to avoid massive GC pauses. > > My use case is to load the data sets from here. > https://www.google.be/url?sa=t&source=web&rct=j&url=http:// > proba-v.vgt.vito.be/sites/default/files/Product_User_ > Manual.pdf&ved=0ahUKEwjwlOG-4L7QAhWBniwKHZVmDZcQFggpMAI&usg= > AFQjCNGRME9ZyHWQ8yCPgAQBDi1PUmzhbQ&sig2=eyaT4DlWCTjqUdQGBhFY0w > I've used that type of data before, a long time ago. I consider that tiled / on-demand block loading is the way to go for those. Work with the header as long as possible, stream tiles if you need to work on the full data set. There is a good chance that: 1- You're memory bound for anything you compute with them 2- I/O times dominates, or become low enough to don't care (very fast SSDs) 3- It's very rare that you need full random access on the complete array 4- GC doesn't matter Stream computing is your solution! This is how the raster GIS are implemented. What is hard for me is manipulating a very large graph, or a sparse very large structure, like a huge Famix model or a FPGA layout model with a full design layed out on top. There, you're randomly accessing the whole of the structure (or at least you see no obvious partition) and the structure is too large for the memory or the GC. This is why I had a long time ago this idea of a in-memory working-set / on-disk full structure with automatic determination of what the working set is. For pointers, have a look at the Graph500 and HPCG benchmarks, especially the efficiency (ratio to peak) of HPCG runs, to see how difficult these cases are. Regards, Thierry

Nov. 23, 2016

Re: [Pharo-dev] STON doesn't produce valid JSON - it shouldn't escape quation mark

by Peter Uhnak

Thanks! P On Sun, Nov 20, 2016 at 08:02:35PM +0100, Sven Van Caekenberghe wrote: > I committed the following in STON (sthub.com, ss3.gemstone.com & github.com) #bleedingEdge: > > === > Name: STON-Core-SvenVanCaekenberghe.78 > Author: SvenVanCaekenberghe > Time: 20 November 2016, 7:48:23.799323 pm > UUID: ee198da8-80e8-4944-ba9b-dffae331c57c > Ancestors: STON-Core-SvenVanCaekenberghe.77 > > In JSON compatibility mode STON should not encode single quotes. (Thanks Peter UhnÃ¡k for reporting this). Patch STONWriter>>#encodeCharacter: > > Fix the encoding of forward slash ($/). This was already in the (external) documentation. Patch STONWriter class>>#initializeSTONCharacters - force STONWriter class>>#initialize to change too > > Fix the representation of Time by adding nanoseconds when there are any. This was already in the (external) documentation. Patch Time>>#stonOn: > > Adjust unit tests accordingly (esp. STONWriterTests>>#testDoubleQuotedString). > > Minor comment changes. > === > Name: STON-Tests-SvenVanCaekenberghe.68 > Author: SvenVanCaekenberghe > Time: 20 November 2016, 7:48:46.115446 pm > UUID: 70402f28-f0e8-44e9-8746-3539c4add09b > Ancestors: STON-Tests-SvenVanCaekenberghe.67 > > In JSON compatibility mode STON should not encode single quotes. (Thanks Peter UhnÃ¡k for reporting this). Patch STONWriter>>#encodeCharacter: > > Fix the encoding of forward slash ($/). This was already in the (external) documentation. Patch STONWriter class>>#initializeSTONCharacters - force STONWriter class>>#initialize to change too > > Fix the representation of Time by adding nanoseconds when there are any. This was already in the (external) documentation. Patch Time>>#stonOn: > > Adjust unit tests accordingly (esp. STONWriterTests>>#testDoubleQuotedString). > > Minor comment changes. > === > > Now you get > > NeoJSONReader fromString: (STON toJsonString: 'single='' & double="'). > "'single='' & double=""'" > > STON toString: { Time now. Date today. DateAndTime now }. > "'[Time[''19:51:15.452023''],Date[''2016-11-20''],DateAndTime[''2016-11-20T19:51:15.454147+01:00'']]'" > > STON toString: 'back-slash=\ & forward-slash=/'. > "'''back-slash=\\ & forward-slash=\/'''" > > Sven > > > On 19 Nov 2016, at 20:56, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > Peter, > > > > You are right, that is a bug. > > > > STON should be a bit more strict when it is in JSON mode. I'll fix it and let you know. > > > > Thanks for reporting this issue. > > > > Sven > > > >> On 19 Nov 2016, at 17:46, <i.uhnak(a)gmail.com> <i.uhnak(a)gmail.com> wrote: > >> > >> \â is not a valid JSON escape character, so STONâs JSON ouput is not a valid JSON string. > >> > >> ston := STON toJsonString: 'element''s'. -> "element\'s" > >> neo := NeoJSONWriter toString: 'element''s'. -> "element's" > >> > >> NeoJSONReader fromString: ston. -> invalid escape character \â > >> > >> Peter > > > >

Nov. 23, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by philippe.back＠highoctane.be

Le 23 nov. 2016 12:07, "Igor Stasenko" <siguctua(a)gmail.com> a Ã©crit : > > > > On 23 November 2016 at 12:41, phil(a)highoctane.be <phil(a)highoctane.be> wrote: >> >> >> >> On Wed, Nov 23, 2016 at 10:51 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: >>> >>> >>> >>> On 23 November 2016 at 10:50, phil(a)highoctane.be <phil(a)highoctane.be> wrote: >>>> >>>> >>>> >>>> On Wed, Nov 23, 2016 at 12:53 AM, Eliot Miranda < eliot.miranda(a)gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> On Tue, Nov 22, 2016 at 10:26 AM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>>> >>>>>> >>>>>> > On 22 Nov 2016, at 19:16, phil(a)highoctane.be wrote: >>>>>> > >>>>>> > >>>>>> > >>>>>> > On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua(a)gmail.com> wrote: >>>>>> > >>>>>> > >>>>>> > On 15 November 2016 at 02:18, Eliot Miranda < eliot.miranda(a)gmail.com> wrote: >>>>>> > Hi Phil, >>>>>> > >>>>>> > On Thu, Nov 10, 2016 at 2:19 AM, phil(a)highoctane.be < phil(a)highoctane.be> wrote: >>>>>> > >>>>>> > >>>>>> > On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov < dionisiydk(a)gmail.com> wrote: >>>>>> > >>>>>> > 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: >>>>>> > Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image. >>>>>> > >>>>>> > Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies. >>>>>> > >>>>>> > Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world. >>>>>> > >>>>>> > What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero. >>>>>> > >>>>>> > Basically that is what happens with Spark. >>>>>> > >>>>>> > http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo >>>>>> > https://0x0fff.com/spark-misconceptions/ >>>>>> > >>>>>> > While global GC may not be useful for big-data scavenging probably will be for any non-trivial query. But I think I see a misconception here. The large RAM on a multiword machine would be divided up between the cores. It makes no sense to run a single Smalltalk across lots of cores (we're a long way from having a thread-safe class library). It makes much more sense to have one Smalltalk per core. So that brings the heap sizes down and makes GC less scary. >>>>>> > >>>>>> > yep, that approach what we're tried in HydraVM >>>>>> > >>>>>> > >>>>>> > and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs. >>>>>> > >>>>>> > Sure. We have all the facilities we need to do this. We can add and remove code at runtime so we can keep live instances running, and send the code to them along with the data we want them to crunch. >>>>>> > >>>>>> > >>>>>> > Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't). >>>>>> > >>>>>> > Or have policies which one can switch. There are quite a few levers into the GC from the image and one can easily switch off global GC with the right levers. One doesn't need a VM that doesn't contain a GC. One needs an image that is using the right policy. >>>>>> > >>>>>> > or just mark whole data (sub)graphs with some bit, telling GC to skip over this so it won't attempt to scan it treating them as always alive.. >>>>>> > this is where we getting back to my idea of heap spaces, where you can toss a subgraph into a special heap space that has such policy, that it is never scanned/GCed automatically and can be triggered only manually or something like that. >>>>>> > >>>>>> > Could be very useful for all kinds of large binary data, like videos and sounds that we can load once and keep in the heap space. >>>>>> > >>>>>> > How hard would it be to get something like that? >>>>>> >>>>>> Large binary data poses no problem (as long as it's not a copying GC). Since a binary blob contains no subpointers, no work needs to be done. A 1M or 1G ByteArray is the same amount of GC work. >>>>> >>>>> >>>>> +1 >>>> >>>> >>>> Amen to that. But a dataset made of a gazillion of composites is not the same, right? >>>> >>> yep, as soon as you have references in your data, you add more work for GC >> >> >> That's what I tought. I have seen Craig Latta marking some objects with special flags in the object headers. Could there be some generic mechanism there now that we have 64-bit, super large headers? Like setting/resetting a kind of bitmask to let some spaces be GC'd or left alone? Things that we could manage image side? >> > well, adding bit(s) is just a simplest part of story. the main one is implement GC discipline to not walk over marked object(s), but as well, is by having a mechanism to ensure that marked object(s) form a closed subgraph (i.e. there's no references coming outside of it) > scanning+marking a graph is usually a simple matter, you just need to provide a root(s). I had experiments with it in HydraVM, with a process we called mytosis - but it has slightly different purpose: > - i implemented two primitives, the one that scans graph and reports if it fully isolated > and another one is to basically clone the graph into separate memory region to start it as an image in own thread etc. > But in our scenario, i imagine, that you cannot fully avoid external references - the most obvious one is instance->class references. In that case, we need some kind of mechanism to ensure that class objects that > referenced by object(s) in desired data set are kept in a system as long as our blob is unchanges. That could be solved by simply declaring a 'fixed' set of external references per a subgraph which live as a normal object(s) in a system, with only exception, like i mentioned, that it need to ensured it won't be GCed, or even better won't be moved as long as our isolated graph is in use. > Then the only what is left is to set the whole graph into read-only mode and you're ready to go.. > And then, as you can imagine, having such mechanism opens even more interesting opportunities, like offloading graph on disk and/or (re)loading it on demand etc. Which is closely related to my flame-topic in this thread :) > But the point is, that identifying subgraph(s) and designating it cannot be automated - this will always be a responsibility of user(s), because only user knows best, what he wants to be used as a static data and what are not etc etc. Yes, I understand the implications and the root object thing. I also read about Mariano's work on Marea which could do the disk piece. Maybe a package manifest can help for doing the specification fo what should stay put. Is there any way we could get a grant or something for such a project? It is really important to have such features to avoid massive GC pauses. My use case is to load the data sets from here. https://www.google.be/url?sa=t&source=web&rct=j&url=http://proba-v.vgt.vito… Phil > >> >> (damn, I need more money in the bank to let me work on these things for a long stretch, it is so frustrating </end of rant>). >> >> Phil >> >>> >>>> >>>> Phil >>>>> >>>>> >>>>> _,,,^..^,,,_ >>>>> best, Eliot >>>> >>>> >>> >>> >>> >>> -- >>> Best regards, >>> Igor Stasenko. >> >> > > > > -- > Best regards, > Igor Stasenko.

Nov. 23, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Igor Stasenko

On 23 November 2016 at 12:41, phil(a)highoctane.be <phil(a)highoctane.be> wrote: > > > On Wed, Nov 23, 2016 at 10:51 AM, Igor Stasenko <siguctua(a)gmail.com> > wrote: > >> >> >> On 23 November 2016 at 10:50, phil(a)highoctane.be <phil(a)highoctane.be> >> wrote: >> >>> >>> >>> On Wed, Nov 23, 2016 at 12:53 AM, Eliot Miranda <eliot.miranda(a)gmail.com >>> > wrote: >>> >>>> >>>> >>>> On Tue, Nov 22, 2016 at 10:26 AM, Sven Van Caekenberghe <sven(a)stfx.eu> >>>> wrote: >>>> >>>>> >>>>> > On 22 Nov 2016, at 19:16, phil(a)highoctane.be wrote: >>>>> > >>>>> > >>>>> > >>>>> > On Tue, Nov 22, 2016 at 5:57 PM, Igor Stasenko <siguctua(a)gmail.com> >>>>> wrote: >>>>> > >>>>> > >>>>> > On 15 November 2016 at 02:18, Eliot Miranda <eliot.miranda(a)gmail.com> >>>>> wrote: >>>>> > Hi Phil, >>>>> > >>>>> > On Thu, Nov 10, 2016 at 2:19 AM, phil(a)highoctane.be < >>>>> phil(a)highoctane.be> wrote: >>>>> > >>>>> > >>>>> > On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov < >>>>> dionisiydk(a)gmail.com> wrote: >>>>> > >>>>> > 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: >>>>> > Ah, but then it may be more interesting to have a data image (maybe >>>>> a lot of these) and a front end image. >>>>> > >>>>> > Isn't Seamless something that could help us here? No need to bring >>>>> the data back, just manipulate it through proxies. >>>>> > >>>>> > Problem that server image will anyway perform GC. And it will be >>>>> slow if server image is big which will stop all world. >>>>> > >>>>> > What if we asked it to not do any GC at all? Like if we have tons of >>>>> RAM, why bother? Especially if what it is used to is to keep datasets: load >>>>> them, save image to disk. When needed trash the loaded stuff and reload >>>>> from zero. >>>>> > >>>>> > Basically that is what happens with Spark. >>>>> > >>>>> > http://sujee.net/2015/01/22/understanding-spark-caching/#.WC >>>>> RIgy0rKpo >>>>> > https://0x0fff.com/spark-misconceptions/ >>>>> > >>>>> > While global GC may not be useful for big-data scavenging probably >>>>> will be for any non-trivial query. But I think I see a misconception >>>>> here. The large RAM on a multiword machine would be divided up between the >>>>> cores. It makes no sense to run a single Smalltalk across lots of cores >>>>> (we're a long way from having a thread-safe class library). It makes much >>>>> more sense to have one Smalltalk per core. So that brings the heap sizes >>>>> down and makes GC less scary. >>>>> > >>>>> > yep, that approach what we're tried in HydraVM >>>>> > >>>>> > >>>>> > and Tachyon/Alluxio is kind of solving this kind of issue (may be >>>>> nice to have that interacting with Pharo image). >>>>> http://www.alluxio.org/ This thing basically keeps stuff in memory in >>>>> case one needs to reuse the data between workload runs. >>>>> > >>>>> > Sure. We have all the facilities we need to do this. We can add >>>>> and remove code at runtime so we can keep live instances running, and send >>>>> the code to them along with the data we want them to crunch. >>>>> > >>>>> > >>>>> > Or have an object memory for work and one for datasets (first one >>>>> gets GC'd, the other one isn't). >>>>> > >>>>> > Or have policies which one can switch. There are quite a few levers >>>>> into the GC from the image and one can easily switch off global GC with the >>>>> right levers. One doesn't need a VM that doesn't contain a GC. One needs >>>>> an image that is using the right policy. >>>>> > >>>>> > or just mark whole data (sub)graphs with some bit, telling GC to >>>>> skip over this so it won't attempt to scan it treating them as always >>>>> alive.. >>>>> > this is where we getting back to my idea of heap spaces, where you >>>>> can toss a subgraph into a special heap space that has such policy, that it >>>>> is never scanned/GCed automatically and can be triggered only manually or >>>>> something like that. >>>>> > >>>>> > Could be very useful for all kinds of large binary data, like videos >>>>> and sounds that we can load once and keep in the heap space. >>>>> > >>>>> > How hard would it be to get something like that? >>>>> >>>>> Large binary data poses no problem (as long as it's not a copying GC). >>>>> Since a binary blob contains no subpointers, no work needs to be done. A 1M >>>>> or 1G ByteArray is the same amount of GC work. >>>>> >>>> >>>> +1 >>>> >>> >>> Amen to that. But a dataset made of a gazillion of composites is not the >>> same, right? >>> >>> yep, as soon as you have references in your data, you add more work for >> GC >> > > That's what I tought. I have seen Craig Latta marking some objects with > special flags in the object headers. Could there be some generic mechanism > there now that we have 64-bit, super large headers? Like setting/resetting > a kind of bitmask to let some spaces be GC'd or left alone? Things that we > could manage image side? > > well, adding bit(s) is just a simplest part of story. the main one is implement GC discipline to not walk over marked object(s), but as well, is by having a mechanism to ensure that marked object(s) form a closed subgraph (i.e. there's no references coming outside of it) scanning+marking a graph is usually a simple matter, you just need to provide a root(s). I had experiments with it in HydraVM, with a process we called mytosis - but it has slightly different purpose: - i implemented two primitives, the one that scans graph and reports if it fully isolated and another one is to basically clone the graph into separate memory region to start it as an image in own thread etc. But in our scenario, i imagine, that you cannot fully avoid external references - the most obvious one is instance->class references. In that case, we need some kind of mechanism to ensure that class objects that referenced by object(s) in desired data set are kept in a system as long as our blob is unchanges. That could be solved by simply declaring a 'fixed' set of external references per a subgraph which live as a normal object(s) in a system, with only exception, like i mentioned, that it need to ensured it won't be GCed, or even better won't be moved as long as our isolated graph is in use. Then the only what is left is to set the whole graph into read-only mode and you're ready to go.. And then, as you can imagine, having such mechanism opens even more interesting opportunities, like offloading graph on disk and/or (re)loading it on demand etc. Which is closely related to my flame-topic in this thread :) But the point is, that identifying subgraph(s) and designating it cannot be automated - this will always be a responsibility of user(s), because only user knows best, what he wants to be used as a static data and what are not etc etc. > (damn, I need more money in the bank to let me work on these things for a > long stretch, it is so frustrating </end of rant>). > > Phil > > >> >>> Phil >>> >>>> >>>> _,,,^..^,,,_ >>>> best, Eliot >>>> >>> >>> >> >> >> -- >> Best regards, >> Igor Stasenko. >> > > -- Best regards, Igor Stasenko.

Nov. 23, 2016