Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit
Igor wrote:
Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally.
The actual problem is of course having to work with all that data before you understand the structure. Or highly interconnected structures with unpredictable access patterns. Partial graphs are nice, once you understand how to partition. Needing to understand how to partition first is a dependency I'd rather avoid.
Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory,
That's often not what is happening. The large data is mostly static, so gets moved out of new space very quickly. Otherwise working with large data quickly becomes annoying indeed. I fully agree with you on that. Stephan
On 10 November 2016 at 19:58, Stephan Eggermont <stephan@stack.nl> wrote:
Igor wrote:
Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally.
The actual problem is of course having to work with all that data before you understand the structure. Or highly interconnected structures with unpredictable access patterns. Partial graphs are nice, once you understand how to partition. Needing to understand how to partition first is a dependency I'd rather avoid.
No, no, no! This is simply not true. It is you, who writes the code that generates a lot of statistical data/analysis data, and its output is fairly predictable.. else you are not collecting any data, but just a random noise, isn't? Those graphs are far from being unpredictable, because they are product of a software you wrote. Its not unpredictable, unless you claim that code you write is unpredictable, then i wonder, what are you doing in a field of data analysis, if you admit that your data is nothing but just a dice roll? If you cannot tame & reason about the complexity of own code, then maybe better to change occupation and go work in casino? :) I mean, Doru is light years ahead of me and many others in field of data analysis.. so what i can advise to him on his playground? You absolutely right, that the most hardest part, you identified, is find the way how you dissect the graph data on smaller chunks. And storing such dissected graph in chunks on a hard drive outside of image and loading in case of need, is just nothing compared to the first part. And if Doru can't handle this, then who else can? Me? I have nothing comparing to his experience in that field. I had very little/occasional experience in my career where i had to deal with such domain. Cmon..
Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory,
That's often not what is happening. The large data is mostly static, so gets moved out of new space very quickly. Otherwise working with large data quickly becomes annoying indeed. I fully agree with you on that.
Stephan
-- Best regards, Igor Stasenko.
On 10/11/16 21:35, Igor Stasenko wrote:
No, no, no! This is simply not true. It is you, who writes the code that generates a lot of statistical data/analysis data, and its output is fairly predictable.. else you are not collecting any data, but just a random noise, isn't?
That would be green field development. In brown field development, I only get in when people start noticing there is a problem (why do we need more than 4GBytes for this?). At that point I want to be able to load everything they can give me in an image so I can start analyzing and structuring it.
I mean, Doru is light years ahead of me and many others in field of data analysis.. so what i can advise to him on his playground?
Well, the current FAMIX model implementation is clearly not well structured for analyzing large code bases. And it is difficult to partition because of unpredictable access patterns and high interconnection. Stephan
Hi, The discussion on this thread had nothing to do with FAMIX or Moose. It had to do with peopleâs ability to load larger pieces of data in the image without the VM imposing a low limit on it. There are clear scenarios where this is desirable, and I do not understand why this is a topic of conversation these days. The move to 64bit is a significant advantage in this area and we should applaud it. Doru
On Nov 11, 2016, at 11:29 AM, Stephan Eggermont <stephan@stack.nl> wrote:
On 10/11/16 21:35, Igor Stasenko wrote:
No, no, no! This is simply not true. It is you, who writes the code that generates a lot of statistical data/analysis data, and its output is fairly predictable.. else you are not collecting any data, but just a random noise, isn't?
That would be green field development. In brown field development, I only get in when people start noticing there is a problem (why do we need more than 4GBytes for this?). At that point I want to be able to load everything they can give me in an image so I can start analyzing and structuring it.
I mean, Doru is light years ahead of me and many others in field of data analysis.. so what i can advise to him on his playground?
Well, the current FAMIX model implementation is clearly not well structured for analyzing large code bases. And it is difficult to partition because of unpredictable access patterns and high interconnection.
Stephan
-- www.tudorgirba.com www.feenk.com "Some battles are better lost than fought."
Le 11/11/2016 à 11:29, Stephan Eggermont a écrit :
On 10/11/16 21:35, Igor Stasenko wrote:
No, no, no! This is simply not true. It is you, who writes the code that generates a lot of statistical data/analysis data, and its output is fairly predictable.. else you are not collecting any data, but just a random noise, isn't?
That would be green field development. In brown field development, I only get in when people start noticing there is a problem (why do we need more than 4GBytes for this?). At that point I want to be able to load everything they can give me in an image so I can start analyzing and structuring it.
I mean, Doru is light years ahead of me and many others in field of data analysis.. so what i can advise to him on his playground?
Well, the current FAMIX model implementation is clearly not well structured for analyzing large code bases. And it is difficult to partition because of unpredictable access patterns and high interconnection.
This is why you look for a general purpose, efficient off-loading scheme, trying to optimize a general case and get reasonable performance out of it (a.k.a fuel, but designed for partial unloading / loading: allow dangling references in a unit of load, focus on per-page units to match the underlying storage layer or network). I wrote one such layer for VW a long time ago, but didn't had time to experiment / qualify some of the techniques in it. There was an interesting attempt (IMHO ... wasn't qualified) at combining paging and automatic refinement of application working set, based on previous experience implementing a hierarchical 2D object access scheme for large datasets on slow medium (decreased access time from 30 minutes to about a few seconds). The other approach I would look is take some of the support code for such an automatic layer and use it to unload parts of my model;, and I'm pretty sure that, if I don't bench intensively, I'll get the partitioning wrong :( Overall, an interesting subject, totally not valid from a scientific point of view (the database guys have already solved everything). Only valid as a hobby, or if a company is ready to pay for a solution. Thierry
Hi thierry did you happen to have a techreport or any description of your work? Stef Le 11/11/16 à 11:44, Thierry Goubier a écrit :
Le 11/11/2016 à 11:29, Stephan Eggermont a écrit :
On 10/11/16 21:35, Igor Stasenko wrote:
No, no, no! This is simply not true. It is you, who writes the code that generates a lot of statistical data/analysis data, and its output is fairly predictable.. else you are not collecting any data, but just a random noise, isn't?
That would be green field development. In brown field development, I only get in when people start noticing there is a problem (why do we need more than 4GBytes for this?). At that point I want to be able to load everything they can give me in an image so I can start analyzing and structuring it.
I mean, Doru is light years ahead of me and many others in field of data analysis.. so what i can advise to him on his playground?
Well, the current FAMIX model implementation is clearly not well structured for analyzing large code bases. And it is difficult to partition because of unpredictable access patterns and high interconnection.
This is why you look for a general purpose, efficient off-loading scheme, trying to optimize a general case and get reasonable performance out of it (a.k.a fuel, but designed for partial unloading / loading: allow dangling references in a unit of load, focus on per-page units to match the underlying storage layer or network).
I wrote one such layer for VW a long time ago, but didn't had time to experiment / qualify some of the techniques in it. There was an interesting attempt (IMHO ... wasn't qualified) at combining paging and automatic refinement of application working set, based on previous experience implementing a hierarchical 2D object access scheme for large datasets on slow medium (decreased access time from 30 minutes to about a few seconds).
The other approach I would look is take some of the support code for such an automatic layer and use it to unload parts of my model;, and I'm pretty sure that, if I don't bench intensively, I'll get the partitioning wrong :(
Overall, an interesting subject, totally not valid from a scientific point of view (the database guys have already solved everything). Only valid as a hobby, or if a company is ready to pay for a solution.
Thierry
Le 14/11/2016 à 21:51, stepharo a écrit :
Hi thierry
did you happen to have a techreport or any description of your work?
It's a chapter in my PhD thesis ;) Thierry
Stef
Le 11/11/16 à 11:44, Thierry Goubier a écrit :
Le 11/11/2016 à 11:29, Stephan Eggermont a écrit :
On 10/11/16 21:35, Igor Stasenko wrote:
No, no, no! This is simply not true. It is you, who writes the code that generates a lot of statistical data/analysis data, and its output is fairly predictable.. else you are not collecting any data, but just a random noise, isn't?
That would be green field development. In brown field development, I only get in when people start noticing there is a problem (why do we need more than 4GBytes for this?). At that point I want to be able to load everything they can give me in an image so I can start analyzing and structuring it.
I mean, Doru is light years ahead of me and many others in field of data analysis.. so what i can advise to him on his playground?
Well, the current FAMIX model implementation is clearly not well structured for analyzing large code bases. And it is difficult to partition because of unpredictable access patterns and high interconnection.
This is why you look for a general purpose, efficient off-loading scheme, trying to optimize a general case and get reasonable performance out of it (a.k.a fuel, but designed for partial unloading / loading: allow dangling references in a unit of load, focus on per-page units to match the underlying storage layer or network).
I wrote one such layer for VW a long time ago, but didn't had time to experiment / qualify some of the techniques in it. There was an interesting attempt (IMHO ... wasn't qualified) at combining paging and automatic refinement of application working set, based on previous experience implementing a hierarchical 2D object access scheme for large datasets on slow medium (decreased access time from 30 minutes to about a few seconds).
The other approach I would look is take some of the support code for such an automatic layer and use it to unload parts of my model;, and I'm pretty sure that, if I don't bench intensively, I'll get the partitioning wrong :(
Overall, an interesting subject, totally not valid from a scientific point of view (the database guys have already solved everything). Only valid as a hobby, or if a company is ready to pay for a solution.
Thierry
participants (5)
-
Igor Stasenko -
Stephan Eggermont -
stepharo -
Thierry Goubier -
Tudor Girba