Re: [Pharo-project] experience with large images?
Igor Stasenko wrote:
I think you looking for solution in a wrong direction. Just ask yourself, how much of that data you need to keep in operative memory at single moment of time to efficiently compute results.
All of it. And then of course the annotations and resulting output model. I know practically nothing about the data at first (about 700 files, about 700 MB). When trying to understand the data, it is crucial that I can test hypotheses fast. Efficiency is about my time, not computer time.
If today you need to deal with >500Mb data sets, tomorrow you may need to deal with multigigabyte datasets, which can easily surpass the amount of operative memory your computer has.
16 GB DIMMS are about 250 Euro. For a commercial project it is easy to justify half a terabyte of ram.
I know, it is easier to find cheap solution, without spending time implementing own data caching scheme, but you just delaying inevitable.
Delaying is crucial. If I wait long enough, PC's will have enough memory. Stephan
On 25 November 2011 01:28, Stephan Eggermont <stephan@stack.nl> wrote:
Igor Stasenko wrote:
I think you looking for solution in a wrong direction. Just ask yourself, how much of that data you need to keep in operative memory at single moment of time to efficiently compute results.
All of it. And then of course the annotations and resulting output model. I know practically nothing about the data at first (about 700 files, about 700 MB). When trying to understand the data, it is crucial that I can test hypotheses fast. Efficiency is about my time, not computer time.
If today you need to deal with >500Mb data sets, tomorrow you may need to deal with multigigabyte datasets, which can easily surpass the amount of operative memory your computer has.
16 GB DIMMS are about 250 Euro. For a commercial project it is easy to justify half a terabyte of ram.
ok. then how about investing a bit to make sure VMs could conserve that much memory? :)
I know, it is easier to find cheap solution, without spending time implementing own data caching scheme, but you just delaying inevitable.
Delaying is crucial. If I wait long enough, PC's will have enough memory.
Stephan
-- Best regards, Igor Stasenko.
Am 25.11.2011 um 00:45 schrieb Igor Stasenko:
On 25 November 2011 01:28, Stephan Eggermont <stephan@stack.nl> wrote:
Igor Stasenko wrote:
I think you looking for solution in a wrong direction. Just ask yourself, how much of that data you need to keep in operative memory at single moment of time to efficiently compute results.
All of it. And then of course the annotations and resulting output model. I know practically nothing about the data at first (about 700 files, about 700 MB). When trying to understand the data, it is crucial that I can test hypotheses fast. Efficiency is about my time, not computer time.
If today you need to deal with >500Mb data sets, tomorrow you may need to deal with multigigabyte datasets, which can easily surpass the amount of operative memory your computer has.
16 GB DIMMS are about 250 Euro. For a commercial project it is easy to justify half a terabyte of ram.
ok. then how about investing a bit to make sure VMs could conserve that much memory? :)
A very good reason to bring 64bit back into focus, don't you think? Norbert
I know, it is easier to find cheap solution, without spending time implementing own data caching scheme, but you just delaying inevitable.
Delaying is crucial. If I wait long enough, PC's will have enough memory.
Stephan
-- Best regards, Igor Stasenko.
On 25 Nov 2011, at 10:31, Norbert Hartl wrote:
A very good reason to bring 64bit back into focus, don't you think?
I too think that big images are important: as other have said, RAM is cheap so there is no reason not to use it. One should not get sloppy and waiste memory, so good engineering and algorithms are very important, but if someone wants to take huge amounts of data in RAM for whatever reason, that should be possible. Being able to use a lot of memory amplifies the power of Smalltalk. At least a 32-bit VM should offer a usable memory space for the image as close as possible to its theoretical maximum. And indeed, next is a 64-bit VM/image combination. Sven
participants (4)
-
Igor Stasenko -
Norbert Hartl -
Stephan Eggermont -
Sven Van Caekenberghe