Am 25.11.2011 um 00:45 schrieb Igor Stasenko:
On 25 November 2011 01:28, Stephan Eggermont <stephan@stack.nl> wrote:
Igor Stasenko wrote:
I think you looking for solution in a wrong direction. Just ask yourself, how much of that data you need to keep in operative memory at single moment of time to efficiently compute results.
All of it. And then of course the annotations and resulting output model. I know practically nothing about the data at first (about 700 files, about 700 MB). When trying to understand the data, it is crucial that I can test hypotheses fast. Efficiency is about my time, not computer time.
If today you need to deal with >500Mb data sets, tomorrow you may need to deal with multigigabyte datasets, which can easily surpass the amount of operative memory your computer has.
16 GB DIMMS are about 250 Euro. For a commercial project it is easy to justify half a terabyte of ram.
ok. then how about investing a bit to make sure VMs could conserve that much memory? :)
A very good reason to bring 64bit back into focus, don't you think? Norbert
I know, it is easier to find cheap solution, without spending time implementing own data caching scheme, but you just delaying inevitable.
Delaying is crucial. If I wait long enough, PC's will have enough memory.
Stephan
-- Best regards, Igor Stasenko.