Am 01.07.2013 um 14:47 schrieb Stephan Eggermont <stephan@stack.nl>:

Esteban wrote:

...

512 GB is "reasonable" maximum for a single threaded app? Certainly it is "reasonable" high! :-)

Engineering is about trade-offs. 10K euro's: ram, or engineering time.

...

Altough I think 4GB might be small, for me it is better to think about scaling out, instead of up.

Latency & bandwidth are no solved problems, AFAIK. As long as we connect images through HTTP, the amount of copying involved is going to seriously affect scalability. There is a reason Gemstone uses shared pages

GemStone uses a shared page cache in order to avoid disk read/writes. It is only a cache and coordinator for everything read/write that needs to have access to disks. Well, it coordinates transactions as well to have a consistent set of pages in a concurrent environment. Each client in the shared memory situation still copies all of the data needed into its private memory section. Apart from that I think the use case dictates the best approach. For a web application or any application that has low memory profile per process but highly concurrent usage horizontal scaling is much easier. A few bucks put into can give some good results. In your use case it is completely different because you operate with one application on a huge graph. In this scenario the copying would be my smallest problem. I first would like to find a reliable way to manage a graph over multiple images :) In this case scaling vertical is most of the time the only way to go. Norbert