[Pharo-project] experience with large images?
Hi! Apparently, without configuring anything, the Pharo image cannot go over 500 Mb. Can this limit be moved up, to, let's say, 4Gb? Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
It depends on the OS. The 500mb limit should be for Windows. Unix and Mac should be more (kind of 2gb). Search in the history a thread between Tudor and Igor about a windows VM with more than 500mb for Moose. cheers On Wed, Nov 23, 2011 at 6:45 PM, Alexandre Bergel <alexandre.bergel@me.com>wrote:
Hi!
Apparently, without configuring anything, the Pharo image cannot go over 500 Mb. Can this limit be moved up, to, let's say, 4Gb?
Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- Mariano http://marianopeck.wordpress.com
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically. On 24 November 2011 00:55, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
It depends on the OS. The 500mb limit should be for Windows. Unix and Mac should be more (kind of 2gb). Search in the history a thread between Tudor and Igor about a windows VM with more than 500mb for Moose.
cheers
On Wed, Nov 23, 2011 at 6:45 PM, Alexandre Bergel <alexandre.bergel@me.com> wrote:
Hi!
Apparently, without configuring anything, the Pharo image cannot go over 500 Mb. Can this limit be moved up, to, let's say, 4Gb?
Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel  http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko.
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB? On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
I guess Alex talks about a Moose image. With Moose we can easily get to quite large images with millions of objects. For Windows, we indeed had a problem with the VM that Igor fixed. At that time, the problem appeared in images of over 200MB that could not be reopened (could be built, but once closed could not be reopened). I do not know what the status is with 500 MB. But, I also guess that Alex refers to the default values for memory when running the VM. Is that correct Alex? Cheers, Doru On 24 Nov 2011, at 07:27, Francois Stephany wrote:
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB?
On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
-- www.tudorgirba.com "Speaking louder won't make the point worthier."
But, I also guess that Alex refers to the default values for memory when running the VM. Is that correct Alex?
Yes. Alexandre
On 24 Nov 2011, at 07:27, Francois Stephany wrote:
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB?
On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
-- www.tudorgirba.com
"Speaking louder won't make the point worthier."
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
I think you looking for solution in a wrong direction. Just ask yourself, how much of that data you need to keep in operative memory at single moment of time to efficiently compute results. If today you need to deal with >500Mb data sets, tomorrow you may need to deal with multigigabyte datasets, which can easily surpass the amount of operative memory your computer has. I know, it is easier to find cheap solution, without spending time implementing own data caching scheme, but you just delaying inevitable. With things like Fuel, i think it won't take too much effort to do it. On 24 November 2011 14:06, Alexandre Bergel <alexandre.bergel@me.com> wrote:
But, I also guess that Alex refers to the default values for memory when running the VM. Is that correct Alex?
Yes.
Alexandre
On 24 Nov 2011, at 07:27, Francois Stephany wrote:
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB?
On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
-- www.tudorgirba.com
"Speaking louder won't make the point worthier."
-- Best regards, Igor Stasenko.
As Stefan put it, the current system forces us to make early decisions, which are likely to be wrong on some point. Since there is no other option, we are thinking very hard on what do we really need. We are implementing ad-hoc caching as you suggested, but this clearly puts some strong constraints on what can be done. Alexandre On 24 Nov 2011, at 13:13, Igor Stasenko wrote:
I think you looking for solution in a wrong direction. Just ask yourself, how much of that data you need to keep in operative memory at single moment of time to efficiently compute results. If today you need to deal with >500Mb data sets, tomorrow you may need to deal with multigigabyte datasets, which can easily surpass the amount of operative memory your computer has.
I know, it is easier to find cheap solution, without spending time implementing own data caching scheme, but you just delaying inevitable.
With things like Fuel, i think it won't take too much effort to do it.
On 24 November 2011 14:06, Alexandre Bergel <alexandre.bergel@me.com> wrote:
But, I also guess that Alex refers to the default values for memory when running the VM. Is that correct Alex?
Yes.
Alexandre
On 24 Nov 2011, at 07:27, Francois Stephany wrote:
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB?
On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
-- www.tudorgirba.com
"Speaking louder won't make the point worthier."
-- Best regards, Igor Stasenko.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Much larger images can be supported. This may be of interest: <http://lists.squeakfoundation.org/pipermail/vm-dev/2010-November/005731.html> In practical use, the current garbage collector will probably be the limiting factor. Dave On Thu, Nov 24, 2011 at 01:29:26PM -0300, Alexandre Bergel wrote:
As Stefan put it, the current system forces us to make early decisions, which are likely to be wrong on some point.
Since there is no other option, we are thinking very hard on what do we really need. We are implementing ad-hoc caching as you suggested, but this clearly puts some strong constraints on what can be done.
Alexandre
On 24 Nov 2011, at 13:13, Igor Stasenko wrote:
I think you looking for solution in a wrong direction. Just ask yourself, how much of that data you need to keep in operative memory at single moment of time to efficiently compute results. If today you need to deal with >500Mb data sets, tomorrow you may need to deal with multigigabyte datasets, which can easily surpass the amount of operative memory your computer has.
I know, it is easier to find cheap solution, without spending time implementing own data caching scheme, but you just delaying inevitable.
With things like Fuel, i think it won't take too much effort to do it.
On 24 November 2011 14:06, Alexandre Bergel <alexandre.bergel@me.com> wrote:
But, I also guess that Alex refers to the default values for memory when running the VM. Is that correct Alex?
Yes.
Alexandre
On 24 Nov 2011, at 07:27, Francois Stephany wrote:
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB?
On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
-- www.tudorgirba.com
"Speaking louder won't make the point worthier."
-- Best regards, Igor Stasenko.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Having a reification in Moose of 100 versions of Mondrian for example :-) Just answering the question 'Which classes and methods of Mondrian have changed more than 10 times since the day Mondrian was born?' cannot be easily done without a lot of memory Alexandre On 24 Nov 2011, at 03:27, Francois Stephany wrote:
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB?
On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
On 24/11/11 05:05, Alexandre Bergel wrote:
Having a reification in Moose of 100 versions of Mondrian for example :-)
Just answering the question 'Which classes and methods of Mondrian have changed more than 10 times since the day Mondrian was born?' cannot be easily done without a lot of memory
Alexandre
Oh ok, I hadnt Moose in mind ;)
Alex two remarks: - what is the level of reification because we reify a lot and it can be discarded. - did you check the orion model because with orion you only represent the delta and this is a big difference. Stef On Nov 24, 2011, at 2:05 PM, Alexandre Bergel wrote:
Having a reification in Moose of 100 versions of Mondrian for example :-)
Just answering the question 'Which classes and methods of Mondrian have changed more than 10 times since the day Mondrian was born?' cannot be easily done without a lot of memory
Alexandre
On 24 Nov 2011, at 03:27, Francois Stephany wrote:
I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is. Alex, what is your use case (in practice!) for more than 500MB?
On 23/11/11 18:25, Igor Stasenko wrote:
It is problematic, and requires different memory management than we currently have. I think if you need really big data sets, then use gemstone, which is developed to deal with that specifically.
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
participants (7)
-
Alexandre Bergel -
David T. Lewis -
Francois Stephany -
Igor Stasenko -
Mariano Martinez Peck -
Stéphane Ducasse -
Tudor Girba