[Pharo-project] Fwd: [Vm-dev] Re: out of memory - cog on windows
---------- Forwarded message ---------- From: Andreas Raab <andreas.raab@gmx.de> Date: Fri, Apr 22, 2011 at 3:01 PM Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows To: vm-dev@lists.squeakfoundation.org FWIW, the reason for the 512MB limit has to do with Windows memory layout. When you're running applications that load dynamic libraries (directly or indirectly such as when using ODBC which loads further DLLs) Windows needs (sometimes a lot of) address space to map these DLLs in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY from the OS which will consequently not be available to map DLLs into. We have found that depending on the libraries in use and depending on the overall system utilization, loading DLLs would fail seemingly "at random" which, after further investigation, we were able to track to reserving too much address space upfront. We were able to show experimentally, that changing the limit from 1GB to 512MB would on some machines make all the difference. [*] This is true in particular for libraries that create more threads as the default Windows policy is to create threads with the stack size of the application executable. Thus a 1MB default stack in the application will by default create all further threads to be allocated with a 1MB stack size. Of course all of this is subject to various other conditions. In the end we concluded that 512MB is a reasonable size for most apps and with 512MB we've never seen these random failures. You can increase the limit by recompiling the VM, but if you ship your app in diverse environments you should be aware of the potential issues. Cheers, - Andreas On 4/21/2011 22:44, Alain_Rastoul wrote: Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps of 128M until it succeeds. so I suppose it could be possible to allocate 2Gb and see how it runs on a 1Gb system and if this is not too much stress for the system (thinking of the pagefile) ? Tudor is your vm a cog or non cog vm ? "Mariano Martinez Peck" <marianopeck@gmail.com> a écrit dans le message de news: BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com... ------------------------------ On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba <tudor.girba@gmail.com> wrote:
Hi,
Should I to understand that the only way to enable more memory is to recompile the VM? Does that mean that there is no way to pass this information as a parameter like we can on Mac?
As far as I know you can pass a parameter, but even so, it won't be able to allocate more than 512MB. I can compile the VM for you with this change in 5 minutes. But I am not sure that such simple code would make it work. I think such limit is there because of something. Otherwise, it sounds stupid imposing a limit just because. The problem is that I cannot recompile the VM because I have no access to a
Windows machine. Is there one available that provides more memory?
I don't think so, but start cc'ing the VM mailing list. You'd probably receive more help. Cheers Mariano
Cheers, Doru
On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
Hi Tudor,
There is a constant in sqWin32Alloc.h (platforms\win32\vm) : #define MAX_VIRTUAL_MEMORY 512*1024*1024 you can change it to whatever you want and rebuild the vm, for exzmple give all the available memory less 256 M .
HTH
Regards Alain
"Tudor Girba" <tudor.girba@gmail.com> a écrit dans le message de news: 03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com ... Hi,
We have no specific startUp: methods in Moose. In any case, the issue with opening the image does not seem to be related to startUp:.
Is it really true that the Cog VM is limited to 512MB of memory?
Cheers, Doru
On 21 Apr 2011, at 14:27, Luc Fabresse wrote:
Hi Doru,
2011/4/21 Tudor Girba <tudor.girba@gmail.com> Hi,
On Apr 21, 2011, at 14:06, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba <tudor.girba@gmail.com> wrote:
Hi again,
I did not say what the problem was :). The problem was that when opening the image on Windows, he got a Space is low message and the image was not usable (see attachment).
That's weird. Does moose have something on the startup list?
something
that can be bothering there?
Not that I know of. Is there a way to check this?
Classes should be registered using Smalltlak addToStartUpList: aClass Then aClass class>>#startUp: is executed at startup. So implementors of #startUp: on Moose classes should give you the answer.
#Luc
Actually, using exactly the same process some days ago he produced an image of 190MB. This one works just fine on Windows. The only difference between the two is the size of the loaded data.
It would be really bad news if the Windows vm would be so severely limited in terms of memory.
Cheers, Doru
On Mac it worked just fine.
Cheers, Doru
On 21 Apr 2011, at 12:52, Tudor Girba wrote:
Hi,
I received a question from someone running a 200MB image on Windows using Cog 2361.
If I open the image on Mac, it works just fine. Unfortunately, I do not have a Windows machine around, and I cannot test but I believe it should be solvable by increasing the allocated memory.
On Mac, I would run it with: ./Croquet -memory 1500m
Can anyone help me with the right incantation for Windows?
Cheers, Doru
-- www.tudorgirba.com
"What we can governs what we wish."
<Space is low.png>
-- www.tudorgirba.com
"Yesterday is a fact. Tomorrow is a possibility. Today is a challenge."
-- Mariano http://marianopeck.wordpress.com
-- www.tudorgirba.com
"Beauty is where we see it."
-- www.tudorgirba.com
"If you interrupt the barber while he is cutting your hair, you will end up with a messy haircut."
-- Mariano http://marianopeck.wordpress.com -- Mariano http://marianopeck.wordpress.com
Does that mean that, even if your app uses 128mb of RAM, the VM will allocate 512mb of memory regardless? It seems a bit strange that just 1gb worth of allocations would cause problems. What about the load address of the VM and the way the memory chunk is allocated? Has the 32 bit VM been compiled to indicate the app is 32 bit address aware (if not, address space is limited to 2gb)? On 4/22/11 14:44 , Mariano Martinez Peck wrote:
---------- Forwarded message ---------- From: *Andreas Raab* <andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> Date: Fri, Apr 22, 2011 at 3:01 PM Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows To: vm-dev@lists.squeakfoundation.org <mailto:vm-dev@lists.squeakfoundation.org>
FWIW, the reason for the 512MB limit has to do with Windows memory layout. When you're running applications that load dynamic libraries (directly or indirectly such as when using ODBC which loads further DLLs) Windows needs (sometimes a lot of) address space to map these DLLs in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY from the OS which will consequently not be available to map DLLs into. We have found that depending on the libraries in use and depending on the overall system utilization, loading DLLs would fail seemingly "at random" which, after further investigation, we were able to track to reserving too much address space upfront. We were able to show experimentally, that changing the limit from 1GB to 512MB would on some machines make all the difference.
[*] This is true in particular for libraries that create more threads as the default Windows policy is to create threads with the stack size of the application executable. Thus a 1MB default stack in the application will by default create all further threads to be allocated with a 1MB stack size. Of course all of this is subject to various other conditions.
In the end we concluded that 512MB is a reasonable size for most apps and with 512MB we've never seen these random failures. You can increase the limit by recompiling the VM, but if you ship your app in diverse environments you should be aware of the potential issues.
Cheers, - Andreas
On 4/21/2011 22:44, Alain_Rastoul wrote:
Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps of 128M until it succeeds. so I suppose it could be possible to allocate 2Gb and see how it runs on a 1Gb system and if this is not too much stress for the system (thinking of the pagefile) ? Tudor is your vm a cog or non cog vm ?
"Mariano Martinez Peck" <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> a écrit dans le message de news: BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com <mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com>...
------------------------------------------------------------------------
On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote:
Hi,
Should I to understand that the only way to enable more memory is to recompile the VM? Does that mean that there is no way to pass this information as a parameter like we can on Mac?
As far as I know you can pass a parameter, but even so, it won't be able to allocate more than 512MB. I can compile the VM for you with this change in 5 minutes. But I am not sure that such simple code would make it work. I think such limit is there because of something. Otherwise, it sounds stupid imposing a limit just because.
The problem is that I cannot recompile the VM because I have no access to a Windows machine. Is there one available that provides more memory?
I don't think so, but start cc'ing the VM mailing list. You'd probably receive more help.
Cheers
Mariano
Cheers, Doru
On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
> Hi Tudor, > > There is a constant in sqWin32Alloc.h (platforms\win32\vm) : > #define MAX_VIRTUAL_MEMORY 512*1024*1024 > you can change it to whatever you want and rebuild the vm, > for exzmple give all the available memory less 256 M . > > HTH > > Regards > Alain > > "Tudor Girba" <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> a écrit > dans le message de news: 03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com <mailto:03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com>... > Hi, > > We have no specific startUp: methods in Moose. In any case, the issue with > opening the image does not seem to be related to startUp:. > > Is it really true that the Cog VM is limited to 512MB of memory? > > Cheers, > Doru > > > On 21 Apr 2011, at 14:27, Luc Fabresse wrote: > >> Hi Doru, >> >> 2011/4/21 Tudor Girba >> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> >> Hi, >> >> >> >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck >> <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote: >> >>> >>> >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba >>> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote: >>>> Hi again, >>>> >>>> I did not say what the problem was :). The problem was that when >>>> opening the image on Windows, he got a Space is low message and the >>>> image was not usable (see attachment). >>> >>> That's weird. Does moose have something on the startup list? something >>> that can be bothering there? >> >> Not that I know of. Is there a way to check this? >> >> Classes should be registered using Smalltlak addToStartUpList: aClass >> Then aClass class>>#startUp: is executed at startup. >> So implementors of #startUp: on Moose classes should give you the answer. >> >> #Luc >> >> >> Actually, using exactly the same process some days ago he produced an >> image of 190MB. This one works just fine on Windows. The only difference >> between the two is the size of the loaded data. >> >> It would be really bad news if the Windows vm would be so severely limited >> in terms of memory. >> >> Cheers, >> Doru >> >> >>> >>> On Mac it worked just fine. >>>> >>>> Cheers, >>>> Doru >>>> >>>> >>> >>>> >>>> >>>> >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I received a question from someone running a 200MB image on Windows >>>>> using Cog 2361. >>>>> >>>>> If I open the image on Mac, it works just fine. Unfortunately, I do >>>>> not have a Windows machine around, and I cannot test but I believe it >>>>> should be solvable by increasing the allocated memory. >>>>> >>>>> On Mac, I would run it with: ./Croquet -memory 1500m >>>>> >>>>> Can anyone help me with the right incantation for Windows? >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> >>>>> -- >>>>> www.tudorgirba.com <http://www.tudorgirba.com> >>>>> >>>>> "What we can governs what we wish." >>>>> >>>>> >>>>> >>>>> <Space is low.png> >>>> >>>> -- >>>> www.tudorgirba.com <http://www.tudorgirba.com> >>>> >>>> "Yesterday is a fact. >>>> Tomorrow is a possibility. >>>> Today is a challenge." >>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> Mariano >>> http://marianopeck.wordpress.com >>> >> > > -- > www.tudorgirba.com <http://www.tudorgirba.com> > > "Beauty is where we see it." > > > > > > > >
-- www.tudorgirba.com <http://www.tudorgirba.com>
"If you interrupt the barber while he is cutting your hair, you will end up with a messy haircut."
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
The code for memory allocation is in sqWin32Alloc.c There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm (512MB) maxReserved = MAX_VIRTUAL_MEMORY; do { pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS); if(!pageBase) { if(maxReserved == nowReserved) break; /* make it smaller in steps of 128MB */ maxReserved -= 128*1024*1024; if(maxReserved < nowReserved) maxReserved = nowReserved; } } while(!pageBase); I don't know gcc compilation flag for address space > 2G for 32b apps on 64 bits ? ((MS) /LARGEADDRESSEPACE gcc equivalent ?) Do you know it ? "Andres Valloud" <avalloud@smalltalk.comcastbiz.net> a écrit dans le message de news: 4DB247E5.7020404@smalltalk.comcastbiz.net...
Does that mean that, even if your app uses 128mb of RAM, the VM will allocate 512mb of memory regardless? It seems a bit strange that just 1gb worth of allocations would cause problems. What about the load address of the VM and the way the memory chunk is allocated? Has the 32 bit VM been compiled to indicate the app is 32 bit address aware (if not, address space is limited to 2gb)?
On 4/22/11 14:44 , Mariano Martinez Peck wrote:
---------- Forwarded message ---------- From: *Andreas Raab* <andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> Date: Fri, Apr 22, 2011 at 3:01 PM Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows To: vm-dev@lists.squeakfoundation.org <mailto:vm-dev@lists.squeakfoundation.org>
FWIW, the reason for the 512MB limit has to do with Windows memory layout. When you're running applications that load dynamic libraries (directly or indirectly such as when using ODBC which loads further DLLs) Windows needs (sometimes a lot of) address space to map these DLLs in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY from the OS which will consequently not be available to map DLLs into. We have found that depending on the libraries in use and depending on the overall system utilization, loading DLLs would fail seemingly "at random" which, after further investigation, we were able to track to reserving too much address space upfront. We were able to show experimentally, that changing the limit from 1GB to 512MB would on some machines make all the difference.
[*] This is true in particular for libraries that create more threads as the default Windows policy is to create threads with the stack size of the application executable. Thus a 1MB default stack in the application will by default create all further threads to be allocated with a 1MB stack size. Of course all of this is subject to various other conditions.
In the end we concluded that 512MB is a reasonable size for most apps and with 512MB we've never seen these random failures. You can increase the limit by recompiling the VM, but if you ship your app in diverse environments you should be aware of the potential issues.
Cheers, - Andreas
On 4/21/2011 22:44, Alain_Rastoul wrote:
Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps of 128M until it succeeds. so I suppose it could be possible to allocate 2Gb and see how it runs on a 1Gb system and if this is not too much stress for the system (thinking of the pagefile) ? Tudor is your vm a cog or non cog vm ?
"Mariano Martinez Peck" <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> a écrit dans le message de news:
BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com
<mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com>...
------------------------------------------------------------------------
On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote:
Hi,
Should I to understand that the only way to enable more memory is to recompile the VM? Does that mean that there is no way to pass this information as a parameter like we can on Mac?
As far as I know you can pass a parameter, but even so, it won't be able to allocate more than 512MB. I can compile the VM for you with this change in 5 minutes. But I am not sure that such simple code would make it work. I think such limit is there because of something. Otherwise, it sounds stupid imposing a limit just because.
The problem is that I cannot recompile the VM because I have no access to a Windows machine. Is there one available that provides more memory?
I don't think so, but start cc'ing the VM mailing list. You'd probably receive more help.
Cheers
Mariano
Cheers, Doru
On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
> Hi Tudor, > > There is a constant in sqWin32Alloc.h (platforms\win32\vm) : > #define MAX_VIRTUAL_MEMORY 512*1024*1024 > you can change it to whatever you want and rebuild the vm, > for exzmple give all the available memory less 256 M . > > HTH > > Regards > Alain > > "Tudor Girba" <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> a écrit > dans le message de news:
03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com
<mailto:03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com>... > Hi, > > We have no specific startUp: methods in Moose. In any case, the issue with > opening the image does not seem to be related to startUp:. > > Is it really true that the Cog VM is limited to 512MB of memory? > > Cheers, > Doru > > > On 21 Apr 2011, at 14:27, Luc Fabresse wrote: > >> Hi Doru, >> >> 2011/4/21 Tudor Girba >> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> >> Hi, >> >> >> >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck >> <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote: >> >>> >>> >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba >>> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote: >>>> Hi again, >>>> >>>> I did not say what the problem was :). The problem was that when >>>> opening the image on Windows, he got a Space is low message and the >>>> image was not usable (see attachment). >>> >>> That's weird. Does moose have something on the startup list? something >>> that can be bothering there? >> >> Not that I know of. Is there a way to check this? >> >> Classes should be registered using Smalltlak addToStartUpList: aClass >> Then aClass class>>#startUp: is executed at startup. >> So implementors of #startUp: on Moose classes should give you the answer. >> >> #Luc >> >> >> Actually, using exactly the same process some days ago he produced an >> image of 190MB. This one works just fine on Windows. The only difference >> between the two is the size of the loaded data. >> >> It would be really bad news if the Windows vm would be so severely limited >> in terms of memory. >> >> Cheers, >> Doru >> >> >>> >>> On Mac it worked just fine. >>>> >>>> Cheers, >>>> Doru >>>> >>>> >>> >>>> >>>> >>>> >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I received a question from someone running a 200MB image on Windows >>>>> using Cog 2361. >>>>> >>>>> If I open the image on Mac, it works just fine. Unfortunately, I do >>>>> not have a Windows machine around, and I cannot test but I believe it >>>>> should be solvable by increasing the allocated memory. >>>>> >>>>> On Mac, I would run it with: ./Croquet -memory 1500m >>>>> >>>>> Can anyone help me with the right incantation for Windows? >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> >>>>> -- >>>>> www.tudorgirba.com <http://www.tudorgirba.com> >>>>> >>>>> "What we can governs what we wish." >>>>> >>>>> >>>>> >>>>> <Space is low.png> >>>> >>>> -- >>>> www.tudorgirba.com <http://www.tudorgirba.com> >>>> >>>> "Yesterday is a fact. >>>> Tomorrow is a possibility. >>>> Today is a challenge." >>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> Mariano >>> http://marianopeck.wordpress.com >>> >> > > -- > www.tudorgirba.com <http://www.tudorgirba.com> > > "Beauty is where we see it." > > > > > > > >
-- www.tudorgirba.com <http://www.tudorgirba.com>
"If you interrupt the barber while he is cutting your hair, you will end up with a messy haircut."
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 23 April 2011 13:19, Alain_Rastoul <alr.dev@free.fr> wrote:
The code for memory allocation is in sqWin32Alloc.c There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm (512MB) Â maxReserved = MAX_VIRTUAL_MEMORY; Â do { Â Â pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS); Â Â if(!pageBase) { Â Â Â if(maxReserved == nowReserved) break; Â Â Â /* make it smaller in steps of 128MB */ Â Â Â maxReserved -= 128*1024*1024; Â Â Â if(maxReserved < nowReserved) maxReserved = nowReserved; Â Â } Â } while(!pageBase);
I don't know gcc compilation flag for address space > 2G Â for 32b apps on 64 bits ? ((MS) /LARGEADDRESSEPACE gcc equivalent ?) Do you know it ?
No idea :)
"Andres Valloud" <avalloud@smalltalk.comcastbiz.net> a écrit dans le message de news: 4DB247E5.7020404@smalltalk.comcastbiz.net...
Does that mean that, even if your app uses 128mb of RAM, the VM will allocate 512mb of memory regardless? Â It seems a bit strange that just 1gb worth of allocations would cause problems. Â What about the load address of the VM and the way the memory chunk is allocated? Â Has the 32 bit VM been compiled to indicate the app is 32 bit address aware (if not, address space is limited to 2gb)?
On 4/22/11 14:44 , Mariano Martinez Peck wrote:
---------- Forwarded message ---------- From: *Andreas Raab* <andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> Date: Fri, Apr 22, 2011 at 3:01 PM Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows To: vm-dev@lists.squeakfoundation.org <mailto:vm-dev@lists.squeakfoundation.org>
FWIW, the reason for the 512MB limit has to do with Windows memory layout. When you're running applications that load dynamic libraries (directly or indirectly such as when using ODBC which loads further DLLs) Windows needs (sometimes a lot of) address space to map these DLLs in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY from the OS which will consequently not be available to map DLLs into. We have found that depending on the libraries in use and depending on the overall system utilization, loading DLLs would fail seemingly "at random" which, after further investigation, we were able to track to reserving too much address space upfront. We were able to show experimentally, that changing the limit from 1GB to 512MB would on some machines make all the difference.
[*] This is true in particular for libraries that create more threads as the default Windows policy is to create threads with the stack size of the application executable. Thus a 1MB default stack in the application will by default create all further threads to be allocated with a 1MB stack size. Of course all of this is subject to various other conditions.
In the end we concluded that 512MB is a reasonable size for most apps and with 512MB we've never seen these random failures. You can increase the limit by recompiling the VM, but if you ship your app in diverse environments you should be aware of the potential issues.
Cheers, Â Â - Andreas
On 4/21/2011 22:44, Alain_Rastoul wrote:
Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps of 128M until it succeeds. so I suppose it could be possible to allocate 2Gb and see how it runs on a 1Gb system and if this is not too much stress for the system (thinking of the pagefile) ? Tudor is your vm a cog or non cog vm ?
  "Mariano Martinez Peck" <marianopeck@gmail.com   <mailto:marianopeck@gmail.com>> a écrit dans le message de news:
BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com
<mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com>...
  ------------------------------------------------------------------------
  On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba   <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote:
    Hi,
    Should I to understand that the only way to enable more memory     is to recompile the VM? Does that mean that there is no way to     pass this information as a parameter like we can on Mac?
  As far as I know you can pass a parameter, but even so, it won't   be able to allocate more than 512MB.   I can compile the VM for you with this change in 5 minutes. But I   am not sure that such simple code would make it work. I think such   limit is there because of something. Otherwise, it sounds stupid   imposing a limit just because.
    The problem is that I cannot recompile the VM because I have     no access to a Windows machine. Is there one available that     provides more memory?
  I don't think so, but start cc'ing the VM mailing list. You'd   probably receive more help.
  Cheers
  Mariano
    Cheers,     Doru
    On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
    > Hi Tudor,     >     > There is a constant in sqWin32Alloc.h (platforms\win32\vm) :     > #define MAX_VIRTUAL_MEMORY 512*1024*1024     > you can change it to whatever you want and rebuild the vm,     > for exzmple give all the available memory less 256 M .     >     > HTH     >     > Regards     > Alain     >     > "Tudor Girba" <tudor.girba@gmail.com     <mailto:tudor.girba@gmail.com>> a écrit     > dans le message de news:
03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com
<mailto:03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com>...     > Hi,     >     > We have no specific startUp: methods in Moose. In any case,     the issue with     > opening the image does not seem to be related to startUp:.     >     > Is it really true that the Cog VM is limited to 512MB of memory?     >     > Cheers,     > Doru     >     >     > On 21 Apr 2011, at 14:27, Luc Fabresse wrote:     >     >> Hi Doru,     >>     >> 2011/4/21 Tudor Girba     >> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>>     >> Hi,     >>     >>     >>     >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck     >> <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote:     >>     >>>     >>>     >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba     >>> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote:     >>>> Hi again,     >>>>     >>>> I did not say what the problem was :). The problem was     that when     >>>> opening the image on Windows, he got a Space is low     message and the     >>>> image was not usable (see attachment).     >>>     >>> That's weird. Does moose have something on the startup     list?  something     >>> that can be bothering there?     >>     >> Not that I know of. Is there a way to check this?     >>     >> Classes should be registered using Smalltlak     addToStartUpList: aClass     >> Then aClass class>>#startUp: is executed at startup.     >> So implementors of #startUp: on Moose classes should give     you the answer.     >>     >> #Luc     >>     >>     >> Actually, using exactly the same process some days ago he     produced an     >> image of 190MB. This one works just fine on Windows. The     only difference     >> between the two is the size of the loaded data.     >>     >> It would be really bad news if the Windows vm would be so     severely limited     >> in terms of memory.     >>     >> Cheers,     >> Doru     >>     >>     >>>     >>> On Mac it worked just fine.     >>>>     >>>> Cheers,     >>>> Doru     >>>>     >>>>     >>>     >>>>     >>>>     >>>>     >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote:     >>>>     >>>>> Hi,     >>>>>     >>>>> I received a question from someone running a 200MB image     on Windows     >>>>> using Cog 2361.     >>>>>     >>>>> If I open the image on Mac, it works just fine.     Unfortunately, I do     >>>>> not have a Windows machine around, and I cannot test but     I believe it     >>>>> should be solvable by increasing the allocated memory.     >>>>>     >>>>> On Mac, I would run it with: ./Croquet -memory 1500m     >>>>>     >>>>> Can anyone help me with the right incantation for Windows?     >>>>>     >>>>> Cheers,     >>>>> Doru     >>>>>     >>>>>     >>>>> --     >>>>> www.tudorgirba.com <http://www.tudorgirba.com>     >>>>>     >>>>> "What we can governs what we wish."     >>>>>     >>>>>     >>>>>     >>>>> <Space is low.png>     >>>>     >>>> --     >>>> www.tudorgirba.com <http://www.tudorgirba.com>     >>>>     >>>> "Yesterday is a fact.     >>>> Tomorrow is a possibility.     >>>> Today is a challenge."     >>>>     >>>>     >>>>     >>>     >>>     >>>     >>>     >>> --     >>> Mariano     >>> http://marianopeck.wordpress.com     >>>     >>     >     > --     > www.tudorgirba.com <http://www.tudorgirba.com>     >     > "Beauty is where we see it."     >     >     >     >     >     >     >     >
    --     www.tudorgirba.com <http://www.tudorgirba.com>
    "If you interrupt the barber while he is cutting your hair,     you will end up with a messy haircut."
  --   Mariano   http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko AKA sig.
Some seemingly useful items to start tracing through Google appear here... http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/b1548cd5e7c... On 4/23/11 6:17 , Igor Stasenko wrote:
On 23 April 2011 13:19, Alain_Rastoul<alr.dev@free.fr> wrote:
The code for memory allocation is in sqWin32Alloc.c There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm (512MB) maxReserved = MAX_VIRTUAL_MEMORY; do { pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS); if(!pageBase) { if(maxReserved == nowReserved) break; /* make it smaller in steps of 128MB */ maxReserved -= 128*1024*1024; if(maxReserved< nowReserved) maxReserved = nowReserved; } } while(!pageBase);
I don't know gcc compilation flag for address space> 2G for 32b apps on 64 bits ? ((MS) /LARGEADDRESSEPACE gcc equivalent ?) Do you know it ?
No idea :)
"Andres Valloud" <avalloud@smalltalk.comcastbiz.net> a écrit dans le message de news: 4DB247E5.7020404@smalltalk.comcastbiz.net...
Does that mean that, even if your app uses 128mb of RAM, the VM will allocate 512mb of memory regardless? It seems a bit strange that just 1gb worth of allocations would cause problems. What about the load address of the VM and the way the memory chunk is allocated? Has the 32 bit VM been compiled to indicate the app is 32 bit address aware (if not, address space is limited to 2gb)?
On 4/22/11 14:44 , Mariano Martinez Peck wrote:
---------- Forwarded message ---------- From: *Andreas Raab*<andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> Date: Fri, Apr 22, 2011 at 3:01 PM Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows To: vm-dev@lists.squeakfoundation.org <mailto:vm-dev@lists.squeakfoundation.org>
FWIW, the reason for the 512MB limit has to do with Windows memory layout. When you're running applications that load dynamic libraries (directly or indirectly such as when using ODBC which loads further DLLs) Windows needs (sometimes a lot of) address space to map these DLLs in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY from the OS which will consequently not be available to map DLLs into. We have found that depending on the libraries in use and depending on the overall system utilization, loading DLLs would fail seemingly "at random" which, after further investigation, we were able to track to reserving too much address space upfront. We were able to show experimentally, that changing the limit from 1GB to 512MB would on some machines make all the difference.
[*] This is true in particular for libraries that create more threads as the default Windows policy is to create threads with the stack size of the application executable. Thus a 1MB default stack in the application will by default create all further threads to be allocated with a 1MB stack size. Of course all of this is subject to various other conditions.
In the end we concluded that 512MB is a reasonable size for most apps and with 512MB we've never seen these random failures. You can increase the limit by recompiling the VM, but if you ship your app in diverse environments you should be aware of the potential issues.
Cheers, - Andreas
On 4/21/2011 22:44, Alain_Rastoul wrote:
Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps of 128M until it succeeds. so I suppose it could be possible to allocate 2Gb and see how it runs on a 1Gb system and if this is not too much stress for the system (thinking of the pagefile) ? Tudor is your vm a cog or non cog vm ?
"Mariano Martinez Peck" <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> a écrit dans le message de news:
BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com
<mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com>...
------------------------------------------------------------------------
On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote:
Hi,
Should I to understand that the only way to enable more memory is to recompile the VM? Does that mean that there is no way to pass this information as a parameter like we can on Mac?
As far as I know you can pass a parameter, but even so, it won't be able to allocate more than 512MB. I can compile the VM for you with this change in 5 minutes. But I am not sure that such simple code would make it work. I think such limit is there because of something. Otherwise, it sounds stupid imposing a limit just because.
The problem is that I cannot recompile the VM because I have no access to a Windows machine. Is there one available that provides more memory?
I don't think so, but start cc'ing the VM mailing list. You'd probably receive more help.
Cheers
Mariano
Cheers, Doru
On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
> Hi Tudor, > > There is a constant in sqWin32Alloc.h (platforms\win32\vm) : > #define MAX_VIRTUAL_MEMORY 512*1024*1024 > you can change it to whatever you want and rebuild the vm, > for exzmple give all the available memory less 256 M . > > HTH > > Regards > Alain > > "Tudor Girba" <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> a écrit > dans le message de news:
03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com
<mailto:03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com>... > Hi, > > We have no specific startUp: methods in Moose. In any case, the issue with > opening the image does not seem to be related to startUp:. > > Is it really true that the Cog VM is limited to 512MB of memory? > > Cheers, > Doru > > > On 21 Apr 2011, at 14:27, Luc Fabresse wrote: > >> Hi Doru, >> >> 2011/4/21 Tudor Girba >> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> >> Hi, >> >> >> >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck >> <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote: >> >>> >>> >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba >>> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote: >>>> Hi again, >>>> >>>> I did not say what the problem was :). The problem was that when >>>> opening the image on Windows, he got a Space is low message and the >>>> image was not usable (see attachment). >>> >>> That's weird. Does moose have something on the startup list? something >>> that can be bothering there? >> >> Not that I know of. Is there a way to check this? >> >> Classes should be registered using Smalltlak addToStartUpList: aClass >> Then aClass class>>#startUp: is executed at startup. >> So implementors of #startUp: on Moose classes should give you the answer. >> >> #Luc >> >> >> Actually, using exactly the same process some days ago he produced an >> image of 190MB. This one works just fine on Windows. The only difference >> between the two is the size of the loaded data. >> >> It would be really bad news if the Windows vm would be so severely limited >> in terms of memory. >> >> Cheers, >> Doru >> >> >>> >>> On Mac it worked just fine. >>>> >>>> Cheers, >>>> Doru >>>> >>>> >>> >>>> >>>> >>>> >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I received a question from someone running a 200MB image on Windows >>>>> using Cog 2361. >>>>> >>>>> If I open the image on Mac, it works just fine. Unfortunately, I do >>>>> not have a Windows machine around, and I cannot test but I believe it >>>>> should be solvable by increasing the allocated memory. >>>>> >>>>> On Mac, I would run it with: ./Croquet -memory 1500m >>>>> >>>>> Can anyone help me with the right incantation for Windows? >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> >>>>> -- >>>>> www.tudorgirba.com<http://www.tudorgirba.com> >>>>> >>>>> "What we can governs what we wish." >>>>> >>>>> >>>>> >>>>> <Space is low.png> >>>> >>>> -- >>>> www.tudorgirba.com<http://www.tudorgirba.com> >>>> >>>> "Yesterday is a fact. >>>> Tomorrow is a possibility. >>>> Today is a challenge." >>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> Mariano >>> http://marianopeck.wordpress.com >>> >> > > -- > www.tudorgirba.com<http://www.tudorgirba.com> > > "Beauty is where we see it." > > > > > > > >
-- www.tudorgirba.com<http://www.tudorgirba.com>
"If you interrupt the barber while he is cutting your hair, you will end up with a messy haircut."
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko AKA sig.
I thought I'd mention... some supported Windows configurations require additional work to allow apps to use more than 2gb. So, even if you compile with (IIRC) /LARGEADDRESSAWARE, it may not always be enough. On 4/23/11 11:48 , Andres Valloud wrote:
Some seemingly useful items to start tracing through Google appear here...
http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/b1548cd5e7c...
On 4/23/11 6:17 , Igor Stasenko wrote:
On 23 April 2011 13:19, Alain_Rastoul<alr.dev@free.fr> wrote:
The code for memory allocation is in sqWin32Alloc.c There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm (512MB) maxReserved = MAX_VIRTUAL_MEMORY; do { pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS); if(!pageBase) { if(maxReserved == nowReserved) break; /* make it smaller in steps of 128MB */ maxReserved -= 128*1024*1024; if(maxReserved< nowReserved) maxReserved = nowReserved; } } while(!pageBase);
I don't know gcc compilation flag for address space> 2G for 32b apps on 64 bits ? ((MS) /LARGEADDRESSEPACE gcc equivalent ?) Do you know it ?
No idea :)
"Andres Valloud" <avalloud@smalltalk.comcastbiz.net> a écrit dans le message de news: 4DB247E5.7020404@smalltalk.comcastbiz.net...
Does that mean that, even if your app uses 128mb of RAM, the VM will allocate 512mb of memory regardless? It seems a bit strange that just 1gb worth of allocations would cause problems. What about the load address of the VM and the way the memory chunk is allocated? Has the 32 bit VM been compiled to indicate the app is 32 bit address aware (if not, address space is limited to 2gb)?
On 4/22/11 14:44 , Mariano Martinez Peck wrote:
---------- Forwarded message ---------- From: *Andreas Raab*<andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> Date: Fri, Apr 22, 2011 at 3:01 PM Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows To: vm-dev@lists.squeakfoundation.org <mailto:vm-dev@lists.squeakfoundation.org>
FWIW, the reason for the 512MB limit has to do with Windows memory layout. When you're running applications that load dynamic libraries (directly or indirectly such as when using ODBC which loads further DLLs) Windows needs (sometimes a lot of) address space to map these DLLs in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY from the OS which will consequently not be available to map DLLs into. We have found that depending on the libraries in use and depending on the overall system utilization, loading DLLs would fail seemingly "at random" which, after further investigation, we were able to track to reserving too much address space upfront. We were able to show experimentally, that changing the limit from 1GB to 512MB would on some machines make all the difference.
[*] This is true in particular for libraries that create more threads as the default Windows policy is to create threads with the stack size of the application executable. Thus a 1MB default stack in the application will by default create all further threads to be allocated with a 1MB stack size. Of course all of this is subject to various other conditions.
In the end we concluded that 512MB is a reasonable size for most apps and with 512MB we've never seen these random failures. You can increase the limit by recompiling the VM, but if you ship your app in diverse environments you should be aware of the potential issues.
Cheers, - Andreas
On 4/21/2011 22:44, Alain_Rastoul wrote:
Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps of 128M until it succeeds. so I suppose it could be possible to allocate 2Gb and see how it runs on a 1Gb system and if this is not too much stress for the system (thinking of the pagefile) ? Tudor is your vm a cog or non cog vm ?
"Mariano Martinez Peck" <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> a écrit dans le message de news:
BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com
<mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com>...
------------------------------------------------------------------------
On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote:
Hi,
Should I to understand that the only way to enable more memory is to recompile the VM? Does that mean that there is no way to pass this information as a parameter like we can on Mac?
As far as I know you can pass a parameter, but even so, it won't be able to allocate more than 512MB. I can compile the VM for you with this change in 5 minutes. But I am not sure that such simple code would make it work. I think such limit is there because of something. Otherwise, it sounds stupid imposing a limit just because.
The problem is that I cannot recompile the VM because I have no access to a Windows machine. Is there one available that provides more memory?
I don't think so, but start cc'ing the VM mailing list. You'd probably receive more help.
Cheers
Mariano
Cheers, Doru
On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
> Hi Tudor, > > There is a constant in sqWin32Alloc.h (platforms\win32\vm) : > #define MAX_VIRTUAL_MEMORY 512*1024*1024 > you can change it to whatever you want and rebuild the vm, > for exzmple give all the available memory less 256 M . > > HTH > > Regards > Alain > > "Tudor Girba" <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> a écrit > dans le message de news:
03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com
<mailto:03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com>... > Hi, > > We have no specific startUp: methods in Moose. In any case, the issue with > opening the image does not seem to be related to startUp:. > > Is it really true that the Cog VM is limited to 512MB of memory? > > Cheers, > Doru > > > On 21 Apr 2011, at 14:27, Luc Fabresse wrote: > >> Hi Doru, >> >> 2011/4/21 Tudor Girba >> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> >> Hi, >> >> >> >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck >> <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote: >> >>> >>> >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba >>> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote: >>>> Hi again, >>>> >>>> I did not say what the problem was :). The problem was that when >>>> opening the image on Windows, he got a Space is low message and the >>>> image was not usable (see attachment). >>> >>> That's weird. Does moose have something on the startup list? something >>> that can be bothering there? >> >> Not that I know of. Is there a way to check this? >> >> Classes should be registered using Smalltlak addToStartUpList: aClass >> Then aClass class>>#startUp: is executed at startup. >> So implementors of #startUp: on Moose classes should give you the answer. >> >> #Luc >> >> >> Actually, using exactly the same process some days ago he produced an >> image of 190MB. This one works just fine on Windows. The only difference >> between the two is the size of the loaded data. >> >> It would be really bad news if the Windows vm would be so severely limited >> in terms of memory. >> >> Cheers, >> Doru >> >> >>> >>> On Mac it worked just fine. >>>> >>>> Cheers, >>>> Doru >>>> >>>> >>> >>>> >>>> >>>> >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I received a question from someone running a 200MB image on Windows >>>>> using Cog 2361. >>>>> >>>>> If I open the image on Mac, it works just fine. Unfortunately, I do >>>>> not have a Windows machine around, and I cannot test but I believe it >>>>> should be solvable by increasing the allocated memory. >>>>> >>>>> On Mac, I would run it with: ./Croquet -memory 1500m >>>>> >>>>> Can anyone help me with the right incantation for Windows? >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> >>>>> -- >>>>> www.tudorgirba.com<http://www.tudorgirba.com> >>>>> >>>>> "What we can governs what we wish." >>>>> >>>>> >>>>> >>>>> <Space is low.png> >>>> >>>> -- >>>> www.tudorgirba.com<http://www.tudorgirba.com> >>>> >>>> "Yesterday is a fact. >>>> Tomorrow is a possibility. >>>> Today is a challenge." >>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> Mariano >>> http://marianopeck.wordpress.com >>> >> > > -- > www.tudorgirba.com<http://www.tudorgirba.com> > > "Beauty is where we see it." > > > > > > > >
-- www.tudorgirba.com<http://www.tudorgirba.com>
"If you interrupt the barber while he is cutting your hair, you will end up with a messy haircut."
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
-- Best regards, Igor Stasenko AKA sig.
On 23 April 2011 21:48, Andres Valloud <avalloud@smalltalk.comcastbiz.net> wrote:
Some seemingly useful items to start tracing through Google appear here...
http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/b1548cd5e7c...
so we can add '-Wl,--large-address-aware'. option to linker. -- Best regards, Igor Stasenko AKA sig.
You can determine the switches better than me... for us, it was /LARGEADDRESSAWARE, also a linker switch: http://msdn.microsoft.com/en-us/library/wz223b1z.aspx Now, keep in mind that expanding the address space capability of the app doesn't mean Windows will give it 4gb. Under some Windows configurations, that won't happen unless you change how Windows boots. Under some versions, 4gb is just not possible. See here: http://msdn.microsoft.com/en-us/library/aa366778%28v=VS.85%29.aspx On 4/23/11 15:40 , Igor Stasenko wrote:
On 23 April 2011 21:48, Andres Valloud <avalloud@smalltalk.comcastbiz.net> wrote:
Some seemingly useful items to start tracing through Google appear here...
http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/b1548cd5e7c...
so we can add '-Wl,--large-address-aware'. option to linker.
Interesting bit from the docs here: "Addresses near the 2-GB boundary are typically used by various system DLLs. Therefore, a 32-bit process cannot allocate more than 2 GB of contiguous memory, even if the entire 4-GB address space is available." http://msdn.microsoft.com/en-us/library/bb613473%28v=VS.85%29.aspx Maybe more fundamental changes are needed, such as the ability to change the image's memory footprint dynamically. On 4/23/11 17:34 , Andres Valloud wrote:
You can determine the switches better than me... for us, it was /LARGEADDRESSAWARE, also a linker switch:
http://msdn.microsoft.com/en-us/library/wz223b1z.aspx
Now, keep in mind that expanding the address space capability of the app doesn't mean Windows will give it 4gb. Under some Windows configurations, that won't happen unless you change how Windows boots. Under some versions, 4gb is just not possible. See here:
http://msdn.microsoft.com/en-us/library/aa366778%28v=VS.85%29.aspx
On 4/23/11 15:40 , Igor Stasenko wrote:
On 23 April 2011 21:48, Andres Valloud <avalloud@smalltalk.comcastbiz.net> wrote:
Some seemingly useful items to start tracing through Google appear here...
http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/b1548cd5e7c...
so we can add '-Wl,--large-address-aware'. option to linker.
Not off the top of my head, we use VisualStudio for VisualWorks. On 4/23/11 3:19 , Alain_Rastoul wrote:
The code for memory allocation is in sqWin32Alloc.c There is MAX_VIRTUAL_MEMORYMB reserved in virtual address space for each vm (512MB) maxReserved = MAX_VIRTUAL_MEMORY; do { pageBase = VirtualAlloc(NULL,maxReserved,MEM_RESERVE, PAGE_NOACCESS); if(!pageBase) { if(maxReserved == nowReserved) break; /* make it smaller in steps of 128MB */ maxReserved -= 128*1024*1024; if(maxReserved< nowReserved) maxReserved = nowReserved; } } while(!pageBase);
I don't know gcc compilation flag for address space> 2G for 32b apps on 64 bits ? ((MS) /LARGEADDRESSEPACE gcc equivalent ?) Do you know it ?
"Andres Valloud" <avalloud@smalltalk.comcastbiz.net> a écrit dans le message de news: 4DB247E5.7020404@smalltalk.comcastbiz.net...
Does that mean that, even if your app uses 128mb of RAM, the VM will allocate 512mb of memory regardless? It seems a bit strange that just 1gb worth of allocations would cause problems. What about the load address of the VM and the way the memory chunk is allocated? Has the 32 bit VM been compiled to indicate the app is 32 bit address aware (if not, address space is limited to 2gb)?
On 4/22/11 14:44 , Mariano Martinez Peck wrote:
---------- Forwarded message ---------- From: *Andreas Raab*<andreas.raab@gmx.de <mailto:andreas.raab@gmx.de>> Date: Fri, Apr 22, 2011 at 3:01 PM Subject: Re: [Vm-dev] Re: [Pharo-project] out of memory - cog on windows To: vm-dev@lists.squeakfoundation.org <mailto:vm-dev@lists.squeakfoundation.org>
FWIW, the reason for the 512MB limit has to do with Windows memory layout. When you're running applications that load dynamic libraries (directly or indirectly such as when using ODBC which loads further DLLs) Windows needs (sometimes a lot of) address space to map these DLLs in order to load them[*]. When the VM starts it grabs MAX_VIRTUAL_MEMORY from the OS which will consequently not be available to map DLLs into. We have found that depending on the libraries in use and depending on the overall system utilization, loading DLLs would fail seemingly "at random" which, after further investigation, we were able to track to reserving too much address space upfront. We were able to show experimentally, that changing the limit from 1GB to 512MB would on some machines make all the difference.
[*] This is true in particular for libraries that create more threads as the default Windows policy is to create threads with the stack size of the application executable. Thus a 1MB default stack in the application will by default create all further threads to be allocated with a 1MB stack size. Of course all of this is subject to various other conditions.
In the end we concluded that 512MB is a reasonable size for most apps and with 512MB we've never seen these random failures. You can increase the limit by recompiling the VM, but if you ship your app in diverse environments you should be aware of the potential issues.
Cheers, - Andreas
On 4/21/2011 22:44, Alain_Rastoul wrote:
Apparently, the vm allocates MAX_VIRTUAL_MEMORY and reduces by steps of 128M until it succeeds. so I suppose it could be possible to allocate 2Gb and see how it runs on a 1Gb system and if this is not too much stress for the system (thinking of the pagefile) ? Tudor is your vm a cog or non cog vm ?
"Mariano Martinez Peck" <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> a écrit dans le message de news:
BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com
<mailto:BANLkTi=Tved13_KL7quOzemOXPnCcfWw+g@mail.gmail.com>...
------------------------------------------------------------------------
On Thu, Apr 21, 2011 at 10:21 PM, Tudor Girba <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote:
Hi,
Should I to understand that the only way to enable more memory is to recompile the VM? Does that mean that there is no way to pass this information as a parameter like we can on Mac?
As far as I know you can pass a parameter, but even so, it won't be able to allocate more than 512MB. I can compile the VM for you with this change in 5 minutes. But I am not sure that such simple code would make it work. I think such limit is there because of something. Otherwise, it sounds stupid imposing a limit just because.
The problem is that I cannot recompile the VM because I have no access to a Windows machine. Is there one available that provides more memory?
I don't think so, but start cc'ing the VM mailing list. You'd probably receive more help.
Cheers
Mariano
Cheers, Doru
On 21 Apr 2011, at 22:09, Alain_Rastoul wrote:
> Hi Tudor, > > There is a constant in sqWin32Alloc.h (platforms\win32\vm) : > #define MAX_VIRTUAL_MEMORY 512*1024*1024 > you can change it to whatever you want and rebuild the vm, > for exzmple give all the available memory less 256 M . > > HTH > > Regards > Alain > > "Tudor Girba" <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> a écrit > dans le message de news:
03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com
<mailto:03B9389F-C719-44D0-B106-2AC78B120F56@gmail.com>... > Hi, > > We have no specific startUp: methods in Moose. In any case, the issue with > opening the image does not seem to be related to startUp:. > > Is it really true that the Cog VM is limited to 512MB of memory? > > Cheers, > Doru > > > On 21 Apr 2011, at 14:27, Luc Fabresse wrote: > >> Hi Doru, >> >> 2011/4/21 Tudor Girba >> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> >> Hi, >> >> >> >> On Apr 21, 2011, at 14:06, Mariano Martinez Peck >> <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote: >> >>> >>> >>> On Thu, Apr 21, 2011 at 1:58 PM, Tudor Girba >>> <tudor.girba@gmail.com <mailto:tudor.girba@gmail.com>> wrote: >>>> Hi again, >>>> >>>> I did not say what the problem was :). The problem was that when >>>> opening the image on Windows, he got a Space is low message and the >>>> image was not usable (see attachment). >>> >>> That's weird. Does moose have something on the startup list? something >>> that can be bothering there? >> >> Not that I know of. Is there a way to check this? >> >> Classes should be registered using Smalltlak addToStartUpList: aClass >> Then aClass class>>#startUp: is executed at startup. >> So implementors of #startUp: on Moose classes should give you the answer. >> >> #Luc >> >> >> Actually, using exactly the same process some days ago he produced an >> image of 190MB. This one works just fine on Windows. The only difference >> between the two is the size of the loaded data. >> >> It would be really bad news if the Windows vm would be so severely limited >> in terms of memory. >> >> Cheers, >> Doru >> >> >>> >>> On Mac it worked just fine. >>>> >>>> Cheers, >>>> Doru >>>> >>>> >>> >>>> >>>> >>>> >>>> On 21 Apr 2011, at 12:52, Tudor Girba wrote: >>>> >>>>> Hi, >>>>> >>>>> I received a question from someone running a 200MB image on Windows >>>>> using Cog 2361. >>>>> >>>>> If I open the image on Mac, it works just fine. Unfortunately, I do >>>>> not have a Windows machine around, and I cannot test but I believe it >>>>> should be solvable by increasing the allocated memory. >>>>> >>>>> On Mac, I would run it with: ./Croquet -memory 1500m >>>>> >>>>> Can anyone help me with the right incantation for Windows? >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> >>>>> -- >>>>> www.tudorgirba.com<http://www.tudorgirba.com> >>>>> >>>>> "What we can governs what we wish." >>>>> >>>>> >>>>> >>>>> <Space is low.png> >>>> >>>> -- >>>> www.tudorgirba.com<http://www.tudorgirba.com> >>>> >>>> "Yesterday is a fact. >>>> Tomorrow is a possibility. >>>> Today is a challenge." >>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> Mariano >>> http://marianopeck.wordpress.com >>> >> > > -- > www.tudorgirba.com<http://www.tudorgirba.com> > > "Beauty is where we see it." > > > > > > > >
-- www.tudorgirba.com<http://www.tudorgirba.com>
"If you interrupt the barber while he is cutting your hair, you will end up with a messy haircut."
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
.
participants (4)
-
Alain_Rastoul -
Andres Valloud -
Igor Stasenko -
Mariano Martinez Peck