September 2010 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Memory problems on Windows

by Jimmie Houchin

On 9/4/2010 2:28 PM, Andreas Raab wrote: > > There is indeed a hard limit of 512MB in the Windows VM currently in > effect, which is the result of debugging some extremely strange effects > when loading DLLs. What we found was that on some systems, under some > circumstances (related to memory load etc) DLLs would not be loaded if > we'd reserve more than 512MB of memory leading the application either to > not start at all, or to cause weird effects at later points etc. > > Cheers, > - Andreas Thanks Andreas for the explanation. When I re-read the 100 Million Objects thread, you mentioned that limit but I thought that you had removed it. Which may be why the 3.7.1 vm with the 3.8 image went up to 800+ mb of ram. But then it started acting strange. Unfortunately I have to interface with Windows COM libraries for business reasons. So currently Windows is required. Later I hope to upgrade to using the Java libraries which would release me from Windows and allow me to use Linux or OS X. Presently I am using Python to interface the COM libraries and then communicate with Squeak/Pharo via http. Since I know I have a hard limit, I'll quit trying to get it to work. :) Thanks again. Jimmie Houchin > On 9/4/2010 2:14 AM, Mariano Martinez Peck wrote: >> You can also ask in VM mailing list. I cc'ed them. >> >> On Sat, Sep 4, 2010 at 4:52 AM, Jimmie Houchin <jlhouchin(a)gmail.com> >> wrote: >> >> Hello, >> >> Sorry for the delay in reply. For some reason the first time I >> looked at your message in my newsreader (Thunderbird/GMane) the >> message was in French and written to someone else. Apparently the >> software did something strange. >> >> To answer your question. Yes. I tried this in both the standard vm >> and the standard Pharo 1.1 image. >> >> I start the vm/image with -memory: 1000 >> and open a new Workspace, then copy the below memory settings and doit. >> I then do a := Array new: 100000000. >> >> That seemingly succeeds but only takes me up to a little below 500mb >> ram. If I attempt to do another array of another 100,000,000 >> objects, I get the Low Space error. >> >> With the standard vm and image, it just sits there consuming cpu >> with simply doing the array initialization. I am on a quad core >> server with 6gm ram and it has currently consumed over 36 cpu >> minutes of one of the cores. At this point I quit the image. I could >> do things in the UI but it is not very responsive. >> >> In the Cog vm/image, the array creation returns almost immediately >> and waits for other instructions. >> >> As I write this I have to go back to Squeak 3.7.1vm and either the >> 3.8 or 3.10 image, for it to be successful. Outside of that I can't >> get past 500mb of ram. This server generally sits at only using 30% >> of ram. >> >> For enterprise/business endeavors, I thing Pharo really needs to be >> able to use all the memory the OS will allow it. I know that it >> being a 32 bit app does create some limits on some OSes. But neither >> of my Vista machines imposes a limit that I can't live with at the >> moment. >> >> Thanks for your reply. I am neither a vm nor Smalltalk expert so I >> don't know how to proceed from here outside of reducing my >> applications memory needs by putting more into the database and only >> having the data in memory that is absolutely necessary for the >> analysis I am attempting. >> >> Jimmie >> >> >> On 9/1/2010 4:48 AM, StÃ©phane Ducasse wrote: >> >> do you have the same problem with the normal VM? >> >> On Sep 1, 2010, at 5:25 AM, Jimmie Houchin wrote: >> >> Hello, >> >> I am developing an application which processes and generates >> a large amount of data. In a recent attempt I encountered an >> Space is Low error. >> >> This is occurring in a Pharo 1.1 image using the latest >> Pharo and Cog VMs. I am opening the vm with the -memory: >> 1000 parameter. >> The below code I have applied the from the Squeak list from >> the 100 Million Objects thread. But the problem occurs at >> about 500mb of ram on a computer with 3 (or 6) gb of ram, >> with only 65% of physical ram in use. The os is Vista. >> >> Any help in using more memory, as much as necessary for the >> app, would be greatly appreciated. >> >> initializeMemorySettings >> "Initialize the memory and GC settings to be more in line >> with QF requirements" >> >> "The following settings affect the rate incremental GCs and >> tenuring" >> >> "Limit incremental GC activity to run every 40k allocations" >> SmalltalkImage current vmParameterAt: 5 put: 40000. >> "allocations between GCs (default: 4000)" >> "Limit tenuring threshold to only tenure w/> 10k survivors" >> SmalltalkImage current vmParameterAt: 6 put: 10000. >> "tenuring threshold (default: 2000)" >> >> "These settings affect overall memory usage" >> >> "Only give memory back to the OS when we have more than 16MB >> free" >> SmalltalkImage current vmParameterAt: 24 put: >> 16*1024*1024. "default 8MB" >> "Try to keep 8MB headroom at all times" >> SmalltalkImage current vmParameterAt: 25 put: 8*1024*1024. >> "default 4MB" >> "These settings describe what to do when we're close to >> running out of free space" >> >> "Tell the VM that we'd rather grow than spinning in tight GC >> loops" >> SmalltalkImage current gcBiasToGrow: true. "default: false" >> "Tell the VM to do a fullGC for good measure if the above >> growth exceeded 16MB" >> SmalltalkImage current gcBiasToGrowLimit: 16*1024*1024. >> "default: 0" >> >> Thanks, >> >> Jimmie Houchin >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> <mailto:Pharo-project@lists.gforge.inria.fr> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> <mailto:Pharo-project@lists.gforge.inria.fr> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >

Sept. 4, 2010

Re: [Pharo-project] [Vm-dev] Re: Memory problems on Windows

by Andreas Raab

There is indeed a hard limit of 512MB in the Windows VM currently in effect, which is the result of debugging some extremely strange effects when loading DLLs. What we found was that on some systems, under some circumstances (related to memory load etc) DLLs would not be loaded if we'd reserve more than 512MB of memory leading the application either to not start at all, or to cause weird effects at later points etc. Cheers, - Andreas On 9/4/2010 2:14 AM, Mariano Martinez Peck wrote: > > > > > You can also ask in VM mailing list. I cc'ed them. > > On Sat, Sep 4, 2010 at 4:52 AM, Jimmie Houchin <jlhouchin(a)gmail.com > <mailto:jlhouchin@gmail.com>> wrote: > > Hello, > > Sorry for the delay in reply. For some reason the first time I > looked at your message in my newsreader (Thunderbird/GMane) the > message was in French and written to someone else. Apparently the > software did something strange. > > To answer your question. Yes. I tried this in both the standard vm > and the standard Pharo 1.1 image. > > I start the vm/image with -memory: 1000 > and open a new Workspace, then copy the below memory settings and doit. > I then do a := Array new: 100000000. > > That seemingly succeeds but only takes me up to a little below 500mb > ram. If I attempt to do another array of another 100,000,000 > objects, I get the Low Space error. > > With the standard vm and image, it just sits there consuming cpu > with simply doing the array initialization. I am on a quad core > server with 6gm ram and it has currently consumed over 36 cpu > minutes of one of the cores. At this point I quit the image. I could > do things in the UI but it is not very responsive. > > In the Cog vm/image, the array creation returns almost immediately > and waits for other instructions. > > As I write this I have to go back to Squeak 3.7.1vm and either the > 3.8 or 3.10 image, for it to be successful. Outside of that I can't > get past 500mb of ram. This server generally sits at only using 30% > of ram. > > For enterprise/business endeavors, I thing Pharo really needs to be > able to use all the memory the OS will allow it. I know that it > being a 32 bit app does create some limits on some OSes. But neither > of my Vista machines imposes a limit that I can't live with at the > moment. > > Thanks for your reply. I am neither a vm nor Smalltalk expert so I > don't know how to proceed from here outside of reducing my > applications memory needs by putting more into the database and only > having the data in memory that is absolutely necessary for the > analysis I am attempting. > > Jimmie > > > On 9/1/2010 4:48 AM, StÃ©phane Ducasse wrote: > > do you have the same problem with the normal VM? > > On Sep 1, 2010, at 5:25 AM, Jimmie Houchin wrote: > > Hello, > > I am developing an application which processes and generates > a large amount of data. In a recent attempt I encountered an > Space is Low error. > > This is occurring in a Pharo 1.1 image using the latest > Pharo and Cog VMs. I am opening the vm with the -memory: > 1000 parameter. > The below code I have applied the from the Squeak list from > the 100 Million Objects thread. But the problem occurs at > about 500mb of ram on a computer with 3 (or 6) gb of ram, > with only 65% of physical ram in use. The os is Vista. > > Any help in using more memory, as much as necessary for the > app, would be greatly appreciated. > > initializeMemorySettings > "Initialize the memory and GC settings to be more in line > with QF requirements" > > "The following settings affect the rate incremental GCs and > tenuring" > > "Limit incremental GC activity to run every 40k allocations" > SmalltalkImage current vmParameterAt: 5 put: 40000. > "allocations between GCs (default: 4000)" > "Limit tenuring threshold to only tenure w/> 10k survivors" > SmalltalkImage current vmParameterAt: 6 put: 10000. > "tenuring threshold (default: 2000)" > > "These settings affect overall memory usage" > > "Only give memory back to the OS when we have more than 16MB > free" > SmalltalkImage current vmParameterAt: 24 put: > 16*1024*1024. "default 8MB" > "Try to keep 8MB headroom at all times" > SmalltalkImage current vmParameterAt: 25 put: 8*1024*1024. > "default 4MB" > "These settings describe what to do when we're close to > running out of free space" > > "Tell the VM that we'd rather grow than spinning in tight GC > loops" > SmalltalkImage current gcBiasToGrow: true. "default: false" > "Tell the VM to do a fullGC for good measure if the above > growth exceeded 16MB" > SmalltalkImage current gcBiasToGrowLimit: 16*1024*1024. > "default: 0" > > Thanks, > > Jimmie Houchin > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > <mailto:Pharo-project@lists.gforge.inria.fr> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > <mailto:Pharo-project@lists.gforge.inria.fr> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > >

Sept. 4, 2010

Re: [Pharo-project] Sockets in Pharo CollaborActive Book

by Schwab,Wilhelm K

Noury, First, please don't assume that you have to change course just because we offered to help. If you are close to having a major improvement, then it might be best to just power through it. If there are things holding you up, we might be able to help there. Using Alien concerns me a little, as the last time I checked, it was not available on Linux?? Failure to support Linux would be a complete show-stopper for me. Regardless, I would like to see your code as I have read some comments suggesting that Alien is cumbersome to use; you might show me that all is well. My interest in Alien is primarily to support callbacks for numerical analysis. So far, I have been adding functions to a shared library, which is easy enough, ultimately translates formulas using C++ (which is better at it than Smalltalk), likely makes things go faster than they would with Smalltalk in the loop, but is far from elegant. Until Alien exists on Linux, it does not exist :( Back to sockets, if the current situation were to stand, for whatever reason, I would consider tackling the problem using a plugin to get OS threads, and use them to achieve the non-blocking operation that is essential for interactive software. In short, everything blocks its calling Smalltalk Process; nothing blocks the entire image. Hopefully you recognize that a server listening with timeouts is hideous design; just start and stop the server as needed, and it should be harmless to the image. If the server is not harmless to the image, it needs to be fixed. Clients having timeouts isn't much better. If you want to add helper methods with timeouts, who am I to tell others what to do, but we must not force timeouts on clients because there is no correct answer to how long something should wait, so we should not look for one. What we should do is make sure that we never block the entire image, and let people make their own decisions. Why would I consider a plugin? In large part because there are hints that I would find examples to follow, and because OpenSSL has a very cumbersome non-blocking interface, at least that was my impression of it. A plugin that I would write would hopefully support TCP and OpenSSL sockets, and hopefully IrDA. Running those things on OS threads and signaling semaphores in the image is likely to be simpler than working out the details of non-blocking calls in the separate worlds - I think. I can understand why you are not thrilled about creating a plugin. The sad part of this is that the current sockets system appears to demonstrate all of the essential pieces to make something that really works; it just hasn't been done. On attempting it, I would no doubt run across things that you found and led you to create a new system. Creating something new allows it to coexist with the current code until it is ready, at which point it could be integrated into Pharo. Thanks for working on it. Is the code on Squeaksource representative of your current progress? Please do not let my interest slow your efforts. If you are indeed doing all of the right things, it might be that my efforts should be directed toward getting Alien going on Linux. It might also be that I can be of most help simply by doing to your code what I did to DCOM long ago. Our network was a harsh landscape, and I got fairly good bringing code to its knees on intranets. This involves creating forked processes doing active communication across a hub that coincidentally gets unpowered w/o warning :) Threads must block or give errors, but the images must continue to be responsive and other threads must run as expected. Until we reliably pass that test, this is all just a game. Bill ________________________________________ From: pharo-project-bounces(a)lists.gforge.inria.fr [pharo-project-bounces(a)lists.gforge.inria.fr] On Behalf Of Noury Bouraqadi [bouraqadi(a)gmail.com] Sent: Saturday, September 04, 2010 5:31 AM To: Pharo-project(a)lists.gforge.inria.fr Subject: Re: [Pharo-project] Sockets in Pharo CollaborActive Book Hi Bill, On 1 sept. 2010, at 05:59, Schwab,Wilhelm K wrote: > The deeper question is whether we want to document how things are, or they they *should be* - which is VERY much different from the eventless and timeout ridden design that we have now. It desperately needs to be redesigned. > I agree that the current status of the networking libraries is horrible. But, to learn the current status and think of the alternatives, I felt the need to understand how thinks work now. As StÃ©f said, Luc and I started the Ocean project. Thanks you Bill and to Guille for your proposal to help. It's a pitty that you don't attend to the ESUG conf (Your names are not on the registrations list) otherwise we could discuss this better. Cause Luc and I will give a short presentation describing the current status of Ocean and where it is headed. Now, we need to think how to split the project in order to have other contributions. Note BTW, our goal is to have a low-level Pharo wrappers of the networking facilities of the OS. We started using Alien to avoid using a plugin and ease portability. Once we get this low-level layer, we'll go on and use it in an OOP middleware like rST and UbiquiTalk. Which is after all, the right level to work with since we want to deal with objects and message sends. A direct use of Sockets isn't really OO most of the time. Noury _______________________________________________ Pharo-project mailing list Pharo-project(a)lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Sept. 4, 2010

Re: [Pharo-project] Single instance executables

by Schwab,Wilhelm K

You mean like the change log? :) If that will work, then we should look close to home and take the hint. ________________________________________ From: pharo-project-bounces(a)lists.gforge.inria.fr [pharo-project-bounces(a)lists.gforge.inria.fr] On Behalf Of Denis Kudriashov [dionisiydk(a)gmail.com] Sent: Saturday, September 04, 2010 3:25 AM To: Pharo-project(a)lists.gforge.inria.fr Subject: Re: [Pharo-project] Single instance executables You could open some file for write at image startup. When its failed you know that another image already started 2010/9/2, Schwab,Wilhelm K <bschwab(a)anest.ufl.edu>: > Hello all, > > I realize that I hold an unpopular position on this, but it *will* come back > to haunt us: we need to have a way to warn the user if a second copy of > Pharo is launched. Random meltdowns and lost work will result from the > current behavior. Let me be more bold: lost work and random corrrupted > images *are resulting* from the current behavior; they are either > unrecognized, or simply ignored, but it is happening. > > Is there a way to fix it? On Windows, I recall creating an OS semaphore > with a given name and if that exists, bail out before the session even gets > going. We would need a cross-platform way to do that (or at least a > Linux/Mac way to do it - Windows has a mechanism), and then a place to abort > startup early enough to prevent problems. Any ideas? > > Bill > > > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > _______________________________________________ Pharo-project mailing list Pharo-project(a)lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Sept. 4, 2010

[Pharo-project] [update 1.2] #12124

by StÃ©phane Ducasse

12124 ----- - Issue 2910: Should include Updater. - Issue 2891: Some other ifNil instead of isNil iftrue: . Thanks Guillermo Polito and Nicolas Cellier.

Sept. 4, 2010

Re: [Pharo-project] systemNavigation cleaning

by StÃ©phane Ducasse

On Sep 4, 2010, at 1:08 PM, Lukas Renggli wrote: >>> I agree that SystemNavigation is ugly, but *a lot* of existing code >>> depends on it. >> >> Well if I start forking all the classes that we should fix because there are used it will be endless. >> I was thinking to start also to look at senders and use deprecate. >> >> do you have a lot of external tools that use it? > > Yes, all tools use it: > > Monticello, Paragraph Editor (new and old), OmniBrowser, > eCompletion, Refactoring Engine. > > Now I removed its use from the BrowserEnvironment in the refactoring > engine itself. There are still a couple of references though, where > people used the wrong abstraction layer when implementing stuff > (platform code instead of the abstraction of the refactoring engine). > >> So what do you suggest? >> Separating BrowserEnvironment and its superclass from RB and starting to >> - check its api >> - improve it >> - migrate caller of systemNavigation -> BrowserEnvironment (rename browserEnvironment) > > Yeah, maybe one step would be to split the refactoring engine into > smaller and independent packages: > > AST-Core (that already is separate) > Refactoring-Environment (this is currently part of yes this would be good so that we can start do the idea with the integration related to systemDictionary been a potential leave of the composition tree we talked last time you visit us. > Refactoring-Core, but would work independently) > Refactoring-Changes (this is currently part of Refactoring-Core, > but would work independently) yes I'm curious to see if/how we could use changes to represent changeset. > Refactoring-Refactorings (this is currently part of > Refactoring-Core and depends on all of the above) > Refactoring-Lint (this is currently part of Refactoring-Core and > depends on all of the above) > > Like this Pharo could integrate Refactoring-Environment and > Refactoring-Changes independently. :) > I'll see if I can repackage the refactoring browser a bit, might take > a while though because the tests are not that nicely separated. I imagine. Now incremental changes is the only way for us :) > > Lukas > > -- > Lukas Renggli > www.lukas-renggli.ch > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Sept. 4, 2010

Re: [Pharo-project] systemNavigation cleaning

by Mariano Martinez Peck

On Sat, Sep 4, 2010 at 1:08 PM, Lukas Renggli <renggli(a)gmail.com> wrote: > >> I agree that SystemNavigation is ugly, but *a lot* of existing code > >> depends on it. > > > > Well if I start forking all the classes that we should fix because there > are used it will be endless. > > I was thinking to start also to look at senders and use deprecate. > > > > do you have a lot of external tools that use it? > > Yes, all tools use it: > > Monticello, Paragraph Editor (new and old), OmniBrowser, > eCompletion, Refactoring Engine. > > Now I removed its use from the BrowserEnvironment in the refactoring > engine itself. There are still a couple of references though, where > people used the wrong abstraction layer when implementing stuff > (platform code instead of the abstraction of the refactoring engine). > > > So what do you suggest? > > Separating BrowserEnvironment and its superclass from RB and starting to > > - check its api > > - improve it > > - migrate caller of systemNavigation -> BrowserEnvironment (rename > browserEnvironment) > > Yeah, maybe one step would be to split the refactoring engine into > smaller and independent packages: > > AST-Core (that already is separate) > Refactoring-Environment (this is currently part of > Refactoring-Core, but would work independently) > Refactoring-Changes (this is currently part of Refactoring-Core, > but would work independently) > Refactoring-Refactorings (this is currently part of > Refactoring-Core and depends on all of the above) > Refactoring-Lint (this is currently part of Refactoring-Core and > depends on all of the above) > > Like this Pharo could integrate Refactoring-Environment and > Refactoring-Changes independently. > > I'll see if I can repackage the refactoring browser a bit, might take > a while though because the tests are not that nicely separated. > > If you finally do this Lukas, let me know so that I update ConfigurationOf ... Lukas > > -- > Lukas Renggli > www.lukas-renggli.ch > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >

Sept. 4, 2010

Re: [Pharo-project] systemNavigation cleaning

by Lukas Renggli

>> I agree that SystemNavigation is ugly, but *a lot* of existing code >> depends on it. > > Well if I start forking all the classes that we should fix because there are used it will be endless. > I was thinking to start also to look at senders and use deprecate. > > do you have a lot of external tools that use it? Yes, all tools use it: Monticello, Paragraph Editor (new and old), OmniBrowser, eCompletion, Refactoring Engine. Now I removed its use from the BrowserEnvironment in the refactoring engine itself. There are still a couple of references though, where people used the wrong abstraction layer when implementing stuff (platform code instead of the abstraction of the refactoring engine). > So what do you suggest? > Separating BrowserEnvironment and its superclass from RB and starting to > Â Â Â Â - check its api > Â Â Â Â - improve it > Â Â Â Â - migrate caller of systemNavigation -> BrowserEnvironment (rename browserEnvironment) Yeah, maybe one step would be to split the refactoring engine into smaller and independent packages: AST-Core (that already is separate) Refactoring-Environment (this is currently part of Refactoring-Core, but would work independently) Refactoring-Changes (this is currently part of Refactoring-Core, but would work independently) Refactoring-Refactorings (this is currently part of Refactoring-Core and depends on all of the above) Refactoring-Lint (this is currently part of Refactoring-Core and depends on all of the above) Like this Pharo could integrate Refactoring-Environment and Refactoring-Changes independently. I'll see if I can repackage the refactoring browser a bit, might take a while though because the tests are not that nicely separated. Lukas -- Lukas Renggli www.lukas-renggli.ch

Sept. 4, 2010

[Pharo-project] Loop while debugging #expectedFailures

by Mariano Martinez Peck

Hi. There is a loop when trying to debug #expectedFailures. To reproduce, just put a "self halt" at the beggining of a #expectedFailures method and then run of the tests of that class. The problem is that when you put the self halt, in an #expectedFailures, it brings a debugger, of course. But the debuggers needs to send the printOn to all the stuff in the context. TestResult implements printOn: in this way: printOn: aStream aStream nextPutAll: self runCount printString; nextPutAll: ' run, '; nextPutAll: self expectedPassCount printString; nextPutAll: ' passes, '; nextPutAll: self expectedDefectCount printString; nextPutAll:' expected failures, '; nextPutAll: self unexpectedFailureCount printString; nextPutAll: ' failures, '; nextPutAll: self unexpectedErrorCount printString; nextPutAll:' errors, '; nextPutAll: self unexpectedPassCount printString; nextPutAll:' unexpected passes'. which...as you can see sends the message #expectedPassCount which at the ends...sends #expectedFailure.....sooo....loop :) I am still thinking which could be the better solution. I will try to come up with it later. Cheers Mariano

Sept. 4, 2010

Re: [Pharo-project] Single instance executables

by Douglas Brebner

On 04/09/2010 08:25, Denis Kudriashov wrote: > You could open some file for write at image startup. When its failed > you know that another image already started > That only works on Windows. Unix will allow you to open a file for writing multiple times simultaneously.

Sept. 4, 2010