- Pharo-dev - lists.pharo.org

Re: [Pharo-project] [Vm-dev] Re: out of memory - cog on windows

by Tudor Girba

Hi, On 18 May 2011, at 16:30, Levente Uzonyi wrote: > On Wed, 18 May 2011, Tudor Girba wrote: > >> >> Hi, >> >> Is there a possibility to have any sort of answer on this topic? >> >> It looks to me like this bug is critical given that it prevents us to work with images larger than some 200M. > > IMHO it's a problem on the image side. Change the behavior of #lowSpaceWatcher to fix the problem. How would you do it? :) And if it is an image problem, why does it only appear on Windows? Cheers, Doru > > Levente > >> >> Cheers, >> Tudor >> >> >> >> On 16 May 2011, at 11:06, Tudor Girba wrote: >> >>> Hi Igor, >>> >>> Thanks a lot for looking into this. For us (around Moose), this is a critical issue because we play with our models in memory and regularly have images of several hundred megabytes. While most of us work on Mac without a problem, some other people work on Windows and this is a bottleneck. >>> >>> Unfortunately, I do not have enough know-how to dive into the VM, but I would be gladly help in any way I can. >>> >>> Cheers, >>> Doru >>> >>> >>> On 16 May 2011, at 10:50, Igor Stasenko wrote: >>> >>>> On 10 May 2011 09:34, Tudor Girba <tudor.girba(a)gmail.com> wrote: >>>>> Hi, >>>>> >>>>> I am back with some more input on the matter. We played with it, and found that basically any image that goes beyond something like 200MB limit (or maybe it's the number of objects), will not open on Windows. >>>>> >>>>> I would need some help on this. Has nobody else hit this wall, or is it that I am doing something wrong? >>>>> >>>>> One scenario that we used to reproduce the problem looks like this: >>>>> 1. Take a Moose image: >>>>> http://hudson.moosetechnology.org/job/moose-latest-dev/lastSuccessfulBuild/… >>>>> 2. Run: >>>>> 1 to: 3 do: [ :i | MooseScripts createModelForConfigurationOfMoose ] >>>>> This will basically create some 850000 objects and will get the image to some 400+MB >>>>> 3. Save and quit the image >>>>> 4. Reopen it >>>>> >>>>> A ready-made image with the result of this can be found here: >>>>> http://dl.dropbox.com/u/18323746/Tmp/moose-case-study-windows-problem.zip >>>>> >>>>> This works well on Mac, but on Windows when you reopen the image, you get the out of memory message. >>>>> >>>>> Cheers, >>>>> Doru >>>>> >>>>> >>>> >>>> I using latest VMs built by cmake on windows, and everytime i run this image, >>>> it opens a 'Space is low' dialog and then image not reacting on anything. >>>> VM not crashing however and responds to windows events.. But because >>>> UI process is broken >>>> an image simply not reacts to anything. >>>> >>>> >>>> I think that the problem is that this dialog are shown at early image >>>> startup time, >>>> before everything is properly initialized, and because of that UI >>>> stalls somewhere. >>>> >>>> >>>> Next things which i tried is to increase a virtual memory limit in >>>> sqWin32Alloc.h >>>> >>>> #ifndef MAX_VIRTUAL_MEMORY >>>> #define MAX_VIRTUAL_MEMORY 512*1024*1024 >>>> #endif >>>> >>>> first i raised it to 768 Mbytes.. same behavior. >>>> then i raised it to 1Gb and again same behavior.. >>>> >>>> So, either this constant is overridden somewhere or it actually >>>> doesn't affecting the low-space detection mechanism as i would expect. >>>> Any suggestions? >>>> >>>> >>>> I will continue looking , what happens on VM side, >>>> but in addition to that, i think we should do something on image side as well: >>>> - a low space watcher should not pop up before passing image startup, >>>> because if preempted process is UI process (and in 99.99% cases it is), >>>> then it means that low space watcher blocks UI process and as a consequence, >>>> your image stops handling events. >>>> >>>> >>>> Also, i thinking about different approach for signaling a low space condition. >>>> Instead of signaling a low space semaphore, what VM could do is to >>>> fail an allocation primitive >>>> and set the error code to "low memory warning" >>>> then a primitive failure code could actually throw an exception, which >>>> then could be handled as usual... >>>> so you could write a code, like: >>>> >>>> [ self allocateSomethingReallyHuge ] on: LowMemoryWarning do: [:err | >>>> self shouldWeReallyCare ifFalse: [ self tellVMThatItsOk. err resume ] >>>> ifTrue: [ err pass ] >>>> ] >>>> >>>> Because by preempting a currently active process, which "possibly" >>>> causing a memory problems is not a solution, >>>> as you can see, if it happens during startup phase, then you it just >>>> stucks and image hangs. >>>> But if we could use exceptions, we could just ignore this warning (or >>>> do something else) during image startup, >>>> instead of blocking UI process , showing a useless popup message and >>>> letting image hang like that. >>>> >>>> -- >>>> Best regards, >>>> Igor Stasenko AKA sig. >>> >>> -- >>> www.tudorgirba.com >>> >>> "If you interrupt the barber while he is cutting your hair, >>> you will end up with a messy haircut." >>> >> >> -- >> www.tudorgirba.com >> >> "Obvious things are difficult to teach." >> >> >> >> -- www.tudorgirba.com "Next time you see your life passing by, say 'hi' and get to know her."

May 18, 2011

Re: [Pharo-project] OCompletion?

by Alexandre Bergel

> In which Pharo? 1.3? I do not seem to bump into it. In what context do you get it? When I try to edit code within a debugger. It does not systematically appear. Just once in a while. Alexandre > > > On 18 May 2011, at 16:19, Alexandre Bergel wrote: > >> Hi! >> >> Am I the only one to bump all the time into error raised by OCompletion? >> >> in OController>>handleKeystrokeBefore: kbEvent editor: theEditor >> I just switched: >> self isMenuOpen ifFalse: [ self openMenuFor: theEditor ]. >> >> into: >> self isMenuOpen ifFalse: [ [self openMenuFor: theEditor] on: Error do: [] ]. >> >> >> Cheers, >> Alexandre >> -- >> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >> Alexandre Bergel http://www.bergel.eu >> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. >> >> >> >> >> >> > > -- > www.tudorgirba.com > > "Next time you see your life passing by, say 'hi' and get to know her." > > > > -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

May 18, 2011

Re: [Pharo-project] OCompletion?

by Tudor Girba

Hi, In which Pharo? 1.3? I do not seem to bump into it. In what context do you get it? Cheers, Doru On 18 May 2011, at 16:19, Alexandre Bergel wrote: > Hi! > > Am I the only one to bump all the time into error raised by OCompletion? > > in OController>>handleKeystrokeBefore: kbEvent editor: theEditor > I just switched: > self isMenuOpen ifFalse: [ self openMenuFor: theEditor ]. > > into: > self isMenuOpen ifFalse: [ [self openMenuFor: theEditor] on: Error do: [] ]. > > > Cheers, > Alexandre > -- > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > Alexandre Bergel http://www.bergel.eu > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > > -- www.tudorgirba.com "Next time you see your life passing by, say 'hi' and get to know her."

May 18, 2011

[Pharo-project] OCompletion?

by Alexandre Bergel

Hi! Am I the only one to bump all the time into error raised by OCompletion? in OController>>handleKeystrokeBefore: kbEvent editor: theEditor I just switched: self isMenuOpen ifFalse: [ self openMenuFor: theEditor ]. into: self isMenuOpen ifFalse: [ [self openMenuFor: theEditor] on: Error do: [] ]. Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

May 18, 2011

Re: [Pharo-project] Circle is gone

by Alexandre Bergel

I also bumped into the same problem. For the case of Mondrian, I will copy the missing classes into Mondrian. These classes were removed quite some times ago from Pharo 1.3 Cheers, Alexandre On 18 May 2011, at 08:54, Tudor Girba wrote: > Hi, > > Circle seems to be gone from Pharo 1.3. > > What is now the preferred way to specify a circle? > > Cheers, > Doru > > > -- > www.tudorgirba.com > > "When people care, great things can happen." > > > > -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

May 18, 2011

Re: [Pharo-project] Basic tricks for improving a serializer?

by Henrik Sperre Johansen

On 18.05.2011 14:53, Sven Van Caekenberghe wrote: > Mariano, > > On 18 May 2011, at 14:29, Mariano Martinez Peck wrote: > >> Sven, I improved quite a lot using your buffered write stream. >> What I don't understans is why MultiByteFileStream doesn't perform as good as them. >> If I see StandardFileStream has an internal buffer called 'buffer1'. >> >> So why is that much difference between using MultiByteFileStream directly and >> a WriteStream (ByteArray new writeSteam ) or ZnBufferedWriteStream ? Because actually writing to file is expensive. MBFS does it on every nextPut/nextPutAll: send, ZnBuffered only when asked to/buffer fills. > I just looked briefly at the hierarchy, it seems that buffer1 is an output buffer of size 1 ! > > Have a look at > > StandardFileStream>>#nextPut: char > "Write the given character to this file." > > rwmode ifFalse: [^ self error: 'Cannot write a read-only file']. > collection ifNotNil: [ > position< readLimit ifTrue: [ self flushReadBuffer ] ]. > buffer1 at: 1 put: char. > self primWrite: fileID from: buffer1 startingAt: 1 count: 1. > ^ char > > There is simply no buffering going on. > buffer1 is a mystery to me ;-) > > Sven The primitive requires an object of variableByte type. If you didn't have a buffer1, you'd need to create a new 1-element ByteArray/String on every call to nextPut: ;) As for the primitive call always flushing to file, that is a decision which have arguments both in favor and against it. I wasn't there when it was written, so have no idea why they weighed one over the other. If I were to guess, I'd say PLS ("But I wrote it to file! Why isnt it there yet when I open it??") and deterministic behaviour won over performance. Changing it now would probably mean quite a few changes to f.ex. source management if you want the same level of recoverability you are ensured now. That does not mean a buffering of write stream should not be available, of course. There are so many ways to do that though it quickly turns into painting a bikeshed. Cheers, Henry

May 18, 2011

Re: [Pharo-project] ConfigurationOfOmnibrowser

by Mariano Martinez Peck

On Wed, May 18, 2011 at 3:26 PM, Alexandre Bergel <alexandre.bergel(a)me.com>wrote: > Hi! > > Apparently, baseline13: does not specific where to grab OB from. I added a > line "spec repository: 'http://source.lukas-renggli.ch/omnibrowser'." and > it works for me. > That's not correct. We are using 'http://www.squeaksource.com/PharoOB' And it is correctly set. Don't check in spec for: #squeakCommon do: [ but in spec for: #pharo do: [ (go down). cheers But I am wondering why we did not bump into this problem before? > Cheers, > Alexandre > -- > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > Alexandre Bergel http://www.bergel.eu > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > > > > > > -- Mariano http://marianopeck.wordpress.com

May 18, 2011

[Pharo-project] ConfigurationOfOmnibrowser

by Alexandre Bergel

Hi! Apparently, baseline13: does not specific where to grab OB from. I added a line "spec repository: 'http://source.lukas-renggli.ch/omnibrowser'." and it works for me. But I am wondering why we did not bump into this problem before? Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

May 18, 2011

[Pharo-project] Wishes for new ClassBuilder (toon?)

by Mariano Martinez Peck

Hi guys. As part of the Fuel project, one of the things we want to do (in fact, it is done) is to serialize classes. There want to have 3 possibilities: 1) Do something like DiskProxy where you only save its global name and some selector to run when deserialing it 2) Really searialize the whole class (following method dictionary, compiled methods, class pools and friend, etc) so that you can deserialize it in another image where such class is not present. 3) Do the same as 2 but validating and installing the class in the system. For 2) what we want to do is just to serialize the class as any other object. As a result, we get a Class instance. Nothing is validated, nothing is added in Smalltalk globals, nothing is compiled. We do not use ClassBuilder at all. For 3) we want to do 2) plus validating and installing the class in the system. For validating I mean ALL the validations that the class builder does when creating or modifying a class. And for "installing in the system" I mean notify, put in Smalltalk globals, re compile what is needed, etc... ClassBuilder seems to do 2) and 3) together. So, what I would love to have in a new class builder is to have those two things separated. So that I can say to an object ClassManager validate: aClass or ClassManager installClassOnSystem: aClass. Do you think such thing is possible? Right now going through ClassBuilder and understand all the validations it does is really complicated :( Thanks in advance, -- Mariano http://marianopeck.wordpress.com

May 18, 2011

Re: [Pharo-project] out of memory - cog on windows

by Tudor Girba

Hi, Is there a possibility to have any sort of answer on this topic? It looks to me like this bug is critical given that it prevents us to work with images larger than some 200M. Cheers, Tudor On 16 May 2011, at 11:06, Tudor Girba wrote: > Hi Igor, > > Thanks a lot for looking into this. For us (around Moose), this is a critical issue because we play with our models in memory and regularly have images of several hundred megabytes. While most of us work on Mac without a problem, some other people work on Windows and this is a bottleneck. > > Unfortunately, I do not have enough know-how to dive into the VM, but I would be gladly help in any way I can. > > Cheers, > Doru > > > On 16 May 2011, at 10:50, Igor Stasenko wrote: > >> On 10 May 2011 09:34, Tudor Girba <tudor.girba(a)gmail.com> wrote: >>> Hi, >>> >>> I am back with some more input on the matter. We played with it, and found that basically any image that goes beyond something like 200MB limit (or maybe it's the number of objects), will not open on Windows. >>> >>> I would need some help on this. Has nobody else hit this wall, or is it that I am doing something wrong? >>> >>> One scenario that we used to reproduce the problem looks like this: >>> 1. Take a Moose image: >>> http://hudson.moosetechnology.org/job/moose-latest-dev/lastSuccessfulBuild/… >>> 2. Run: >>> 1 to: 3 do: [ :i | MooseScripts createModelForConfigurationOfMoose ] >>> This will basically create some 850000 objects and will get the image to some 400+MB >>> 3. Save and quit the image >>> 4. Reopen it >>> >>> A ready-made image with the result of this can be found here: >>> http://dl.dropbox.com/u/18323746/Tmp/moose-case-study-windows-problem.zip >>> >>> This works well on Mac, but on Windows when you reopen the image, you get the out of memory message. >>> >>> Cheers, >>> Doru >>> >>> >> >> I using latest VMs built by cmake on windows, and everytime i run this image, >> it opens a 'Space is low' dialog and then image not reacting on anything. >> VM not crashing however and responds to windows events.. But because >> UI process is broken >> an image simply not reacts to anything. >> >> >> I think that the problem is that this dialog are shown at early image >> startup time, >> before everything is properly initialized, and because of that UI >> stalls somewhere. >> >> >> Next things which i tried is to increase a virtual memory limit in >> sqWin32Alloc.h >> >> #ifndef MAX_VIRTUAL_MEMORY >> #define MAX_VIRTUAL_MEMORY 512*1024*1024 >> #endif >> >> first i raised it to 768 Mbytes.. same behavior. >> then i raised it to 1Gb and again same behavior.. >> >> So, either this constant is overridden somewhere or it actually >> doesn't affecting the low-space detection mechanism as i would expect. >> Any suggestions? >> >> >> I will continue looking , what happens on VM side, >> but in addition to that, i think we should do something on image side as well: >> - a low space watcher should not pop up before passing image startup, >> because if preempted process is UI process (and in 99.99% cases it is), >> then it means that low space watcher blocks UI process and as a consequence, >> your image stops handling events. >> >> >> Also, i thinking about different approach for signaling a low space condition. >> Instead of signaling a low space semaphore, what VM could do is to >> fail an allocation primitive >> and set the error code to "low memory warning" >> then a primitive failure code could actually throw an exception, which >> then could be handled as usual... >> so you could write a code, like: >> >> [ self allocateSomethingReallyHuge ] on: LowMemoryWarning do: [:err | >> self shouldWeReallyCare ifFalse: [ self tellVMThatItsOk. err resume ] >> ifTrue: [ err pass ] >> ] >> >> Because by preempting a currently active process, which "possibly" >> causing a memory problems is not a solution, >> as you can see, if it happens during startup phase, then you it just >> stucks and image hangs. >> But if we could use exceptions, we could just ignore this warning (or >> do something else) during image startup, >> instead of blocking UI process , showing a useless popup message and >> letting image hang like that. >> >> -- >> Best regards, >> Igor Stasenko AKA sig. > > -- > www.tudorgirba.com > > "If you interrupt the barber while he is cutting your hair, > you will end up with a messy haircut." > -- www.tudorgirba.com "Obvious things are difficult to teach."

May 18, 2011