- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [Moose-dev] Too frequent crashes :-(

by Alexandre Bergel

Hi Ronie, Below you said: > The problem with asForm, is that the form returned can be used after the cairo surface which holds the pixels is garbage collected. I cannot reproduce this case. You gave a script: > surface := AthensCairoSurface extent: 640@480. > surface drawDuring: [ :canvas | > surface clear: Color blue. > ]. > > surface asForm asMorph openInWindow I see a blue window. Cheers, Alexandre > How about changing AthensCairoSurface >> asForm into the following definition?: > asForm > > "create a form and copy an image data there" > | form | > self checkSession. > > self flush. > form := Form extent: (self width@self height) depth: 32. > form unhibernate. > LibC memCopy: self getDataPtr to: form bits size: self width*self height*4. > ^ form > > This involves a whole copy, but it removes completely the dependency on the surface plugin. > > If we want to keep using the surface plugin with Cairo, the old definition of asForm still has a bug, which can be reproduced with the following snippet: > > Smalltalk compiler evaluate: ' > | surface | > surface := AthensCairoSurface extent: 640@480. > surface drawDuring: [ :canvas | > surface clear: Color blue. > ]. > > surface asForm asMorph openInWindow > ' > > The problem with asForm, is that the form returned can be used after the cairo surface which holds the pixels is garbage collected.

March 14, 2017

Re: [Pharo-dev] image not opening.

by stepharong

thanks eliot for your analysis. Stef > > > On Sun, Mar 12, 2017 at 3:04 AM, Guillermo Polito > <guillermopolito(a)gmail.com> wrote: >> Yeh, to be more precise,* the problem is that the image somehow got >> it's display size modified to a very small one.* And that happenned >> while building an image in the CI in headless mode.* If the same image >> is built locally in a vm launched in non-headless mode, the issue >> cannot be reproduced. >> >> So my conjecture is that >> - either the image is changing the display size while loading pillar >> (less likely and not reproduceable locally) >> - either the latest stable VM messes the display size in the image >> header when running on headless (this is yet to reproduce) > > I'm pretty confident this is to do with bugs in the Athens surface code > which assumes that callbacks can be made in the existing copyBits and > warpBits primitive. They can't do >this safely because a GC (scavenge) > can happen during a callback, which then causes chaos when the copyBits > primitive tries to access objects that have been moved under >its feet. > > I've done work to fix callbacks so that when there is a failure it is > the copyBits primitive that fails, instead of apparently the callback > return primitive. One of the apparent effects of >this fix is to stop > the screen opening up too small; another is getting the background > colour right, and yet another is eliminating bogus pixels in the > VGTigerDemo demo. But >more work is required to fix the copyBits and > warpBits primitives. There are a few approaches one might take: > > a) fixing the primitive so that it saves and restores oops around the > callbacks using the external pop table. That's a pain but possible. > > b) fixing the primitive so that it pins the objects it needs before ever > invoking a callback > > c) fixing the primitive so that it uses the scavenge and fullGC counters > in the VM to detect if a GC occurred during one of the callbacks and > would fail the primitive. The primitive >would then simply be retried. > d) ? > > I like c) as it's very lightweight, but it has issues. It is fine to > use for callbacks *before* cop[yBits and warpBits move any bits (the > lockSurface and querySurface functions). But >it's potentially > erroneous after the unlockSurface primitive. For example, a primitive > which does an xor with the screen can't simply be retried as the first, > falling pass, would have >updated the destination bits but not displayed > them via unlockSurface. But I think it could be arranged that no > objects are accessed after unlockSurface, which should naturally >be the > last call in the primitive (or do I mean showSurface?). So the approach > would be to check for GCs occurring during querySurface and lockSurface, > failing if so, and then >caching any and all state needed by > unlockSurface and showSurface in local variables. This way no object > state is accessed to make the unlockSurface and showSurface >calls, and > no bits are moved before the queryDurface and lockSurface calls. > > If we used a failure code such as #'object may move' then the primitives > could answer this when a GC during callbacks is detected and then the > primitive could be retried only >when required. > > >> >> On Sun, Mar 12, 2017 at 10:52 AM, stepharong <stepharong(a)free.fr> wrote: >>> guillermo was suspecting a problem with the interpretatin of the image >>> metadata >>> >>>> Maybe related? >>>> http://forum.world.st/BUG-A-problem-with-callbacks-that-shows-up-in-64bits-… >>>> >>>> On Sat, Mar 11, 2017 at 2:22 AM, Cyril Ferlicot >>>> <cyril.ferlicot(a)gmail.com> wrote: >>>>> >>>>> On ven. 10 mars 2017 at 18:42, stepharong <stepharong(a)free.fr> wrote: >>>>>> >>>>>> We found the problem with Guillermo and this may be a problem of >>>>>> the last >>>>>> VM. >>>>>> >>>>>> The image got its starting window size really small: some pixels on >>>>>> some >>>>>> pixles. >>>>>> So may be the metadata of the image are systematically corrupted. >>>>>> >>>>>> Hi >>>>>> >>>>>> I have a jenkins that produce an image that does not start >>>>>> when I try to open it locally. >>>>>> >>>>>> Now if I take the image and load the "same" configuration I get a >>>>>> working >>>>>> image >>>>>> >>>>>> I have not idea how to debug that. >>>>>> >>>>>> Stef >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Using Opera's mail client: http://www.opera.com/mail/ >>>>> >>>>> >>>>> If it can help: here is the build: >>>>> >>>>> https://ci.inria.fr/pharo-contribution/job/OOnoz/ >>>>> >>>>> It is happening in almost all Pharo 6 builds >>>>> -- >>>>> Cheers >>>>> Cyril Ferlicot >>>> >>>>>>> >>> >>>>>> -- >>> Using Opera's mail client: http://www.opera.com/mail/ >>> >> > > > > -- >> _,,,^..^,,,_ > best, Eliot -- Using Opera's mail client: http://www.opera.com/mail/

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Eliot Miranda

On Tue, Mar 14, 2017 at 8:56 AM, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > 2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > >> >> Hi Esteban, Hi Igor, Hi All, >> >> On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >> wrote: >> >>> >>> Hi, >>> >>> Iâm tumbling into an error in Pharo, because we use callbacks >>> intensively, in Athens(cairo)-to-World conversion in particular, and people >>> is sending always their crash reportsâ¦ we made the whole conversion a lot >>> more robust since problems started to arise, but now I hit a wall I cannot >>> solve: I think problem is in something in callbacks. >>> >>> And problem is showing very easy on 64bits (while in 32bits it takes >>> time and is more random). >>> >> >> I responded in the "image not opening" thread, but it's the same >> problem. I really want to hear what y'all think because I'll happily >> implement a fix, but I want to know which one y'all think is a good idea. >> Here's my reply (edits between [ & ] to add information): >> >> >> On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> >> wrote: >> >> I'm pretty confident [I know] this is to do with bugs in the Athens >> surface code which assumes that callbacks can be made in the existing >> copyBits and warpBits primitive. They can't do this safely because a GC >> (scavenge) can happen during a callback, which then causes chaos when the >> copyBits primitive tries to access objects that have been moved under its >> feet. >> >> I've done work to fix callbacks so that when there is a failure it is the >> copyBits primitive that fails, instead of apparently the callback return >> primitive. One of the apparent effects of this fix is to stop the screen >> opening up too small; another is getting the background colour right, and >> yet another is eliminating bogus pixels in the VGTigerDemo demo. But more >> work is required to fix the copyBits and warpBits primitives. There are a >> few approaches one might take: >> >> a) fixing the primitive so that it saves and restores oops around the >> callbacks using the external oop table [InterpreterProxy>>addGCRoot: & >> removeGCRoot:]. That's a pain but possible. [It's a pain because all the >> derived pointers (the start of the destForm, sourceForm, halftoneForm and >> colorMapTable) must be recomputed also, and of course most of the time the >> objects don't move; we only scavenge about once every 2 seconds in normal >> running] >> >> b) fixing the primitive so that it pins the objects it needs before ever >> invoking a callback [this is a pain because pinning an object causes it to >> be tenured to old space if it is in new space; objects can't be pinned in >> new space, so instead the pin operation forwards the new space object to an >> old space copy if required and answers its location in old space, so a >> putative withPinnedObjectsDo: operation for the copyBits primitive looks >> like >> withPinnedFormsDo: aBlock >> <inline: #always> >> self cppIf: SPURVM & false >> ifTrue: >> [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | >> (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: >> [bitBltOop := interpreterProxy pinObject: bitBltOop]. >> (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: >> [destForm := interpreterProxy pinObject: destForm]. >> (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: >> [sourceForm := interpreterProxy pinObject: sourceForm]. >> (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: >> [halftoneForm := interpreterProxy pinObject: halftoneForm]. >> aBlock value. >> bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. >> destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. >> sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. >> halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] >> ifFalse: [aBlock value] >> and tenuring objects to old space is not ideal because they are only >> collected by a full GC, so doing this would at least tenure the bitBltOop >> which is very likely to be in new space] >> >> c) fixing the primitive so that it uses the scavenge and fullGC counters >> in the VM to detect if a GC occurred during one of the callbacks and would >> fail the primitive [if it detected that a GC had occurred in any of the >> surface functions]. The primitive would then simply be retried. >> >> d) ? >> > > Wouldn't it be possible to just pause the GC (scavange) when entering a > primitive ? > I don't think so. There is a callback occurring. If the computation executed by the callback requires a GC the application will abort if a GC cannot be done. Right? This is the case here. I like c) as it's very lightweight, but it has issues. It is fine to use >> for callbacks *before* cop[yBits and warpBits move any bits (the >> lockSurface and querySurface functions). But it's potentially erroneous >> after the unlockSurface primitive. For example, a primitive which does an >> xor with the screen can't simply be retried as the first, falling pass, >> would have updated the destination bits but not displayed them via >> unlockSurface. But I think it could be arranged that no objects are >> accessed after unlockSurface, which should naturally be the last call in >> the primitive (or do I mean showSurface?). So the approach would be to >> check for GCs occurring during querySurface and lockSurface, failing if so, >> and then caching any and all state needed by unlockSurface and showSurface >> in local variables. This way no object state is accessed to make the >> unlockSurface and showSurface calls, and no bits are moved before the >> queryDurface and lockSurface calls. >> >> If we used a failure code such as #'object may move' then the primitives >> could answer this when a GC during callbacks is detected and then the >> primitive could be retried only when required. >> >> >> [Come on folks, please comment. I want to know which idea you like >> best. We could fix this quickly. But right now it feels like I'm talking >> to myself.] >> >> >>> Here is the easiest way to reproduce it (in mac): >>> >>> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >>> wget files.pharo.org/get-files/60/pharo64.zip >>> wget files.pharo.org/get-files/60/sources.zip >>> unzip pharo64-mac-latest.zip >>> unzip pharo64.zip >>> unzip sources.zip >>> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval >>> "VGTigerDemo runDemo" >>> >>> eventually (like 5-6 seconds after, if not immediately), you will have a >>> stack like this: >>> >>> SmallInteger(Object)>>primitiveFailed: >>> SmallInteger(Object)>>primitiveFailed >>> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >>> GrafPort>>copyBits >>> GrafPort>>image:at:sourceRect:rule: >>> FormCanvas>>image:at:sourceRect:rule: >>> FormCanvas(Canvas)>>drawImage:at:sourceRect: >>> FormCanvas(Canvas)>>drawImage:at: >>> VGTigerDemo>>runDemo >>> VGTigerDemo class>>runDemo >>> UndefinedObject>>DoIt >>> OpalCompiler>>evaluate >>> OpalCompiler(AbstractCompiler)>>evaluate: >>> [ result := Smalltalk compiler evaluate: aStream. >>> self hasSessionChanged >>> ifFalse: [ self stdout >>> print: result; >>> lf ] ] in EvaluateCommandLineHandler>>evaluate: >>> in Block: [ result := Smalltalk compiler evaluate: aStream.... >>> BlockClosure>>on:do: >>> EvaluateCommandLineHandler>>evaluate: >>> EvaluateCommandLineHandler>>evaluateArguments >>> EvaluateCommandLineHandler>>activate >>> EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ aCommandLinehandler activateWith: commandLine ] in >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> in Block: [ aCommandLinehandler activateWith: commandLine ] >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >>> [ self >>> handleArgument: >>> (self arguments >>> ifEmpty: [ '' ] >>> ifNotEmpty: [ :arguments | arguments first ]) ] >>> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: >>> [ self... >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >>> PharoCommandLineHandler>>activate >>> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >>> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >>> >>> Any idea? >>> >>> thanks! >>> Esteban >> >> >> >> >> -- >> _,,,^..^,,,_ >> best, Eliot >> >> > > -- _,,,^..^,,,_ best, Eliot

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by ClÃ©ment Bera

On Tue, Mar 14, 2017 at 8:57 AM, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > 2017-03-14 16:56 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: > >> >> >> 2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: >> >>> >>> Hi Esteban, Hi Igor, Hi All, >>> >>> On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> Iâm tumbling into an error in Pharo, because we use callbacks >>>> intensively, in Athens(cairo)-to-World conversion in particular, and people >>>> is sending always their crash reportsâ¦ we made the whole conversion a lot >>>> more robust since problems started to arise, but now I hit a wall I cannot >>>> solve: I think problem is in something in callbacks. >>>> >>>> And problem is showing very easy on 64bits (while in 32bits it takes >>>> time and is more random). >>>> >>> >>> I responded in the "image not opening" thread, but it's the same >>> problem. I really want to hear what y'all think because I'll happily >>> implement a fix, but I want to know which one y'all think is a good idea. >>> Here's my reply (edits between [ & ] to add information): >>> >>> >>> On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> >>> wrote: >>> >>> I'm pretty confident [I know] this is to do with bugs in the Athens >>> surface code which assumes that callbacks can be made in the existing >>> copyBits and warpBits primitive. They can't do this safely because a GC >>> (scavenge) can happen during a callback, which then causes chaos when the >>> copyBits primitive tries to access objects that have been moved under its >>> feet. >>> >>> I've done work to fix callbacks so that when there is a failure it is >>> the copyBits primitive that fails, instead of apparently the callback >>> return primitive. One of the apparent effects of this fix is to stop the >>> screen opening up too small; another is getting the background colour >>> right, and yet another is eliminating bogus pixels in the VGTigerDemo >>> demo. But more work is required to fix the copyBits and warpBits >>> primitives. There are a few approaches one might take: >>> >>> a) fixing the primitive so that it saves and restores oops around the >>> callbacks using the external oop table [InterpreterProxy>>addGCRoot: & >>> removeGCRoot:]. That's a pain but possible. [It's a pain because all the >>> derived pointers (the start of the destForm, sourceForm, halftoneForm and >>> colorMapTable) must be recomputed also, and of course most of the time the >>> objects don't move; we only scavenge about once every 2 seconds in normal >>> running] >>> >>> b) fixing the primitive so that it pins the objects it needs before ever >>> invoking a callback [this is a pain because pinning an object causes it to >>> be tenured to old space if it is in new space; objects can't be pinned in >>> new space, so instead the pin operation forwards the new space object to an >>> old space copy if required and answers its location in old space, so a >>> putative withPinnedObjectsDo: operation for the copyBits primitive looks >>> like >>> withPinnedFormsDo: aBlock >>> <inline: #always> >>> self cppIf: SPURVM & false >>> ifTrue: >>> [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | >>> (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: >>> [bitBltOop := interpreterProxy pinObject: bitBltOop]. >>> (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: >>> [destForm := interpreterProxy pinObject: destForm]. >>> (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: >>> [sourceForm := interpreterProxy pinObject: sourceForm]. >>> (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: >>> [halftoneForm := interpreterProxy pinObject: halftoneForm]. >>> aBlock value. >>> bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. >>> destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. >>> sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. >>> halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] >>> ifFalse: [aBlock value] >>> and tenuring objects to old space is not ideal because they are only >>> collected by a full GC, so doing this would at least tenure the bitBltOop >>> which is very likely to be in new space] >>> >>> c) fixing the primitive so that it uses the scavenge and fullGC counters >>> in the VM to detect if a GC occurred during one of the callbacks and would >>> fail the primitive [if it detected that a GC had occurred in any of the >>> surface functions]. The primitive would then simply be retried. >>> >>> That's clearly the best solution unless someone figures out a better d) solution. > d) ? >>> >> >> Wouldn't it be possible to just pause the GC (scavange) when entering a >> primitive ? >> > > Wouldn't it be possible to just *disable* the GC (scavange) when entering > a primitive ? > If you disable only the scavenges and some of the objects are in old space and a fullGC happens, then the same problem happens. During the call-backs, an infinite amount of objects can be allocated. Where do you allocate such objects if you disable the GC ? > >> >> >> >> >>> >>> I like c) as it's very lightweight, but it has issues. It is fine to >>> use for callbacks *before* cop[yBits and warpBits move any bits (the >>> lockSurface and querySurface functions). But it's potentially erroneous >>> after the unlockSurface primitive. For example, a primitive which does an >>> xor with the screen can't simply be retried as the first, falling pass, >>> would have updated the destination bits but not displayed them via >>> unlockSurface. But I think it could be arranged that no objects are >>> accessed after unlockSurface, which should naturally be the last call in >>> the primitive (or do I mean showSurface?). So the approach would be to >>> check for GCs occurring during querySurface and lockSurface, failing if so, >>> and then caching any and all state needed by unlockSurface and showSurface >>> in local variables. This way no object state is accessed to make the >>> unlockSurface and showSurface calls, and no bits are moved before the >>> queryDurface and lockSurface calls. >>> >>> If we used a failure code such as #'object may move' then the primitives >>> could answer this when a GC during callbacks is detected and then the >>> primitive could be retried only when required. >>> >>> >>> [Come on folks, please comment. I want to know which idea you like >>> best. We could fix this quickly. But right now it feels like I'm talking >>> to myself.] >>> >>> >>>> Here is the easiest way to reproduce it (in mac): >>>> >>>> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >>>> wget files.pharo.org/get-files/60/pharo64.zip >>>> wget files.pharo.org/get-files/60/sources.zip >>>> unzip pharo64-mac-latest.zip >>>> unzip pharo64.zip >>>> unzip sources.zip >>>> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval >>>> "VGTigerDemo runDemo" >>>> >>>> eventually (like 5-6 seconds after, if not immediately), you will have >>>> a stack like this: >>>> >>>> SmallInteger(Object)>>primitiveFailed: >>>> SmallInteger(Object)>>primitiveFailed >>>> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >>>> GrafPort>>copyBits >>>> GrafPort>>image:at:sourceRect:rule: >>>> FormCanvas>>image:at:sourceRect:rule: >>>> FormCanvas(Canvas)>>drawImage:at:sourceRect: >>>> FormCanvas(Canvas)>>drawImage:at: >>>> VGTigerDemo>>runDemo >>>> VGTigerDemo class>>runDemo >>>> UndefinedObject>>DoIt >>>> OpalCompiler>>evaluate >>>> OpalCompiler(AbstractCompiler)>>evaluate: >>>> [ result := Smalltalk compiler evaluate: aStream. >>>> self hasSessionChanged >>>> ifFalse: [ self stdout >>>> print: result; >>>> lf ] ] in EvaluateCommandLineHandler>>evaluate: >>>> in Block: [ result := Smalltalk compiler evaluate: aStream.... >>>> BlockClosure>>on:do: >>>> EvaluateCommandLineHandler>>evaluate: >>>> EvaluateCommandLineHandler>>evaluateArguments >>>> EvaluateCommandLineHandler>>activate >>>> EvaluateCommandLineHandler class(CommandLineHandler >>>> class)>>activateWith: >>>> [ aCommandLinehandler activateWith: commandLine ] in >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>>> in Block: [ aCommandLinehandler activateWith: commandLine ] >>>> BlockClosure>>on:do: >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >>>> [ self >>>> handleArgument: >>>> (self arguments >>>> ifEmpty: [ '' ] >>>> ifNotEmpty: [ :arguments | arguments first ]) ] >>>> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in >>>> Block: [ self... >>>> BlockClosure>>on:do: >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >>>> PharoCommandLineHandler>>activate >>>> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >>>> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >>>> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >>>> >>>> Any idea? >>>> >>>> thanks! >>>> Esteban >>> >>> >>> >>> >>> -- >>> _,,,^..^,,,_ >>> best, Eliot >>> >>> >> > >

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Nicolai Hess

2017-03-14 16:56 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: > > > 2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > >> >> Hi Esteban, Hi Igor, Hi All, >> >> On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >> wrote: >> >>> >>> Hi, >>> >>> Iâm tumbling into an error in Pharo, because we use callbacks >>> intensively, in Athens(cairo)-to-World conversion in particular, and people >>> is sending always their crash reportsâ¦ we made the whole conversion a lot >>> more robust since problems started to arise, but now I hit a wall I cannot >>> solve: I think problem is in something in callbacks. >>> >>> And problem is showing very easy on 64bits (while in 32bits it takes >>> time and is more random). >>> >> >> I responded in the "image not opening" thread, but it's the same >> problem. I really want to hear what y'all think because I'll happily >> implement a fix, but I want to know which one y'all think is a good idea. >> Here's my reply (edits between [ & ] to add information): >> >> >> On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> >> wrote: >> >> I'm pretty confident [I know] this is to do with bugs in the Athens >> surface code which assumes that callbacks can be made in the existing >> copyBits and warpBits primitive. They can't do this safely because a GC >> (scavenge) can happen during a callback, which then causes chaos when the >> copyBits primitive tries to access objects that have been moved under its >> feet. >> >> I've done work to fix callbacks so that when there is a failure it is the >> copyBits primitive that fails, instead of apparently the callback return >> primitive. One of the apparent effects of this fix is to stop the screen >> opening up too small; another is getting the background colour right, and >> yet another is eliminating bogus pixels in the VGTigerDemo demo. But more >> work is required to fix the copyBits and warpBits primitives. There are a >> few approaches one might take: >> >> a) fixing the primitive so that it saves and restores oops around the >> callbacks using the external oop table [InterpreterProxy>>addGCRoot: & >> removeGCRoot:]. That's a pain but possible. [It's a pain because all the >> derived pointers (the start of the destForm, sourceForm, halftoneForm and >> colorMapTable) must be recomputed also, and of course most of the time the >> objects don't move; we only scavenge about once every 2 seconds in normal >> running] >> >> b) fixing the primitive so that it pins the objects it needs before ever >> invoking a callback [this is a pain because pinning an object causes it to >> be tenured to old space if it is in new space; objects can't be pinned in >> new space, so instead the pin operation forwards the new space object to an >> old space copy if required and answers its location in old space, so a >> putative withPinnedObjectsDo: operation for the copyBits primitive looks >> like >> withPinnedFormsDo: aBlock >> <inline: #always> >> self cppIf: SPURVM & false >> ifTrue: >> [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | >> (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: >> [bitBltOop := interpreterProxy pinObject: bitBltOop]. >> (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: >> [destForm := interpreterProxy pinObject: destForm]. >> (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: >> [sourceForm := interpreterProxy pinObject: sourceForm]. >> (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: >> [halftoneForm := interpreterProxy pinObject: halftoneForm]. >> aBlock value. >> bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. >> destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. >> sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. >> halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] >> ifFalse: [aBlock value] >> and tenuring objects to old space is not ideal because they are only >> collected by a full GC, so doing this would at least tenure the bitBltOop >> which is very likely to be in new space] >> >> c) fixing the primitive so that it uses the scavenge and fullGC counters >> in the VM to detect if a GC occurred during one of the callbacks and would >> fail the primitive [if it detected that a GC had occurred in any of the >> surface functions]. The primitive would then simply be retried. >> >> d) ? >> > > Wouldn't it be possible to just pause the GC (scavange) when entering a > primitive ? > Wouldn't it be possible to just *disable* the GC (scavange) when entering a primitive ? > > > > >> >> I like c) as it's very lightweight, but it has issues. It is fine to use >> for callbacks *before* cop[yBits and warpBits move any bits (the >> lockSurface and querySurface functions). But it's potentially erroneous >> after the unlockSurface primitive. For example, a primitive which does an >> xor with the screen can't simply be retried as the first, falling pass, >> would have updated the destination bits but not displayed them via >> unlockSurface. But I think it could be arranged that no objects are >> accessed after unlockSurface, which should naturally be the last call in >> the primitive (or do I mean showSurface?). So the approach would be to >> check for GCs occurring during querySurface and lockSurface, failing if so, >> and then caching any and all state needed by unlockSurface and showSurface >> in local variables. This way no object state is accessed to make the >> unlockSurface and showSurface calls, and no bits are moved before the >> queryDurface and lockSurface calls. >> >> If we used a failure code such as #'object may move' then the primitives >> could answer this when a GC during callbacks is detected and then the >> primitive could be retried only when required. >> >> >> [Come on folks, please comment. I want to know which idea you like >> best. We could fix this quickly. But right now it feels like I'm talking >> to myself.] >> >> >>> Here is the easiest way to reproduce it (in mac): >>> >>> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >>> wget files.pharo.org/get-files/60/pharo64.zip >>> wget files.pharo.org/get-files/60/sources.zip >>> unzip pharo64-mac-latest.zip >>> unzip pharo64.zip >>> unzip sources.zip >>> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval >>> "VGTigerDemo runDemo" >>> >>> eventually (like 5-6 seconds after, if not immediately), you will have a >>> stack like this: >>> >>> SmallInteger(Object)>>primitiveFailed: >>> SmallInteger(Object)>>primitiveFailed >>> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >>> GrafPort>>copyBits >>> GrafPort>>image:at:sourceRect:rule: >>> FormCanvas>>image:at:sourceRect:rule: >>> FormCanvas(Canvas)>>drawImage:at:sourceRect: >>> FormCanvas(Canvas)>>drawImage:at: >>> VGTigerDemo>>runDemo >>> VGTigerDemo class>>runDemo >>> UndefinedObject>>DoIt >>> OpalCompiler>>evaluate >>> OpalCompiler(AbstractCompiler)>>evaluate: >>> [ result := Smalltalk compiler evaluate: aStream. >>> self hasSessionChanged >>> ifFalse: [ self stdout >>> print: result; >>> lf ] ] in EvaluateCommandLineHandler>>evaluate: >>> in Block: [ result := Smalltalk compiler evaluate: aStream.... >>> BlockClosure>>on:do: >>> EvaluateCommandLineHandler>>evaluate: >>> EvaluateCommandLineHandler>>evaluateArguments >>> EvaluateCommandLineHandler>>activate >>> EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ aCommandLinehandler activateWith: commandLine ] in >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> in Block: [ aCommandLinehandler activateWith: commandLine ] >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >>> [ self >>> handleArgument: >>> (self arguments >>> ifEmpty: [ '' ] >>> ifNotEmpty: [ :arguments | arguments first ]) ] >>> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: >>> [ self... >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >>> PharoCommandLineHandler>>activate >>> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >>> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >>> >>> Any idea? >>> >>> thanks! >>> Esteban >> >> >> >> >> -- >> _,,,^..^,,,_ >> best, Eliot >> >> >

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Nicolai Hess

2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > > Hi Esteban, Hi Igor, Hi All, > > On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> > wrote: > >> >> Hi, >> >> Iâm tumbling into an error in Pharo, because we use callbacks >> intensively, in Athens(cairo)-to-World conversion in particular, and people >> is sending always their crash reportsâ¦ we made the whole conversion a lot >> more robust since problems started to arise, but now I hit a wall I cannot >> solve: I think problem is in something in callbacks. >> >> And problem is showing very easy on 64bits (while in 32bits it takes time >> and is more random). >> > > I responded in the "image not opening" thread, but it's the same > problem. I really want to hear what y'all think because I'll happily > implement a fix, but I want to know which one y'all think is a good idea. > Here's my reply (edits between [ & ] to add information): > > > On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> > wrote: > > I'm pretty confident [I know] this is to do with bugs in the Athens > surface code which assumes that callbacks can be made in the existing > copyBits and warpBits primitive. They can't do this safely because a GC > (scavenge) can happen during a callback, which then causes chaos when the > copyBits primitive tries to access objects that have been moved under its > feet. > > I've done work to fix callbacks so that when there is a failure it is the > copyBits primitive that fails, instead of apparently the callback return > primitive. One of the apparent effects of this fix is to stop the screen > opening up too small; another is getting the background colour right, and > yet another is eliminating bogus pixels in the VGTigerDemo demo. But more > work is required to fix the copyBits and warpBits primitives. There are a > few approaches one might take: > > a) fixing the primitive so that it saves and restores oops around the > callbacks using the external oop table [InterpreterProxy>>addGCRoot: & > removeGCRoot:]. That's a pain but possible. [It's a pain because all the > derived pointers (the start of the destForm, sourceForm, halftoneForm and > colorMapTable) must be recomputed also, and of course most of the time the > objects don't move; we only scavenge about once every 2 seconds in normal > running] > > b) fixing the primitive so that it pins the objects it needs before ever > invoking a callback [this is a pain because pinning an object causes it to > be tenured to old space if it is in new space; objects can't be pinned in > new space, so instead the pin operation forwards the new space object to an > old space copy if required and answers its location in old space, so a > putative withPinnedObjectsDo: operation for the copyBits primitive looks > like > withPinnedFormsDo: aBlock > <inline: #always> > self cppIf: SPURVM & false > ifTrue: > [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | > (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: > [bitBltOop := interpreterProxy pinObject: bitBltOop]. > (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: > [destForm := interpreterProxy pinObject: destForm]. > (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: > [sourceForm := interpreterProxy pinObject: sourceForm]. > (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: > [halftoneForm := interpreterProxy pinObject: halftoneForm]. > aBlock value. > bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. > destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. > sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. > halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] > ifFalse: [aBlock value] > and tenuring objects to old space is not ideal because they are only > collected by a full GC, so doing this would at least tenure the bitBltOop > which is very likely to be in new space] > > c) fixing the primitive so that it uses the scavenge and fullGC counters > in the VM to detect if a GC occurred during one of the callbacks and would > fail the primitive [if it detected that a GC had occurred in any of the > surface functions]. The primitive would then simply be retried. > > d) ? > Wouldn't it be possible to just pause the GC (scavange) when entering a primitive ? > > I like c) as it's very lightweight, but it has issues. It is fine to use > for callbacks *before* cop[yBits and warpBits move any bits (the > lockSurface and querySurface functions). But it's potentially erroneous > after the unlockSurface primitive. For example, a primitive which does an > xor with the screen can't simply be retried as the first, falling pass, > would have updated the destination bits but not displayed them via > unlockSurface. But I think it could be arranged that no objects are > accessed after unlockSurface, which should naturally be the last call in > the primitive (or do I mean showSurface?). So the approach would be to > check for GCs occurring during querySurface and lockSurface, failing if so, > and then caching any and all state needed by unlockSurface and showSurface > in local variables. This way no object state is accessed to make the > unlockSurface and showSurface calls, and no bits are moved before the > queryDurface and lockSurface calls. > > If we used a failure code such as #'object may move' then the primitives > could answer this when a GC during callbacks is detected and then the > primitive could be retried only when required. > > > [Come on folks, please comment. I want to know which idea you like best. > We could fix this quickly. But right now it feels like I'm talking to > myself.] > > >> Here is the easiest way to reproduce it (in mac): >> >> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >> wget files.pharo.org/get-files/60/pharo64.zip >> wget files.pharo.org/get-files/60/sources.zip >> unzip pharo64-mac-latest.zip >> unzip pharo64.zip >> unzip sources.zip >> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval "VGTigerDemo >> runDemo" >> >> eventually (like 5-6 seconds after, if not immediately), you will have a >> stack like this: >> >> SmallInteger(Object)>>primitiveFailed: >> SmallInteger(Object)>>primitiveFailed >> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >> GrafPort>>copyBits >> GrafPort>>image:at:sourceRect:rule: >> FormCanvas>>image:at:sourceRect:rule: >> FormCanvas(Canvas)>>drawImage:at:sourceRect: >> FormCanvas(Canvas)>>drawImage:at: >> VGTigerDemo>>runDemo >> VGTigerDemo class>>runDemo >> UndefinedObject>>DoIt >> OpalCompiler>>evaluate >> OpalCompiler(AbstractCompiler)>>evaluate: >> [ result := Smalltalk compiler evaluate: aStream. >> self hasSessionChanged >> ifFalse: [ self stdout >> print: result; >> lf ] ] in EvaluateCommandLineHandler>>evaluate: >> in Block: [ result := Smalltalk compiler evaluate: aStream.... >> BlockClosure>>on:do: >> EvaluateCommandLineHandler>>evaluate: >> EvaluateCommandLineHandler>>evaluateArguments >> EvaluateCommandLineHandler>>activate >> EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: >> [ aCommandLinehandler activateWith: commandLine ] in >> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: in >> Block: [ aCommandLinehandler activateWith: commandLine ] >> BlockClosure>>on:do: >> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >> [ self >> handleArgument: >> (self arguments >> ifEmpty: [ '' ] >> ifNotEmpty: [ :arguments | arguments first ]) ] >> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: >> [ self... >> BlockClosure>>on:do: >> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >> PharoCommandLineHandler>>activate >> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >> >> Any idea? >> >> thanks! >> Esteban > > > > > -- > _,,,^..^,,,_ > best, Eliot > >

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Eliot Miranda

Hi Esteban, Hi Igor, Hi All, On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > Hi, > > Iâm tumbling into an error in Pharo, because we use callbacks intensively, > in Athens(cairo)-to-World conversion in particular, and people is sending > always their crash reportsâ¦ we made the whole conversion a lot more robust > since problems started to arise, but now I hit a wall I cannot solve: I > think problem is in something in callbacks. > > And problem is showing very easy on 64bits (while in 32bits it takes time > and is more random). > I responded in the "image not opening" thread, but it's the same problem. I really want to hear what y'all think because I'll happily implement a fix, but I want to know which one y'all think is a good idea. Here's my reply (edits between [ & ] to add information): On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: I'm pretty confident [I know] this is to do with bugs in the Athens surface code which assumes that callbacks can be made in the existing copyBits and warpBits primitive. They can't do this safely because a GC (scavenge) can happen during a callback, which then causes chaos when the copyBits primitive tries to access objects that have been moved under its feet. I've done work to fix callbacks so that when there is a failure it is the copyBits primitive that fails, instead of apparently the callback return primitive. One of the apparent effects of this fix is to stop the screen opening up too small; another is getting the background colour right, and yet another is eliminating bogus pixels in the VGTigerDemo demo. But more work is required to fix the copyBits and warpBits primitives. There are a few approaches one might take: a) fixing the primitive so that it saves and restores oops around the callbacks using the external oop table [InterpreterProxy>>addGCRoot: & removeGCRoot:]. That's a pain but possible. [It's a pain because all the derived pointers (the start of the destForm, sourceForm, halftoneForm and colorMapTable) must be recomputed also, and of course most of the time the objects don't move; we only scavenge about once every 2 seconds in normal running] b) fixing the primitive so that it pins the objects it needs before ever invoking a callback [this is a pain because pinning an object causes it to be tenured to old space if it is in new space; objects can't be pinned in new space, so instead the pin operation forwards the new space object to an old space copy if required and answers its location in old space, so a putative withPinnedObjectsDo: operation for the copyBits primitive looks like withPinnedFormsDo: aBlock <inline: #always> self cppIf: SPURVM & false ifTrue: [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: [bitBltOop := interpreterProxy pinObject: bitBltOop]. (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: [destForm := interpreterProxy pinObject: destForm]. (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: [sourceForm := interpreterProxy pinObject: sourceForm]. (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: [halftoneForm := interpreterProxy pinObject: halftoneForm]. aBlock value. bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] ifFalse: [aBlock value] and tenuring objects to old space is not ideal because they are only collected by a full GC, so doing this would at least tenure the bitBltOop which is very likely to be in new space] c) fixing the primitive so that it uses the scavenge and fullGC counters in the VM to detect if a GC occurred during one of the callbacks and would fail the primitive [if it detected that a GC had occurred in any of the surface functions]. The primitive would then simply be retried. d) ? I like c) as it's very lightweight, but it has issues. It is fine to use for callbacks *before* cop[yBits and warpBits move any bits (the lockSurface and querySurface functions). But it's potentially erroneous after the unlockSurface primitive. For example, a primitive which does an xor with the screen can't simply be retried as the first, falling pass, would have updated the destination bits but not displayed them via unlockSurface. But I think it could be arranged that no objects are accessed after unlockSurface, which should naturally be the last call in the primitive (or do I mean showSurface?). So the approach would be to check for GCs occurring during querySurface and lockSurface, failing if so, and then caching any and all state needed by unlockSurface and showSurface in local variables. This way no object state is accessed to make the unlockSurface and showSurface calls, and no bits are moved before the queryDurface and lockSurface calls. If we used a failure code such as #'object may move' then the primitives could answer this when a GC during callbacks is detected and then the primitive could be retried only when required. [Come on folks, please comment. I want to know which idea you like best. We could fix this quickly. But right now it feels like I'm talking to myself.] > Here is the easiest way to reproduce it (in mac): > > wget files.pharo.org/get-files/60/pharo64-mac-latest.zip > wget files.pharo.org/get-files/60/pharo64.zip > wget files.pharo.org/get-files/60/sources.zip > unzip pharo64-mac-latest.zip > unzip pharo64.zip > unzip sources.zip > ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval "VGTigerDemo > runDemo" > > eventually (like 5-6 seconds after, if not immediately), you will have a > stack like this: > > SmallInteger(Object)>>primitiveFailed: > SmallInteger(Object)>>primitiveFailed > SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: > GrafPort>>copyBits > GrafPort>>image:at:sourceRect:rule: > FormCanvas>>image:at:sourceRect:rule: > FormCanvas(Canvas)>>drawImage:at:sourceRect: > FormCanvas(Canvas)>>drawImage:at: > VGTigerDemo>>runDemo > VGTigerDemo class>>runDemo > UndefinedObject>>DoIt > OpalCompiler>>evaluate > OpalCompiler(AbstractCompiler)>>evaluate: > [ result := Smalltalk compiler evaluate: aStream. > self hasSessionChanged > ifFalse: [ self stdout > print: result; > lf ] ] in EvaluateCommandLineHandler>>evaluate: > in Block: [ result := Smalltalk compiler evaluate: aStream.... > BlockClosure>>on:do: > EvaluateCommandLineHandler>>evaluate: > EvaluateCommandLineHandler>>evaluateArguments > EvaluateCommandLineHandler>>activate > EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: > [ aCommandLinehandler activateWith: commandLine ] in > PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: in > Block: [ aCommandLinehandler activateWith: commandLine ] > BlockClosure>>on:do: > PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: > PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand > PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: > [ self > handleArgument: > (self arguments > ifEmpty: [ '' ] > ifNotEmpty: [ :arguments | arguments first ]) ] in > PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: [ > self... > BlockClosure>>on:do: > PharoCommandLineHandler(BasicCommandLineHandler)>>activate > PharoCommandLineHandler>>activate > PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: > [ super activateWith: aCommandLine ] in PharoCommandLineHandler > class>>activateWith: in Block: [ super activateWith: aCommandLine ] > > Any idea? > > thanks! > Esteban -- _,,,^..^,,,_ best, Eliot

March 14, 2017

Re: [Pharo-dev] How to listen for windows messages?

by Eliot Miranda

Hi Phil, Hi All, On Tue, Feb 28, 2017 at 4:29 AM, phil(a)highoctane.be <phil(a)highoctane.be> wrote: > Hi, > > Pharo is itself a Windows application (under Windows of course). > > So Window event loop is picking up the events and transforming them into > Pharo events. > Right, but this is *not good enough* :-) It is a fundamentally different architecture. The processing of Windows events in the Windows VM transforms *event callbacks* into an *event queue*. This is fine for events such as mouse clicks, keyboard presses, etc. But it is fundamentally broken for events such as those that ask an application to quit because the OS is about to exit, or events that try to obtain mouse feedback while moving a native window, etc, etc. So one either needs to extend the VM event queue so that one can install callbacks for certain kinds of events still providing a cross-platform interface), or, as Vassili did for Newspeak native windows (which was fully working in 2008, with the ability to switch a window between emulated (Morphic) and native at will or on image startup), interface to the native Windows event pump via callbacks. > You can check this in https://github.com/pharo-project/pharo-vm > > in opensmalltalk-vm\platforms\win32\vm\sqWin32Window.c > > https://github.com/pharo-project/pharo-vm/blob/master/ > opensmalltalk-vm/platforms/win32/vm/sqWin32Window.c > > Like L235... > > LRESULT CALLBACK MainWndProcW(HWND hwnd, > UINT message, > WPARAM wParam, > LPARAM lParam) > > You'll see that we stick a lot of stuff into evt->... in various ways. > > There is a default fallback at the end. > > default: > /* Unprocessed messages may be processed outside the current > module. If firstMessageHook is non-NULL and returns a non > zero value, the message has been successfully processed */ > if(firstMessageHook) > if((*firstMessageHook)(hwnd, message, wParam, lParam)) > return 1; > return DefWindowProcW(hwnd,message,wParam,lParam); > > { > > > So, if firstMessageHook exists, one can do whatever. > > Now, a Windows manager may need other stuff and so on. > What you are looking at is how to hook this VM side with more of the > general Windows system. > > These hooks are globals, so should be accessible. > > /*********************************************************** > *****************/ > /* Message Processing */ > /*********************************************************** > *****************/ > /* The last dispatched event. It is used for the event processing > mechanism. */ > MSG *lastMessage = NULL; > /* The entry to the message hooks called from the window procedure. > If another module requires to process messages by itself, it should > put its message procedure in this place. */ > messageHook firstMessageHook = 0; > /* The entry to a pre-message hook. Can be used to intercept any messages > to the squeak main window. Useful for modules that wish to be notified > about certain messages before they are processed. */ > messageHook preMessageHook = 0; > > Never played with these but any Windows integration is of interest to me. > > So, go ahead. Nothing would resist focused work. > > Be aware that there is another VM underway with a cleanup of all of this > but this is not released yet. > > Keep us posted. > Phil > > > > > On Tue, Feb 28, 2017 at 9:33 AM, Torsten Bergmann <astares(a)gmx.de> wrote: > > > > Hi, > > > > I guess what you want to achieve will not be an easy task for you if you > are new to programming. > > But it is always good to have a goal and if you have time to learn I'm > pretty sure you will master > > it. > > > > Even when OS-Windows inspire you for automizing tasks in other Windows > processes (like autoit) you > > should not start with my OS-Windows project directly as contributing to > it requires knowledge > > on your side on Smalltalk as well as Win32 C programming. > > > > First start with learning Smalltalk and Pharo > > - http://files.pharo.org/books/ > > - http://pharo.pharocloud.com/pharobooks > > - http://stephane.ducasse.free.fr/FreeBooks.html > > > > After having an idea about Smalltalk and Pharo you should try to learn > UFFI which > > is Pharos unified foreign function interface and ability to call C DLL's. > > > > https://ci.inria.fr/pharo-contribution/view/Books/job/ > PharoBookWorkInProgress/lastSuccessfulBuild/artifact/ > book-result/UnifiedFFI/ > > > > Sideways start learning about the Win32 API and C as this helps to > better understand > > how things are done on the Windows side. There are many, many tutorials > out there. > > Try to call or wrap some simple DLL functions on you own first. > > > > There is a nice and tiny C compiler if you want to try out some C samples > > http://www.pellesc.de/index.php?page=download&lang=en > > > > With this initial knowledge you can try to understand why and how > OS-Windows was done. > > I wrote many tests - just add a breakpoint and run them to debug through > the code to understand. > > Google for the API descriptions of the Win32 functions that are called > to get an understanding > > how Pharo and C get connected. > > > > Also check how you work with callbacks in UFFI (as this would be > required for a hook). > > > > If you are then still interested in interception windows messages you > should start reading here > > https://msdn.microsoft.com/en-us/library/windows/desktop/ > ms632589(v=vs.85).aspx > > > > Primarily one has to wrap SetWindowsHookEx API function and friends and > wrap a good API > > for them in Pharo. > > > > Hope that helps to get started. > > > > Regards > > Torsten > > > > > Gesendet: Dienstag, 28. Februar 2017 um 06:04 Uhr > > > Von: lw1990 <lukewallace1990(a)gmail.com> > > > An: pharo-dev(a)lists.pharo.org > > > Betreff: [Pharo-dev] How to listen for windows messages? > > > > > > In the OS-Windows package, there exists the ability to use the windows > api in > > > Pharo. > > > This is very powerful, and I intend to add more of the windows api > into it > > > (it's only partially implemented). > > > > > > This appears to all have been done with DllCalls (FFI calls) so far. > > > > > > A big part of interacting with windows is listening to/intercepting and > > > responding to windows messages. > > > For example, every time a user presses a key on the keyboard, a windows > > > message will happen in the background for that key. If Pharo was aware > of > > > these messages, then Pharo could do things in response to hotkeys > pressed on > > > Windows (outside of a pharo window). > > > > > > It would also make reacting to events potentially nicer. Like making a > > > windows-desktop-manager in Pharo that can tell when a new window is > created > > > by windows (maybe they opened notepad). It would certainly be better > than an > > > infinite loop or timer of 'get active window and compare to last active > > > window and see if it changed'. Instead it would be 'when receive the > > > WM_MESSAGE for new window created, notify Pharo so it can react'. > > > > > > How can I set up a windows message hook in Pharo? > > > Please keep in mind I'm very new to programming, but I work from home > and > > > have lots of time to learn :-) > > > > > > > > > > > > -- > > > View this message in context: http://forum.world.st/How-to- > listen-for-windows-messages-tp4936285.html > > > Sent from the Pharo Smalltalk Developers mailing list archive at > Nabble.com. > > > > > > > > > -- _,,,^..^,,,_ best, Eliot

March 14, 2017

Re: [Pharo-dev] How to listen for microsoft os windows messages (WM_MESSAGE's)?

by Esteban Lorenzano

you cannot use QT out of the box because is C++ I have some experiments with GTK3 that were working but event loop was kinda failing (no time to go back to it soon). It would be very interesting to have native windows support, I may take a look eventually :) Esteban > On 14 Mar 2017, at 13:37, Dimitris Chloupis <kilon.alios(a)gmail.com> wrote: > > > > There are other cross platform APIs e.g. wxWidgets and Gtk > > There used to be a version of squeak with wx bindings wxSqueak but the > code seems not to be on line now. > > > GTK is not a very good choice, it works well on Linux but on MacOS and Windows has many issues and a lot less developers bug fixing. > > wxWidgets is a nice GUI API but again it does not come remotely close to the feature set and stability of QT. Some of its areas also are abandoned or very much lagging behind. In my case a big turn off was the outdated python wrappers which sent me directly back to QT. > > When it comes to guis No1 is Native GUIs , No2 is QT. > > But QT needs a commercial license for developing pro applications. Its free license is GLP which limits the usage on commercial projects. > > It does however offer a free LGPL version with more limited features. LGPL allows to close code as long as the code is distributed via dynamically linked libraries. > > Still it may be tricky to balance on the LGPL without falling down. Which is why many like us prefer MIT over LGPL. > > There are a ton of open source GUI APIs out there, but most of them are immature, limited or plain abandoned. So you have plenty of choice, but I think you will have little reason to prefer something that is not QT, especially if we are talking about commercial projects that need to run on many platforms.

March 14, 2017

[pharo-project/pharo-core]

by GitHub

Branch: refs/tags/60441 Home: https://github.com/pharo-project/pharo-core

March 14, 2017