March 2017 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] How to listen for windows messages?

by phil＠highoctane.be

On Tue, Mar 14, 2017 at 4:27 PM, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > Hi Phil, Hi All, > > On Tue, Feb 28, 2017 at 4:29 AM, phil(a)highoctane.be <phil(a)highoctane.be> > wrote: > >> Hi, >> >> Pharo is itself a Windows application (under Windows of course). >> >> So Window event loop is picking up the events and transforming them into >> Pharo events. >> > > Right, but this is *not good enough* :-) It is a fundamentally different > architecture. The processing of Windows events in the Windows VM > transforms *event callbacks* into an *event queue*. This is fine for > events such as mouse clicks, keyboard presses, etc. But it is > fundamentally broken for events such as those that ask an application to > quit because the OS is about to exit, or events that try to obtain mouse > feedback while moving a native window, etc, etc. > > So one either needs to extend the VM event queue so that one can install > callbacks for certain kinds of events still providing a cross-platform > interface), or, as Vassili did for Newspeak native windows (which was fully > working in 2008, with the ability to switch a window between emulated > (Morphic) and native at will or on image startup), interface to the native > Windows event pump via callbacks. > Yes, I understand that. But there is some bit rot in the basics already, so, first things first. And Pharo lowcode VM is also replacing a lot of this with other ways (which kind of bury the Windows thing even deeper in way I guess). Would the VM core be a libray, we could have more options. Look at V8 ending up in Chrome, NodeJS and Electron. Or Lua getting all over the place. Tcl also is pretty cool on that front. Phil > > > >> You can check this in https://github.com/pharo-project/pharo-vm >> >> in opensmalltalk-vm\platforms\win32\vm\sqWin32Window.c >> >> https://github.com/pharo-project/pharo-vm/blob/master/opensm >> alltalk-vm/platforms/win32/vm/sqWin32Window.c >> >> Like L235... >> >> LRESULT CALLBACK MainWndProcW(HWND hwnd, >> UINT message, >> WPARAM wParam, >> LPARAM lParam) >> >> You'll see that we stick a lot of stuff into evt->... in various ways. >> >> There is a default fallback at the end. >> >> default: >> /* Unprocessed messages may be processed outside the current >> module. If firstMessageHook is non-NULL and returns a non >> zero value, the message has been successfully processed */ >> if(firstMessageHook) >> if((*firstMessageHook)(hwnd, message, wParam, lParam)) >> return 1; >> return DefWindowProcW(hwnd,message,wParam,lParam); >> >> { >> >> >> So, if firstMessageHook exists, one can do whatever. >> >> Now, a Windows manager may need other stuff and so on. >> What you are looking at is how to hook this VM side with more of the >> general Windows system. >> >> These hooks are globals, so should be accessible. >> >> /*********************************************************** >> *****************/ >> /* Message Processing */ >> /*********************************************************** >> *****************/ >> /* The last dispatched event. It is used for the event processing >> mechanism. */ >> MSG *lastMessage = NULL; >> /* The entry to the message hooks called from the window procedure. >> If another module requires to process messages by itself, it should >> put its message procedure in this place. */ >> messageHook firstMessageHook = 0; >> /* The entry to a pre-message hook. Can be used to intercept any messages >> to the squeak main window. Useful for modules that wish to be notified >> about certain messages before they are processed. */ >> messageHook preMessageHook = 0; >> >> Never played with these but any Windows integration is of interest to me. >> >> So, go ahead. Nothing would resist focused work. >> >> Be aware that there is another VM underway with a cleanup of all of this >> but this is not released yet. >> >> Keep us posted. >> Phil >> >> >> >> >> On Tue, Feb 28, 2017 at 9:33 AM, Torsten Bergmann <astares(a)gmx.de> wrote: >> > >> > Hi, >> > >> > I guess what you want to achieve will not be an easy task for you if >> you are new to programming. >> > But it is always good to have a goal and if you have time to learn I'm >> pretty sure you will master >> > it. >> > >> > Even when OS-Windows inspire you for automizing tasks in other Windows >> processes (like autoit) you >> > should not start with my OS-Windows project directly as contributing to >> it requires knowledge >> > on your side on Smalltalk as well as Win32 C programming. >> > >> > First start with learning Smalltalk and Pharo >> > - http://files.pharo.org/books/ >> > - http://pharo.pharocloud.com/pharobooks >> > - http://stephane.ducasse.free.fr/FreeBooks.html >> > >> > After having an idea about Smalltalk and Pharo you should try to learn >> UFFI which >> > is Pharos unified foreign function interface and ability to call C >> DLL's. >> > >> > https://ci.inria.fr/pharo-contribution/view/Books/job/PharoB >> ookWorkInProgress/lastSuccessfulBuild/artifact/book-result/UnifiedFFI/ >> > >> > Sideways start learning about the Win32 API and C as this helps to >> better understand >> > how things are done on the Windows side. There are many, many tutorials >> out there. >> > Try to call or wrap some simple DLL functions on you own first. >> > >> > There is a nice and tiny C compiler if you want to try out some C >> samples >> > http://www.pellesc.de/index.php?page=download&lang=en >> > >> > With this initial knowledge you can try to understand why and how >> OS-Windows was done. >> > I wrote many tests - just add a breakpoint and run them to debug >> through the code to understand. >> > Google for the API descriptions of the Win32 functions that are called >> to get an understanding >> > how Pharo and C get connected. >> > >> > Also check how you work with callbacks in UFFI (as this would be >> required for a hook). >> > >> > If you are then still interested in interception windows messages you >> should start reading here >> > https://msdn.microsoft.com/en-us/library/windows/desktop/ms6 >> 32589(v=vs.85).aspx >> > >> > Primarily one has to wrap SetWindowsHookEx API function and friends and >> wrap a good API >> > for them in Pharo. >> > >> > Hope that helps to get started. >> > >> > Regards >> > Torsten >> > >> > > Gesendet: Dienstag, 28. Februar 2017 um 06:04 Uhr >> > > Von: lw1990 <lukewallace1990(a)gmail.com> >> > > An: pharo-dev(a)lists.pharo.org >> > > Betreff: [Pharo-dev] How to listen for windows messages? >> > > >> > > In the OS-Windows package, there exists the ability to use the >> windows api in >> > > Pharo. >> > > This is very powerful, and I intend to add more of the windows api >> into it >> > > (it's only partially implemented). >> > > >> > > This appears to all have been done with DllCalls (FFI calls) so far. >> > > >> > > A big part of interacting with windows is listening to/intercepting >> and >> > > responding to windows messages. >> > > For example, every time a user presses a key on the keyboard, a >> windows >> > > message will happen in the background for that key. If Pharo was >> aware of >> > > these messages, then Pharo could do things in response to hotkeys >> pressed on >> > > Windows (outside of a pharo window). >> > > >> > > It would also make reacting to events potentially nicer. Like making a >> > > windows-desktop-manager in Pharo that can tell when a new window is >> created >> > > by windows (maybe they opened notepad). It would certainly be better >> than an >> > > infinite loop or timer of 'get active window and compare to last >> active >> > > window and see if it changed'. Instead it would be 'when receive the >> > > WM_MESSAGE for new window created, notify Pharo so it can react'. >> > > >> > > How can I set up a windows message hook in Pharo? >> > > Please keep in mind I'm very new to programming, but I work from home >> and >> > > have lots of time to learn :-) >> > > >> > > >> > > >> > > -- >> > > View this message in context: http://forum.world.st/How-to-l >> isten-for-windows-messages-tp4936285.html >> > > Sent from the Pharo Smalltalk Developers mailing list archive at >> Nabble.com. >> > > >> > > >> > >> > > > > -- > _,,,^..^,,,_ > best, Eliot >

March 14, 2017

Type Inference tools

by Gus

Hi, I am working on a project of type inference, and i have the following doubt Are there currently any tools in pharo that use this information (perhaps in any library)? For example: a better autocomplete where you can chose different types, or tool that show you the type of a variable. I've already downloaded Roel Typer but i couldn't find any of those tools. Any help would be appreciated. Thanks -- View this message in context: http://forum.world.st/Type-Inference-tools-tp4938677.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.

March 14, 2017

Re: [Pharo-dev] PharoSpur32Vm

by Nicolas Cellier

2017-03-14 9:30 GMT+01:00 Nicolas Cellier < nicolas.cellier.aka.nice(a)gmail.com>: > > > 2017-03-14 8:58 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: > >> >> >> 2017-03-11 10:01 GMT+01:00 Nicolas Cellier <nicolas.cellier.aka.nice@gmai >> l.com>: >> >>> Hi Nicolai, >>> >>> If you look at appveyor.yml configuration on >>> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/.appveyor.yml, >>> you will see that the pharo brand is not yet in the matrix. >>> >>> It's because builds are based on cygwin, but as I understood, some >>> library used by some plugin required by pharo refuse to compile with >>> cygwin. They appear to work with mingw32. >>> Cygwin provides headers for legacy directx, some distribution of mingw32 >>> did in the past, but it does not seem the case anymore. >>> Using the directx headers from Microsoft SDK is a problem. They are not >>> redistributable and can't be found anymore on the net (too old). We cannot >>> seriously base the builds on something so fragile (both technically and >>> legally) in the long term. >>> >>> Also, the 64 bits VM does only work with clang, and we don't have >>> anything available as a 64bits Microsoft SDK... So pharo has to fix that >>> too. >>> >>> In the interim, you should look at https://github.com/pharo-proje >>> ct/pharo-vm/blob/master/.appveyor.yml and follow the scripts there. >>> >> >> Ok, thank you. >> >> > > I gave it a shot on sunday, because it was particularly rainy in Nantes, > and I almost succeeded in compiling all the dependencies with cygwin. > Well, I mean with autotools cmake libtool pkg-config and I surely forget a > few other niceties that some not so well informed programmers committed > with the faith that it would make their life "easier". It certainly does > not make mine simpler... > Almost, because gcc 5.4.0 failed to compile cairo with ssize_t: it seems > that the workaround of Igor does not work anymore. > ssize_t, WTF??? > Maybe I'll be able to fix it tonight. Or tomorrow. In which case I'll > publish the branch to see how far appveyor goes. > > > So I solved the ssize_t problem by removing the hack from Igor which is not necessary anymore... But got another problem soon after while building the tests... There are trailing lines generated at end of tests/cairo-test-constructors.c that make the compilation fail: i686-w64-mingw32-gcc -DHAVE_CONFIG_H -I. -I.. -I. -I./pdiff -I../boilerplate -I../util/cairo-missing -I../util/cairo-script -I../src -I../src -D_REENTRANT -I/cygdrive/y/Smalltalk/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/pixman-1 -I/cygdrive/y/Smalltalk/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/libpng16 -Wall -Wextra -Wmissing-declarations -Werror-implicit-function-declaration -Wpointer-arith -Wwrite-strings -Wsign-compare -Wpacked -Wswitch-enum -Wmissing-format-attribute -Wvolatile-register-var -Wstrict-aliasing=2 -Winit-self -Wunsafe-loop-optimizations -Wno-missing-field-initializers -Wno-unused-parameter -Wno-attributes -Wno-long-long -Winline -fno-strict-aliasing -fno-common -Wp,-D_FORTIFY_SOURCE=2 -Wno-unused-but-set-variable -D_REENTRANT -m32 -static-libgcc -static-libstdc++ -I/cygdrive/y/Smalltalk/opensmalltalk-vm/.thirdparty-cache/windows/i386/include -march=pentium4 -c -o cairo_test_suite-cairo-test-constructors.o `test -f 'cairo-test-constructors.c' || echo './'`cairo-test-constructors.c cairo-test-constructors.c:1118:1: attention : la dÃ©finition de donnÃ©es n'a pas de type ni de classe de stockage oning (); ^ cairo-test-constructors.c:1118:1: attention : type defaults to âintâ in declaration of âoningâ [-Wimplicit-int] cairo-test-constructors.c:1119:5: attention : la dÃ©finition de donnÃ©es n'a pas de type ni de classe de stockage _register_ft_show_glyphs_table (); ^ And the file looks like it has obviously been overwritten, but not truncated !!! ... extern void _register_multi_page (void); extern void _register_fallback_resolution (void); void _cairo_test_runner_register_tests (void) { _register_a1_bug (); _register_a1_clip_paint (); ... _register_multi_page (); _register_fallback_resolution (); } *oning (); _register_ft_show_glyphs_table (); _register_ft_text_vertical_layout_type1 (); _register_ft_text_vertical_layout_type3 (); _register_ft_text_antialias_none (); _register_ps_eps (); _register_ps_features (); _register_ps_surface_source (); _register_svg_surface (); _register_svg_clip (); _register_svg_surface_source (); _register_multi_page (); _register_fallback_resolution ();}* This file is generated by a shell script test/make-cairo-test-constructors.sh I can't find any reference of the bug, and upgrade to version 1.14.8 does not solve the issue. So it will wait until tomorrow... Nicolas >>> I hope that Esteban will find time to resolve all these problems and >>> have pharo brand back on opensmalltalk-vm. I guess that any form of help is >>> welcome. >>> >>> Nicolas >>> >>> >>> 2017-03-11 8:33 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: >>> >>>> I still have problems building a vm on windows. >>>> can you give me some hints how to start ? >>>> I cloned the recent pharo-vm project, >>>> in opensmalltalk-vm\build.win32x86\pharo.cog.spur\ >>>> run >>>> mvm >>>> >>>> But I got a couple of problems (mingw-32 compiler commands not found, I >>>> had to adjust the include path for finding directx header, missing variable >>>> replacement for git-versions). >>>> So I may miss some important steps. >>>> Is there a repository where I can clone the mingw environment used to >>>> build the win32-pharo-vm, the environment used on the build-server ? >>>> >>>> Thanks >>>> Nicolai >>>> >>>> 2017-02-04 1:50 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: >>>> >>>>> >>>>> >>>>> 2017-02-04 1:44 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: >>>>> >>>>>> >>>>>> >>>>>> 2017-01-23 8:59 GMT+01:00 Esteban Lorenzano <estebanlm(a)gmail.com>: >>>>>> >>>>>>> >>>>>>> On 22 Jan 2017, at 13:19, Nicolai Hess <nicolaihess(a)gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2017-01-22 10:21 GMT+01:00 ClÃ©ment Bera <bera.clement(a)gmail.com>: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I believe they're built from* https://github.com/OpenSmalltalk/vm >>>>>>>> <https://github.com/OpenSmalltalk/vm>* using travis and appveyor. >>>>>>>> On the gitbhub readme there are relevant links. All built artifacts are >>>>>>>> also kept on bintray for history. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Thank you! >>>>>>> >>>>>>> >>>>>>> no, they arenât :) >>>>>>> instead, they are built here: https://github.com/pharo >>>>>>> -project/pharo-vm >>>>>>> >>>>>>> (README still not updated) >>>>>>> >>>>>> >>>>>> what did changed ? I am not able to build the vm on windows anymore >>>>>> (something wrong with generating the generator.image, I'll now reset my >>>>>> local pharo-vm build directory and see if it works afterwards). >>>>>> >>>>> >>>>> see attached the stderr log : >>>>> >>>>> 'Errors in script loaded from u:\github\pharo-vm\scripts\Loa >>>>> dVMMaker.st' >>>>> [31mMessageNotUnderstood: receiver of "default:" is nil >>>>> [0mUndefinedObject(Object)>>doesNotUnderstand: #default: >>>>> BaseSoundSystem class>>initialize >>>>> MCMethodDefinition>>postloadOver: >>>>> >>>>> .... >>>>> >>>>> >>>>>> Are we still working with branch spur-64, or are we back on master ? >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Esteban >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Sun, Jan 22, 2017 at 9:25 AM, Nicolai Hess < >>>>>>>> nicolaihess(a)gmail.com> wrote: >>>>>>>> >>>>>>>>> Where are the latest Pharo-spur-vms (32bit) are built? >>>>>>>>> I don't see them on the build server, only the buildresults at >>>>>>>>> http://files.pharo.org/vm/pharo-spur32/linux/ >>>>>>>>> >>>>>>>>> The latest builds on the buildserver are from the last year only. >>>>>>>>> >>>>>>>>> nicolai >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >

March 14, 2017

Re: [Pharo-dev] [Moose-dev] Too frequent crashes :-(

by Alexandre Bergel

Hi Ronie, Below you said: > The problem with asForm, is that the form returned can be used after the cairo surface which holds the pixels is garbage collected. I cannot reproduce this case. You gave a script: > surface := AthensCairoSurface extent: 640@480. > surface drawDuring: [ :canvas | > surface clear: Color blue. > ]. > > surface asForm asMorph openInWindow I see a blue window. Cheers, Alexandre > How about changing AthensCairoSurface >> asForm into the following definition?: > asForm > > "create a form and copy an image data there" > | form | > self checkSession. > > self flush. > form := Form extent: (self width@self height) depth: 32. > form unhibernate. > LibC memCopy: self getDataPtr to: form bits size: self width*self height*4. > ^ form > > This involves a whole copy, but it removes completely the dependency on the surface plugin. > > If we want to keep using the surface plugin with Cairo, the old definition of asForm still has a bug, which can be reproduced with the following snippet: > > Smalltalk compiler evaluate: ' > | surface | > surface := AthensCairoSurface extent: 640@480. > surface drawDuring: [ :canvas | > surface clear: Color blue. > ]. > > surface asForm asMorph openInWindow > ' > > The problem with asForm, is that the form returned can be used after the cairo surface which holds the pixels is garbage collected.

March 14, 2017

Re: [Pharo-dev] image not opening.

by stepharong

thanks eliot for your analysis. Stef > > > On Sun, Mar 12, 2017 at 3:04 AM, Guillermo Polito > <guillermopolito(a)gmail.com> wrote: >> Yeh, to be more precise,* the problem is that the image somehow got >> it's display size modified to a very small one.* And that happenned >> while building an image in the CI in headless mode.* If the same image >> is built locally in a vm launched in non-headless mode, the issue >> cannot be reproduced. >> >> So my conjecture is that >> - either the image is changing the display size while loading pillar >> (less likely and not reproduceable locally) >> - either the latest stable VM messes the display size in the image >> header when running on headless (this is yet to reproduce) > > I'm pretty confident this is to do with bugs in the Athens surface code > which assumes that callbacks can be made in the existing copyBits and > warpBits primitive. They can't do >this safely because a GC (scavenge) > can happen during a callback, which then causes chaos when the copyBits > primitive tries to access objects that have been moved under >its feet. > > I've done work to fix callbacks so that when there is a failure it is > the copyBits primitive that fails, instead of apparently the callback > return primitive. One of the apparent effects of >this fix is to stop > the screen opening up too small; another is getting the background > colour right, and yet another is eliminating bogus pixels in the > VGTigerDemo demo. But >more work is required to fix the copyBits and > warpBits primitives. There are a few approaches one might take: > > a) fixing the primitive so that it saves and restores oops around the > callbacks using the external pop table. That's a pain but possible. > > b) fixing the primitive so that it pins the objects it needs before ever > invoking a callback > > c) fixing the primitive so that it uses the scavenge and fullGC counters > in the VM to detect if a GC occurred during one of the callbacks and > would fail the primitive. The primitive >would then simply be retried. > d) ? > > I like c) as it's very lightweight, but it has issues. It is fine to > use for callbacks *before* cop[yBits and warpBits move any bits (the > lockSurface and querySurface functions). But >it's potentially > erroneous after the unlockSurface primitive. For example, a primitive > which does an xor with the screen can't simply be retried as the first, > falling pass, would have >updated the destination bits but not displayed > them via unlockSurface. But I think it could be arranged that no > objects are accessed after unlockSurface, which should naturally >be the > last call in the primitive (or do I mean showSurface?). So the approach > would be to check for GCs occurring during querySurface and lockSurface, > failing if so, and then >caching any and all state needed by > unlockSurface and showSurface in local variables. This way no object > state is accessed to make the unlockSurface and showSurface >calls, and > no bits are moved before the queryDurface and lockSurface calls. > > If we used a failure code such as #'object may move' then the primitives > could answer this when a GC during callbacks is detected and then the > primitive could be retried only >when required. > > >> >> On Sun, Mar 12, 2017 at 10:52 AM, stepharong <stepharong(a)free.fr> wrote: >>> guillermo was suspecting a problem with the interpretatin of the image >>> metadata >>> >>>> Maybe related? >>>> http://forum.world.st/BUG-A-problem-with-callbacks-that-shows-up-in-64bits-… >>>> >>>> On Sat, Mar 11, 2017 at 2:22 AM, Cyril Ferlicot >>>> <cyril.ferlicot(a)gmail.com> wrote: >>>>> >>>>> On ven. 10 mars 2017 at 18:42, stepharong <stepharong(a)free.fr> wrote: >>>>>> >>>>>> We found the problem with Guillermo and this may be a problem of >>>>>> the last >>>>>> VM. >>>>>> >>>>>> The image got its starting window size really small: some pixels on >>>>>> some >>>>>> pixles. >>>>>> So may be the metadata of the image are systematically corrupted. >>>>>> >>>>>> Hi >>>>>> >>>>>> I have a jenkins that produce an image that does not start >>>>>> when I try to open it locally. >>>>>> >>>>>> Now if I take the image and load the "same" configuration I get a >>>>>> working >>>>>> image >>>>>> >>>>>> I have not idea how to debug that. >>>>>> >>>>>> Stef >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Using Opera's mail client: http://www.opera.com/mail/ >>>>> >>>>> >>>>> If it can help: here is the build: >>>>> >>>>> https://ci.inria.fr/pharo-contribution/job/OOnoz/ >>>>> >>>>> It is happening in almost all Pharo 6 builds >>>>> -- >>>>> Cheers >>>>> Cyril Ferlicot >>>> >>>>>>> >>> >>>>>> -- >>> Using Opera's mail client: http://www.opera.com/mail/ >>> >> > > > > -- >> _,,,^..^,,,_ > best, Eliot -- Using Opera's mail client: http://www.opera.com/mail/

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Eliot Miranda

On Tue, Mar 14, 2017 at 8:56 AM, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > 2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > >> >> Hi Esteban, Hi Igor, Hi All, >> >> On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >> wrote: >> >>> >>> Hi, >>> >>> Iâm tumbling into an error in Pharo, because we use callbacks >>> intensively, in Athens(cairo)-to-World conversion in particular, and people >>> is sending always their crash reportsâ¦ we made the whole conversion a lot >>> more robust since problems started to arise, but now I hit a wall I cannot >>> solve: I think problem is in something in callbacks. >>> >>> And problem is showing very easy on 64bits (while in 32bits it takes >>> time and is more random). >>> >> >> I responded in the "image not opening" thread, but it's the same >> problem. I really want to hear what y'all think because I'll happily >> implement a fix, but I want to know which one y'all think is a good idea. >> Here's my reply (edits between [ & ] to add information): >> >> >> On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> >> wrote: >> >> I'm pretty confident [I know] this is to do with bugs in the Athens >> surface code which assumes that callbacks can be made in the existing >> copyBits and warpBits primitive. They can't do this safely because a GC >> (scavenge) can happen during a callback, which then causes chaos when the >> copyBits primitive tries to access objects that have been moved under its >> feet. >> >> I've done work to fix callbacks so that when there is a failure it is the >> copyBits primitive that fails, instead of apparently the callback return >> primitive. One of the apparent effects of this fix is to stop the screen >> opening up too small; another is getting the background colour right, and >> yet another is eliminating bogus pixels in the VGTigerDemo demo. But more >> work is required to fix the copyBits and warpBits primitives. There are a >> few approaches one might take: >> >> a) fixing the primitive so that it saves and restores oops around the >> callbacks using the external oop table [InterpreterProxy>>addGCRoot: & >> removeGCRoot:]. That's a pain but possible. [It's a pain because all the >> derived pointers (the start of the destForm, sourceForm, halftoneForm and >> colorMapTable) must be recomputed also, and of course most of the time the >> objects don't move; we only scavenge about once every 2 seconds in normal >> running] >> >> b) fixing the primitive so that it pins the objects it needs before ever >> invoking a callback [this is a pain because pinning an object causes it to >> be tenured to old space if it is in new space; objects can't be pinned in >> new space, so instead the pin operation forwards the new space object to an >> old space copy if required and answers its location in old space, so a >> putative withPinnedObjectsDo: operation for the copyBits primitive looks >> like >> withPinnedFormsDo: aBlock >> <inline: #always> >> self cppIf: SPURVM & false >> ifTrue: >> [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | >> (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: >> [bitBltOop := interpreterProxy pinObject: bitBltOop]. >> (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: >> [destForm := interpreterProxy pinObject: destForm]. >> (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: >> [sourceForm := interpreterProxy pinObject: sourceForm]. >> (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: >> [halftoneForm := interpreterProxy pinObject: halftoneForm]. >> aBlock value. >> bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. >> destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. >> sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. >> halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] >> ifFalse: [aBlock value] >> and tenuring objects to old space is not ideal because they are only >> collected by a full GC, so doing this would at least tenure the bitBltOop >> which is very likely to be in new space] >> >> c) fixing the primitive so that it uses the scavenge and fullGC counters >> in the VM to detect if a GC occurred during one of the callbacks and would >> fail the primitive [if it detected that a GC had occurred in any of the >> surface functions]. The primitive would then simply be retried. >> >> d) ? >> > > Wouldn't it be possible to just pause the GC (scavange) when entering a > primitive ? > I don't think so. There is a callback occurring. If the computation executed by the callback requires a GC the application will abort if a GC cannot be done. Right? This is the case here. I like c) as it's very lightweight, but it has issues. It is fine to use >> for callbacks *before* cop[yBits and warpBits move any bits (the >> lockSurface and querySurface functions). But it's potentially erroneous >> after the unlockSurface primitive. For example, a primitive which does an >> xor with the screen can't simply be retried as the first, falling pass, >> would have updated the destination bits but not displayed them via >> unlockSurface. But I think it could be arranged that no objects are >> accessed after unlockSurface, which should naturally be the last call in >> the primitive (or do I mean showSurface?). So the approach would be to >> check for GCs occurring during querySurface and lockSurface, failing if so, >> and then caching any and all state needed by unlockSurface and showSurface >> in local variables. This way no object state is accessed to make the >> unlockSurface and showSurface calls, and no bits are moved before the >> queryDurface and lockSurface calls. >> >> If we used a failure code such as #'object may move' then the primitives >> could answer this when a GC during callbacks is detected and then the >> primitive could be retried only when required. >> >> >> [Come on folks, please comment. I want to know which idea you like >> best. We could fix this quickly. But right now it feels like I'm talking >> to myself.] >> >> >>> Here is the easiest way to reproduce it (in mac): >>> >>> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >>> wget files.pharo.org/get-files/60/pharo64.zip >>> wget files.pharo.org/get-files/60/sources.zip >>> unzip pharo64-mac-latest.zip >>> unzip pharo64.zip >>> unzip sources.zip >>> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval >>> "VGTigerDemo runDemo" >>> >>> eventually (like 5-6 seconds after, if not immediately), you will have a >>> stack like this: >>> >>> SmallInteger(Object)>>primitiveFailed: >>> SmallInteger(Object)>>primitiveFailed >>> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >>> GrafPort>>copyBits >>> GrafPort>>image:at:sourceRect:rule: >>> FormCanvas>>image:at:sourceRect:rule: >>> FormCanvas(Canvas)>>drawImage:at:sourceRect: >>> FormCanvas(Canvas)>>drawImage:at: >>> VGTigerDemo>>runDemo >>> VGTigerDemo class>>runDemo >>> UndefinedObject>>DoIt >>> OpalCompiler>>evaluate >>> OpalCompiler(AbstractCompiler)>>evaluate: >>> [ result := Smalltalk compiler evaluate: aStream. >>> self hasSessionChanged >>> ifFalse: [ self stdout >>> print: result; >>> lf ] ] in EvaluateCommandLineHandler>>evaluate: >>> in Block: [ result := Smalltalk compiler evaluate: aStream.... >>> BlockClosure>>on:do: >>> EvaluateCommandLineHandler>>evaluate: >>> EvaluateCommandLineHandler>>evaluateArguments >>> EvaluateCommandLineHandler>>activate >>> EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ aCommandLinehandler activateWith: commandLine ] in >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> in Block: [ aCommandLinehandler activateWith: commandLine ] >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >>> [ self >>> handleArgument: >>> (self arguments >>> ifEmpty: [ '' ] >>> ifNotEmpty: [ :arguments | arguments first ]) ] >>> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: >>> [ self... >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >>> PharoCommandLineHandler>>activate >>> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >>> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >>> >>> Any idea? >>> >>> thanks! >>> Esteban >> >> >> >> >> -- >> _,,,^..^,,,_ >> best, Eliot >> >> > > -- _,,,^..^,,,_ best, Eliot

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by ClÃ©ment Bera

On Tue, Mar 14, 2017 at 8:57 AM, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > 2017-03-14 16:56 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: > >> >> >> 2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: >> >>> >>> Hi Esteban, Hi Igor, Hi All, >>> >>> On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> Iâm tumbling into an error in Pharo, because we use callbacks >>>> intensively, in Athens(cairo)-to-World conversion in particular, and people >>>> is sending always their crash reportsâ¦ we made the whole conversion a lot >>>> more robust since problems started to arise, but now I hit a wall I cannot >>>> solve: I think problem is in something in callbacks. >>>> >>>> And problem is showing very easy on 64bits (while in 32bits it takes >>>> time and is more random). >>>> >>> >>> I responded in the "image not opening" thread, but it's the same >>> problem. I really want to hear what y'all think because I'll happily >>> implement a fix, but I want to know which one y'all think is a good idea. >>> Here's my reply (edits between [ & ] to add information): >>> >>> >>> On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> >>> wrote: >>> >>> I'm pretty confident [I know] this is to do with bugs in the Athens >>> surface code which assumes that callbacks can be made in the existing >>> copyBits and warpBits primitive. They can't do this safely because a GC >>> (scavenge) can happen during a callback, which then causes chaos when the >>> copyBits primitive tries to access objects that have been moved under its >>> feet. >>> >>> I've done work to fix callbacks so that when there is a failure it is >>> the copyBits primitive that fails, instead of apparently the callback >>> return primitive. One of the apparent effects of this fix is to stop the >>> screen opening up too small; another is getting the background colour >>> right, and yet another is eliminating bogus pixels in the VGTigerDemo >>> demo. But more work is required to fix the copyBits and warpBits >>> primitives. There are a few approaches one might take: >>> >>> a) fixing the primitive so that it saves and restores oops around the >>> callbacks using the external oop table [InterpreterProxy>>addGCRoot: & >>> removeGCRoot:]. That's a pain but possible. [It's a pain because all the >>> derived pointers (the start of the destForm, sourceForm, halftoneForm and >>> colorMapTable) must be recomputed also, and of course most of the time the >>> objects don't move; we only scavenge about once every 2 seconds in normal >>> running] >>> >>> b) fixing the primitive so that it pins the objects it needs before ever >>> invoking a callback [this is a pain because pinning an object causes it to >>> be tenured to old space if it is in new space; objects can't be pinned in >>> new space, so instead the pin operation forwards the new space object to an >>> old space copy if required and answers its location in old space, so a >>> putative withPinnedObjectsDo: operation for the copyBits primitive looks >>> like >>> withPinnedFormsDo: aBlock >>> <inline: #always> >>> self cppIf: SPURVM & false >>> ifTrue: >>> [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | >>> (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: >>> [bitBltOop := interpreterProxy pinObject: bitBltOop]. >>> (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: >>> [destForm := interpreterProxy pinObject: destForm]. >>> (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: >>> [sourceForm := interpreterProxy pinObject: sourceForm]. >>> (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: >>> [halftoneForm := interpreterProxy pinObject: halftoneForm]. >>> aBlock value. >>> bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. >>> destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. >>> sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. >>> halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] >>> ifFalse: [aBlock value] >>> and tenuring objects to old space is not ideal because they are only >>> collected by a full GC, so doing this would at least tenure the bitBltOop >>> which is very likely to be in new space] >>> >>> c) fixing the primitive so that it uses the scavenge and fullGC counters >>> in the VM to detect if a GC occurred during one of the callbacks and would >>> fail the primitive [if it detected that a GC had occurred in any of the >>> surface functions]. The primitive would then simply be retried. >>> >>> That's clearly the best solution unless someone figures out a better d) solution. > d) ? >>> >> >> Wouldn't it be possible to just pause the GC (scavange) when entering a >> primitive ? >> > > Wouldn't it be possible to just *disable* the GC (scavange) when entering > a primitive ? > If you disable only the scavenges and some of the objects are in old space and a fullGC happens, then the same problem happens. During the call-backs, an infinite amount of objects can be allocated. Where do you allocate such objects if you disable the GC ? > >> >> >> >> >>> >>> I like c) as it's very lightweight, but it has issues. It is fine to >>> use for callbacks *before* cop[yBits and warpBits move any bits (the >>> lockSurface and querySurface functions). But it's potentially erroneous >>> after the unlockSurface primitive. For example, a primitive which does an >>> xor with the screen can't simply be retried as the first, falling pass, >>> would have updated the destination bits but not displayed them via >>> unlockSurface. But I think it could be arranged that no objects are >>> accessed after unlockSurface, which should naturally be the last call in >>> the primitive (or do I mean showSurface?). So the approach would be to >>> check for GCs occurring during querySurface and lockSurface, failing if so, >>> and then caching any and all state needed by unlockSurface and showSurface >>> in local variables. This way no object state is accessed to make the >>> unlockSurface and showSurface calls, and no bits are moved before the >>> queryDurface and lockSurface calls. >>> >>> If we used a failure code such as #'object may move' then the primitives >>> could answer this when a GC during callbacks is detected and then the >>> primitive could be retried only when required. >>> >>> >>> [Come on folks, please comment. I want to know which idea you like >>> best. We could fix this quickly. But right now it feels like I'm talking >>> to myself.] >>> >>> >>>> Here is the easiest way to reproduce it (in mac): >>>> >>>> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >>>> wget files.pharo.org/get-files/60/pharo64.zip >>>> wget files.pharo.org/get-files/60/sources.zip >>>> unzip pharo64-mac-latest.zip >>>> unzip pharo64.zip >>>> unzip sources.zip >>>> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval >>>> "VGTigerDemo runDemo" >>>> >>>> eventually (like 5-6 seconds after, if not immediately), you will have >>>> a stack like this: >>>> >>>> SmallInteger(Object)>>primitiveFailed: >>>> SmallInteger(Object)>>primitiveFailed >>>> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >>>> GrafPort>>copyBits >>>> GrafPort>>image:at:sourceRect:rule: >>>> FormCanvas>>image:at:sourceRect:rule: >>>> FormCanvas(Canvas)>>drawImage:at:sourceRect: >>>> FormCanvas(Canvas)>>drawImage:at: >>>> VGTigerDemo>>runDemo >>>> VGTigerDemo class>>runDemo >>>> UndefinedObject>>DoIt >>>> OpalCompiler>>evaluate >>>> OpalCompiler(AbstractCompiler)>>evaluate: >>>> [ result := Smalltalk compiler evaluate: aStream. >>>> self hasSessionChanged >>>> ifFalse: [ self stdout >>>> print: result; >>>> lf ] ] in EvaluateCommandLineHandler>>evaluate: >>>> in Block: [ result := Smalltalk compiler evaluate: aStream.... >>>> BlockClosure>>on:do: >>>> EvaluateCommandLineHandler>>evaluate: >>>> EvaluateCommandLineHandler>>evaluateArguments >>>> EvaluateCommandLineHandler>>activate >>>> EvaluateCommandLineHandler class(CommandLineHandler >>>> class)>>activateWith: >>>> [ aCommandLinehandler activateWith: commandLine ] in >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>>> in Block: [ aCommandLinehandler activateWith: commandLine ] >>>> BlockClosure>>on:do: >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >>>> [ self >>>> handleArgument: >>>> (self arguments >>>> ifEmpty: [ '' ] >>>> ifNotEmpty: [ :arguments | arguments first ]) ] >>>> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in >>>> Block: [ self... >>>> BlockClosure>>on:do: >>>> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >>>> PharoCommandLineHandler>>activate >>>> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >>>> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >>>> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >>>> >>>> Any idea? >>>> >>>> thanks! >>>> Esteban >>> >>> >>> >>> >>> -- >>> _,,,^..^,,,_ >>> best, Eliot >>> >>> >> > >

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Nicolai Hess

2017-03-14 16:56 GMT+01:00 Nicolai Hess <nicolaihess(a)gmail.com>: > > > 2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > >> >> Hi Esteban, Hi Igor, Hi All, >> >> On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >> wrote: >> >>> >>> Hi, >>> >>> Iâm tumbling into an error in Pharo, because we use callbacks >>> intensively, in Athens(cairo)-to-World conversion in particular, and people >>> is sending always their crash reportsâ¦ we made the whole conversion a lot >>> more robust since problems started to arise, but now I hit a wall I cannot >>> solve: I think problem is in something in callbacks. >>> >>> And problem is showing very easy on 64bits (while in 32bits it takes >>> time and is more random). >>> >> >> I responded in the "image not opening" thread, but it's the same >> problem. I really want to hear what y'all think because I'll happily >> implement a fix, but I want to know which one y'all think is a good idea. >> Here's my reply (edits between [ & ] to add information): >> >> >> On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> >> wrote: >> >> I'm pretty confident [I know] this is to do with bugs in the Athens >> surface code which assumes that callbacks can be made in the existing >> copyBits and warpBits primitive. They can't do this safely because a GC >> (scavenge) can happen during a callback, which then causes chaos when the >> copyBits primitive tries to access objects that have been moved under its >> feet. >> >> I've done work to fix callbacks so that when there is a failure it is the >> copyBits primitive that fails, instead of apparently the callback return >> primitive. One of the apparent effects of this fix is to stop the screen >> opening up too small; another is getting the background colour right, and >> yet another is eliminating bogus pixels in the VGTigerDemo demo. But more >> work is required to fix the copyBits and warpBits primitives. There are a >> few approaches one might take: >> >> a) fixing the primitive so that it saves and restores oops around the >> callbacks using the external oop table [InterpreterProxy>>addGCRoot: & >> removeGCRoot:]. That's a pain but possible. [It's a pain because all the >> derived pointers (the start of the destForm, sourceForm, halftoneForm and >> colorMapTable) must be recomputed also, and of course most of the time the >> objects don't move; we only scavenge about once every 2 seconds in normal >> running] >> >> b) fixing the primitive so that it pins the objects it needs before ever >> invoking a callback [this is a pain because pinning an object causes it to >> be tenured to old space if it is in new space; objects can't be pinned in >> new space, so instead the pin operation forwards the new space object to an >> old space copy if required and answers its location in old space, so a >> putative withPinnedObjectsDo: operation for the copyBits primitive looks >> like >> withPinnedFormsDo: aBlock >> <inline: #always> >> self cppIf: SPURVM & false >> ifTrue: >> [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | >> (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: >> [bitBltOop := interpreterProxy pinObject: bitBltOop]. >> (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: >> [destForm := interpreterProxy pinObject: destForm]. >> (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: >> [sourceForm := interpreterProxy pinObject: sourceForm]. >> (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: >> [halftoneForm := interpreterProxy pinObject: halftoneForm]. >> aBlock value. >> bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. >> destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. >> sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. >> halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] >> ifFalse: [aBlock value] >> and tenuring objects to old space is not ideal because they are only >> collected by a full GC, so doing this would at least tenure the bitBltOop >> which is very likely to be in new space] >> >> c) fixing the primitive so that it uses the scavenge and fullGC counters >> in the VM to detect if a GC occurred during one of the callbacks and would >> fail the primitive [if it detected that a GC had occurred in any of the >> surface functions]. The primitive would then simply be retried. >> >> d) ? >> > > Wouldn't it be possible to just pause the GC (scavange) when entering a > primitive ? > Wouldn't it be possible to just *disable* the GC (scavange) when entering a primitive ? > > > > >> >> I like c) as it's very lightweight, but it has issues. It is fine to use >> for callbacks *before* cop[yBits and warpBits move any bits (the >> lockSurface and querySurface functions). But it's potentially erroneous >> after the unlockSurface primitive. For example, a primitive which does an >> xor with the screen can't simply be retried as the first, falling pass, >> would have updated the destination bits but not displayed them via >> unlockSurface. But I think it could be arranged that no objects are >> accessed after unlockSurface, which should naturally be the last call in >> the primitive (or do I mean showSurface?). So the approach would be to >> check for GCs occurring during querySurface and lockSurface, failing if so, >> and then caching any and all state needed by unlockSurface and showSurface >> in local variables. This way no object state is accessed to make the >> unlockSurface and showSurface calls, and no bits are moved before the >> queryDurface and lockSurface calls. >> >> If we used a failure code such as #'object may move' then the primitives >> could answer this when a GC during callbacks is detected and then the >> primitive could be retried only when required. >> >> >> [Come on folks, please comment. I want to know which idea you like >> best. We could fix this quickly. But right now it feels like I'm talking >> to myself.] >> >> >>> Here is the easiest way to reproduce it (in mac): >>> >>> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >>> wget files.pharo.org/get-files/60/pharo64.zip >>> wget files.pharo.org/get-files/60/sources.zip >>> unzip pharo64-mac-latest.zip >>> unzip pharo64.zip >>> unzip sources.zip >>> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval >>> "VGTigerDemo runDemo" >>> >>> eventually (like 5-6 seconds after, if not immediately), you will have a >>> stack like this: >>> >>> SmallInteger(Object)>>primitiveFailed: >>> SmallInteger(Object)>>primitiveFailed >>> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >>> GrafPort>>copyBits >>> GrafPort>>image:at:sourceRect:rule: >>> FormCanvas>>image:at:sourceRect:rule: >>> FormCanvas(Canvas)>>drawImage:at:sourceRect: >>> FormCanvas(Canvas)>>drawImage:at: >>> VGTigerDemo>>runDemo >>> VGTigerDemo class>>runDemo >>> UndefinedObject>>DoIt >>> OpalCompiler>>evaluate >>> OpalCompiler(AbstractCompiler)>>evaluate: >>> [ result := Smalltalk compiler evaluate: aStream. >>> self hasSessionChanged >>> ifFalse: [ self stdout >>> print: result; >>> lf ] ] in EvaluateCommandLineHandler>>evaluate: >>> in Block: [ result := Smalltalk compiler evaluate: aStream.... >>> BlockClosure>>on:do: >>> EvaluateCommandLineHandler>>evaluate: >>> EvaluateCommandLineHandler>>evaluateArguments >>> EvaluateCommandLineHandler>>activate >>> EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ aCommandLinehandler activateWith: commandLine ] in >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> in Block: [ aCommandLinehandler activateWith: commandLine ] >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >>> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >>> [ self >>> handleArgument: >>> (self arguments >>> ifEmpty: [ '' ] >>> ifNotEmpty: [ :arguments | arguments first ]) ] >>> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: >>> [ self... >>> BlockClosure>>on:do: >>> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >>> PharoCommandLineHandler>>activate >>> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >>> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >>> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >>> >>> Any idea? >>> >>> thanks! >>> Esteban >> >> >> >> >> -- >> _,,,^..^,,,_ >> best, Eliot >> >> >

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Nicolai Hess

2017-03-14 16:46 GMT+01:00 Eliot Miranda <eliot.miranda(a)gmail.com>: > > Hi Esteban, Hi Igor, Hi All, > > On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> > wrote: > >> >> Hi, >> >> Iâm tumbling into an error in Pharo, because we use callbacks >> intensively, in Athens(cairo)-to-World conversion in particular, and people >> is sending always their crash reportsâ¦ we made the whole conversion a lot >> more robust since problems started to arise, but now I hit a wall I cannot >> solve: I think problem is in something in callbacks. >> >> And problem is showing very easy on 64bits (while in 32bits it takes time >> and is more random). >> > > I responded in the "image not opening" thread, but it's the same > problem. I really want to hear what y'all think because I'll happily > implement a fix, but I want to know which one y'all think is a good idea. > Here's my reply (edits between [ & ] to add information): > > > On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> > wrote: > > I'm pretty confident [I know] this is to do with bugs in the Athens > surface code which assumes that callbacks can be made in the existing > copyBits and warpBits primitive. They can't do this safely because a GC > (scavenge) can happen during a callback, which then causes chaos when the > copyBits primitive tries to access objects that have been moved under its > feet. > > I've done work to fix callbacks so that when there is a failure it is the > copyBits primitive that fails, instead of apparently the callback return > primitive. One of the apparent effects of this fix is to stop the screen > opening up too small; another is getting the background colour right, and > yet another is eliminating bogus pixels in the VGTigerDemo demo. But more > work is required to fix the copyBits and warpBits primitives. There are a > few approaches one might take: > > a) fixing the primitive so that it saves and restores oops around the > callbacks using the external oop table [InterpreterProxy>>addGCRoot: & > removeGCRoot:]. That's a pain but possible. [It's a pain because all the > derived pointers (the start of the destForm, sourceForm, halftoneForm and > colorMapTable) must be recomputed also, and of course most of the time the > objects don't move; we only scavenge about once every 2 seconds in normal > running] > > b) fixing the primitive so that it pins the objects it needs before ever > invoking a callback [this is a pain because pinning an object causes it to > be tenured to old space if it is in new space; objects can't be pinned in > new space, so instead the pin operation forwards the new space object to an > old space copy if required and answers its location in old space, so a > putative withPinnedObjectsDo: operation for the copyBits primitive looks > like > withPinnedFormsDo: aBlock > <inline: #always> > self cppIf: SPURVM & false > ifTrue: > [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | > (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: > [bitBltOop := interpreterProxy pinObject: bitBltOop]. > (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: > [destForm := interpreterProxy pinObject: destForm]. > (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: > [sourceForm := interpreterProxy pinObject: sourceForm]. > (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: > [halftoneForm := interpreterProxy pinObject: halftoneForm]. > aBlock value. > bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. > destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. > sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. > halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] > ifFalse: [aBlock value] > and tenuring objects to old space is not ideal because they are only > collected by a full GC, so doing this would at least tenure the bitBltOop > which is very likely to be in new space] > > c) fixing the primitive so that it uses the scavenge and fullGC counters > in the VM to detect if a GC occurred during one of the callbacks and would > fail the primitive [if it detected that a GC had occurred in any of the > surface functions]. The primitive would then simply be retried. > > d) ? > Wouldn't it be possible to just pause the GC (scavange) when entering a primitive ? > > I like c) as it's very lightweight, but it has issues. It is fine to use > for callbacks *before* cop[yBits and warpBits move any bits (the > lockSurface and querySurface functions). But it's potentially erroneous > after the unlockSurface primitive. For example, a primitive which does an > xor with the screen can't simply be retried as the first, falling pass, > would have updated the destination bits but not displayed them via > unlockSurface. But I think it could be arranged that no objects are > accessed after unlockSurface, which should naturally be the last call in > the primitive (or do I mean showSurface?). So the approach would be to > check for GCs occurring during querySurface and lockSurface, failing if so, > and then caching any and all state needed by unlockSurface and showSurface > in local variables. This way no object state is accessed to make the > unlockSurface and showSurface calls, and no bits are moved before the > queryDurface and lockSurface calls. > > If we used a failure code such as #'object may move' then the primitives > could answer this when a GC during callbacks is detected and then the > primitive could be retried only when required. > > > [Come on folks, please comment. I want to know which idea you like best. > We could fix this quickly. But right now it feels like I'm talking to > myself.] > > >> Here is the easiest way to reproduce it (in mac): >> >> wget files.pharo.org/get-files/60/pharo64-mac-latest.zip >> wget files.pharo.org/get-files/60/pharo64.zip >> wget files.pharo.org/get-files/60/sources.zip >> unzip pharo64-mac-latest.zip >> unzip pharo64.zip >> unzip sources.zip >> ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval "VGTigerDemo >> runDemo" >> >> eventually (like 5-6 seconds after, if not immediately), you will have a >> stack like this: >> >> SmallInteger(Object)>>primitiveFailed: >> SmallInteger(Object)>>primitiveFailed >> SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: >> GrafPort>>copyBits >> GrafPort>>image:at:sourceRect:rule: >> FormCanvas>>image:at:sourceRect:rule: >> FormCanvas(Canvas)>>drawImage:at:sourceRect: >> FormCanvas(Canvas)>>drawImage:at: >> VGTigerDemo>>runDemo >> VGTigerDemo class>>runDemo >> UndefinedObject>>DoIt >> OpalCompiler>>evaluate >> OpalCompiler(AbstractCompiler)>>evaluate: >> [ result := Smalltalk compiler evaluate: aStream. >> self hasSessionChanged >> ifFalse: [ self stdout >> print: result; >> lf ] ] in EvaluateCommandLineHandler>>evaluate: >> in Block: [ result := Smalltalk compiler evaluate: aStream.... >> BlockClosure>>on:do: >> EvaluateCommandLineHandler>>evaluate: >> EvaluateCommandLineHandler>>evaluateArguments >> EvaluateCommandLineHandler>>activate >> EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: >> [ aCommandLinehandler activateWith: commandLine ] in >> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: in >> Block: [ aCommandLinehandler activateWith: commandLine ] >> BlockClosure>>on:do: >> PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: >> PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand >> PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: >> [ self >> handleArgument: >> (self arguments >> ifEmpty: [ '' ] >> ifNotEmpty: [ :arguments | arguments first ]) ] >> in PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: >> [ self... >> BlockClosure>>on:do: >> PharoCommandLineHandler(BasicCommandLineHandler)>>activate >> PharoCommandLineHandler>>activate >> PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: >> [ super activateWith: aCommandLine ] in PharoCommandLineHandler >> class>>activateWith: in Block: [ super activateWith: aCommandLine ] >> >> Any idea? >> >> thanks! >> Esteban > > > > > -- > _,,,^..^,,,_ > best, Eliot > >

March 14, 2017

Re: [Pharo-dev] [Vm-dev] BUG? A problem with callbacks that shows up in 64bits (but is on 32bits too)

by Eliot Miranda

Hi Esteban, Hi Igor, Hi All, On Fri, Mar 10, 2017 at 7:35 AM, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > Hi, > > Iâm tumbling into an error in Pharo, because we use callbacks intensively, > in Athens(cairo)-to-World conversion in particular, and people is sending > always their crash reportsâ¦ we made the whole conversion a lot more robust > since problems started to arise, but now I hit a wall I cannot solve: I > think problem is in something in callbacks. > > And problem is showing very easy on 64bits (while in 32bits it takes time > and is more random). > I responded in the "image not opening" thread, but it's the same problem. I really want to hear what y'all think because I'll happily implement a fix, but I want to know which one y'all think is a good idea. Here's my reply (edits between [ & ] to add information): On Mon, Mar 13, 2017 at 9:11 PM, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: I'm pretty confident [I know] this is to do with bugs in the Athens surface code which assumes that callbacks can be made in the existing copyBits and warpBits primitive. They can't do this safely because a GC (scavenge) can happen during a callback, which then causes chaos when the copyBits primitive tries to access objects that have been moved under its feet. I've done work to fix callbacks so that when there is a failure it is the copyBits primitive that fails, instead of apparently the callback return primitive. One of the apparent effects of this fix is to stop the screen opening up too small; another is getting the background colour right, and yet another is eliminating bogus pixels in the VGTigerDemo demo. But more work is required to fix the copyBits and warpBits primitives. There are a few approaches one might take: a) fixing the primitive so that it saves and restores oops around the callbacks using the external oop table [InterpreterProxy>>addGCRoot: & removeGCRoot:]. That's a pain but possible. [It's a pain because all the derived pointers (the start of the destForm, sourceForm, halftoneForm and colorMapTable) must be recomputed also, and of course most of the time the objects don't move; we only scavenge about once every 2 seconds in normal running] b) fixing the primitive so that it pins the objects it needs before ever invoking a callback [this is a pain because pinning an object causes it to be tenured to old space if it is in new space; objects can't be pinned in new space, so instead the pin operation forwards the new space object to an old space copy if required and answers its location in old space, so a putative withPinnedObjectsDo: operation for the copyBits primitive looks like withPinnedFormsDo: aBlock <inline: #always> self cppIf: SPURVM & false ifTrue: [| bitBltOopWasPinned destWasPinned sourceWasPinned halftoneWasPinned | (bitBltOopWasPinned := interpreterProxy isPinned: bitBltOop) ifFalse: [bitBltOop := interpreterProxy pinObject: bitBltOop]. (destWasPinned := interpreterProxy isPinned: destForm) ifFalse: [destForm := interpreterProxy pinObject: destForm]. (sourceWasPinned := interpreterProxy isPinned: sourceForm) ifFalse: [sourceForm := interpreterProxy pinObject: sourceForm]. (halftoneWasPinned := interpreterProxy isPinned: halftoneForm) ifFalse: [halftoneForm := interpreterProxy pinObject: halftoneForm]. aBlock value. bitBltOopWasPinned ifFalse: [interpreterProxy unpinObject: bitBltOop]. destWasPinned ifFalse: [interpreterProxy unpinObject: destForm]. sourceWasPinned ifFalse: [interpreterProxy unpinObject: sourceForm]. halftoneWasPinned ifFalse: [interpreterProxy unpinObject: halftoneForm]] ifFalse: [aBlock value] and tenuring objects to old space is not ideal because they are only collected by a full GC, so doing this would at least tenure the bitBltOop which is very likely to be in new space] c) fixing the primitive so that it uses the scavenge and fullGC counters in the VM to detect if a GC occurred during one of the callbacks and would fail the primitive [if it detected that a GC had occurred in any of the surface functions]. The primitive would then simply be retried. d) ? I like c) as it's very lightweight, but it has issues. It is fine to use for callbacks *before* cop[yBits and warpBits move any bits (the lockSurface and querySurface functions). But it's potentially erroneous after the unlockSurface primitive. For example, a primitive which does an xor with the screen can't simply be retried as the first, falling pass, would have updated the destination bits but not displayed them via unlockSurface. But I think it could be arranged that no objects are accessed after unlockSurface, which should naturally be the last call in the primitive (or do I mean showSurface?). So the approach would be to check for GCs occurring during querySurface and lockSurface, failing if so, and then caching any and all state needed by unlockSurface and showSurface in local variables. This way no object state is accessed to make the unlockSurface and showSurface calls, and no bits are moved before the queryDurface and lockSurface calls. If we used a failure code such as #'object may move' then the primitives could answer this when a GC during callbacks is detected and then the primitive could be retried only when required. [Come on folks, please comment. I want to know which idea you like best. We could fix this quickly. But right now it feels like I'm talking to myself.] > Here is the easiest way to reproduce it (in mac): > > wget files.pharo.org/get-files/60/pharo64-mac-latest.zip > wget files.pharo.org/get-files/60/pharo64.zip > wget files.pharo.org/get-files/60/sources.zip > unzip pharo64-mac-latest.zip > unzip pharo64.zip > unzip sources.zip > ./Pharo.app/Contents/MacOS/Pharo ./Pharo64-60438.image eval "VGTigerDemo > runDemo" > > eventually (like 5-6 seconds after, if not immediately), you will have a > stack like this: > > SmallInteger(Object)>>primitiveFailed: > SmallInteger(Object)>>primitiveFailed > SmallInteger(VMCallbackContext64)>>primSignal:andReturnAs:fromContext: > GrafPort>>copyBits > GrafPort>>image:at:sourceRect:rule: > FormCanvas>>image:at:sourceRect:rule: > FormCanvas(Canvas)>>drawImage:at:sourceRect: > FormCanvas(Canvas)>>drawImage:at: > VGTigerDemo>>runDemo > VGTigerDemo class>>runDemo > UndefinedObject>>DoIt > OpalCompiler>>evaluate > OpalCompiler(AbstractCompiler)>>evaluate: > [ result := Smalltalk compiler evaluate: aStream. > self hasSessionChanged > ifFalse: [ self stdout > print: result; > lf ] ] in EvaluateCommandLineHandler>>evaluate: > in Block: [ result := Smalltalk compiler evaluate: aStream.... > BlockClosure>>on:do: > EvaluateCommandLineHandler>>evaluate: > EvaluateCommandLineHandler>>evaluateArguments > EvaluateCommandLineHandler>>activate > EvaluateCommandLineHandler class(CommandLineHandler class)>>activateWith: > [ aCommandLinehandler activateWith: commandLine ] in > PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: in > Block: [ aCommandLinehandler activateWith: commandLine ] > BlockClosure>>on:do: > PharoCommandLineHandler(BasicCommandLineHandler)>>activateSubCommand: > PharoCommandLineHandler(BasicCommandLineHandler)>>handleSubcommand > PharoCommandLineHandler(BasicCommandLineHandler)>>handleArgument: > [ self > handleArgument: > (self arguments > ifEmpty: [ '' ] > ifNotEmpty: [ :arguments | arguments first ]) ] in > PharoCommandLineHandler(BasicCommandLineHandler)>>activate in Block: [ > self... > BlockClosure>>on:do: > PharoCommandLineHandler(BasicCommandLineHandler)>>activate > PharoCommandLineHandler>>activate > PharoCommandLineHandler class(CommandLineHandler class)>>activateWith: > [ super activateWith: aCommandLine ] in PharoCommandLineHandler > class>>activateWith: in Block: [ super activateWith: aCommandLine ] > > Any idea? > > thanks! > Esteban -- _,,,^..^,,,_ best, Eliot

March 14, 2017