May 2016 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] OS-Windows and UnifiedFFI

by Nicolai Hess

2016-04-25 10:17 GMT+02:00 Nicolai Hess <nicolaihess(a)gmail.com>: > > > 2016-04-25 9:28 GMT+02:00 Esteban Lorenzano <estebanlm(a)gmail.com>: > >> Hi Nicolai, >> >> On 25 Apr 2016, at 09:14, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >> >> >> >> 2016-04-24 15:26 GMT+02:00 Torsten Bergmann <astares(a)gmx.de>: >> >>> Hi Nicolai, >>> >>> missed this thread - sorry. To my knowledge there was a call in NB for >>> CreateWindow >>> but there was no example in opening a new window. For X-Windows there >>> was an example >>> if I remember correctly. >>> >> >> Hi Torsten, >> >> yes there is an "example" in NBOpenGL. On Windows, it registers and >> creates window for getting an opengl context. >> The code and classes for registering and creating a window are similiar >> to what is now in your os-windows. >> I needed to define some more pool variables for some style attributes, >> all that is working. >> What did not work is storing the hInstance attribute in the WndClass >> struct. >> >> >> what and how it does not work? >> can you paste the code? >> > > > Sure. > Load stable version of Torstens OSWindows. > Change WinProcess class>>getVMModuleHandle (it has to use HINSTANCE > instead of HMODULE (or just define a new pool variable HMODULE in WinTypes)) > > getVMModuleHandle > "The GetModuleHandle function retrieves a module handle for the > specified module if the file has been mapped into the address space of the > calling process." > > ^ self ffiCall: #( HINSTANCE GetModuleHandleA (0)) module: #Kernel32 > > > Now create a new WinWndClassEx and set the hInstance field: > > |hInst wndClass| > hInst := WinProcess getVMModuleHandle. > wndClass := WinWndClassEx new. > wndClass hInstance: hInst. > > > -> Message not understood SmallInteger>>isExternalAddress > > > The equivalent code with NB that was working: > > |hInst wndClass| > hInst := NativeBoostWin32 getVMModuleHandle. > wndClass := NBWndClassEx new. > wndClass hInstance: hInst. > wndClass > > Any more ideas? > > >> >> cheers! >> Esteban >> >> With NBOpenGL (and non-spur resp. NativeBoost) this works. >> With the new UnifiedFFI this gives an error from: >> >> pointerAt: byteOffset put: value >> "Store a pointer object at the given byte address" >> value isExternalAddress >> >> >> >>> >>> Basically I would like to add support for this into OS-Windows, yes. >>> Some parts are there and UFFI now has easy callbacks. Havent looked >>> at OSWindow yet - but I guess the goal is to create native windows as >>> well. >>> Also I would like to add more support in manipulating/quering other >>> external windows (which can be done using the already built in support >>> for process and window handles). >>> >>> In Win32 to create a window one has to register a window class first and >>> use the window class name afterwards in the CreateWindow API function. >>> >>> No time to look into this this week ... but can give a try afterwards. >>> The WNDCLASS/WNDCLASSEX structure needs to be wrapped, RegisterClass >>> called and CreateWindowEx with a callback for a function to process >>> window messages. >>> >>> Dont know how if this interferes with the VM's own processing loop ... >>> but we should give a try. If you catch me on Slack next week we can >>> try together. >>> >>> Bye >>> Torsten >>> >>> >>> Gesendet: Sonntag, 24. April 2016 um 12:17 Uhr >>> Von: "Nicolai Hess" <nicolaihess(a)gmail.com> >>> An: "Pharo Development List" <pharo-dev(a)lists.pharo.org> >>> Betreff: Re: [Pharo-dev] OS-Windows and UnifiedFFI >>> >>> >>> >>> 2016-04-15 21:58 GMT+02:00 Nicolai Hess <nicolaihess(a)gmail.com>: >>> >>> Hi Torsten, Esteban, >>> I try to use OS-Windows (not OSWindow :) ) with latest spur VM and >>> UnifiedFFI.As far as I can see, you (Torsten) already ported some parts to >>> UnifiedFFI - great, thanks! >>> I now try to open a new window withWinWindow>>createWindowEx ....as I >>> took this example originally from NBOpenGL, it seems I have to define >>> a Window Class first, with >>> WinWndClassEx >>> This seems to work in the old NBOpenGL/NBWin32 code but not with >>> OS-Windows >>> WinWndClassEx >>> first, NBOpenGL uses >>> NativeBoostWin32 getVMModuleHandle for an HModule instance >>> there is a similar call in OS-Windows, WinProcess getVMModuleHandle >>> (I needed to define a missing type, HMODULE, but that works). >>> >>> What does not work is this part: >>> >>> | hInst wndClass | >>> hInst := WinProcess getVMModuleHandle. >>> wndClass := WinWndClassEx new. >>> wndClass hInstance: hInst. >>> >>> FF complains about >>> pointerAt: byteOffset put: value >>> "Store a pointer object at the given byte address" >>> value isExternalAddress >>> >>> here value is not a handle, but a small integer, and it does not define >>> isExternalAddress >>> >>> I tried to put the handle instead of the handles value, but this does >>> not work either. >>> >>> Any Idea what is missing here, I need a way to store the HMODULE handle >>> in the structure of the WinWNdClassEx >>> >>> Thanks in advance. >>> >>> Anyone? >>> >>> >> >> >

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by Martin McClure

On 05/18/2016 10:42 PM, Sven Van Caekenberghe wrote: > Martin, > > Would you be willing to contribute your implementation to Pharo ? Possibly with some tests ? I'll ask GemTalk management if they're OK with that. I imagine it'll be OK. But you might be better off taking the Squeak implementation, or reading and understanding the CMWC papers and implementing from scratch. My implementation is based on reference implementations and constants for 32-bit numbers in C. This works well if your SmallIntegers are 64-bit, as they are in GemStone. A straight port to Pharo would be handling large integers all the time. This might be OK, but is something to think about. Transposing it to 29-bit numbers would be possible, but would require finding new prime numbers with the appropriate properties. I consider that fairly heavy lifting, but perhaps you've got somebody with better numerical methods chops than I've got. My API is a bit unconventional, too, though *I* think it makes sense. :-) Example code: | random foo | random := Random new. foo := random integerBetween: 4 and: 27. "works until you get close to 2**32" foo := random integer. "[0..2**32-1]" foo := random float. "Double in [0..1)" Tests: I don't recall what automated tests we have. Some, I'm sure. The tough thing is testing a generator for randomness -- you need something like 650GB of free disk space and a couple of days to generate the random numbers and run the tests over them. I did all that manually. Those tests come from here: http://simul.iro.umontreal.ca/testu01/tu01.html They're the most complete randomness tests of which I'm aware. Regards, -Martin

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by Sven Van Caekenberghe

> On 19 May 2016, at 02:07, Martin McClure <martin(a)hand2mouse.com> wrote: > > On 05/18/2016 04:05 PM, Peter UhnÃ¡k wrote: >> My questions: >> 1) do we really want to have global fixed seed? > > No! > >> 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value > > Yes. > >> 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? >> > Yes. > > Though this only works on Unix, so on Windows it uses the current time as a seed. It might get better results using #microsecondClockValue instead of #millisecondClockValue. And I'd think about just taking the microsecond value and sending it #hashMultiply instead of the weird things it's doing now. (bitXor with the identity hash of the UUIDGenerator instance? That depends on the randomness of identity hashes, which may or may not be very good.) > It seems likely that Windows has a built-in random number generator, as well, which would probably be better. > > While we're at it, the class Random is a Park-Miller generator, which has quite poor randomness by modern standards. Most other Smalltalks have upgraded -- GemStone uses CMWC (pure Smalltalk, very simple and quite fast; I wrote that one), VW I believe uses Lagged Fibonacci, and Squeak has moved to Mersenne Twister last I heard. I've tested all three against statistical tests of randomness -- generated about 650GB of random bytes from each, ran them through the tests (which require that much data to get good statistics). > > All three are pretty good, only failing a few tests. The only generator I tested that passed *all* tests was Linux /dev/urandom. Which is also fast. If you're on Linux, and don't need a repeatable random sequence, I'd use /dev/urandom. If you do need to be able to have a repeatable sequence with good randomness properties, I'd use one of the three generators I mentioned above. Of the three, the simplest and fastest is CMWC. Martin, Would you be willing to contribute your implementation to Pharo ? Possibly with some tests ? Sven > Regards, > > -Martin >

May 19, 2016

Re: [Pharo-dev] TEXT drag and drop

by Esteban Lorenzano

I would vote yes :) > On 18 May 2016, at 22:02, Torsten Bergmann <astares(a)gmx.de> wrote: > > Hi Liang Bing, > > nice! Could be a nice addition to the standard image if others like it too. > Did you already sign as contributor? > > Thx > T. > > Gesendet: Mittwoch, 18. Mai 2016 um 19:09 Uhr > Von: lb <liangbing64(a)126.com> > An: pharo-dev(a)lists.pharo.org > Betreff: [Pharo-dev] TEXT drag and drop > Hi, > I have made a little tool for draging and dropping text selection. like windows Word. > It can work in Pharo 40 50 and 60. > > Have a fun! > > LiangBing > > MCHttpRepository > location: 'http://www.smalltalkhub.com/mc/liangbing/Text-DragDrop/main' <http://www.smalltalkhub.com/mc/liangbing/Text-DragDrop/main'> > user: '' > password: '' > > >

May 19, 2016

Re: [Pharo-dev] Problem with delay waiting (OSSubprocess code) on Pharo 5.0

by Martin McClure

On 05/18/2016 07:39 PM, Ben Coman wrote: > Waiting in the wings for Pharo 6 I have changes that should help: > * have #save/#restoreResumptionTimes*only* called from timing > priority event loop (i.e. #handleTimerEvent) > * shutDown/startUp suspends/resumes the timing priority event loop, > instead of trying to block signals to timingSemaphore > Yeah, that sounds better, thanks. -Martin

May 19, 2016

Re: [Pharo-dev] Problem with delay waiting (OSSubprocess code) on Pharo 5.0

by Ben Coman

On Thu, May 19, 2016 at 8:49 AM, Martin McClure <martin(a)hand2mouse.com> wrote: > On 05/18/2016 03:17 PM, Martin McClure wrote: >> >> On 05/18/2016 08:49 AM, Mariano Martinez Peck wrote: >>> >>> Hi guys, >>> >>> I am seeing a problem in Pharo 5.0 regarding Delay >> wait. I cannot >>> explain how this could happened but it does, and it happened to me a couple >>> of times (but not fully reproducible). >>> >> >> Hmm. The schedulerResumptionTime is, somehow, being (approximately) >> doubled. It's not clear how that can happen, but I'll look a little more. >> > > Mario, is there any chance that you might be saving the image during one of > these Delays? > > > This one smells like a race condition, and I think I see something that > *might* explain it. But I don't have any more time to spend on this one, so > I'll leave the rest to someone else. I hope this is helpful: > > The only way I immediately see for the schedulerResumptionTime to become > approximately doubled is if the Delay's resumption time is adjusted by > #restoreResumptionTimes without previously having been adjusted by > #saveResumptionTimes. > > The only time either of those are sent, that I can see, is on saving the > image. Both are normally sent, (save before the snapshot, restore > afterwards), but there may be a hole there. > > #saveResumptionTimes is only sent (by this scheduler class) when the > accessProtect semaphore is held, but #handleTimerEvent: is executed in the > timing Process *without* the protection of accessProtect, in the case of the > VM signaling the timingSemaphore. If the VM signals the timingSemaphore, > #handleTimerEvent: could run in the middle of #saveResumptionTimes. If some > Delay expires because of that timer event, our Delay could move from being > the first suspended delay to being the active delay. If that happens after > we've adjusted the active delay, but before we've processed the suspended > delays, that Delay will not get adjusted, and will show the symptoms that > Mariano is seeing. A quick experiment to test this might be in shutDown/#startUp trying... [ self saveResumptionTimes ] valueAt: Processor timingPriority [ self resumeResumptionTimes ] valueAt: Processor timingPriority > > Also, I'm not sure how the Heap that holds the suspendedDelays will react to > being modified in the middle of an enumeration. That might open a larger > window for the problem. > > Regards, > > -Martin > Even if not directly related to Mariano's problem, I agree with your general assessment. I'm not comfortable with the way that #save/#restoreResumptionTimes (which manipulate suspendedDelays) are called from user priority code via #shutDown/#startUp. Per the original code**, accessProtect can't be used inside the timing priority #handleTimerEvent since accessProtect is held by the user priority #schedule when it uses "timingSemaphore signal" to invoke invokes #handleTimerEvent. accessProtect never protected timingPriority manipulation of suspendedDelays by #handleTimerEvent, nor expired delays waking up. But ahhh... the disabling of accessProtect previously prevented new delays being scheduled between a #save and #restore. If a new delay is scheduled after the #save, when it is #restore'd its resumptionTime would be wrong. Waiting in the wings for Pharo 6 I have changes that should help: * have #save/#restoreResumptionTimes *only* called from timing priority event loop (i.e. #handleTimerEvent) * shutDown/startUp suspends/resumes the timing priority event loop, instead of trying to block signals to timingSemaphore I haven't touched it for a few months so I'll need to chase it up to provide a preview. Mariano, can you try DelayMillisecondScheduler (which however is missing some fixes for other issues). cheers -ben

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by Andres Valloud

> Though this only works on Unix, so on Windows it uses the current time > as a seed. It might get better results using #microsecondClockValue > instead of #millisecondClockValue. And I'd think about just taking the > microsecond value and sending it #hashMultiply instead of the weird > things it's doing now. (bitXor with the identity hash of the > UUIDGenerator instance? That depends on the randomness of identity > hashes, which may or may not be very good.) I agree less magic is better. And, more specifically, why would applying basically a linear transformation on a RNG seed produce "better random"? Note both xor and hashMultiply are essentially linear transformations. What is going on with how the RNG is using the seed to set itself up? If the RNG seeds are assumed / preferred to be uniformly distributed over some range of values, I'd consider crypto-pre-digesting the seed candidate (because crypto hashes are engineered to uniformly spread small bit changes over the entire output, roughly speaking --- so one might even use consecutive values starting at 1 without penalty). Or, if /dev/urandom or equivalent is available, how about using it for the initial seed as well? Andres.

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by David T. Lewis

On Thu, May 19, 2016 at 07:57:37AM +0800, Ben Coman wrote: > On Thu, May 19, 2016 at 7:05 AM, Peter Uhn??k <i.uhnak(a)gmail.com> wrote: > > > > Not so random??? not random at all. > > Obligitory cartoon... > https://xkcd.com/221/ > Perfect :-)

May 19, 2016

Re: [Pharo-dev] Problem with delay waiting (OSSubprocess code) on Pharo 5.0

by Martin McClure

On 05/18/2016 03:17 PM, Martin McClure wrote: > On 05/18/2016 08:49 AM, Mariano Martinez Peck wrote: >> Hi guys, >> >> I am seeing a problem in Pharo 5.0 regarding Delay >> wait. I cannot >> explain how this could happened but it does, and it happened to me a >> couple of times (but not fully reproducible). >> > > Hmm. The schedulerResumptionTime is, somehow, being (approximately) > doubled. It's not clear how that can happen, but I'll look a little more. > Mario, is there any chance that you might be saving the image during one of these Delays? This one smells like a race condition, and I think I see something that *might* explain it. But I don't have any more time to spend on this one, so I'll leave the rest to someone else. I hope this is helpful: The only way I immediately see for the schedulerResumptionTime to become approximately doubled is if the Delay's resumption time is adjusted by #restoreResumptionTimes without previously having been adjusted by #saveResumptionTimes. The only time either of those are sent, that I can see, is on saving the image. Both are normally sent, (save before the snapshot, restore afterwards), but there may be a hole there. #saveResumptionTimes is only sent (by this scheduler class) when the accessProtect semaphore is held, but #handleTimerEvent: is executed in the timing Process *without* the protection of accessProtect, in the case of the VM signaling the timingSemaphore. If the VM signals the timingSemaphore, #handleTimerEvent: could run in the middle of #saveResumptionTimes. If some Delay expires because of that timer event, our Delay could move from being the first suspended delay to being the active delay. If that happens after we've adjusted the active delay, but before we've processed the suspended delays, that Delay will not get adjusted, and will show the symptoms that Mariano is seeing. Also, I'm not sure how the Heap that holds the suspendedDelays will react to being modified in the middle of an enumeration. That might open a larger window for the problem. Regards, -Martin

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by Ron Teitelbaum

Hi Peter, You are absolutely correct. There is a lot that can be done to improve gathering entropy on cryptography. We have discussed this a number of times. At 3DICC use the croquet plugin and #gatherEntropy: by Andreas. That works great for us. We have very specific random needs including replicated randoms. it uses /dev/urandom on linux and MacOS and SystemFunction036 in Advapi32.dll on windows In general you hit on the issue. We should have an implementation for #gatherEntropy: in crypto. It should fire off for seeding PRNG during startup, The makeSeed in UUIDGenerator seems insufficient unless you are on linux. It queries the sound system and then defaults to a time calculation. How good that is for you really depends on your specific needs for random. 1) do we really want to have global fixed seed? [Ron Teitelbaum] No 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value [Ron Teitelbaum] Yes it should call gatherEntropy to seed the PRNG 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? [Ron Teitelbaum] No we should either implement a proper gatherEntropy method or use SystemFunction036 in Advapi32.dll for windows like we do in CroquetPlugin. Chris previously pointed out the gatherEntropy requirements from http://www.amazon.com/Practical-Cryptography-Niels-Ferguson/dp/0471223573/r… <http://www.amazon.com/Practical-Cryptography-Niels-Ferguson/dp/0471223573/r…> &qid=1385152889&sr=8-1&keywords=schneier+practical+cryptography that would be a good place to start or we should support the croquet plugin method gatherEntropy in cryptography. All the best, Ron Teitelbaum From: Pharo-dev [mailto:pharo-dev-bounces@lists.pharo.org] On Behalf Of Robert Withers Sent: Wednesday, May 18, 2016 7:26 PM To: Peter UhnÃ¡k; Pharo Development List; Cryptography Mailing List Subject: Re: [Pharo-dev] Random is not random at startup Thanks for cc-ing me. I don't have much exposure to random, so I am cc-ing the Cryptography list, in hopes they might help. This doesn't very random, you're right. Here were Peter's questions: My questions: 1) do we really want to have global fixed seed? 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? Rob On 05/18/2016 07:05 PM, Peter UhnÃ¡k wrote: Hi, (cc-ing Robert Withers as he seems to be working with cryptography and security... as this seems related and may have some implications, but I am likely wrong about the implications) yesterday I've encountered a very surprising behavior I executed the same script `10 atRandom` on the same image without saving it and got the same output: while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom' done 10 10 10 10 10 10 Not so randomâ¦ not random at all. Apparently the default random generator uses SharedRandom pool, that is initialized only onceâ¦ so every time you start an image you get the EXACT same random seed... I think this is stupid, and I am not sure what are the security implications of this (e.g. when opening an SSL connectionâ¦ having fixed world-wide initial seed seems like an awful, awful idea), but whateverâ¦ So instead I tried to explicitly specify the Random generatorâ¦ which I can do while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom: Random new' done 5 5 5 5 5 Still not randomâ¦ what? while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval 'Random new instVarNamed: #seed' done 426306047 426305545 426305546 426306010 So the seed is different but thanks to the magic of masking the seed, I always get the same first several bitsâ¦ thus the same result for small numbers. So if I actually want what seems like a random valueâ¦ I have to at least once run the generatorâ¦ while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom: (Random new next; yourself)' done 7 3 4 9 6 7 Once I start to use it the properties of the algo kick in so it's pseudo-randomâ¦ but I need to run it once to initialize it, which is wtf. My questions: 1) do we really want to have global fixed seed? 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? Peter

May 19, 2016