- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Random is not random at startup

by Andres Valloud

> Though this only works on Unix, so on Windows it uses the current time > as a seed. It might get better results using #microsecondClockValue > instead of #millisecondClockValue. And I'd think about just taking the > microsecond value and sending it #hashMultiply instead of the weird > things it's doing now. (bitXor with the identity hash of the > UUIDGenerator instance? That depends on the randomness of identity > hashes, which may or may not be very good.) I agree less magic is better. And, more specifically, why would applying basically a linear transformation on a RNG seed produce "better random"? Note both xor and hashMultiply are essentially linear transformations. What is going on with how the RNG is using the seed to set itself up? If the RNG seeds are assumed / preferred to be uniformly distributed over some range of values, I'd consider crypto-pre-digesting the seed candidate (because crypto hashes are engineered to uniformly spread small bit changes over the entire output, roughly speaking --- so one might even use consecutive values starting at 1 without penalty). Or, if /dev/urandom or equivalent is available, how about using it for the initial seed as well? Andres.

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by David T. Lewis

On Thu, May 19, 2016 at 07:57:37AM +0800, Ben Coman wrote: > On Thu, May 19, 2016 at 7:05 AM, Peter Uhn??k <i.uhnak(a)gmail.com> wrote: > > > > Not so random??? not random at all. > > Obligitory cartoon... > https://xkcd.com/221/ > Perfect :-)

May 19, 2016

Re: [Pharo-dev] Problem with delay waiting (OSSubprocess code) on Pharo 5.0

by Martin McClure

On 05/18/2016 03:17 PM, Martin McClure wrote: > On 05/18/2016 08:49 AM, Mariano Martinez Peck wrote: >> Hi guys, >> >> I am seeing a problem in Pharo 5.0 regarding Delay >> wait. I cannot >> explain how this could happened but it does, and it happened to me a >> couple of times (but not fully reproducible). >> > > Hmm. The schedulerResumptionTime is, somehow, being (approximately) > doubled. It's not clear how that can happen, but I'll look a little more. > Mario, is there any chance that you might be saving the image during one of these Delays? This one smells like a race condition, and I think I see something that *might* explain it. But I don't have any more time to spend on this one, so I'll leave the rest to someone else. I hope this is helpful: The only way I immediately see for the schedulerResumptionTime to become approximately doubled is if the Delay's resumption time is adjusted by #restoreResumptionTimes without previously having been adjusted by #saveResumptionTimes. The only time either of those are sent, that I can see, is on saving the image. Both are normally sent, (save before the snapshot, restore afterwards), but there may be a hole there. #saveResumptionTimes is only sent (by this scheduler class) when the accessProtect semaphore is held, but #handleTimerEvent: is executed in the timing Process *without* the protection of accessProtect, in the case of the VM signaling the timingSemaphore. If the VM signals the timingSemaphore, #handleTimerEvent: could run in the middle of #saveResumptionTimes. If some Delay expires because of that timer event, our Delay could move from being the first suspended delay to being the active delay. If that happens after we've adjusted the active delay, but before we've processed the suspended delays, that Delay will not get adjusted, and will show the symptoms that Mariano is seeing. Also, I'm not sure how the Heap that holds the suspendedDelays will react to being modified in the middle of an enumeration. That might open a larger window for the problem. Regards, -Martin

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by Ron Teitelbaum

Hi Peter, You are absolutely correct. There is a lot that can be done to improve gathering entropy on cryptography. We have discussed this a number of times. At 3DICC use the croquet plugin and #gatherEntropy: by Andreas. That works great for us. We have very specific random needs including replicated randoms. it uses /dev/urandom on linux and MacOS and SystemFunction036 in Advapi32.dll on windows In general you hit on the issue. We should have an implementation for #gatherEntropy: in crypto. It should fire off for seeding PRNG during startup, The makeSeed in UUIDGenerator seems insufficient unless you are on linux. It queries the sound system and then defaults to a time calculation. How good that is for you really depends on your specific needs for random. 1) do we really want to have global fixed seed? [Ron Teitelbaum] No 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value [Ron Teitelbaum] Yes it should call gatherEntropy to seed the PRNG 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? [Ron Teitelbaum] No we should either implement a proper gatherEntropy method or use SystemFunction036 in Advapi32.dll for windows like we do in CroquetPlugin. Chris previously pointed out the gatherEntropy requirements from http://www.amazon.com/Practical-Cryptography-Niels-Ferguson/dp/0471223573/r… <http://www.amazon.com/Practical-Cryptography-Niels-Ferguson/dp/0471223573/r…> &qid=1385152889&sr=8-1&keywords=schneier+practical+cryptography that would be a good place to start or we should support the croquet plugin method gatherEntropy in cryptography. All the best, Ron Teitelbaum From: Pharo-dev [mailto:pharo-dev-bounces@lists.pharo.org] On Behalf Of Robert Withers Sent: Wednesday, May 18, 2016 7:26 PM To: Peter UhnÃ¡k; Pharo Development List; Cryptography Mailing List Subject: Re: [Pharo-dev] Random is not random at startup Thanks for cc-ing me. I don't have much exposure to random, so I am cc-ing the Cryptography list, in hopes they might help. This doesn't very random, you're right. Here were Peter's questions: My questions: 1) do we really want to have global fixed seed? 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? Rob On 05/18/2016 07:05 PM, Peter UhnÃ¡k wrote: Hi, (cc-ing Robert Withers as he seems to be working with cryptography and security... as this seems related and may have some implications, but I am likely wrong about the implications) yesterday I've encountered a very surprising behavior I executed the same script `10 atRandom` on the same image without saving it and got the same output: while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom' done 10 10 10 10 10 10 Not so randomâ¦ not random at all. Apparently the default random generator uses SharedRandom pool, that is initialized only onceâ¦ so every time you start an image you get the EXACT same random seed... I think this is stupid, and I am not sure what are the security implications of this (e.g. when opening an SSL connectionâ¦ having fixed world-wide initial seed seems like an awful, awful idea), but whateverâ¦ So instead I tried to explicitly specify the Random generatorâ¦ which I can do while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom: Random new' done 5 5 5 5 5 Still not randomâ¦ what? while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval 'Random new instVarNamed: #seed' done 426306047 426305545 426305546 426306010 So the seed is different but thanks to the magic of masking the seed, I always get the same first several bitsâ¦ thus the same result for small numbers. So if I actually want what seems like a random valueâ¦ I have to at least once run the generatorâ¦ while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom: (Random new next; yourself)' done 7 3 4 9 6 7 Once I start to use it the properties of the algo kick in so it's pseudo-randomâ¦ but I need to run it once to initialize it, which is wtf. My questions: 1) do we really want to have global fixed seed? 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? Peter

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by Martin McClure

On 05/18/2016 04:05 PM, Peter UhnÃ¡k wrote: > My questions: > 1) do we really want to have global fixed seed? No! > 2) Random new should actually setup a usable seed so I don't need to > first run it N times before I can use the value Yes. > 3) Should we switch to what UUIDGenerator is usingâ¦ reading > /dev/urandom for the initial seed setup? > Yes. Though this only works on Unix, so on Windows it uses the current time as a seed. It might get better results using #microsecondClockValue instead of #millisecondClockValue. And I'd think about just taking the microsecond value and sending it #hashMultiply instead of the weird things it's doing now. (bitXor with the identity hash of the UUIDGenerator instance? That depends on the randomness of identity hashes, which may or may not be very good.) It seems likely that Windows has a built-in random number generator, as well, which would probably be better. While we're at it, the class Random is a Park-Miller generator, which has quite poor randomness by modern standards. Most other Smalltalks have upgraded -- GemStone uses CMWC (pure Smalltalk, very simple and quite fast; I wrote that one), VW I believe uses Lagged Fibonacci, and Squeak has moved to Mersenne Twister last I heard. I've tested all three against statistical tests of randomness -- generated about 650GB of random bytes from each, ran them through the tests (which require that much data to get good statistics). All three are pretty good, only failing a few tests. The only generator I tested that passed *all* tests was Linux /dev/urandom. Which is also fast. If you're on Linux, and don't need a repeatable random sequence, I'd use /dev/urandom. If you do need to be able to have a repeatable sequence with good randomness properties, I'd use one of the three generators I mentioned above. Of the three, the simplest and fastest is CMWC. Regards, -Martin

May 19, 2016

Re: [Pharo-dev] Random is not random at startup

by Ben Coman

On Thu, May 19, 2016 at 7:05 AM, Peter UhnÃ¡k <i.uhnak(a)gmail.com> wrote: > Hi, > > (cc-ing Robert Withers as he seems to be working with cryptography and > security... as this seems related and may have some implications, but I am > likely wrong about the implications) > > yesterday I've encountered a very surprising behavior > > I executed the same script `10 atRandom` on the same image without saving it > and got the same output: > > while true; do > pharo-vm --nodisplay latest.image --no-default-preferences eval '10 > atRandom' > done > 10 > 10 > 10 > 10 > 10 > 10 > > Not so randomâ¦ not random at all. Obligitory cartoon... https://xkcd.com/221/ cheers -ben

May 18, 2016

Re: [Pharo-dev] Random is not random at startup

by Robert Withers

Thanks for cc-ing me. I don't have much exposure to random, so I am cc-ing the Cryptography list, in hopes they might help. This doesn't very random, you're right. Here were Peter's questions: My questions: 1) do we really want to have global fixed seed? 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? Rob On 05/18/2016 07:05 PM, Peter UhnÃ¡k wrote: > Hi, > > (cc-ing Robert Withers as he seems to be working with cryptography and > security... as this seems related and may have some implications, but > I am likely wrong about the implications) > > yesterday I've encountered a very surprising behavior > > I executed the same script `10 atRandom` on the same image without > saving it and got the same output: > > while true; do > pharo-vm --nodisplay latest.image --no-default-preferences > eval '10 atRandom' > done > 10 > 10 > 10 > 10 > 10 > 10 > > Not so randomâ¦ not random at all. > > Apparently the default random generator uses SharedRandom pool, that > is initialized only onceâ¦ so every time you start an image you get the > EXACT same random seed... I think this is stupid, and I am not sure > what are the security implications of this (e.g. when opening an SSL > connectionâ¦ having fixed world-wide initial seed seems like an awful, > awful idea), but whateverâ¦ > > So instead I tried to explicitly specify the Random generatorâ¦ which I > can do > > while true; do > pharo-vm --nodisplay latest.image --no-default-preferences > eval '10 atRandom: Random new' > done > 5 > 5 > 5 > 5 > 5 > > Still not randomâ¦ what? > > while true; do > pharo-vm --nodisplay latest.image --no-default-preferences > eval 'Random new instVarNamed: #seed' > done > 426306047 > 426305545 > 426305546 > 426306010 > > So the seed is different but thanks to the magic of masking the seed, > I always get the same first several bitsâ¦ thus the same result for > small numbers. > > So if I actually want what seems like a random valueâ¦ I have to at > least once run the generatorâ¦ > > while true; do > pharo-vm --nodisplay latest.image --no-default-preferences > eval '10 atRandom: (Random new next; yourself)' > done > 7 > 3 > 4 > 9 > 6 > 7 > > Once I start to use it the properties of the algo kick in so it's > pseudo-randomâ¦ but I need to run it once to initialize it, which is wtf. > > My questions: > 1) do we really want to have global fixed seed? > 2) Random new should actually setup a usable seed so I don't need to > first run it N times before I can use the value > 3) Should we switch to what UUIDGenerator is usingâ¦ reading > /dev/urandom for the initial seed setup? > > Peter

May 18, 2016

Random is not random at startup

by Peter UhnÃ¡k

Hi, (cc-ing Robert Withers as he seems to be working with cryptography and security... as this seems related and may have some implications, but I am likely wrong about the implications) yesterday I've encountered a very surprising behavior I executed the same script `10 atRandom` on the same image without saving it and got the same output: while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom' done 10 10 10 10 10 10 Not so randomâ¦ not random at all. Apparently the default random generator uses SharedRandom pool, that is initialized only onceâ¦ so every time you start an image you get the EXACT same random seed... I think this is stupid, and I am not sure what are the security implications of this (e.g. when opening an SSL connectionâ¦ having fixed world-wide initial seed seems like an awful, awful idea), but whateverâ¦ So instead I tried to explicitly specify the Random generatorâ¦ which I can do while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom: Random new' done 5 5 5 5 5 Still not randomâ¦ what? while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval 'Random new instVarNamed: #seed' done 426306047 426305545 426305546 426306010 So the seed is different but thanks to the magic of masking the seed, I always get the same first several bitsâ¦ thus the same result for small numbers. So if I actually want what seems like a random valueâ¦ I have to at least once run the generatorâ¦ while true; do pharo-vm --nodisplay latest.image --no-default-preferences eval '10 atRandom: (Random new next; yourself)' done 7 3 4 9 6 7 Once I start to use it the properties of the algo kick in so it's pseudo-randomâ¦ but I need to run it once to initialize it, which is wtf. My questions: 1) do we really want to have global fixed seed? 2) Random new should actually setup a usable seed so I don't need to first run it N times before I can use the value 3) Should we switch to what UUIDGenerator is usingâ¦ reading /dev/urandom for the initial seed setup? Peter

May 18, 2016

Re: [Pharo-dev] Problem with delay waiting (OSSubprocess code) on Pharo 5.0

by Mariano Martinez Peck

On Wed, May 18, 2016 at 7:28 PM, Martin McClure <martin(a)hand2mouse.com> wrote: > On 05/18/2016 02:46 PM, Paul DeBruicker wrote: > >> the reason I asked about OSProcess in Pharo 5 was that >> >> OSProcess waitForCommand: 'ls' >> >> >> never returned. >> >> >> Maybe its another instance of this problem. And it happens everytime. >> >> There is a 50ms delay in the #waitForCommand: method >> >> I'm currently having a problem with an OSProcess of a chmod never > returning. And that's in Pharo 4! Never seen a problem there before. > Investigating. > > Thanks Martin. I cannot tell about 4.0 as OSSubprocess only works in 5.0... What I know is that in 5.0 there have been changes in the wait delay (that's why Ben answered). > Regards, > > -Martin > > -- Mariano http://marianopeck.wordpress.com

May 18, 2016

Re: [Pharo-dev] Problem with delay waiting (OSSubprocess code) on Pharo 5.0

by Mariano Martinez Peck

On Wed, May 18, 2016 at 6:46 PM, Paul DeBruicker <pdebruic(a)gmail.com> wrote: > the reason I asked about OSProcess in Pharo 5 was that > > OSProcess waitForCommand: 'ls' > > > never returned. > > > Maybe its another instance of this problem. And it happens everytime. > > There is a 50ms delay in the #waitForCommand: method > It looks quite similar indeed. And... both, OSProcess and OSSubprocess have a similar child watcher (I based mine in OSProcess one). Paul could you inspect the child process from the process browser tool, do an inspect, and check the suspended process? Check the wait time to see if it is 50ms or 20ms etc...that way we can see which part is the "hunging". > > > hope this helps > > Paul > > > > > Mariano Martinez Peck wrote > > On Wed, May 18, 2016 at 2:10 PM, Ben Coman < > > > btc@ > > > > wrote: > > > >> I don't have time to look at this properly until tomorrow. > >> In the meantime, could you just post the result of... > >> > >> Delay delaySchedulerClass > >> it should be "DelayExperimentalSpinScheduler" *** > >> > >> > > Yes, it is. > > > > > >> Also could you try a few of the other options from > >> World > System > Settings > System ... > >> > >> > > Do you know which is the "Safest" to use? Or at least which was used in > > say, 4.0? > > > > > > > > > >> > >> ***whoops, the Experimental tag should have been dropped for Pharo 5 > >> Release. Its been fine for months (unless this is a fault :) ] > >> > >> > > Yes, I found that strange too. > > > > > >> > > >> > Maybe it is wrong to keep and re-use same instance of Delay? (note it > >> is > >> > outside the closure) (this was not a problem in the past). > >> > >> Not sure. I've heard others say Delays should only be used once. To > >> experiment, could you try creating a delay each time. > >> > >> > > Ok, I will try this. But again, it would take some some time to test and > > it > > happens to me every in while only. > > > > Thanks!! > > > > > > > > -- > > Mariano > > http://marianopeck.wordpress.com > > > > > > -- > View this message in context: > http://forum.world.st/Problem-with-delay-waiting-OSSubprocess-code-on-Pharo… > Sent from the Pharo Smalltalk Developers mailing list archive at > Nabble.com. > > -- Mariano http://marianopeck.wordpress.com

May 18, 2016