- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] WorkingSession UUID looks "sketchy"

by Sven Van Caekenberghe

> On 6 Feb 2017, at 23:01, werner kassens <wkassens(a)libello.com> wrote: > > On 02/06/2017 09:00 PM, Sven Van Caekenberghe wrote: >> Regarding the poor seed at startup: >>> 1k outside runs of 'NeoUUIDGenerator new nextRandom16' (on a fresh image) gives me only 116 unique values, compared to the expected 990-1000 >>> In the above it's already second run of the generator, for (random initial) counter, there was only 69 unique values out of 1000. >> But that is not proper use of a random generator, you create a new instance every time. You basically test seeding, which is different from random number generator proper. >> > Hi, isnt it that, what the thread originally was about? as far as i understand it, it started with yuriy saying: > > The idea is that each time an image is started a new session is created and assigned a new UUID... Certainly for NeoUUIDGenerator, that will be the case, AFAIK. > werner

Feb. 6, 2017

Re: [Pharo-dev] WorkingSession UUID looks "sketchy"

by Sven Van Caekenberghe

> On 6 Feb 2017, at 22:59, Peter Uhnak <i.uhnak(a)gmail.com> wrote: > > >>> On Windows (10) (with latest 6 image and VM) `Time microsecondClockValue` returns microseconds, but (presumably the system) cannot give precision beyond 1 second - this will imho need a VM fix; >> >> I find that quite hard to believe and I did not know that. Are you really sure ? That would be terrible. A solution might be to use another clock primitive. >> >>> I can also generate about 1.2M UUIDs per second (limited by single core I guess), which means that about every 600-1200 consequential UUIDs will have the same clock value. >> >> I can't follow your reasoning here: if the clock precision would be 1 second, you would get 1.2M consecutive UUIDs with the same clock value, right ? > > Whoops, I meant the precision on a millisecond, not second... thus the ~1000k consecutive clock values. > The returned value is still in microseconds, but the last 3 digits are fixed (they are different on startup, but same during execution). > > I faintly remember having the same issue in a different language years ago, because they had different APIs or some bullshit like that. > > So when I generate e.g. 10 random UUIDs I get: > > > (1 to: 10) collect: [ :i | NeoUUIDGenerator next ]. "an Array( > an UUID('349b2ccc-4404-0d00-a8e7-5c9c0ef2da37') > an UUID('349b2ccc-4404-0d00-a8e8-2f3a0ef2da37') > an UUID('349b2ccc-4404-0d00-a8e9-9e0b0ef2da37') > an UUID('349b2ccc-4404-0d00-a8ea-15520ef2da37') > an UUID('349b2ccc-4404-0d00-a8eb-fa5c0ef2da37') > an UUID('349b2ccc-4404-0d00-a8ec-d25f0ef2da37') > an UUID('349b2ccc-4404-0d00-a8ed-6a590ef2da37') > an UUID('349b2ccc-4404-0d00-a8ee-3a8d0ef2da37') > an UUID('349b2ccc-4404-0d00-a8ef-0ad20ef2da37') > an UUID('349b2ccc-4404-0d00-a8f0-7bda0ef2da37'))" I don't see any problem here, they are all different. Of course the clock part is the same during the same millisecond. >> In the unit tests this is verified (that the time value goes forward on consecutive calls). >> > > testTwo*Generator only tests if they are close, not that one is higher than the other, > > if I add > > self assert: time2 equals: time1. > > the test will still pass (but that is already obvious from the list above). > > > >>> 9 & 10 bytes are counter, but 9th bit is rewritten with variant, so the counter is actually 0-255 and not 0-65536. >> >> Yes, but more correctly, the top 2 bits are set to 10, making the range 1 to 2^14 (16384) (instead of 2^16). > > Welp, I can't do bit math. > >> >>> And finally 11 & 12 are random bits (assuming the seeding isn't broken). >> >> The seeding is one aspect (the quality of the seed), the algorithm is another. >> Note that in the current Random class, the seed is initialised using the clock as well. > > The problem with Random is not the time-based initialization, but masking of the initial value (which is mentioned in the originally linked thread on randomness, which fixes itself upon further generations). > >> >>> So on Windows, the conditional probability that nth and n+256th UUIDs will be identical is imho 1/65536 (assuming they are in the same second, which is easy). >> >> I am pretty/quite sure this is not correct but it would take me much more time to come up with a more correct calculation. > > With the actual counter I would have to generate 16384 UUID/milisecond, which unlike 256/ms I cannot do, so it is safe again. :) (well... until Pharo goes multicore) > >> >> BTW, you also have to define the context of being the same: the same instance of one NeoUUIDGenerator generating the same UUID, or several different generator instances, in the same image, in different images, on different runs, the same or different machines/networks, ... >> >>> On Linux my understanding is that clash can only happen if NTP adjusts my clock during UUID generation (at which point it is same as Windows). >>> >>> Can UUID clash be achieved on Linux if you deploy copies of the same image and let them all generate UUIDs? (It should be again 1/65536). > > Yeah, across machines the last four bytes behave randomly too. > > Maybe I can get clash if I run two random generators in forks (to fix 14 bits of counter randomness :) ... but no time for that now. :) > > Peter >

Feb. 6, 2017

Re: [Pharo-dev] FileStreams and encoding ( opening a readStream on a FileRerefence)

by Sven Van Caekenberghe

> On 6 Feb 2017, at 22:33, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > 2017-02-04 13:40 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > > On 4 Feb 2017, at 13:01, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > > > > > 2017-02-04 12:49 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > Hi Nicolai, > > > > The FileSystem API is a bit inconsistent, yes. > > > > This is how you can use it: > > > > (FileLocator temp / 'foo.txt') writeStreamDo: [ :out | > > out binary. > > (ZnCharacterWriteStream on: out encoding: #utf8) << 'Ã©lÃ¨ve' ]. > > > > (FileLocator temp / 'foo.txt') readStreamDo: [ :in | > > in binary. > > ZnCharacterReadStream on: in encoding: #utf8) upToEnd ]. > > > > (FileLocator temp / 'foo.txt') binaryReadStreamDo: [ :in | > > (ZnCharacterReadStream on: in encoding: #utf8) upToEnd ]. > > > > There is no #binaryWriteStreamDo: > > > > The API around File is more correct, IMHO. > > > > Does this help ? > > > > Yes, thanks for the fast response. > > > > What exactly is your question ? > > > > I am looking at the issues with FileList, there are some parts that don't work anymore (see FileList>>#contents, it calls some unimplemted methods), and it uses TextConverter and somoe parts of the older File api. > > It looks like most other (newer) parts are using ZnCharacterReadstream for encoding, but I couldn't find a way to use it together with FileReferences (most parts of the FileList already operate with the newer FileSystem API. > > OK. > > From my standpoint and understanding, I would always use binary streams with explicit Zn converters, this combination is much easier to understand and better implemented with more features. > > If you encounter any problem or have any questions, I will gladly try to help you. > > :-) > Is there a way to list all supported encodings, similar to TextConverter class >> #allEncodingNames ? > I only found > ZnSimplifiedByteEncoder class >> #knownEncodingIdentifiers > and > ZnByteEncoder class >> #knownEncodingIdentifiers > and nothing for all supported utf-8/16 ... encodings. Ah, yes, you are right, I should add a #knownEncodingIdentifiers at the level of ZnCharacterEncoder that returns the union of all of them. I will add that tomorrow. > nicolai > > > > Sven > > > > > On 4 Feb 2017, at 12:09, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > > > Hi > > > How can I specify the character encoding when opening a readStream on a FileRerefence. > > > > > > I found this, that works: > > > > > > | readStream fileContent | > > > readStream := (File named: aFileName) openForRead. > > > fileContent := ZnCharacterReadStream on: readStream encoding: encoding. > > > fileContent upToEnd asString. > > > > > > But if I try to do the same with a readStream from a FileReference > > > > > > | readStream fileContent | > > > readStream := aFileName asFileReference readStream. > > > fileContent := ZnCharacterReadStream on: readStream encoding: encoding. > > > fileContent upToEnd asString. > > > > > > I get an error SmallInteger DNU #asciiValue, > > > > > > this is because, in the first method, we create a binary filestream, and if we > > > use readStream from a FileReference, the stream is a MultibyteFileStream. > > > > > > How can I us ZnEncoder for a readstream from a FileReference? > > > > > > (and is it on purpose that both readStream method (openForRead/readStream) > > > return different kinds of binary streams?) > > > > > > > > > nicolai

Feb. 6, 2017

Re: [Pharo-dev] WorkingSession UUID looks "sketchy"

by werner kassens

On 02/06/2017 09:00 PM, Sven Van Caekenberghe wrote: > Regarding the poor seed at startup: >> 1k outside runs of 'NeoUUIDGenerator new nextRandom16' (on a fresh image) gives me only 116 unique values, compared to the expected 990-1000 >> In the above it's already second run of the generator, for (random initial) counter, there was only 69 unique values out of 1000. > But that is not proper use of a random generator, you create a new instance every time. You basically test seeding, which is different from random number generator proper. > Hi, isnt it that, what the thread originally was about? as far as i understand it, it started with yuriy saying: The idea is that each time an image is started a new session is created and assigned a new UUID... werner

Feb. 6, 2017

Re: [Pharo-dev] WorkingSession UUID looks "sketchy"

by Peter Uhnak

> > On Windows (10) (with latest 6 image and VM) `Time microsecondClockValue` returns microseconds, but (presumably the system) cannot give precision beyond 1 second - this will imho need a VM fix; > > I find that quite hard to believe and I did not know that. Are you really sure ? That would be terrible. A solution might be to use another clock primitive. > > > I can also generate about 1.2M UUIDs per second (limited by single core I guess), which means that about every 600-1200 consequential UUIDs will have the same clock value. > > I can't follow your reasoning here: if the clock precision would be 1 second, you would get 1.2M consecutive UUIDs with the same clock value, right ? Whoops, I meant the precision on a millisecond, not second... thus the ~1000k consecutive clock values. The returned value is still in microseconds, but the last 3 digits are fixed (they are different on startup, but same during execution). I faintly remember having the same issue in a different language years ago, because they had different APIs or some bullshit like that. So when I generate e.g. 10 random UUIDs I get: (1 to: 10) collect: [ :i | NeoUUIDGenerator next ]. "an Array( an UUID('349b2ccc-4404-0d00-a8e7-5c9c0ef2da37') an UUID('349b2ccc-4404-0d00-a8e8-2f3a0ef2da37') an UUID('349b2ccc-4404-0d00-a8e9-9e0b0ef2da37') an UUID('349b2ccc-4404-0d00-a8ea-15520ef2da37') an UUID('349b2ccc-4404-0d00-a8eb-fa5c0ef2da37') an UUID('349b2ccc-4404-0d00-a8ec-d25f0ef2da37') an UUID('349b2ccc-4404-0d00-a8ed-6a590ef2da37') an UUID('349b2ccc-4404-0d00-a8ee-3a8d0ef2da37') an UUID('349b2ccc-4404-0d00-a8ef-0ad20ef2da37') an UUID('349b2ccc-4404-0d00-a8f0-7bda0ef2da37'))" > > In the unit tests this is verified (that the time value goes forward on consecutive calls). > testTwo*Generator only tests if they are close, not that one is higher than the other, if I add self assert: time2 equals: time1. the test will still pass (but that is already obvious from the list above). > > 9 & 10 bytes are counter, but 9th bit is rewritten with variant, so the counter is actually 0-255 and not 0-65536. > > Yes, but more correctly, the top 2 bits are set to 10, making the range 1 to 2^14 (16384) (instead of 2^16). Welp, I can't do bit math. > > > And finally 11 & 12 are random bits (assuming the seeding isn't broken). > > The seeding is one aspect (the quality of the seed), the algorithm is another. > Note that in the current Random class, the seed is initialised using the clock as well. The problem with Random is not the time-based initialization, but masking of the initial value (which is mentioned in the originally linked thread on randomness, which fixes itself upon further generations). > > > So on Windows, the conditional probability that nth and n+256th UUIDs will be identical is imho 1/65536 (assuming they are in the same second, which is easy). > > I am pretty/quite sure this is not correct but it would take me much more time to come up with a more correct calculation. With the actual counter I would have to generate 16384 UUID/milisecond, which unlike 256/ms I cannot do, so it is safe again. :) (well... until Pharo goes multicore) > > BTW, you also have to define the context of being the same: the same instance of one NeoUUIDGenerator generating the same UUID, or several different generator instances, in the same image, in different images, on different runs, the same or different machines/networks, ... > > > On Linux my understanding is that clash can only happen if NTP adjusts my clock during UUID generation (at which point it is same as Windows). > > > > Can UUID clash be achieved on Linux if you deploy copies of the same image and let them all generate UUIDs? (It should be again 1/65536). Yeah, across machines the last four bytes behave randomly too. Maybe I can get clash if I run two random generators in forks (to fix 14 bits of counter randomness :) ... but no time for that now. :) Peter

Feb. 6, 2017

Re: [Pharo-dev] FileStreams and encoding ( opening a readStream on a FileRerefence)

by Nicolai Hess

2017-02-04 13:40 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > > On 4 Feb 2017, at 13:01, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > > > > > 2017-02-04 12:49 GMT+01:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > Hi Nicolai, > > > > The FileSystem API is a bit inconsistent, yes. > > > > This is how you can use it: > > > > (FileLocator temp / 'foo.txt') writeStreamDo: [ :out | > > out binary. > > (ZnCharacterWriteStream on: out encoding: #utf8) << 'Ã©lÃ¨ve' ]. > > > > (FileLocator temp / 'foo.txt') readStreamDo: [ :in | > > in binary. > > ZnCharacterReadStream on: in encoding: #utf8) upToEnd ]. > > > > (FileLocator temp / 'foo.txt') binaryReadStreamDo: [ :in | > > (ZnCharacterReadStream on: in encoding: #utf8) upToEnd ]. > > > > There is no #binaryWriteStreamDo: > > > > The API around File is more correct, IMHO. > > > > Does this help ? > > > > Yes, thanks for the fast response. > > > > What exactly is your question ? > > > > I am looking at the issues with FileList, there are some parts that > don't work anymore (see FileList>>#contents, it calls some unimplemted > methods), and it uses TextConverter and somoe parts of the older File api. > > It looks like most other (newer) parts are using ZnCharacterReadstream > for encoding, but I couldn't find a way to use it together with > FileReferences (most parts of the FileList already operate with the newer > FileSystem API. > > OK. > > From my standpoint and understanding, I would always use binary streams > with explicit Zn converters, this combination is much easier to understand > and better implemented with more features. > > If you encounter any problem or have any questions, I will gladly try to > help you. > :-) Is there a way to list all supported encodings, similar to TextConverter class >> #allEncodingNames ? I only found ZnSimplifiedByteEncoder class >> #knownEncodingIdentifiers and ZnByteEncoder class >> #knownEncodingIdentifiers and nothing for all supported utf-8/16 ... encodings. nicolai > > > Sven > > > > > On 4 Feb 2017, at 12:09, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > > > Hi > > > How can I specify the character encoding when opening a readStream on > a FileRerefence. > > > > > > I found this, that works: > > > > > > | readStream fileContent | > > > readStream := (File named: aFileName) openForRead. > > > fileContent := ZnCharacterReadStream on: readStream encoding: encoding. > > > fileContent upToEnd asString. > > > > > > But if I try to do the same with a readStream from a FileReference > > > > > > | readStream fileContent | > > > readStream := aFileName asFileReference readStream. > > > fileContent := ZnCharacterReadStream on: readStream encoding: encoding. > > > fileContent upToEnd asString. > > > > > > I get an error SmallInteger DNU #asciiValue, > > > > > > this is because, in the first method, we create a binary filestream, > and if we > > > use readStream from a FileReference, the stream is a > MultibyteFileStream. > > > > > > How can I us ZnEncoder for a readstream from a FileReference? > > > > > > (and is it on purpose that both readStream method > (openForRead/readStream) > > > return different kinds of binary streams?) > > > > > > > > > nicolai > > >

Feb. 6, 2017

Re: [Pharo-dev] [QA] Debugger buttons explained

by Andrei Chis

On Mon, Feb 6, 2017 at 9:14 PM, Hilaire <hilaire(a)drgeo.eu> wrote: > Thanks for the updated message. > > Le 06/02/2017 Ã 19:39, Sven Van Caekenberghe a Ã©crit : > > That last one is not correct. It has nothing to do with specific > iteration selectors. It means: stop when execution returns in one of the > argument blocks. It also works for #on:error: #streamContents: or any other > message that accepts blocks as argument - which is pretty cool. > > Strangely step over does not step in the bloc with #do: but does with > #to:do: > > In the later case step over and through behave mostly the same, the > stack looks a bit different though. > > > Another question, I try to locate where the tooltips are set in the > debugger. I can't find it with traditional browser tools (send, class > use) and the finder: > > The finder tells me "Through" is used in SpecThroughDebugAction, but > browser tells me there is no use case of this class. I see some pragma > in class side method, did it... > actionType > <debuggingAction> > <contextMenuDebuggingAction> > > No user of actionType, but I guess user or pragma, how do you guys track > the logic from here > Seems that the logic for changing the tooltips is not that straightforward. If you ping me once you decided for a the text of the tooltip I can add them. If you want to look at it yourself here are some details: Each action in the debugger is modeled as a subclass of DebugAction. The Glamour interface of the debugger uses DebugAction>>#asGlamourAction to transform that into a Glamour action. This action is then rendered in GLMMorphicActionRenderer>>#render:. There the tooltip is set directly by concatenating the title and the shortcut of the action. Best solution would be to add a description attribute to DebugAction and GLMAction and then modify GLMMorphicActionRenderer>>#render: to use it. For the Spec interface of the debugger you need to adapt DebugAction>>#asMenuRegistrationIn: Cheers, Andrei > > > -- > Dr. Geo > http://drgeo.eu > > >

Feb. 6, 2017

Re: [Pharo-dev] [QA] Debugger buttons explained

by Hilaire

Ok, give a look to SpecDebugActionButton, this class has no user, only progma reference, not easy to track the logic. Never mind I test on a halt in its #update method. Nothing happen when opening the debugger. May be there is instance in memory, no idea under which name it is referenced, may be not. Inspecting the debugger window does not help, it is a kind of generic morph. Never mind, can try to look at the menu item "Debug it" in the playground window, I used to find the action there. Oops, get lost there, I see #debugSelection looks promising, see announcement. Get lost again. No idea, someone else should put these tooltips in the appropriate place. Le 06/02/2017 Ã 21:14, Hilaire a Ã©crit : > No user of actionType, but I guess user or pragma, how do you guys track > the logic from here -- Dr. Geo http://drgeo.eu

Feb. 6, 2017

Re: [Pharo-dev] [QA] Debugger buttons explained

by Sean P. DeNigris

How about... Into Step into the highlighted message, i.e. dive the debugger into the method which handles it Over Step over the highlighted message, i.e. execute it, but keep the debugger focused on the current method Through Like stepping "Over", except step through block contents when executed Restart Rewind to the start of the current execution context, resetting all local variables Continue Relinquish debugger control and proceed with normal execution ----- Cheers, Sean -- View this message in context: http://forum.world.st/QA-Debugger-buttons-explained-tp4933212p4933238.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.

Feb. 6, 2017

Re: [Pharo-dev] [QA] Debugger buttons explained

by Hilaire

Thanks for the updated message. Le 06/02/2017 Ã 19:39, Sven Van Caekenberghe a Ã©crit : > That last one is not correct. It has nothing to do with specific iteration selectors. It means: stop when execution returns in one of the argument blocks. It also works for #on:error: #streamContents: or any other message that accepts blocks as argument - which is pretty cool. Strangely step over does not step in the bloc with #do: but does with #to:do: In the later case step over and through behave mostly the same, the stack looks a bit different though. Another question, I try to locate where the tooltips are set in the debugger. I can't find it with traditional browser tools (send, class use) and the finder: The finder tells me "Through" is used in SpecThroughDebugAction, but browser tells me there is no use case of this class. I see some pragma in class side method, did it... actionType <debuggingAction> <contextMenuDebuggingAction> No user of actionType, but I guess user or pragma, how do you guys track the logic from here -- Dr. Geo http://drgeo.eu

Feb. 6, 2017