I tried for one hour yesterday to understand the problem :). This morning my priority was to not lose my code because I noticed the bug a couple of hours after my commits... Thanks I remembered to save my image and that the good old fileout in .st is working! I'll keep trying to reproduce and keep you posted On 02/04/2016 10:21 AM, stepharo wrote:

Hi guille

Esteban got some problems with MC recently so it would be good to have a reproducible case.

Stef

Le 4/2/16 10:07, Guille Polito a écrit :

...

Ok, I am puzzled. I downloaded a new fresh image, in a fresh directory, worked on something else, committed, and my commit is completely broken. Attached screenshot of what monticello shows me.

I'm on debian jessie 64bits. Maybe it has something to do?

On 02/03/2016 03:41 PM, Guille Polito wrote:

...

Hi all,

I'm finally back, rechecking this issue:

https://pharo.fogbugz.com/f/cases/7241/Object-name-should-best-be-removed

I remade the Slice to load in latest Pharo5 with the new Spur changes, plus some fixes proposed by Nicolai. I can load the slice in a new image and everything looks ok. So far so good.

Now, the monkey starts checking the Slice and somehow it cannot load the Slice due to "dependencies to some classes". Of course, the slice I submitted does not depend on the packages on the complaint, and locally it loads well. Moreover, I never had/worked with those classes, nor they were installed in my system. Even, I have a new machine which is clean so I cannot believe I have some interference due to some package cache...

The message of the monkey is here:

https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/26128//artifact/vali...

Did somebody find some similar problem or it is just me? I believe the problem is on the monkey side, but I have no clue... Maybe somebody has a better idea.

Thanks! Guille

But StHub did not change for a long time, no ? Saving to an HTTP MC repo is nothing more than an HTTP PUT. You should be able to do that manually with just ZnClient.

On 04 Feb 2016, at 10:52, Guille Polito <guillermopolito@gmail.com> wrote:

So far, I'm blaming Smalltalkhub:

- I create a new empty package. - I commit it to a local directory, it works ok. - I commit it to a smalltalkhub repository: the file in my package cache is ok, but the file in smalltalkhub is corrupted e.g., http://www.smalltalkhub.com/#!/~Guille/playground/versions/test-GuillermoPol...

The strange thing is that it is a recurrent bug. I cannot commit to smalltalkhub, not even do a push of a version in my package cache. Smalltalkhub always shows a buggy version.

I'll now try with squeaksource 3 to narrow my conclusions.

On 02/04/2016 10:33 AM, Guille Polito wrote:

...

I tried for one hour yesterday to understand the problem :). This morning my priority was to not lose my code because I noticed the bug a couple of hours after my commits... Thanks I remembered to save my image and that the good old fileout in .st is working!

I'll keep trying to reproduce and keep you posted

On 02/04/2016 10:21 AM, stepharo wrote:

...

Hi guille

Esteban got some problems with MC recently so it would be good to have a reproducible case.

Stef

Le 4/2/16 10:07, Guille Polito a écrit :

...

Ok, I am puzzled. I downloaded a new fresh image, in a fresh directory, worked on something else, committed, and my commit is completely broken. Attached screenshot of what monticello shows me.

I'm on debian jessie 64bits. Maybe it has something to do?

On 02/03/2016 03:41 PM, Guille Polito wrote:

...

Hi all,

I'm finally back, rechecking this issue:

https://pharo.fogbugz.com/f/cases/7241/Object-name-should-best-be-removed

I remade the Slice to load in latest Pharo5 with the new Spur changes, plus some fixes proposed by Nicolai. I can load the slice in a new image and everything looks ok. So far so good.

Now, the monkey starts checking the Slice and somehow it cannot load the Slice due to "dependencies to some classes". Of course, the slice I submitted does not depend on the packages on the complaint, and locally it loads well. Moreover, I never had/worked with those classes, nor they were installed in my system. Even, I have a new machine which is clean so I cannot believe I have some interference due to some package cache...

The message of the monkey is here:

https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/26128//artifact/vali...

Did somebody find some similar problem or it is just me? I believe the problem is on the monkey side, but I have no clue... Maybe somebody has a better idea.

Thanks! Guille

And yes, linux VM's are not shipped with the UUID plugin... at least not since 2014/2013... Also, I believe the plugin is not compiled as internal since it is not listed in Smalltalk vm listBuiltinModules. Smalltalk vm listLoadedModules. On 02/04/2016 11:12 AM, Guille Polito wrote:

On 02/04/2016 10:58 AM, Sven Van Caekenberghe wrote:

...

But StHub did not change for a long time, no ? I know...

Saving to an HTTP MC repo is nothing more than an HTTP PUT.

You should be able to do that manually with just ZnClient. Well, yes. But I'm just clicking the save button on the monticello browser...

Now, I tested with squeaksource3:

- I commit in squeaksource 3 using the monticello browser: the file in package-cache is OK, the file in squeaksource3 is OK http://ss3.gemtalksystems.com/ss?_s=WvI7T3UuBKEaYUrR&_k=ukyWjav7hA-MhHIz

- I make a copy of the file from the monticello UI to a smalltalkhub repository. The file in smalltalkhub is not the one I want. http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito....

In both Squeaksource3 and Smalltalkhub I have the same UUID, project name, version number, commit message, but the MCZ file that I download from them is different. Even, making a diff from SmalltalkHub's UI shows me a diff against a completely different package (RoboticTable) that is from Santiago Bragagnolo, and that I guess it is a private project because it is not listed and cannot browsed.

Making a diff on other versions, such as http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito.... , makes a diff against a version of the Makros package, which is also from Santiago Bragagnolo, but this is not a private project.

Aaaand, now I see a problem!

both my test package:

http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito....

and the one of Santiago

http://www.smalltalkhub.com/#!/~sbragagnolo/Makros/versions/Makros-SantiagoB...

have the same UUID!!!!!!

Maybe this is because of my debian 64bits? Hmm maybe the UUID plugin is not loading? I'll check that for now, because I cannot work if I cannot commit :). If somebody has an idea, It is VERY welcome!

Thanks for reading my loud ranting/reasoning, Guille

...

...

On 04 Feb 2016, at 10:52, Guille Polito <guillermopolito@gmail.com> wrote:

So far, I'm blaming Smalltalkhub:

- I create a new empty package. - I commit it to a local directory, it works ok. - I commit it to a smalltalkhub repository: the file in my package cache is ok, but the file in smalltalkhub is corrupted e.g., http://www.smalltalkhub.com/#!/~Guille/playground/versions/test-GuillermoPol...

The strange thing is that it is a recurrent bug. I cannot commit to smalltalkhub, not even do a push of a version in my package cache. Smalltalkhub always shows a buggy version.

I'll now try with squeaksource 3 to narrow my conclusions.

On 02/04/2016 10:33 AM, Guille Polito wrote:

...

I tried for one hour yesterday to understand the problem :). This morning my priority was to not lose my code because I noticed the bug a couple of hours after my commits... Thanks I remembered to save my image and that the good old fileout in .st is working!

I'll keep trying to reproduce and keep you posted

On 02/04/2016 10:21 AM, stepharo wrote:

...

Hi guille

Esteban got some problems with MC recently so it would be good to have a reproducible case.

Stef

Le 4/2/16 10:07, Guille Polito a écrit :

...

Ok, I am puzzled. I downloaded a new fresh image, in a fresh directory, worked on something else, committed, and my commit is completely broken. Attached screenshot of what monticello shows me.

I'm on debian jessie 64bits. Maybe it has something to do?

On 02/03/2016 03:41 PM, Guille Polito wrote:

...

Hi all,

I'm finally back, rechecking this issue:

https://pharo.fogbugz.com/f/cases/7241/Object-name-should-best-be-removed

I remade the Slice to load in latest Pharo5 with the new Spur changes, plus some fixes proposed by Nicolai. I can load the slice in a new image and everything looks ok. So far so good.

Now, the monkey starts checking the Slice and somehow it cannot load the Slice due to "dependencies to some classes". Of course, the slice I submitted does not depend on the packages on the complaint, and locally it loads well. Moreover, I never had/worked with those classes, nor they were installed in my system. Even, I have a new machine which is clean so I cannot believe I have some interference due to some package cache...

The message of the monkey is here:

https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/26128//artifact/vali...

Did somebody find some similar problem or it is just me? I believe the problem is on the monkey side, but I have no clue... Maybe somebody has a better idea.

Thanks! Guille

Well, maybe we do not need a plugin, but then the fallback code is not generating unique enough UUIDs :/ Maybe it is a side effect of Spur? I'll try to patch the UUID generator in the image with the Neo one and test if it solves my problem... I just want to commit :'( On 02/04/2016 11:36 AM, Sven Van Caekenberghe wrote:

It beats the hell out of me why we need a plugin (apart from speed) and why the fallback should not work (i.e. generate duplicates). It is not rocket science or magic, right ?

I once made my own generator: Neo-UUID (http://mc.stfx.eu/Neo or Neo on StHub). It is pretty simple and fast.

...

On 04 Feb 2016, at 11:29, Guille Polito <guillermopolito@gmail.com> wrote:

And yes, linux VM's are not shipped with the UUID plugin... at least not since 2014/2013...

Also, I believe the plugin is not compiled as internal since it is not listed in

Smalltalk vm listBuiltinModules. Smalltalk vm listLoadedModules.

On 02/04/2016 11:12 AM, Guille Polito wrote:

...

On 02/04/2016 10:58 AM, Sven Van Caekenberghe wrote:

...

But StHub did not change for a long time, no ? I know... Saving to an HTTP MC repo is nothing more than an HTTP PUT.

You should be able to do that manually with just ZnClient. Well, yes. But I'm just clicking the save button on the monticello browser...

Now, I tested with squeaksource3:

- I commit in squeaksource 3 using the monticello browser: the file in package-cache is OK, the file in squeaksource3 is OK http://ss3.gemtalksystems.com/ss?_s=WvI7T3UuBKEaYUrR&_k=ukyWjav7hA-MhHIz

- I make a copy of the file from the monticello UI to a smalltalkhub repository. The file in smalltalkhub is not the one I want. http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito....

In both Squeaksource3 and Smalltalkhub I have the same UUID, project name, version number, commit message, but the MCZ file that I download from them is different. Even, making a diff from SmalltalkHub's UI shows me a diff against a completely different package (RoboticTable) that is from Santiago Bragagnolo, and that I guess it is a private project because it is not listed and cannot browsed.

Making a diff on other versions, such as http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito.... , makes a diff against a version of the Makros package, which is also from Santiago Bragagnolo, but this is not a private project.

Aaaand, now I see a problem!

both my test package:

http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito....

and the one of Santiago

http://www.smalltalkhub.com/#!/~sbragagnolo/Makros/versions/Makros-SantiagoB...

have the same UUID!!!!!!

Maybe this is because of my debian 64bits? Hmm maybe the UUID plugin is not loading? I'll check that for now, because I cannot work if I cannot commit :). If somebody has an idea, It is VERY welcome!

Thanks for reading my loud ranting/reasoning, Guille

...

...

On 04 Feb 2016, at 10:52, Guille Polito <guillermopolito@gmail.com> wrote:

So far, I'm blaming Smalltalkhub:

- I create a new empty package. - I commit it to a local directory, it works ok. - I commit it to a smalltalkhub repository: the file in my package cache is ok, but the file in smalltalkhub is corrupted e.g., http://www.smalltalkhub.com/#!/~Guille/playground/versions/test-GuillermoPol...

The strange thing is that it is a recurrent bug. I cannot commit to smalltalkhub, not even do a push of a version in my package cache. Smalltalkhub always shows a buggy version.

I'll now try with squeaksource 3 to narrow my conclusions.

On 02/04/2016 10:33 AM, Guille Polito wrote:

...

I tried for one hour yesterday to understand the problem :). This morning my priority was to not lose my code because I noticed the bug a couple of hours after my commits... Thanks I remembered to save my image and that the good old fileout in .st is working!

I'll keep trying to reproduce and keep you posted

On 02/04/2016 10:21 AM, stepharo wrote:

...

Hi guille

Esteban got some problems with MC recently so it would be good to have a reproducible case.

Stef

Le 4/2/16 10:07, Guille Polito a écrit :

...

Ok, I am puzzled. I downloaded a new fresh image, in a fresh directory, worked on something else, committed, and my commit is completely broken. Attached screenshot of what monticello shows me.

I'm on debian jessie 64bits. Maybe it has something to do?

On 02/03/2016 03:41 PM, Guille Polito wrote:

...

Hi all,

I'm finally back, rechecking this issue:

https://pharo.fogbugz.com/f/cases/7241/Object-name-should-best-be-removed

I remade the Slice to load in latest Pharo5 with the new Spur changes, plus some fixes proposed by Nicolai. I can load the slice in a new image and everything looks ok. So far so good.

Now, the monkey starts checking the Slice and somehow it cannot load the Slice due to "dependencies to some classes". Of course, the slice I submitted does not depend on the packages on the complaint, and locally it loads well. Moreover, I never had/worked with those classes, nor they were installed in my system. Even, I have a new machine which is clean so I cannot believe I have some interference due to some package cache...

The message of the monkey is here:

https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/26128//artifact/vali...

Did somebody find some similar problem or it is just me? I believe the problem is on the monkey side, but I have no clue... Maybe somebody has a better idea.

Thanks! Guille

On 04 Feb 2016, at 11:55, Guille Polito <guillermopolito@gmail.com> wrote:

Good news, so far, installing NeoUUID and replacing UUID class >> new by:

UUID class >> new ^NeoUUIDGenerator new next

seems to work. At it looks that I can trust again my commits.

Thanks Sven :)

OK. Now, the idea is that one instance of the generator is kept around, this is way more efficient, but also more correct, as the random generator should not be created anew each time.

Now I do not know how this should be solved, nor if this is reproducible in a machine other than mine...

I seriously doubt the plugin is better (more random, more unique) then anything that we can do in Pharo itself. I like Pharo code way better, because we all see what it does. Anyone else with an opinion ?

On 02/04/2016 11:41 AM, Guille Polito wrote:

...

Well, maybe we do not need a plugin, but then the fallback code is not generating unique enough UUIDs :/

Maybe it is a side effect of Spur?

I'll try to patch the UUID generator in the image with the Neo one and test if it solves my problem...

I just want to commit :'(

On 02/04/2016 11:36 AM, Sven Van Caekenberghe wrote:

...

It beats the hell out of me why we need a plugin (apart from speed) and why the fallback should not work (i.e. generate duplicates). It is not rocket science or magic, right ?

I once made my own generator: Neo-UUID (http://mc.stfx.eu/Neo or Neo on StHub). It is pretty simple and fast.

...

On 04 Feb 2016, at 11:29, Guille Polito <guillermopolito@gmail.com> wrote:

And yes, linux VM's are not shipped with the UUID plugin... at least not since 2014/2013...

Also, I believe the plugin is not compiled as internal since it is not listed in

Smalltalk vm listBuiltinModules. Smalltalk vm listLoadedModules.

On 02/04/2016 11:12 AM, Guille Polito wrote:

...

On 02/04/2016 10:58 AM, Sven Van Caekenberghe wrote:

...

But StHub did not change for a long time, no ? I know... Saving to an HTTP MC repo is nothing more than an HTTP PUT.

You should be able to do that manually with just ZnClient. Well, yes. But I'm just clicking the save button on the monticello browser...

Now, I tested with squeaksource3:

- I commit in squeaksource 3 using the monticello browser: the file in package-cache is OK, the file in squeaksource3 is OK http://ss3.gemtalksystems.com/ss?_s=WvI7T3UuBKEaYUrR&_k=ukyWjav7hA-MhHIz

- I make a copy of the file from the monticello UI to a smalltalkhub repository. The file in smalltalkhub is not the one I want. http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito....

In both Squeaksource3 and Smalltalkhub I have the same UUID, project name, version number, commit message, but the MCZ file that I download from them is different. Even, making a diff from SmalltalkHub's UI shows me a diff against a completely different package (RoboticTable) that is from Santiago Bragagnolo, and that I guess it is a private project because it is not listed and cannot browsed.

Making a diff on other versions, such as http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito.... , makes a diff against a version of the Makros package, which is also from Santiago Bragagnolo, but this is not a private project.

Aaaand, now I see a problem!

both my test package:

http://www.smalltalkhub.com/#!/~Guille/playground/diff/test-GuillermoPolito....

and the one of Santiago

http://www.smalltalkhub.com/#!/~sbragagnolo/Makros/versions/Makros-SantiagoB...

have the same UUID!!!!!!

Maybe this is because of my debian 64bits? Hmm maybe the UUID plugin is not loading? I'll check that for now, because I cannot work if I cannot commit :). If somebody has an idea, It is VERY welcome!

Thanks for reading my loud ranting/reasoning, Guille

...

...

On 04 Feb 2016, at 10:52, Guille Polito <guillermopolito@gmail.com> wrote:

So far, I'm blaming Smalltalkhub:

- I create a new empty package. - I commit it to a local directory, it works ok. - I commit it to a smalltalkhub repository: the file in my package cache is ok, but the file in smalltalkhub is corrupted e.g., http://www.smalltalkhub.com/#!/~Guille/playground/versions/test-GuillermoPol...

The strange thing is that it is a recurrent bug. I cannot commit to smalltalkhub, not even do a push of a version in my package cache. Smalltalkhub always shows a buggy version.

I'll now try with squeaksource 3 to narrow my conclusions.

On 02/04/2016 10:33 AM, Guille Polito wrote:

...

I tried for one hour yesterday to understand the problem :). This morning my priority was to not lose my code because I noticed the bug a couple of hours after my commits... Thanks I remembered to save my image and that the good old fileout in .st is working!

I'll keep trying to reproduce and keep you posted

On 02/04/2016 10:21 AM, stepharo wrote:

...

Hi guille

Esteban got some problems with MC recently so it would be good to have a reproducible case.

Stef

Le 4/2/16 10:07, Guille Polito a écrit :

...

Ok, I am puzzled. I downloaded a new fresh image, in a fresh directory, worked on something else, committed, and my commit is completely broken. Attached screenshot of what monticello shows me.

I'm on debian jessie 64bits. Maybe it has something to do?

On 02/03/2016 03:41 PM, Guille Polito wrote:

...

Hi all,

I'm finally back, rechecking this issue:

https://pharo.fogbugz.com/f/cases/7241/Object-name-should-best-be-removed

I remade the Slice to load in latest Pharo5 with the new Spur changes, plus some fixes proposed by Nicolai. I can load the slice in a new image and everything looks ok. So far so good.

Now, the monkey starts checking the Slice and somehow it cannot load the Slice due to "dependencies to some classes". Of course, the slice I submitted does not depend on the packages on the complaint, and locally it loads well. Moreover, I never had/worked with those classes, nor they were installed in my system. Even, I have a new machine which is clean so I cannot believe I have some interference due to some package cache...

The message of the monkey is here:

https://ci.inria.fr/pharo/job/Pharo-5.0-Issue-Validator/26128//artifact/vali...

Did somebody find some similar problem or it is just me? I believe the problem is on the monkey side, but I have no clue... Maybe somebody has a better idea.

Thanks! Guille

On 04 Feb 2016, at 14:02, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 04 Feb 2016, at 12:52 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 04 Feb 2016, at 11:55, Guille Polito <guillermopolito@gmail.com> wrote:

Good news, so far, installing NeoUUID and replacing UUID class >> new by:

UUID class >> new ^NeoUUIDGenerator new next

seems to work. At it looks that I can trust again my commits.

Thanks Sven :)

OK.

Now, the idea is that one instance of the generator is kept around, this is way more efficient, but also more correct, as the random generator should not be created anew each time.

...

Now I do not know how this should be solved, nor if this is reproducible in a machine other than mine...

I seriously doubt the plugin is better (more random, more unique) then anything that we can do in Pharo itself. I like Pharo code way better, because we all see what it does.

Anyone else with an opinion ?

More random for certain, at least much faster than we could do in Pharo, is certainly possible with a plugin. More unique... well, depends on whether you can trust the source RNG (seeded from data with enough entropy, period much greater than 2^128, etc).

A single next:into:startingAt: primitive filling random data (obtained using platform primitives and/or RDRAND/RDSEED) into a buffer would be much nicer/more flexible than the current UUID primitive though. (the linked UUID library is basically just a thin wrapper around the platform primitives, plus setting variant/version bits in the returned UUID bytes)

As a stopgap, one could instead use another, more general primitive (but with a less flexible api than next:into:startingAt:) in an otherwise unrelated plugin (but that is built internal to all Cog VM's, at least): rand: aBuffer <primitive: 'primitiveGatherEntropy' module: 'CroquetPlugin'> ^self primitiveFail

(which, iirc, resolves to the same platform primitives as UUID lib)

Also, the meaning of different UUID versions are well defined in the RFC, marking NeoUUID's as being a version 4 UUID seems somewhat ... disingenuous.

Yes, but to 99.99% of developers, a UUID is something special/magic where some effort was made to generate something as unique as possible. If we insist on it being just 128 random bits, then why all the fuss ? Why do we even need so called standards ? Why do we need a plugin, an implementation even ? It makes no sense. Just read 16 bytes from /dev/[u]random, or the equivalent use of a RNG and be done with it. But if you tell developers that their database IDs are just plain random numbers, they will probably feels less good about them. To me UUID is a specific concept: << The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". >> Not including any 'node' information (identifying process, image, VM, machine, network endpoint), nor 'counter' does not feel right.

Cheers, Henry

I detach the conversation from the Object>>name issue. So, to summarize the conversation: For some reason, the UUIDs generated by my Pharo in my new machine (Debian Jessie 64bit) are not unique at all. They keep clashing with UUIDs of existing packages in Smalltalkhub. I mean with this that it is not an isolated case or a random thing. I can reproduce it every time I commit. I opened a bug for this [1]. And I marked it as show stopper, as I believe we cannot release pharo5 with this bug that prevents people from doing a commit. Then, I tested the NeoUUIDGenerator and it works better in that sense. Or at least it gives me better guarantees. I propose to replace the existing UUID generator by NeoUUID generator. The latter is not only better, but it also includes tests. What do you think? Guille [1] https://pharo.fogbugz.com/f/cases/17544/UUIDs-are-not-so-unique -------- Forwarded Message -------- Subject: Re: [Pharo-dev] [Need help with Monkey] Removing Object>>name Date: Fri, 5 Feb 2016 14:12:10 +0100 From: Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> Reply-To: Pharo Development List <pharo-dev@lists.pharo.org> To: Pharo Development List <pharo-dev@lists.pharo.org> On Thu, Feb 4, 2016 at 2:59 PM, Sven Van Caekenberghe <sven@stfx.eu <mailto:sven@stfx.eu>> wrote: > On 04 Feb 2016, at 14:02, Henrik Johansen <henrik.s.johansen@veloxit.no <mailto:henrik.s.johansen@veloxit.no>> wrote: > >> >> On 04 Feb 2016, at 12:52 , Sven Van Caekenberghe <sven@stfx.eu <mailto:sven@stfx.eu>> wrote: >> >> >>> On 04 Feb 2016, at 11:55, Guille Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote: >>> >>> Good news, so far, installing NeoUUID and replacing UUID class >> new by: >>> >>> UUID class >> new >>> ^NeoUUIDGenerator new next >>> >>> seems to work. At it looks that I can trust again my commits. >>> >>> Thanks Sven :) >> >> OK. >> >> Now, the idea is that one instance of the generator is kept around, this is way more efficient, but also more correct, as the random generator should not be created anew each time. >> >>> Now I do not know how this should be solved, nor if this is reproducible in a machine other than mine... >> >> I seriously doubt the plugin is better (more random, more unique) then anything that we can do in Pharo itself. I like Pharo code way better, because we all see what it does. >> >> Anyone else with an opinion ? > > More random for certain, at least much faster than we could do in Pharo, is certainly possible with a plugin. > More unique... well, depends on whether you can trust the source RNG (seeded from data with enough entropy, period much greater than 2^128, etc). > > A single next:into:startingAt: primitive filling random data (obtained using platform primitives and/or RDRAND/RDSEED) into a buffer would be much nicer/more flexible than the current UUID primitive though. (the linked UUID library is basically just a thin wrapper around the platform primitives, plus setting variant/version bits in the returned UUID bytes) > > As a stopgap, one could instead use another, more general primitive (but with a less flexible api than next:into:startingAt:) in an otherwise unrelated plugin (but that is built internal to all Cog VM's, at least): > rand: aBuffer > <primitive: 'primitiveGatherEntropy' module: 'CroquetPlugin'> > ^self primitiveFail > > (which, iirc, resolves to the same platform primitives as UUID lib) > > Also, the meaning of different UUID versions are well defined in the RFC, marking NeoUUID's as being a version 4 UUID seems somewhat ... disingenuous. Yes, but to 99.99% of developers, a UUID is something special/magic where some effort was made to generate something as unique as possible. If we insist on it being just 128 random bits, then why all the fuss ? Why do we even need so called standards ? In my mind, you just answered that :) The standard provides 4 well-defined ways to implement something where it's guaranteed that some effort has been made to generate something as unique as possible. Why do we need a plugin, an implementation even ? It makes no sense. Just read 16 bytes from /dev/[u]random, or the equivalent use of a RNG and be done with it. But if you tell developers that their database IDs are just plain random numbers, they will probably feels less good about them. But... that's *exactly* what you tell them by marking UUID's as being of type 4. To me UUID is a specific concept: << The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". >> Not including any 'node' information (identifying process, image, VM, machine, network endpoint), nor 'counter' does not feel right. And I agree Neo-UUID fits that bill nicely, the values it returns *are* practically unique. The two things I disagreed with was: 1) The statement that it is possible to implement purely in Smalltalk UUID's as random as those returned by the plugin (Which, as you say, is basically reading 16 bytes from /dev/random), at least as performant, without consulting an external trusted randomness source. 2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator. Cheers, Henry

The thing is that it is not a monkey problem. The monkey asks Smalltalkhub for the right file, but SmalltalkHub retrieves the wrong mcz. And that's because Smalltalkhub associates a commit to an mcz file by the UUID of the commit. On 02/08/2016 01:21 PM, Ben Coman wrote:

On Mon, Feb 8, 2016 at 4:47 PM, Guille Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote:

I detach the conversation from the Object>>name issue.

So, to summarize the conversation:

For some reason, the UUIDs generated by my Pharo in my new machine (Debian Jessie 64bit) are not unique at all. They keep clashing with UUIDs of existing packages in Smalltalkhub. I mean with this that it is not an isolated case or a random thing. I can reproduce it every time I commit. I opened a bug for this [1]. And I marked it as show stopper, as I believe we cannot release pharo5 with this bug that prevents people from doing a commit.

Should there be any concern that the CI monkey may have picked up any wrong slices?

cheers -ben

Then, I tested the NeoUUIDGenerator and it works better in that sense. Or at least it gives me better guarantees. I propose to replace the existing UUID generator by NeoUUID generator. The latter is not only better, but it also includes tests. What do you think?

Guille

[1] https://pharo.fogbugz.com/f/cases/17544/UUIDs-are-not-so-unique

-------- Forwarded Message -------- Subject: Re: [Pharo-dev] [Need help with Monkey] Removing Object>>name Date: Fri, 5 Feb 2016 14:12:10 +0100 From: Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> <mailto:henrik.s.johansen@veloxit.no> Reply-To: Pharo Development List <pharo-dev@lists.pharo.org> <mailto:pharo-dev@lists.pharo.org> To: Pharo Development List <pharo-dev@lists.pharo.org> <mailto:pharo-dev@lists.pharo.org>

On Thu, Feb 4, 2016 at 2:59 PM, Sven Van Caekenberghe <sven@stfx.eu <mailto:sven@stfx.eu>> wrote:

> On 04 Feb 2016, at 14:02, Henrik Johansen <henrik.s.johansen@veloxit.no <mailto:henrik.s.johansen@veloxit.no>> wrote: > >> >> On 04 Feb 2016, at 12:52 , Sven Van Caekenberghe <sven@stfx.eu <mailto:sven@stfx.eu>> wrote: >> >> >>> On 04 Feb 2016, at 11:55, Guille Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote: >>> >>> Good news, so far, installing NeoUUID and replacing UUID class >> new by: >>> >>> UUID class >> new >>> ^NeoUUIDGenerator new next >>> >>> seems to work. At it looks that I can trust again my commits. >>> >>> Thanks Sven :) >> >> OK. >> >> Now, the idea is that one instance of the generator is kept around, this is way more efficient, but also more correct, as the random generator should not be created anew each time. >> >>> Now I do not know how this should be solved, nor if this is reproducible in a machine other than mine... >> >> I seriously doubt the plugin is better (more random, more unique) then anything that we can do in Pharo itself. I like Pharo code way better, because we all see what it does. >> >> Anyone else with an opinion ? > > More random for certain, at least much faster than we could do in Pharo, is certainly possible with a plugin. > More unique... well, depends on whether you can trust the source RNG (seeded from data with enough entropy, period much greater than 2^128, etc). > > A single next:into:startingAt: primitive filling random data (obtained using platform primitives and/or RDRAND/RDSEED) into a buffer would be much nicer/more flexible than the current UUID primitive though. (the linked UUID library is basically just a thin wrapper around the platform primitives, plus setting variant/version bits in the returned UUID bytes) > > As a stopgap, one could instead use another, more general primitive (but with a less flexible api than next:into:startingAt:) in an otherwise unrelated plugin (but that is built internal to all Cog VM's, at least): > rand: aBuffer > <primitive: 'primitiveGatherEntropy' module: 'CroquetPlugin'> > ^self primitiveFail > > (which, iirc, resolves to the same platform primitives as UUID lib) > > Also, the meaning of different UUID versions are well defined in the RFC, marking NeoUUID's as being a version 4 UUID seems somewhat ... disingenuous.

Yes, but to 99.99% of developers, a UUID is something special/magic where some effort was made to generate something as unique as possible.

If we insist on it being just 128 random bits, then why all the fuss ? Why do we even need so called standards ?

In my mind, you just answered that :) The standard provides 4 well-defined ways to implement something where it's guaranteed that some effort has been made to generate something as unique as possible.

Why do we need a plugin, an implementation even ? It makes no sense. Just read 16 bytes from /dev/[u]random, or the equivalent use of a RNG and be done with it.

But if you tell developers that their database IDs are just plain random numbers, they will probably feels less good about them.

But... that's *exactly* what you tell them by marking UUID's as being of type 4.

To me UUID is a specific concept: << The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". >>

Not including any 'node' information (identifying process, image, VM, machine, network endpoint), nor 'counter' does not feel right.

And I agree Neo-UUID fits that bill nicely, the values it returns *are* practically unique. The two things I disagreed with was: 1) The statement that it is possible to implement purely in Smalltalk UUID's as random as those returned by the plugin (Which, as you say, is basically reading 16 bytes from /dev/random), at least as performant, without consulting an external trusted randomness source. 2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator.

Cheers, Henry

Yes, creepy. However, I do not know if this is reproducible from some other machine/environment or just in mine, as nobody told so... On 02/08/2016 02:22 PM, Stephan Eggermont wrote:

On 08-02-16 14:13, Guille Polito wrote:

...

The thing is that it is not a monkey problem. The monkey asks Smalltalkhub for the right file, but SmalltalkHub retrieves the wrong mcz. And that's because Smalltalkhub associates a commit to an mcz file by the UUID of the commit.

So we lost source code? A slice in the inbox that has a duplicate UUID?

Stephan

On 05 Feb 2016, at 14:12, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:

On Thu, Feb 4, 2016 at 2:59 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 04 Feb 2016, at 14:02, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 04 Feb 2016, at 12:52 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 04 Feb 2016, at 11:55, Guille Polito <guillermopolito@gmail.com> wrote:

Good news, so far, installing NeoUUID and replacing UUID class >> new by:

UUID class >> new ^NeoUUIDGenerator new next

seems to work. At it looks that I can trust again my commits.

Thanks Sven :)

OK.

Now, the idea is that one instance of the generator is kept around, this is way more efficient, but also more correct, as the random generator should not be created anew each time.

...

Now I do not know how this should be solved, nor if this is reproducible in a machine other than mine...

I seriously doubt the plugin is better (more random, more unique) then anything that we can do in Pharo itself. I like Pharo code way better, because we all see what it does.

Anyone else with an opinion ?

More random for certain, at least much faster than we could do in Pharo, is certainly possible with a plugin. More unique... well, depends on whether you can trust the source RNG (seeded from data with enough entropy, period much greater than 2^128, etc).

A single next:into:startingAt: primitive filling random data (obtained using platform primitives and/or RDRAND/RDSEED) into a buffer would be much nicer/more flexible than the current UUID primitive though. (the linked UUID library is basically just a thin wrapper around the platform primitives, plus setting variant/version bits in the returned UUID bytes)

As a stopgap, one could instead use another, more general primitive (but with a less flexible api than next:into:startingAt:) in an otherwise unrelated plugin (but that is built internal to all Cog VM's, at least): rand: aBuffer <primitive: 'primitiveGatherEntropy' module: 'CroquetPlugin'> ^self primitiveFail

(which, iirc, resolves to the same platform primitives as UUID lib)

Also, the meaning of different UUID versions are well defined in the RFC, marking NeoUUID's as being a version 4 UUID seems somewhat ... disingenuous.

Yes, but to 99.99% of developers, a UUID is something special/magic where some effort was made to generate something as unique as possible.

If we insist on it being just 128 random bits, then why all the fuss ? Why do we even need so called standards ? In my mind, you just answered that :) The standard provides 4 well-defined ways to implement something where it's guaranteed that some effort has been made to generate something as unique as possible.

Why do we need a plugin, an implementation even ? It makes no sense. Just read 16 bytes from /dev/[u]random, or the equivalent use of a RNG and be done with it.

But if you tell developers that their database IDs are just plain random numbers, they will probably feels less good about them.

But... that's *exactly* what you tell them by marking UUID's as being of type 4.

To me UUID is a specific concept: << The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". >>

Not including any 'node' information (identifying process, image, VM, machine, network endpoint), nor 'counter' does not feel right.

And I agree Neo-UUID fits that bill nicely, the values it returns *are* practically unique. The two things I disagreed with was: 1) The statement that it is possible to implement purely in Smalltalk UUID's as random as those returned by the plugin (Which, as you say, is basically reading 16 bytes from /dev/random), at least as performant, without consulting an external trusted randomness source.

Agreed.

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator.

OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ?

Cheers, Henry

Here is a new version, in preparation of possible integration in the main image: === Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1 Streamline UUID generation Add a current, shared instance Added better class and method comments Add more tests As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach === The class comments now reads as follows: === I am NeoUUIDGenerator, I generate UUIDs. An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array. The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice. I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access I can generate about 500K UUIDs per second. Implementation: Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier) Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:) Usage: NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next. Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up. See also: http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 === If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me. And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find. Sven

On 08 Feb 2016, at 10:38, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator.

OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ?

My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

I can do the integration too, but I need some people to say go ahead. I vote for replacing everything, there is no need for a plugin.

On 09 Feb 2016, at 09:25, Guille Polito <guillermopolito@gmail.com> wrote:

Sven, just to answer your last question. The UUID generation right now generates the UUID fields like this:

UUIDGenerator>>generateFieldsVersion4

timeLow := self generateRandomBitsOfLength: 32. timeMid := self generateRandomBitsOfLength: 16. timeHiAndVersion := 16r4000 bitOr: (self generateRandomBitsOfLength: 12). clockSeqHiAndReserved := 16r80 bitOr: (self generateRandomBitsOfLength: 6). clockSeqLow := self generateRandomBitsOfLength: 8. node := self generateRandomBitsOfLength: 48.

So... It's basically completely random. There is no part of the UUID that is actually based on the node, the clock or the time. It is actually a random string of bits that are generated using a number from /dev/urandom as seed (in linux).

Does the mac VM include the plugin? (I do not have a mac any more to test fast ^^)

I'll work on the integration of NeoUUID now, I hope this is the kind of issues that got integrated in code-freeze :)

Guille

On 02/08/2016 08:39 PM, Sven Van Caekenberghe wrote:

...

Here is a new version, in preparation of possible integration in the main image:

=== Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1

Streamline UUID generation

Add a current, shared instance

Added better class and method comments

Add more tests

As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach ===

The class comments now reads as follows:

=== I am NeoUUIDGenerator, I generate UUIDs.

An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array.

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice.

I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient

To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access

I can generate about 500K UUIDs per second.

Implementation:

Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier) Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:)

Usage:

NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next.

Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up.

See also:

http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 ===

If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me.

And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find.

Sven

...

On 08 Feb 2016, at 10:38, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator. OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ? My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

Sad and true... :'( On Tue, Feb 9, 2016 at 10:01 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

Beside, you can't make slices ;-)

...

On 09 Feb 2016, at 09:58, Sven Van Caekenberghe <sven@stfx.eu> wrote:

I can do the integration too, but I need some people to say go ahead. I vote for replacing everything, there is no need for a plugin.

...

On 09 Feb 2016, at 09:25, Guille Polito <guillermopolito@gmail.com> wrote:

Sven, just to answer your last question. The UUID generation right now generates the UUID fields like this:

UUIDGenerator>>generateFieldsVersion4

timeLow := self generateRandomBitsOfLength: 32. timeMid := self generateRandomBitsOfLength: 16. timeHiAndVersion := 16r4000 bitOr: (self generateRandomBitsOfLength: 12). clockSeqHiAndReserved := 16r80 bitOr: (self generateRandomBitsOfLength: 6). clockSeqLow := self generateRandomBitsOfLength: 8. node := self generateRandomBitsOfLength: 48.

So... It's basically completely random. There is no part of the UUID that is actually based on the node, the clock or the time. It is actually a random string of bits that are generated using a number from /dev/urandom as seed (in linux).

Does the mac VM include the plugin? (I do not have a mac any more to test fast ^^)

I'll work on the integration of NeoUUID now, I hope this is the kind of issues that got integrated in code-freeze :)

Guille

On 02/08/2016 08:39 PM, Sven Van Caekenberghe wrote:

...

Here is a new version, in preparation of possible integration in the main image:

=== Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1

Streamline UUID generation

Add a current, shared instance

Added better class and method comments

Add more tests

As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach ===

The class comments now reads as follows:

=== I am NeoUUIDGenerator, I generate UUIDs.

An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array.

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice.

I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient

To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access

I can generate about 500K UUIDs per second.

Implementation:

Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier) Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:)

Usage:

NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next.

Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up.

See also:

http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 ===

If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me.

And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find.

Sven

...

On 08 Feb 2016, at 10:38, Henrik Johansen < henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator. OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ? My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

Yes, go on. I think it is the easiest. Right now, to check if a slice is good or not I have to commit, browse Smalltalkhub's UI, and if it not good I have to delete the mcz manually... On 02/09/2016 10:10 AM, Sven Van Caekenberghe wrote:

You want me to do it then ?

I don't want us to do double work ...

...

On 09 Feb 2016, at 10:09, Guillermo Polito <guillermopolito@gmail.com> wrote:

Sad and true... :'(

On Tue, Feb 9, 2016 at 10:01 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Beside, you can't make slices ;-)

...

On 09 Feb 2016, at 09:58, Sven Van Caekenberghe <sven@stfx.eu> wrote:

I can do the integration too, but I need some people to say go ahead. I vote for replacing everything, there is no need for a plugin.

...

On 09 Feb 2016, at 09:25, Guille Polito <guillermopolito@gmail.com> wrote:

Sven, just to answer your last question. The UUID generation right now generates the UUID fields like this:

UUIDGenerator>>generateFieldsVersion4

timeLow := self generateRandomBitsOfLength: 32. timeMid := self generateRandomBitsOfLength: 16. timeHiAndVersion := 16r4000 bitOr: (self generateRandomBitsOfLength: 12). clockSeqHiAndReserved := 16r80 bitOr: (self generateRandomBitsOfLength: 6). clockSeqLow := self generateRandomBitsOfLength: 8. node := self generateRandomBitsOfLength: 48.

So... It's basically completely random. There is no part of the UUID that is actually based on the node, the clock or the time. It is actually a random string of bits that are generated using a number from /dev/urandom as seed (in linux).

Does the mac VM include the plugin? (I do not have a mac any more to test fast ^^)

I'll work on the integration of NeoUUID now, I hope this is the kind of issues that got integrated in code-freeze :)

Guille

On 02/08/2016 08:39 PM, Sven Van Caekenberghe wrote:

...

Here is a new version, in preparation of possible integration in the main image:

=== Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1

Streamline UUID generation

Add a current, shared instance

Added better class and method comments

Add more tests

As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach ===

The class comments now reads as follows:

=== I am NeoUUIDGenerator, I generate UUIDs.

An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array.

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice.

I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient

To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access

I can generate about 500K UUIDs per second.

Implementation:

Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier) Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:)

Usage:

NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next.

Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up.

See also:

http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 ===

If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me.

And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find.

Sven

...

On 08 Feb 2016, at 10:38, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator. OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ? My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

On Tue, Feb 9, 2016 at 9:12 PM, Guille Polito <guillermopolito@gmail.com> wrote:

AAAANNNDDD! It looks I found the real cause of this!

- I tested an image pre-new session manager, and an image post-new session manager. The issue only appeared in the latter.

- Checking, it seems that UUIDGenerator is not subscribed to the new startup list. This means that the UUIDGenerator is not being reinitialized on every startup. This means, moreover, that every person that is loading the latest Pharo image is using the same UUIDGenerator instance, with the same random seed => same generated UUIDs.

So, adding UUIDGenerator to the list would be the simplest solution and the integration of NeoUUIDGenerator can be moved to Pharo6 maybe.

However, now that we found that something was missing in the startup list, we should check for others...

Guille

Good find. How can we make a test for this? Maybe it needs to be done from outside the image? cheers -ben

On 02/09/2016 10:36 AM, Guille Polito wrote:

...

Yes, go on. I think it is the easiest.

Right now, to check if a slice is good or not I have to commit, browse Smalltalkhub's UI, and if it not good I have to delete the mcz manually...

On 02/09/2016 10:10 AM, Sven Van Caekenberghe wrote:

...

You want me to do it then ?

I don't want us to do double work ...

...

On 09 Feb 2016, at 10:09, Guillermo Polito <guillermopolito@gmail.com> wrote:

Sad and true... :'(

On Tue, Feb 9, 2016 at 10:01 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Beside, you can't make slices ;-)

...

On 09 Feb 2016, at 09:58, Sven Van Caekenberghe <sven@stfx.eu> wrote:

I can do the integration too, but I need some people to say go ahead. I vote for replacing everything, there is no need for a plugin.

...

On 09 Feb 2016, at 09:25, Guille Polito <guillermopolito@gmail.com> wrote:

Sven, just to answer your last question. The UUID generation right now generates the UUID fields like this:

UUIDGenerator>>generateFieldsVersion4

timeLow := self generateRandomBitsOfLength: 32. timeMid := self generateRandomBitsOfLength: 16. timeHiAndVersion := 16r4000 bitOr: (self generateRandomBitsOfLength: 12). clockSeqHiAndReserved := 16r80 bitOr: (self generateRandomBitsOfLength: 6). clockSeqLow := self generateRandomBitsOfLength: 8. node := self generateRandomBitsOfLength: 48.

So... It's basically completely random. There is no part of the UUID that is actually based on the node, the clock or the time. It is actually a random string of bits that are generated using a number from /dev/urandom as seed (in linux).

Does the mac VM include the plugin? (I do not have a mac any more to test fast ^^)

I'll work on the integration of NeoUUID now, I hope this is the kind of issues that got integrated in code-freeze :)

Guille

On 02/08/2016 08:39 PM, Sven Van Caekenberghe wrote:

...

Here is a new version, in preparation of possible integration in the main image:

=== Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1

Streamline UUID generation

Add a current, shared instance

Added better class and method comments

Add more tests

As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach ===

The class comments now reads as follows:

=== I am NeoUUIDGenerator, I generate UUIDs.

An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array.

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice.

I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient

To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access

I can generate about 500K UUIDs per second.

Implementation:

Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier) Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:)

Usage:

NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next.

Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up.

See also:

http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 ===

If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me.

And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find.

Sven

...

On 08 Feb 2016, at 10:38, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator.

OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ?

My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

On 09/02/16 14:12, Guille Polito wrote:

AAAANNNDDD! It looks I found the real cause of this!

- I tested an image pre-new session manager, and an image post-new session manager. The issue only appeared in the latter.

- Checking, it seems that UUIDGenerator is not subscribed to the new startup list. This means that the UUIDGenerator is not being reinitialized on every startup. This means, moreover, that every person that is loading the latest Pharo image is using the same UUIDGenerator instance, with the same random seed => same generated UUIDs.

So, adding UUIDGenerator to the list would be the simplest solution and the integration of NeoUUIDGenerator can be moved to Pharo6 maybe.

However, now that we found that something was missing in the startup list, we should check for others...

So, which image versions do we need to mark as broken, don't use? Stephan

Le 9/2/16 09:58, Sven Van Caekenberghe a écrit :

I can do the integration too, but I need some people to say go ahead. I vote for replacing everything, there is no need for a plugin.

me too. Less plugin!

...

On 09 Feb 2016, at 09:25, Guille Polito <guillermopolito@gmail.com> wrote:

Sven, just to answer your last question. The UUID generation right now generates the UUID fields like this:

UUIDGenerator>>generateFieldsVersion4

timeLow := self generateRandomBitsOfLength: 32. timeMid := self generateRandomBitsOfLength: 16. timeHiAndVersion := 16r4000 bitOr: (self generateRandomBitsOfLength: 12). clockSeqHiAndReserved := 16r80 bitOr: (self generateRandomBitsOfLength: 6). clockSeqLow := self generateRandomBitsOfLength: 8. node := self generateRandomBitsOfLength: 48.

So... It's basically completely random. There is no part of the UUID that is actually based on the node, the clock or the time. It is actually a random string of bits that are generated using a number from /dev/urandom as seed (in linux).

Does the mac VM include the plugin? (I do not have a mac any more to test fast ^^)

I'll work on the integration of NeoUUID now, I hope this is the kind of issues that got integrated in code-freeze :)

Guille

On 02/08/2016 08:39 PM, Sven Van Caekenberghe wrote:

...

Here is a new version, in preparation of possible integration in the main image:

=== Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1

Streamline UUID generation

Add a current, shared instance

Added better class and method comments

Add more tests

As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach ===

The class comments now reads as follows:

=== I am NeoUUIDGenerator, I generate UUIDs.

An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array.

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice.

I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient

To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access

I can generate about 500K UUIDs per second.

Implementation:

Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier) Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:)

Usage:

NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next.

Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up.

See also:

http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 ===

If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me.

And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find.

Sven

...

On 08 Feb 2016, at 10:38, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator. OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ? My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

On 09 Feb 2016, at 17:35, Ben Coman <btc@openinworld.com> wrote:

On Tue, Feb 9, 2016 at 11:53 PM, stepharo <stepharo@free.fr> wrote:

...

Le 9/2/16 09:58, Sven Van Caekenberghe a écrit :

...

I can do the integration too, but I need some people to say go ahead. I vote for replacing everything, there is no need for a plugin.

me too. Less plugin!

This is a good general philosophy, but we should benchmark in-image versus plugin so we can make an informed analytical decision on how much performance we are willing to trade for the convenience of it being all in-image, and whether we want to maintain a separate CI job for keeping the plugin tested. Overriding that is providing a quick fix so other development is not blocked, which however Guille's solution seems sufficient for the code freeze of Pharo 5.

From a comment in

=== Speedwise, in Pharo 5 Spur on my machine, the new implementation is just as fast as the primitive, or so it seems: [ UUIDGenerator next ] bench. "'1,235,356 per second'" [ UUID nilUUID primMakeUUID ] bench. "'1,237,213 per second'" === But even if it were slower (like half or quarter speed), there is a *HUGE* benefit to in image code. C code will always win, performance wise, but if that is all that counts, why do we program in Pharo at all. Please read << Design Principles Behind Smalltalk >> http://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.html Principle 1: Personal Mastery: If a system is to serve the creative spirit, it must be entirely comprehensible to a single individual. Further on: Operating System: An operating system is a collection of things that don't fit into a language. There shouldn't be one. Not that this document is an axiom or anything, but it articulates well some very relevant principles. For all these years, you nor I ever knew what took place in that silly plugin (1 method), while a confusing UUIDGenerator made some of us believe that it was used or somehow identical to the plugin (that was probably not true), now you, me and 99% of all other Pharo developers can read clean code. For me, Pharo is much more than the next scripting language that links to OS libraries. Pharo makes software (development) tangible, understandable in one single language/environment. When is the last time you looked into the open source C code of any library of your OS, let alone the kernel itself or one of its drivers ? Probably never. But in Pharo we can all look under the hood, everywhere, seamlessly. Sven

cheers -ben

...

...

...

On 09 Feb 2016, at 09:25, Guille Polito <guillermopolito@gmail.com> wrote:

Sven, just to answer your last question. The UUID generation right now generates the UUID fields like this:

UUIDGenerator>>generateFieldsVersion4

timeLow := self generateRandomBitsOfLength: 32. timeMid := self generateRandomBitsOfLength: 16. timeHiAndVersion := 16r4000 bitOr: (self generateRandomBitsOfLength: 12). clockSeqHiAndReserved := 16r80 bitOr: (self generateRandomBitsOfLength: 6). clockSeqLow := self generateRandomBitsOfLength: 8. node := self generateRandomBitsOfLength: 48.

So... It's basically completely random. There is no part of the UUID that is actually based on the node, the clock or the time. It is actually a random string of bits that are generated using a number from /dev/urandom as seed (in linux).

Does the mac VM include the plugin? (I do not have a mac any more to test fast ^^)

I'll work on the integration of NeoUUID now, I hope this is the kind of issues that got integrated in code-freeze :)

Guille

On 02/08/2016 08:39 PM, Sven Van Caekenberghe wrote:

...

Here is a new version, in preparation of possible integration in the main image:

=== Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1

Streamline UUID generation

Add a current, shared instance

Added better class and method comments

Add more tests

As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach ===

The class comments now reads as follows:

=== I am NeoUUIDGenerator, I generate UUIDs.

An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array.

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique". I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice.

I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient

To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access

I can generate about 500K UUIDs per second.

Implementation:

Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier) Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:)

Usage:

NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next.

Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up.

See also:

http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 ===

If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me.

And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find.

Sven

...

On 08 Feb 2016, at 10:38, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator.

OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ?

My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

I love the class comment! This is because of such class comment that I love also pharo Le 8/2/16 20:39, Sven Van Caekenberghe a écrit :

Here is a new version, in preparation of possible integration in the main image:

=== Name: Neo-UUID-SvenVanCaekenberghe.2 Author: SvenVanCaekenberghe Time: 8 February 2016, 8:33:04.141334 pm UUID: a909453e-35dd-4c25-8273-62a9b2bd982e Ancestors: Neo-UUID-SvenVanCaekenberghe.1

Streamline UUID generation

Add a current, shared instance

Added better class and method comments

Add more tests

As suggested by Henrik Johansen, change to a version 0 UUID to indicate that we follow a custom approach ===

The class comments now reads as follows:

=== I am NeoUUIDGenerator, I generate UUIDs.

An RFC4122 Universally Unique Identifier (UUID) is an opaque 128-bit number that can be used for identification purposes. Concretely, a UUID is a 16 element byte array.

The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. In this context the word unique should be taken to mean "practically unique" rather than "guaranteed unique".

I generate UUIDs similar, in spirit, to those defined in RFC4122, though I use version 0 to indicate that I follow none of the defined versions. This does not matter much, if at all, in practice.

I try to conform to the following aspects: - each 'node' (machine, image, instance) should generate unique UUIDs - even when generating UUIDs at a very fast rate, they should remain unique - be fast and efficient

To achieve this goal, I - take several aspects into account to generate a unique node ID - combine a clock, a counter and some random bits - hold a state, protected for multi user access

I can generate about 500K UUIDs per second.

Implementation:

Although a UUID should be seen as totally opaque, here is the concrete way I generate one: - the first 8 bytes are the millisecond clock value with the smallest quantity first; this means that the later of these 8 bytes will be identical when generated with small(er) timespans; within the same millisecond, the full first 8 bytes will be identical - the next 2 bytes represent a counter with safe overflow, held as protected state inside me; after 2*16 this value will repeat; the counter initalizes with a random value - the next 2 bytes are simply random, based on the system PRNG, Random - the final 4 bytes represent the node ID; the node ID is unique per instance of me, across OS environments where the image might run; the node ID is the MD5 hash of a string that is the concatenation of several elements (see #computeNodeIdentifier)

Some bits are set to some predefined value, to indicate the variant and version (see #setVariantAndVersion:)

Usage:

NeoUUIDGenerator next. NeoUUIDGenerator current next. NeoUUIDGenerator new next.

Sharing an instance is more efficient and correct. Instances should be reset whenever the image comes up.

See also:

http://en.wikipedia.org/wiki/UUID https://tools.ietf.org/html/rfc4122 ===

If we integrate this, I think we should replace the old generator and the use of the primitive/plugin. But that requires at least some support apart from me.

And although I think that we should integrate this generator and get rid of the plugin, I think there is probably an underlying problem here (why did the generator fail ?) that could be important to find.

Sven

...

On 08 Feb 2016, at 10:38, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:

...

On 08 Feb 2016, at 10:29 , Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

2) Misrepresenting the way the UUID was generated (a combination of node identifier + timestamp + random value, similar to type 3, but with differently sized/ordered fields) by marking it as being of type 4, which is defined to be UUID consisting of random bytes. IOW, I think it should be marked as type 0 instead of 4, so for the 1 person in each country who might be found to assume something about the implementation based on the type field, won't later feel he's been duped when checking the generator. OK, I certainly want to change the type. Thing is, I cannot find a reference to type 0 anywhere that I am looking (I mostly used https://en.wikipedia.org/wiki/Universally_unique_identifier). Where did you find a definition of type 0 ? Or would that be a way to say 'no specific type' then ? My rationale was that it is currently unassigned, and the least likely number to be chosen as identifier by new versions of the standard. IOW, for those who care, it might raise a "hmm, this is strange, better check the source", upon which they will discover it is generated in a non-standard fashion (but can verify for themselves it is generated in a way still pretty much guaranteed to be unique), and the rest... well, they can (most probably) keep on living happily without ever seeing a collision.

Cheers, Henry

2016-02-04 11:29 GMT+01:00 Guille Polito <guillermopolito@gmail.com>:

And yes, linux VM's are not shipped with the UUID plugin... at least not since 2014/2013...

Also, I believe the plugin is not compiled as internal since it is not listed in

Smalltalk vm listBuiltinModules. Smalltalk vm listLoadedModules.

It looks like Monticello ignore UUID failure and just use value from ancestor. Could you investigate it? It can be fixed in that case.