voyage/mongo randomly wrong OIDs
Hi, I have a strange problem and before searching for more hours, I want to ask, if this problem is known. If not, I will proceed by myself ;-) Background: I write currencies and countries into mongo with voyage. For database creation, first I create all currencies and countries of the world in the image. One Country has N currencies and one Currency has N countries. After creating the model objects in the image, I save them to mongo with voyage then. Problem: Here something strange happens. SOMETIMES, the OID to the Currencies collection is wrong! It is the OID of the object itself (the country). Please see the example of UnitedArabEmirates below to understand what I mean. The OID of the country UAE is the same as the pointer to the currency(!). But in the model objects in my image, the inst var currencies was NOT pointing to the country itself but to the currency. { "#version": NumberInt(466896549), "_id": ObjectId("a5466c0b0000000000000000"), "countryName": "labelCountryUnitedArabEmirates", "currencies": { "0": { "#collection": "Currencies", "__id": ObjectId("a5466c0b0000000000000000") } }, "iso3166": "ARE" } this is an example of a country which is correct. { "#version": NumberInt(228627064), "_id": ObjectId("b7956c0b0000000000000000"), "countryName": "labelCountryArgentina", "currencies": { "0": { "#collection": "Currencies", "__id": ObjectId("77926c0b0000000000000000") } }, "iso3166": "ARG" } The important point is: this happens randomly. I take the one fresh loaded image, create the objects into the database and close the image without saving. The next time, other countries are wrong. If I delete the wrong objects from the database and re-create them, they are correct. I was trying to save the objects with delays but this did noch change anyting. I want to find out, what the reason for that is and ask you if you had similar problems or if this is a known bug.... any hint appreciated. Sabine -- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi Esteban, All, I was proceeding to seach for the reason of the problem I described yesterday. I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur. That made the whole process slower...and I had an idea.... I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code: |theOld theNew| 100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew] The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?! In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc. Important: I work with windows and so >>makeUnixSeed returns nil... :-) What is your opinion about that? Sabine -- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi, feel free to use & integrate the following fix: Fix-Mongo-OID.cs <http://forum.world.st/file/n4705616/Fix-Mongo-OID.cs> Jan Sabine Knöfel wrote
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
----- Save The World! -- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705619.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
file not found :) On Aug 29, 2013, at 2:44 PM, jan.struz <public+pharo@struz.cz> wrote:
Hi, feel free to use & integrate the following fix:
Fix-Mongo-OID.cs <http://forum.world.st/file/n4705616/Fix-Mongo-OID.cs>
Jan
Sabine Knöfel wrote
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
----- Save The World! -- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705619.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
ok, here again: Fix-Mongo-OID.cs <http://forum.world.st/file/n4705649/Fix-Mongo-OID.cs> :) EstebanLM wrote
file not found :)On Aug 29, 2013, at 2:44 PM, jan.struz <
public+pharo@
> wrote:> Hi,> feel free to use & integrate the following fix:> > Fix-Mongo-OID.cs <http://forum.world.st/file/n4705616/Fix-Mongo-OID.cs>; > > Jan> > > Sabine Knöfel wrote>> Hi Esteban, All,>> >> I was proceeding to seach for the reason of the problem I described>> yesterday.>> >> I added some debugging code into VOMongoSerializer>>ensurePersisted: and>> the problem I described, did NOT occur.>> >> That made the whole process slower...and I had an idea....>> >> I was looking into >>UUIDGenerator default makeSeed.>> Then I tried the following code:>> >> |theOld theNew|>> >> 100000000 timesRepeat: [ >> theNew := UUIDGenerator default makeSeed.>> theNew = theOld ifTrue: [self halt].>> theOld := theNew]>> >> The debugger came up! Doesn't that mean that, if the code is run very>> fast, there are double OIDs generated?!>> >> In my case, the objects for country and currency are very lightweight and>> so, they are created very fast. Also, the error occured much more often>> within my fast production EC2 instance as at my (old and slow) development>> pc. >> >> Important: I work with windows and so >>makeUnixSeed returns nil... :-)>> >> What is your opinion about that?>> >> Sabine> > > > > > -----> Save The World!> --> View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705619.htm...> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.>
----- Save The World! -- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705649.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
hi well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives: 1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :( anyway... I open to ideas :) in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision. Esteban On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On 29 Aug 2013, at 16:51, Esteban Lorenzano <estebanlm@gmail.com> wrote:
hi
well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way. What does the primitive actually do ? Is it different ?
Esteban
On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On Aug 29, 2013, at 5:08 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 29 Aug 2013, at 16:51, Esteban Lorenzano <estebanlm@gmail.com> wrote:
hi
well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way.
What does the primitive actually do ? Is it different ?
the primitive uses the clock ticks to produce an UUID... you shouldn't have repeated numbers that way... but well, it depends on the platform implementation also.
Esteban
On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On 30 Aug 2013, at 16:35, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On Aug 29, 2013, at 5:08 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 29 Aug 2013, at 16:51, Esteban Lorenzano <estebanlm@gmail.com> wrote:
hi
well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way.
What does the primitive actually do ? Is it different ?
the primitive uses the clock ticks to produce an UUID... you shouldn't have repeated numbers that way... but well, it depends on the platform implementation also.
I don't like plugins because it is some much harder for everyone to look at the implementation, while everybody thinks some magic happens there, and often the truth is quite disappointing. Maybe the resolution of #primUTCMicrosecondsClock is not high enough ?
Esteban
On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On Aug 30, 2013, at 4:49 PM, Sven Van Caekenberghe <sven@beta9.be> wrote:
On 30 Aug 2013, at 16:35, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On Aug 29, 2013, at 5:08 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 29 Aug 2013, at 16:51, Esteban Lorenzano <estebanlm@gmail.com> wrote:
hi
well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way.
What does the primitive actually do ? Is it different ?
the primitive uses the clock ticks to produce an UUID... you shouldn't have repeated numbers that way... but well, it depends on the platform implementation also.
I don't like plugins because it is some much harder for everyone to look at the implementation, while everybody thinks some magic happens there, and often the truth is quite disappointing.
Maybe the resolution of #primUTCMicrosecondsClock is not high enough ?
I tried... not enough :(
Esteban
On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On 30 Aug 2013, at 16:54, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On Aug 30, 2013, at 4:49 PM, Sven Van Caekenberghe <sven@beta9.be> wrote:
On 30 Aug 2013, at 16:35, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On Aug 29, 2013, at 5:08 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 29 Aug 2013, at 16:51, Esteban Lorenzano <estebanlm@gmail.com> wrote:
hi
well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way.
What does the primitive actually do ? Is it different ?
the primitive uses the clock ticks to produce an UUID... you shouldn't have repeated numbers that way... but well, it depends on the platform implementation also.
I don't like plugins because it is some much harder for everyone to look at the implementation, while everybody thinks some magic happens there, and often the truth is quite disappointing.
Maybe the resolution of #primUTCMicrosecondsClock is not high enough ?
I tried... not enough :(
I understand: the clock is not fast enough for the request rate. But then what is there against some counter as part of a UUID ? Sorry for the discussion, I find it an interesting subject. I'll have to read a bit about UUIDs. Can anyone point to the plugin implementation code ?
Esteban
On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
"Sorry for the discussion" for me, this discussion is interesting and important because I need a solution. :-) Thank you for the discussion! I tried following ideas but they have been up to 4 times slower than the original: 1) I tried to remember the last seed in an inst var of UUIDGenerator and create it new if it was the same as the last one. 2) wait for 1 millisecond before create new seed. As third, I tried adding a (random nextInt: 100000). This did not work, there have been also doouble numbers. So, perhaps the counter is a good idea but is it fast? Again, I work on windows and so I get nil as answer to makeUnixSeed. makeSeed | seed | seed := self makeUnixSeed. seed ifNotNil: [^seed]. [seed := (Time millisecondClockValue bitAnd: 16r3FFFFFFF) bitXor: self hash. seed := seed bitXor: (Time totalSeconds bitAnd: 16r3FFFFFFF). seed = 0] whileTrue: ["Try again if ever get a seed = 0"]. ^seed On Fri, Aug 30, 2013 at 5:15 PM, Sven Van Caekenberghe-2 [via Smalltalk] < ml-node+s1294792n4705817h22@n4.nabble.com> wrote:
On 30 Aug 2013, at 16:54, Esteban Lorenzano <[hidden email]<http://user/SendEmail.jtp?type=node&node=4705817&i=0>> wrote:
On Aug 30, 2013, at 4:49 PM, Sven Van Caekenberghe <[hidden email]<http://user/SendEmail.jtp?type=node&node=4705817&i=1>>
wrote:
On 30 Aug 2013, at 16:35, Esteban Lorenzano <[hidden email]<http://user/SendEmail.jtp?type=node&node=4705817&i=2>>
wrote:
On Aug 29, 2013, at 5:08 PM, Sven Van Caekenberghe <[hidden email]<http://user/SendEmail.jtp?type=node&node=4705817&i=3>>
wrote:
On 29 Aug 2013, at 16:51, Esteban Lorenzano <[hidden email]<http://user/SendEmail.jtp?type=node&node=4705817&i=4>>
wrote:
hi
well... I've never been happy on using the UUID generator for my
keys, but is the fastest option I found.
There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way.
What does the primitive actually do ? Is it different ?
the primitive uses the clock ticks to produce an UUID... you shouldn't have repeated numbers that way... but well, it depends on the platform implementation also.
I don't like plugins because it is some much harder for everyone to look at the implementation, while everybody thinks some magic happens there, and often the truth is quite disappointing.
Maybe the resolution of #primUTCMicrosecondsClock is not high enough ?
I tried... not enough :(
I understand: the clock is not fast enough for the request rate. But then what is there against some counter as part of a UUID ?
Sorry for the discussion, I find it an interesting subject. I'll have to read a bit about UUIDs. Can anyone point to the plugin implementation code ?
Esteban
On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <[hidden email]<http://user/SendEmail.jtp?type=node&node=4705817&i=5>> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
------------------------------ If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705817.htm... To unsubscribe from voyage/mongo randomly wrong OIDs, click here<http://forum.world.st/template/NamlServlet.jtp?macro=unsubscribe_by_code&nod...> . NAML<http://forum.world.st/template/NamlServlet.jtp?macro=macro_viewer&id=instant...>
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705819.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
yes, is a conversation geeky enough to be fun :) unix: https://github.com/pharo-project/pharovm/blob/master/platforms/unix/plugins/... mac: https://github.com/pharo-project/pharovm/blob/master/platforms/iOS/plugins/U... win: https://github.com/pharo-project/pharovm/blob/master/platforms/win32/plugins... as you can see... it much depends on the platform. I would like to unify that (I'm doing it slooooowly, with certain plugins... but just when I have time, which is not very frequent :), and also it is not always possible) Esteban On Aug 30, 2013, at 5:15 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 30 Aug 2013, at 16:54, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On Aug 30, 2013, at 4:49 PM, Sven Van Caekenberghe <sven@beta9.be> wrote:
On 30 Aug 2013, at 16:35, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On Aug 29, 2013, at 5:08 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 29 Aug 2013, at 16:51, Esteban Lorenzano <estebanlm@gmail.com> wrote:
hi
well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way.
What does the primitive actually do ? Is it different ?
the primitive uses the clock ticks to produce an UUID... you shouldn't have repeated numbers that way... but well, it depends on the platform implementation also.
I don't like plugins because it is some much harder for everyone to look at the implementation, while everybody thinks some magic happens there, and often the truth is quite disappointing.
Maybe the resolution of #primUTCMicrosecondsClock is not high enough ?
I tried... not enough :(
I understand: the clock is not fast enough for the request rate. But then what is there against some counter as part of a UUID ?
Sorry for the discussion, I find it an interesting subject. I'll have to read a bit about UUIDs. Can anyone point to the plugin implementation code ?
Esteban
On Aug 29, 2013, at 11:27 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban, All,
I was proceeding to seach for the reason of the problem I described yesterday.
I added some debugging code into VOMongoSerializer>>ensurePersisted: and the problem I described, did NOT occur.
That made the whole process slower...and I had an idea....
I was looking into >>UUIDGenerator default makeSeed. Then I tried the following code:
|theOld theNew|
100000000 timesRepeat: [ theNew := UUIDGenerator default makeSeed. theNew = theOld ifTrue: [self halt]. theOld := theNew]
The debugger came up! Doesn't that mean that, if the code is run very fast, there are double OIDs generated?!
In my case, the objects for country and currency are very lightweight and so, they are created very fast. Also, the error occured much more often within my fast production EC2 instance as at my (old and slow) development pc.
Important: I work with windows and so >>makeUnixSeed returns nil... :-)
What is your opinion about that?
Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705603.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
So, I will wait and till then, use the solution of Jan. Thank you, Jan for the file in. Sabine -- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705821.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi, One idea (not sure if it will work) You can take Time primUTCMicrosecondsClock when system start, then increase it sequentially. most probably that will ensure uniqueness while providing fast ids. but of course, that depends, you need to check in your system. I'm thinking on provide different ID generators for Voyage, and let people plug what is more convenient for them. (all of this happens because, unlike other db drivers, mongo does not answer the inserted id :( ) Esteban On Aug 30, 2013, at 5:41 PM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
So, I will wait and till then, use the solution of Jan. Thank you, Jan for the file in. Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705821.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi Esteban, yes, Mongo is not very communicative ;-) I replaced OID>>nextOID with my own RKAOIDGenerator nextOID ^self value: (RKAOIDGenerator getNextOID) instead of nextOID ^self value: (UUIDGenerator default makeSeed) RKAOIDGenerator uses Time>>primMillisecondClock at startup and adds +1 each time I getNextOID as you suggested. This works fine. It seems to be much much much faster than the existing solution. [4000 timesRepeat: [UUIDGenerator default makeSeed]] timeToRun ==>> 5817 [4000 timesRepeat: [RKAOIDGenerator getNextOID]] timeToRun ==> 1 (!!) Perhaps a solution like this would be a better solution for all because it is faster and guarantees unique IDs? Sabine On Fri, Aug 30, 2013 at 5:58 PM, EstebanLM [via Smalltalk] < ml-node+s1294792n4705827h32@n4.nabble.com> wrote:
Hi,
One idea (not sure if it will work)
You can take
Time primUTCMicrosecondsClock
when system start, then increase it sequentially. most probably that will ensure uniqueness while providing fast ids.
but of course, that depends, you need to check in your system.
I'm thinking on provide different ID generators for Voyage, and let people plug what is more convenient for them.
(all of this happens because, unlike other db drivers, mongo does not answer the inserted id :( )
Esteban
On Aug 30, 2013, at 5:41 PM, Sabine Knöfel <[hidden email]<http://user/SendEmail.jtp?type=node&node=4705827&i=0>> wrote:
So, I will wait and till then, use the solution of Jan. Thank you, Jan for the file in. Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705821.htm...
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
------------------------------ If you reply to this email, your message will be added to the discussion below:
http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705827.htm... To unsubscribe from voyage/mongo randomly wrong OIDs, click here<http://forum.world.st/template/NamlServlet.jtp?macro=unsubscribe_by_code&nod...> . NAML<http://forum.world.st/template/NamlServlet.jtp?macro=macro_viewer&id=instant...>
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705898.htm... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Sabine what we could do is to propose a "subclass of UUID" and to group several UUID generators. Like that with a couple of classes, we could get a better eco system where people can pick the one they want. stef On Aug 31, 2013, at 11:49 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban,
yes, Mongo is not very communicative ;-)
I replaced OID>>nextOID with my own RKAOIDGenerator
nextOID ^self value: (RKAOIDGenerator getNextOID)
instead of
nextOID ^self value: (UUIDGenerator default makeSeed)
RKAOIDGenerator uses Time>>primMillisecondClock at startup and adds +1 each time I getNextOID as you suggested.
This works fine. It seems to be much much much faster than the existing solution.
[4000 timesRepeat: [UUIDGenerator default makeSeed]] timeToRun ==>> 5817 [4000 timesRepeat: [RKAOIDGenerator getNextOID]] timeToRun ==> 1 (!!)
Perhaps a solution like this would be a better solution for all because it is faster and guarantees unique IDs?
Sabine
On Fri, Aug 30, 2013 at 5:58 PM, EstebanLM [via Smalltalk] <[hidden email]> wrote: Hi,
One idea (not sure if it will work)
You can take
Time primUTCMicrosecondsClock
when system start, then increase it sequentially. most probably that will ensure uniqueness while providing fast ids.
but of course, that depends, you need to check in your system.
I'm thinking on provide different ID generators for Voyage, and let people plug what is more convenient for them.
(all of this happens because, unlike other db drivers, mongo does not answer the inserted id :( )
Esteban
On Aug 30, 2013, at 5:41 PM, Sabine Knöfel <[hidden email]> wrote:
So, I will wait and till then, use the solution of Jan. Thank you, Jan for the file in. Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705821.htm...
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
If you reply to this email, your message will be added to the discussion below: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705827.htm... To unsubscribe from voyage/mongo randomly wrong OIDs, click here. NAML
View this message in context: Re: voyage/mongo randomly wrong OIDs Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On 31 Aug 2013, at 13:47, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Sabine what we could do is to propose a "subclass of UUID" and to group several UUID generators. Like that with a couple of classes, we could get a better eco system where people can pick the one they want.
stef
I just made my own, called NeoUUIDGenerator, http://www.smalltalkhub.com/#!/~SvenVanCaekenberghe/Neo/packages/Neo-UUID @Sabine IMHO what I think a local counter does not, is give you uniques over different machines, images, instances - that is why there is also the concept of node identification. In my implementation I combine the millisecond clock, a small random number, a counter and a node id. The node id is based on several elements, it should be different when running multiple images. This is a hack, not something that I can prove mathematically. But it can't be worse than pure random. I think the speed is also acceptable: | generator | generator := NeoUUIDGenerator new. [ generator next ] bench. '408,000 per second.' | generator | generator := UUIDGenerator new. [ generator generateBytes: UUID nilUUID forVersion: 4 ] bench. '13,300 per second.' Sven
On Aug 31, 2013, at 11:49 AM, Sabine Knöfel <sabine.knoefel@gmail.com> wrote:
Hi Esteban,
yes, Mongo is not very communicative ;-)
I replaced OID>>nextOID with my own RKAOIDGenerator
nextOID ^self value: (RKAOIDGenerator getNextOID)
instead of
nextOID ^self value: (UUIDGenerator default makeSeed)
RKAOIDGenerator uses Time>>primMillisecondClock at startup and adds +1 each time I getNextOID as you suggested.
This works fine. It seems to be much much much faster than the existing solution.
[4000 timesRepeat: [UUIDGenerator default makeSeed]] timeToRun ==>> 5817 [4000 timesRepeat: [RKAOIDGenerator getNextOID]] timeToRun ==> 1 (!!)
Perhaps a solution like this would be a better solution for all because it is faster and guarantees unique IDs?
Sabine
On Fri, Aug 30, 2013 at 5:58 PM, EstebanLM [via Smalltalk] <[hidden email]> wrote: Hi,
One idea (not sure if it will work)
You can take
Time primUTCMicrosecondsClock
when system start, then increase it sequentially. most probably that will ensure uniqueness while providing fast ids.
but of course, that depends, you need to check in your system.
I'm thinking on provide different ID generators for Voyage, and let people plug what is more convenient for them.
(all of this happens because, unlike other db drivers, mongo does not answer the inserted id :( )
Esteban
On Aug 30, 2013, at 5:41 PM, Sabine Knöfel <[hidden email]> wrote:
So, I will wait and till then, use the solution of Jan. Thank you, Jan for the file in. Sabine
-- View this message in context: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705821.htm...
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
If you reply to this email, your message will be added to the discussion below: http://forum.world.st/voyage-mongo-randomly-wrong-OIDs-tp4705396p4705827.htm... To unsubscribe from voyage/mongo randomly wrong OIDs, click here. NAML
View this message in context: Re: voyage/mongo randomly wrong OIDs Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On Aug 31, 2013, at 3:30 , Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 31 Aug 2013, at 13:47, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Sabine what we could do is to propose a "subclass of UUID" and to group several UUID generators. Like that with a couple of classes, we could get a better eco system where people can pick the one they want.
stef
I just made my own, called NeoUUIDGenerator, http://www.smalltalkhub.com/#!/~SvenVanCaekenberghe/Neo/packages/Neo-UUID
@Sabine
IMHO what I think a local counter does not, is give you uniques over different machines, images, instances - that is why there is also the concept of node identification.
In my implementation I combine the millisecond clock, a small random number, a counter and a node id. The node id is based on several elements, it should be different when running multiple images.
This is a hack, not something that I can prove mathematically. But it can't be worse than pure random. I think the speed is also acceptable:
| generator | generator := NeoUUIDGenerator new. [ generator next ] bench. '408,000 per second.'
| generator | generator := UUIDGenerator new. [ generator generateBytes: UUID nilUUID forVersion: 4 ] bench. '13,300 per second.'
Sven
So sorta like UUID type 3/5, but with a custom object identifier scheme, and no hashing? Not sure it'd be fair to call that any kind of UUIDGenerator anymore, as the UUID standard and its encompassing types is a pretty well-defined ;) IIRC, the reason those went out of flavor in favor of type 4, is the fact they do potentially identify the source computer from which they were created, and thus a purely random approach was considered better. (as long as it is just that, which, as this thread illustrates, is another matter) Cheers, Henry
On 02 Sep 2013, at 10:34, Henrik Johansen <henrik.s.johansen@veloxit.no> wrote:
On Aug 31, 2013, at 3:30 , Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 31 Aug 2013, at 13:47, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:
Sabine what we could do is to propose a "subclass of UUID" and to group several UUID generators. Like that with a couple of classes, we could get a better eco system where people can pick the one they want.
stef
I just made my own, called NeoUUIDGenerator, http://www.smalltalkhub.com/#!/~SvenVanCaekenberghe/Neo/packages/Neo-UUID
@Sabine
IMHO what I think a local counter does not, is give you uniques over different machines, images, instances - that is why there is also the concept of node identification.
In my implementation I combine the millisecond clock, a small random number, a counter and a node id. The node id is based on several elements, it should be different when running multiple images.
This is a hack, not something that I can prove mathematically. But it can't be worse than pure random. I think the speed is also acceptable:
| generator | generator := NeoUUIDGenerator new. [ generator next ] bench. '408,000 per second.'
| generator | generator := UUIDGenerator new. [ generator generateBytes: UUID nilUUID forVersion: 4 ] bench. '13,300 per second.'
Sven
So sorta like UUID type 3/5, but with a custom object identifier scheme, and no hashing? Not sure it'd be fair to call that any kind of UUIDGenerator anymore, as the UUID standard and its encompassing types is a pretty well-defined ;)
Yes, it is a hack, mixing type 3/5 elements while pretending to be type 4 ;-)
IIRC, the reason those went out of flavor in favor of type 4, is the fact they do potentially identify the source computer from which they were created, and thus a purely random approach was considered better. (as long as it is just that, which, as this thread illustrates, is another matter)
Somehow, I don't feel like just random data would do. Maybe the chance for repetition is low, but it is not zero across instances, images and machines, and it depends on the quality of a random generator that is hard to control. I have this feeling that adding a counter, the time and a node identification is better. But this is totally unscientific.
Cheers, Henry
On Aug 30, 2013, at 4:35 , Esteban Lorenzano <estebanlm@gmail.com> wrote:
On Aug 29, 2013, at 5:08 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 29 Aug 2013, at 16:51, Esteban Lorenzano <estebanlm@gmail.com> wrote:
hi
well... I've never been happy on using the UUID generator for my keys, but is the fastest option I found. There are, of course, alternatives:
1) Using your own number generator (sequential, or whatever). Problem with that is that is image based, then you need a persistence strategy... then you are slow. 2) then you can use your own procedure in mongo... with same problem than (1) 3) you could use timestamp. but TimeStamps are slow :(
anyway... I open to ideas :)
in the mean time, you can check if your UUID generator is using the primitive or not. In you are not, you have more possibilities of having a collision.
Yes, the Smalltalk code (type 4 UUID) is just a random number that is computed in a complex way.
What does the primitive actually do ? Is it different ?
the primitive uses the clock ticks to produce an UUID... you shouldn't have repeated numbers that way... but well, it depends on the platform implementation also.
Wait, which primitive is this? UUID > primMakeUUID? That code calls libUUID, which (as long as it doesn't crash :P) fetches from dev/urandom (or the windows equivalent RtlGenRandom), there shouldn't ever be a problem getting duplicate results unless those system-provided resources are bugged, in which case your entire system is borked. The fallback code using UUIDGenerator default admittedly has a weak PRNG, and is dog slow: gen := UUIDGenerator new. [gen generateFieldsVersion4] bench '12,900 per second.' [ UUID new ] bench '3,610,000 per second.' "Primitive working, btw" but that hardly means you should run into duplicate sequential UUID's⦠The *only* way sequentially equal UUID's could arise, is if one uses UUIDGenerator directly, and creates a new instance each time, which does indeed run the risk of returning values derived from the same Time initialization , but that should *never* be done⦠Cheers, Henry
participants (8)
-
Esteban A. Maringolo -
Esteban Lorenzano -
Henrik Johansen -
jan.struz -
Sabine Knöfel -
Stéphane Ducasse -
Sven Van Caekenberghe -
Sven Van Caekenberghe