VoyageMongo and cache compaction
Hi guys, I started to run a production load with VoyageMongo and every day the maximum latency of it goes up a bit. Today I looked at the production system and the VOMongoRepository and the VOMongoCache and saw that the cache had around 90 entries in the "objects" WeakValueDictionary all pointing to nil 34k entries in the timestamp dictionary This is a write "heavy" workload but at no point in time more than a couple of objects will be in the cache. So somehow the VOMongoCache>>#compactIfNeeded check does not trigger and/or given the difference in the sizes of the dictionary it is possible that some of the dead objects are not removed from the timestamp cache. I have accidentally restarted one image and flushed the cache on the other but if it is of any help I can wait a day or two and then look at the compactLimit again. But even right now in one image the compactLimit is 3277, the objects dictionary has 142 entries and the timestamp one 272. Is this a known issue? Does this look like "normal"/"good" operation? kind regards holger
On 03 Dec 2015, at 09:58, Holger Freyther <holger@freyther.de> wrote:
Hi guys,
This is a write "heavy" workload but at no point in time more than a couple of objects will be in the cache. So somehow the VOMongoCache>>#compactIfNeeded check does not trigger and/or given the difference in the sizes of the dictionary it is possible that some of the dead objects are not removed from the timestamp cache.
the keyword here is "delete" heavy. VOMongoCache>>#removeValue: is called by the VOMongoRepository>>#remove: but this leaves the timeStamp (pharo v3 version or versions in newer) to have an entry for a OID that was removed. And as >>#performCompact will only remove "dead" keys from timeStamp/versions it will leave many delete entries in the timeStamp. The other thing is that I wonder why the self mutex critical lock is not taken around the removal of the object? Is there anyone willing to resolve this, create a testcase and make a backport to Pharo3? holger
I haven't used VoyageMongo, but just a general comment that maybe the testcase is best produced at your end, since you understand the symptoms and can cut down a slice of your application as a template. Publishing that to the mail list provides an explicit demonstration of the problem making it easier for someone to assist. Indeed, as an extension if all the steps to load up VoyageMongo and set up the environment are included, sometimes I'll even be inclined to investigate such things that are unknown to me as a form of goal-directed learning (although right I'm now tied up with some other things). cheers -ben On Thu, Dec 3, 2015 at 10:02 PM, Holger Freyther <holger@freyther.de> wrote:
On 03 Dec 2015, at 09:58, Holger Freyther <holger@freyther.de> wrote:
Hi guys,
This is a write "heavy" workload but at no point in time more than a couple of objects will be in the cache. So somehow the VOMongoCache>>#compactIfNeeded check does not trigger and/or given the difference in the sizes of the dictionary it is possible that some of the dead objects are not removed from the timestamp cache.
the keyword here is "delete" heavy. VOMongoCache>>#removeValue: is called by the VOMongoRepository>>#remove: but this leaves the timeStamp (pharo v3 version or versions in newer) to have an entry for a OID that was removed.
And as >>#performCompact will only remove "dead" keys from timeStamp/versions it will leave many delete entries in the timeStamp.
The other thing is that I wonder why the self mutex critical lock is not taken around the removal of the object?
Is there anyone willing to resolve this, create a testcase and make a backport to Pharo3?
holger
On 04 Dec 2015, at 00:16, Ben Coman <btc@openinworld.com> wrote:
I haven't used VoyageMongo, but just a general comment that maybe the testcase is best produced at your end, since you understand the symptoms and can cut down a slice of your application as a template. Publishing that to the mail list provides an explicit demonstration of the problem making it easier for someone to assist. Indeed, as an extension if all the steps to load up VoyageMongo and set up the environment are included, sometimes I'll even be inclined to investigate such things that are unknown to me as a form of goal-directed learning (although right I'm now tied up with some other things).
One will need to install a real mongodb to test this and then load VoyageMongo (e.g. through the Configuration Browser) | repo point | repo := VOMongoRepository database: 'cacheIssue'. 1 to: 100 do: [:each | point := Point x: 10 y: 20. repo save: point. repo remove: point. ]. "In Pharo3.0 inspect objects and timeStamps" (repo cache instVarNamed: #objects) size -> (repo cache instVarNamed: #timeStamps) size. "Later versions inspect objects and versions" "(repo cache instVarNamed: #objects) size." (repo cache instVarNamed: #versions) size." The entries in timeStamps/versions will not go away even if one calls: repo cache performCompact The two issues I see are: 1.) VOMongoCache>>#removeValue: is leaving the timeStamp/versions alone and nobody is removing the timestamp entry and compaction will only remove keys from timeStamps/versions that were inside the objects. 2.) removeValue should most likely use self mutex criticial to guard itself. kind regards holger
participants (2)
-
Ben Coman -
Holger Freyther