[Pharo-project] FilePlugin rant
I recently started to hack around Monticello due to very low response times when having a huge package cache. Now interestingly the FilePlugin primitives for listing files are horrible! Example: ----------------------------------------------------------- MC often requests a list of all mzc's cached locally: (FileDirectory default / 'package-cache') entries In my case there are 4300 mczs in the shared package-cache timeToRun yields a nightmarish 2050ms for this, further investigation leads to the fact that most of the time is spent in the file primitive! Now thats some 2ms for a single directory entry! [Just as a side note, ruby1.8 does the same job in 10ms :D] ----------------------------------------------------------- primLookupEntryIn: fullPath index: index <primitive: 'primitiveDirectoryLookup' module: 'FilePlugin'> ----------------------------------------------------------- I think we should definitely add another primitive which directly lists the file under a certain path. Of course this will break compatibility with existing code, but I think the current approach does not scale... best cami
Camillo, On 01 Mar 2012, at 10:54, Camillo Bruni wrote:
I recently started to hack around Monticello due to very low response times when having a huge package cache.
Now interestingly the FilePlugin primitives for listing files are horrible!
Example: ----------------------------------------------------------- MC often requests a list of all mzc's cached locally: (FileDirectory default / 'package-cache') entries
In my case there are 4300 mczs in the shared package-cache timeToRun yields a nightmarish 2050ms for this, further investigation leads to the fact that most of the time is spent in the file primitive!
Now thats some 2ms for a single directory entry! [Just as a side note, ruby1.8 does the same job in 10ms :D] -----------------------------------------------------------
primLookupEntryIn: fullPath index: index <primitive: 'primitiveDirectoryLookup' module: 'FilePlugin'>
-----------------------------------------------------------
I think we should definitely add another primitive which directly lists the file under a certain path.
Of course this will break compatibility with existing code, but I think the current approach does not scale...
best cami
I have (FileDirectory default / 'package-cache') entries size 5053 [ (FileDirectory default / 'package-cache') entries ] timeToRun 2012 But if I run it in the Time Profiler, I get this: How do you manage to track the primitive ? Why/how would an extra primitive or better implementation of #entries break compatibility ? Sven
Hi Sven, I did a quick test with Igor by removing all unnecessary code in FileDirectory >> #directoryContentsFor:do: and you will see that just running #primLookupEntryIn:index: will take 100% of the time. best cami On 2012-03-01, at 11:14, Sven Van Caekenberghe wrote:
Camillo,
On 01 Mar 2012, at 10:54, Camillo Bruni wrote:
I recently started to hack around Monticello due to very low response times when having a huge package cache.
Now interestingly the FilePlugin primitives for listing files are horrible!
Example: ----------------------------------------------------------- MC often requests a list of all mzc's cached locally: (FileDirectory default / 'package-cache') entries
In my case there are 4300 mczs in the shared package-cache timeToRun yields a nightmarish 2050ms for this, further investigation leads to the fact that most of the time is spent in the file primitive!
Now thats some 2ms for a single directory entry! [Just as a side note, ruby1.8 does the same job in 10ms :D] -----------------------------------------------------------
primLookupEntryIn: fullPath index: index <primitive: 'primitiveDirectoryLookup' module: 'FilePlugin'>
-----------------------------------------------------------
I think we should definitely add another primitive which directly lists the file under a certain path.
Of course this will break compatibility with existing code, but I think the current approach does not scale...
best cami
I have
(FileDirectory default / 'package-cache') entries size 5053 [ (FileDirectory default / 'package-cache') entries ] timeToRun 2012
But if I run it in the Time Profiler, I get this:
<Screen Shot 2012-03-01 at 11.12.46.png>
How do you manage to track the primitive ?
Why/how would an extra primitive or better implementation of #entries break compatibility ?
Sven
On 03/01/2012 11:14 AM, Sven Van Caekenberghe wrote:
Camillo,
On 01 Mar 2012, at 10:54, Camillo Bruni wrote:
I recently started to hack around Monticello due to very low response times when having a huge package cache.
Now interestingly the FilePlugin primitives for listing files are horrible!
Example: ----------------------------------------------------------- MC often requests a list of all mzc's cached locally: (FileDirectory default / 'package-cache') entries
In my case there are 4300 mczs in the shared package-cache timeToRun yields a nightmarish 2050ms for this, further investigation leads to the fact that most of the time is spent in the file primitive!
Now thats some 2ms for a single directory entry! [Just as a side note, ruby1.8 does the same job in 10ms :D] -----------------------------------------------------------
primLookupEntryIn: fullPath index: index <primitive: 'primitiveDirectoryLookup' module: 'FilePlugin'>
-----------------------------------------------------------
I think we should definitely add another primitive which directly lists the file under a certain path.
Of course this will break compatibility with existing code, but I think the current approach does not scale...
best cami
I have
(FileDirectory default / 'package-cache') entries size 5053 [ (FileDirectory default / 'package-cache') entries ] timeToRun 2012
But if I run it in the Time Profiler, I get this:
How do you manage to track the primitive ?
You could give SystemProfiler a shot. [1] http://www.squeaksource.com/SystemProfilerPharo.html Cheers Philippe
Hi Cami. This is a useless email but I just wanted to say that I have also experimented the problem of Monticello being unusable with large package-cache. So...whatever you can do to improve it, I will be very very happy. On Thu, Mar 1, 2012 at 10:54 AM, Camillo Bruni <camillobruni@gmail.com>wrote:
I recently started to hack around Monticello due to very low response times when having a huge package cache.
Now interestingly the FilePlugin primitives for listing files are horrible!
Example: ----------------------------------------------------------- MC often requests a list of all mzc's cached locally: (FileDirectory default / 'package-cache') entries
In my case there are 4300 mczs in the shared package-cache timeToRun yields a nightmarish 2050ms for this, further investigation leads to the fact that most of the time is spent in the file primitive!
Now thats some 2ms for a single directory entry! [Just as a side note, ruby1.8 does the same job in 10ms :D] -----------------------------------------------------------
primLookupEntryIn: fullPath index: index <primitive: 'primitiveDirectoryLookup' module: 'FilePlugin'>
-----------------------------------------------------------
I think we should definitely add another primitive which directly lists the file under a certain path.
Of course this will break compatibility with existing code, but I think the current approach does not scale...
best cami
-- Mariano http://marianopeck.wordpress.com
participants (4)
-
Camillo Bruni -
Mariano Martinez Peck -
Philippe Marschall -
Sven Van Caekenberghe