[Pharo-project] Speeding up Finder string search
Hi: The finder is annoyingly slow when doing string searches. Compared to a 'grep' its multiple magnitudes slower, at least from my perception :( Profiling pointed my at RemoteString>>string and its use of readOnlyCopy on the SourceFiles. I put a cache of these read only files into ExpandedSourceFileArray and get a nice speedup. { [ Finder new constructSourceDictionary ] timeToRun. [ Finder new constructSourceDictionary ] timeToRun. [ Finder new constructSourceDictionary ] timeToRun . } Without Cache: #(32079 29034 32274) With Cache: #(10757 10718 11050) While that works for the searching, it does fall apart for other uses of RemoteString>>string. (For instance, filing out the changes...) If anyone has an idea how to avoid the overhead of copying, I would be all ears. Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
Thanks for having a look at this problem. With Camillo, we though about using parallel processes but never go further than an idea :) Ben On Jan 15, 2012, at 6:11 PM, Stefan Marr wrote:
Hi:
The finder is annoyingly slow when doing string searches. Compared to a 'grep' its multiple magnitudes slower, at least from my perception :(
Profiling pointed my at RemoteString>>string and its use of readOnlyCopy on the SourceFiles.
I put a cache of these read only files into ExpandedSourceFileArray and get a nice speedup.
{ [ Finder new constructSourceDictionary ] timeToRun. [ Finder new constructSourceDictionary ] timeToRun. [ Finder new constructSourceDictionary ] timeToRun . }
Without Cache: #(32079 29034 32274) With Cache: #(10757 10718 11050)
While that works for the searching, it does fall apart for other uses of RemoteString>>string. (For instance, filing out the changes...)
If anyone has an idea how to avoid the overhead of copying, I would be all ears.
Thanks Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
I guess having the sources completely in the image is the straight-forward solution to this :/ or using a native tool to grep the source files and then use the results to map back to the methods cami On 2012-01-15, at 18:54, Benjamin wrote:
Thanks for having a look at this problem.
With Camillo, we though about using parallel processes but never go further than an idea :)
Ben
On Jan 15, 2012, at 6:11 PM, Stefan Marr wrote:
Hi:
The finder is annoyingly slow when doing string searches. Compared to a 'grep' its multiple magnitudes slower, at least from my perception :(
Profiling pointed my at RemoteString>>string and its use of readOnlyCopy on the SourceFiles.
I put a cache of these read only files into ExpandedSourceFileArray and get a nice speedup.
{ [ Finder new constructSourceDictionary ] timeToRun. [ Finder new constructSourceDictionary ] timeToRun. [ Finder new constructSourceDictionary ] timeToRun . }
Without Cache: #(32079 29034 32274) With Cache: #(10757 10718 11050)
While that works for the searching, it does fall apart for other uses of RemoteString>>string. (For instance, filing out the changes...)
If anyone has an idea how to avoid the overhead of copying, I would be all ears.
Thanks Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
Hi: On 16 Jan 2012, at 14:31, Camillo Bruni wrote:
I guess having the sources completely in the image is the straight-forward solution to this :/
Well, we could also cache just the resulting source string. Not sure what that means in terms of memory, but going to the file every time a method is asked for its code seems to be very naive. How does Ring do it?
or using a native tool to grep the source files and then use the results to map back to the methods
Ehm, that sounds very brittle, except if you got a nice file representation of your code of course ;) Levente pointed out to me that he introduced a smarter approach to caching in Squeak, with similar performance improvements, and he was thinking about improving that to a process-local variable. Perhaps also an option. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
Hi: On 16 Jan 2012, at 14:50, Stefan Marr wrote:
Hi:
On 16 Jan 2012, at 14:31, Camillo Bruni wrote:
I guess having the sources completely in the image is the straight-forward solution to this :/
Well, we could also cache just the resulting source string. Not sure what that means in terms of memory, but going to the file every time a method is asked for its code seems to be very naive.
I put a cache (IdentityDictionary) methodSource into Class. That requires recompiling all classes which takes a while, and I do not know, whether it should not be in ClassDescription. And guess what: The bad news: the image size doubles from 22.2MB to 39.5MB. The good news: "before using cache" #(43239 23490 22084) #(20581 20238 20207) The 43239 created all dictionaries. "warming up and using cache" #(20177 1428 1393) #(1389 1374 1400) That is a pretty nice speedup, grep is still a faster, but now finder becomes usable! I use the cache like this: CompiledMethod>>getSourceFromFile "Read the source code from file, determining source file index and file position from the last 3 bytes of this method." | position classSrc | classSrc := self methodClass getSourceFor: self selector. classSrc ifNotNil: [ ^ classSrc]. (position := self filePosition) = 0 ifTrue: [^ nil]. classSrc := (RemoteString newFileNumber: self fileIndex position: position) string. self methodClass setSourceFor: self selector to: classSrc. ^ classSrc To me that feels all pretty straight forward and naive, so feel free to have better ideas. Perhaps the cache should get evicted when not used, etc... Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
Stefan, another approach would be to store the source code in the compiled method trailer. That way you don't need the IdentityDictionary nor searching anything. Just store the source in there. Since a couple of releases, CompiledMethodTrailer is reified and you can have custom types of trailers. Check methods like #encodeEmbeddedSourceQCompress and #decodeEmbeddedSourceQCompress etc... not sure if everything is working, but I am sure that the CompiledMethodTrailer "architecture" is working and it should be really easy to be able to store sources there. Cheers PS: I will interested if you do it :) On Mon, Jan 16, 2012 at 10:07 PM, Stefan Marr <smalltalk@stefan-marr.de>wrote:
Hi:
On 16 Jan 2012, at 14:50, Stefan Marr wrote:
Hi:
On 16 Jan 2012, at 14:31, Camillo Bruni wrote:
I guess having the sources completely in the image is the straight-forward solution to this :/
Well, we could also cache just the resulting source string. Not sure what that means in terms of memory, but going to the file every time a method is asked for its code seems to be very naive.
I put a cache (IdentityDictionary) methodSource into Class. That requires recompiling all classes which takes a while, and I do not know, whether it should not be in ClassDescription.
And guess what:
The bad news: the image size doubles from 22.2MB to 39.5MB.
The good news:
"before using cache" #(43239 23490 22084) #(20581 20238 20207) The 43239 created all dictionaries.
"warming up and using cache" #(20177 1428 1393) #(1389 1374 1400)
That is a pretty nice speedup, grep is still a faster, but now finder becomes usable!
I use the cache like this: CompiledMethod>>getSourceFromFile "Read the source code from file, determining source file index and file position from the last 3 bytes of this method." | position classSrc |
classSrc := self methodClass getSourceFor: self selector. classSrc ifNotNil: [ ^ classSrc].
(position := self filePosition) = 0 ifTrue: [^ nil]. classSrc := (RemoteString newFileNumber: self fileIndex position: position) string. self methodClass setSourceFor: self selector to: classSrc.
^ classSrc
To me that feels all pretty straight forward and naive, so feel free to have better ideas. Perhaps the cache should get evicted when not used, etc...
Best regards Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Mariano http://marianopeck.wordpress.com
Hi: On 17 Jan 2012, at 09:47, Mariano Martinez Peck wrote:
Stefan, another approach would be to store the source code in the compiled method trailer. Looks like everything is there already, but it does not seem to work stable enough.
With the code below, this should work: (do not apply the change to getSourceFromFile!!) ------------- |trailer source m | m := (Gofer>>#load). trailer := m trailer. source := m getSource. trailer sourceCode: source. m recompileWithTrailer: trailer. ------------- Here the code: ------------- !Behavior methodsFor: 'compiling' stamp: 'StefanMarr 1/17/2012 15:14' prior: 82391250! recompile: selector from: oldClass withNewTrailer: trailer "Compile the method associated with selector in the receiver's method dictionary." "ar 7/10/1999: Use oldClass compiledMethodAt: not self compiledMethodAt:" | method methodNode | method := oldClass compiledMethodAt: selector. methodNode := self compilerClass new compile: (oldClass sourceCodeAt: selector) in: self notifying: nil ifFail: [^ self]. "Assume OK after proceed from SyntaxError" selector == methodNode selector ifFalse: [self error: 'selector changed!!']. self basicAddSelector: selector withMethod: (methodNode generate: trailer). ! ! !Behavior methodsFor: 'compiling' stamp: 'StefanMarr 1/17/2012 15:15' prior: 82391969! recompile: selector from: oldClass | method | method := oldClass compiledMethodAt: selector. self recompile: selector from: oldClass withNewTrailer: method trailer. ! ! !CompiledMethod methodsFor: 'as yet unclassified' stamp: 'StefanMarr 1/17/2012 15:16' prior: 82392229! recompileWithTrailer: trailer self methodClass recompile: self selector from: self methodClass withNewTrailer: trailer.! ! !CompiledMethod methodsFor: 'source code management' stamp: 'StefanMarr 1/17/2012 15:23' prior: 68352220! getSourceFromFile "Read the source code from file, determining source file index and file position from the last 3 bytes of this method." | position source trailer | trailer := self trailer. source := trailer sourceCode. source ifNotNil: [ ^ source ]. (position := self filePosition) = 0 ifTrue: [^ nil]. source := (RemoteString newFileNumber: self fileIndex position: position) string. trailer sourceCode: source. self recompileWithTrailer: trailer. ^ source.! ! ------------- This is unfortunately undebuggable. At least I do not get a debugger (which seems logical if there is something going wrong and recurses in getSource...) Good that we have a stratified system. Best regards Stefan
PS: I will interested if you do it :)
In return, I would really appreciate if you could learn a bit about etiquette and quoting http://www.gweep.ca/~edmonds/usenet/ml-etiquette.html#SECTION000900000000000... And I am apparently not the only one who would like that. See the recent reference to http://emailcharter.org/ -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
On Tue, Jan 17, 2012 at 5:58 PM, Stefan Marr <smalltalk@stefan-marr.de>wrote:
Hi:
On 17 Jan 2012, at 09:47, Mariano Martinez Peck wrote:
Stefan, another approach would be to store the source code in the compiled method trailer. Looks like everything is there already, but it does not seem to work stable enough.
Yes, that was pretty much the same idea I got when I tried it.
With the code below, this should work: (do not apply the change to getSourceFromFile!!) ------------- |trailer source m | m := (Gofer>>#load). trailer := m trailer. source := m getSource. trailer sourceCode: source. m recompileWithTrailer: trailer. -------------
If you don't bother about being slow, you can also make it much easier: |trailer source m | m := (Gofer>>#load). trailer := m trailer. source := m getSource. trailer sourceCode: source. m becomeForward: (m copyWithTrailerBytes: trailer) and if you want to do it for the whole image you can use one bulk become and it will be fast. I will experiment and see if it is working.
PS: I will interested if you do it :)
In return, I would really appreciate if you could learn a bit about etiquette and quoting http://www.gweep.ca/~edmonds/usenet/ml-etiquette.html#SECTION000900000000000...
99% of the cases I answer where I have to answer (like I am doing now). But sometimes if my answer is obvious, I think it is much easier a top posting than having to quote your email. -- Mariano http://marianopeck.wordpress.com
If you don't bother about being slow, you can also make it much easier:
|trailer source m | m := (Gofer>>#load). trailer := m trailer. source := m getSource. trailer sourceCode: source. m becomeForward: (m copyWithTrailerBytes: trailer)
and if you want to do it for the whole image you can use one bulk become and it will be fast.
I will experiment and see if it is working.
Well...do not try to execute this: | trailer source key dict | dict := IdentityDictionary new. CompiledMethod allInstances do: [:each | trailer := each trailer. source := each getSource. trailer sourceCode: source. each becomeForward: (each copyWithTrailerBytes: trailer). "dict at: each put: (each copyWithTrailerBytes: trailer)." ]. dict keys asArray elementsExchangeIdentityWith: dict values asArray. since it hangs my VM... time to go now... cheers
PS: I will interested if you do it :)
In return, I would really appreciate if you could learn a bit about etiquette and quoting http://www.gweep.ca/~edmonds/usenet/ml-etiquette.html#SECTION000900000000000...
99% of the cases I answer where I have to answer (like I am doing now). But sometimes if my answer is obvious, I think it is much easier a top posting than having to quote your email.
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Hi Mariano: On 17 Jan 2012, at 18:20, Mariano Martinez Peck wrote:
Yes, that was pretty much the same idea I got when I tried it.
After walking home, the reason for the hang became clear. Recompile is obviously using getSource. After fixing that, I get these numbers: #(102851 10781 10798 9511 10335 9883) Not really impressive. I did not immediately see how to avoid the compression. Is there somewhere an encoding for plain text, or do I have to add that? The image size goes currently up by ca. 50%, from 22MB to 31MB. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
And here we go: On 17 Jan 2012, at 19:37, Stefan Marr wrote:
After fixing that, I get these numbers: #(102851 10781 10798 9511 10335 9883) Not really impressive.
I did not immediately see how to avoid the compression. The image size goes currently up by ca. 50%, from 22MB to 31MB.
#(76700 1787 1793 1768 1741 1748) That is space for performance: image size is at 39MB. Not that my version with an extra dictionary is still faster by about 20% (1.4s vs. 1.7s) since it avoids the decoding from the compiled method. Is there any general use for such an embedding, beside making the finder usable? Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
marcus would love to have all the code in the image. Now we should think about it as a community. Stef On Jan 17, 2012, at 10:10 PM, Stefan Marr wrote:
And here we go:
On 17 Jan 2012, at 19:37, Stefan Marr wrote:
After fixing that, I get these numbers: #(102851 10781 10798 9511 10335 9883) Not really impressive.
I did not immediately see how to avoid the compression. The image size goes currently up by ca. 50%, from 22MB to 31MB.
#(76700 1787 1793 1768 1741 1748)
That is space for performance: image size is at 39MB.
<SourceInMethods.1.cs>
Not that my version with an extra dictionary is still faster by about 20% (1.4s vs. 1.7s) since it avoids the decoding from the compiled method.
Is there any general use for such an embedding, beside making the finder usable?
Best regards Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
On Tue, Jan 17, 2012 at 10:10 PM, Stefan Marr <smalltalk@stefan-marr.de>wrote:
And here we go:
On 17 Jan 2012, at 19:37, Stefan Marr wrote:
After fixing that, I get these numbers: #(102851 10781 10798 9511 10335 9883) Not really impressive.
I did not immediately see how to avoid the compression. The image size goes currently up by ca. 50%, from 22MB to 31MB.
#(76700 1787 1793 1768 1741 1748)
That is space for performance: image size is at 39MB.
Not that my version with an extra dictionary is still faster by about 20% (1.4s vs. 1.7s) since it avoids the decoding from the compiled method.
Is there any general use for such an embedding, beside making the finder usable?
I have created a separate thread because under this subject nobody will read it. I will answer there.
Best regards Stefan
-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
-- Mariano http://marianopeck.wordpress.com
participants (5)
-
Benjamin -
Camillo Bruni -
Mariano Martinez Peck -
Stefan Marr -
Stéphane Ducasse