October 2009 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Shared directories: The bug is quicker than the eye

by Schwab,Wilhelm K

Dave, I am pretty certain I am *not* following all of this, but it is something that I will hopefully check in the near future. I have been amazed at Squeak's willingness to suppress information about errors in other situations, so this is worth a careful look. Bill -----Original Message----- From: pharo-project-bounces(a)lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis Sent: Wednesday, October 28, 2009 5:20 PM To: Pharo-project(a)lists.gforge.inria.fr Subject: Re: [Pharo-project] Shared directories: The bug is quicker than the eye On any platform that supports the C library fflush(), there is numeric error number ("errno") that is set if the fflush() call does not succeed (type "man fflush" on your Linux or OS X console for details). John pointed out that the #flush method does not provide you with the errno value in the event of a failure, so you cannot directly check the errno value to see if there was a problem. I was showing a way that this could be done using some OSProcess calls rather than the normal #flush. My assumption was that you have some kind of file stream in your code, and that you would do the OSProcess thing instead of #flush just for purposes of debugging the problem. My personal hunch is that you are not likely to see any failure from the #flush calls, even if you are doing the #flush and the data seems to be flushed down the bit bucket. The reason is that the result of the #flush will appear successful as long as the C library fflush() has handed the data over to the operating system. On networked file systems there is a long way to go between the time the data leaves your program (the VM doing fflush()) and the time that something actually lands on the spinning platter. So you can try checking the result of the #flush calls to be sure that they are not failing, but I'm guessing that you won't see any failures. HTH, Dave On Wed, Oct 28, 2009 at 03:10:00PM -0400, Schwab,Wilhelm K wrote: > Dave, > > I'm not a fully trained penguin herder yet, so I have to ask a dumb question: is the idea to connect to the same file that went wrong and get an error condition after the fact, or are you hoping that opening and flushing will cause an error by itself? I suspect the problem is very real, and more "random" than that could hope to uncover. > > Bill > > > -----Original Message----- > From: pharo-project-bounces(a)lists.gforge.inria.fr > [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of > David T. Lewis > Sent: Sunday, October 25, 2009 3:27 PM > To: Pharo-project(a)lists.gforge.inria.fr; > johnmci(a)smalltalkconsulting.com > Subject: Re: [Pharo-project] Shared directories: The bug is quicker > than the eye > > I don't know if this helps, but for purposes of debugging here is a way to get at the error code from a flush(). No doubt I wrote the primitive by accident, not noticing that there was already a primitiveFileFlush in the FilePlugin. > > aStream := FileStream readOnlyFileNamed: 'foo.tmp'. > result := OSProcess accessor flushExternalStream: aStream fileID. > errorString := OSProcess accessor primErrorMessageAt: result. > > Dave > > > On Fri, Oct 23, 2009 at 11:57:06AM -0700, John M McIntosh wrote: > > If you invoke the flush primitive then it does this on > > unix/linux/mac- osx/iPhone > > http://developer.apple.com/mac/library/documentation/Darwin/Referenc > > e/ > > ManPages/man3/fflush.3.html > > & > > http://developer.apple.com/mac/library/documentation/Darwin/Referenc > > e/ ManPages/man2/write.2.html#//apple_ref/doc/man/2/write > > > > sqInt sqFileFlush(SQFile *f) { > > /* Return the length of the given file. */ > > > > if (!sqFileValid(f)) return interpreterProxy->success(false); > > fflush(getFile(f)); > > return 1; > > } > > > > On windows it does > > http://msdn.microsoft.com/en-us/library/aa364439(VS.85).aspx > > > > sqInt sqFileFlush(SQFile *f) { > > if (!sqFileValid(f)) FAIL(); > > /* note: ignores the return value in case of read-only access */ > > FlushFileBuffers(FILE_HANDLE(f)); > > return 1; > > } > > > > I'll note the api doesn't actually check for errors, give feedback > > or whatever. > > *cough* it could be failing and give you a clue, but you would need > > to resort to FFI to do the file system calls to get the data to > > visualize, or build your own VM with that returns the result. > > > > > > > > On 2009-10-22, at 5:31 PM, Schwab,Wilhelm K wrote: > > > > > Hello all, > > > > > > I'm not sure what to make of this one. I just spent a couple of > > > hours trying to find the "leak" in an algorithm of mine. It was > > > reading roughly 1200 records, claiming to have processed all of > > > them, and yet writing only about 450 rows into an output text file. > > > One clue should have been that the number of output rows was > > > somewhat random; I did not fully appreciate that until I worked > > > around the problme. > > > > > > I tried an explicit #flush - no help. I looked for logic errors > > > and found none. The file was being written to a Windows hosted > > > share mounted by CIFS (which I am learning to view with contempt) > > > from Ubuntu 9.04. You can see where this is going: writing the > > > file locally gave the expected result. > > > > > > Any ideas on how one might further isolate the problem? My trust > > > in Windows is well known<g>; I have never liked shared > > > directories; I _really_ do not like CIFS as compared (reliability > > > wise) to SMBFS; the network between me and the server is in > > > question too (long story). All of that said, file support in > > > Squeak, and hence so far inherited by Pharo, is not the best code > > > I have seen to date, so it is easy to suspect too. Can one argue > > > that since it worked locally, Pharo is not the problem? > > > > > > The little bit that I know of cifs is not encouraging. It sounds > > > as though things moved from an easily killed process into the > > > kernel which shows an almost Windows-like unwillingness to shut > > > down when it cannot see servers. I have found numerous reports of > > > problems copying large files over cifs, and I have enountered them too. > > > > > > Bill > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list Pharo-project(a)lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Oct. 29, 2009

Re: [Pharo-project] Hashed collection improvements: Some code

by Martin McClure

StÃ©phane Ducasse wrote: > Martin > > are these changes related to the graphs you sent? Yes, you should be able to duplicate the graphs after filing in the code I sent. Sorry for not giving the two the same subject line; I was in too much of a hurry that night. But the code I sent isn't quite ready to integrate; see below. > I will have a look after my hospital check. > Andres? Nicolas? > Any feedback? My own feedback to my code: Last night I discovered two problems with the filein I sent: * It's a Unix full of linefeeds, so creates lousy-looking (though fast :-) code when filed in. * There's a typo that leaves an identityHash method in Object's method dictionary, where it does not belong. Andres sent some improvements to my code to the list Sun, 25 Oct 2009 03:29:31 -0700. The major changes to my code are: * His list of 'good primes' is better than mine. I am heartily in favor of this change. * His update procedure is much more careful to not upset a running system. The new code changes hashing, and so requires rehashing all of the existing hashed collections in the system. My code depends on nothing much else going on in the system during a critical part of the upgrade; Andres' updating code is more complex, but is intended to be safe (and as far as I can tell is safe) to be stopped at any point in the upgrade and the system will continue to run. Which approach you want to use depends on how you will package the change for users, I suppose. * I increased (by bitShift: 18) the range of values answered by #identityHash, and introduced a new #primIdentityHash method that answers the old limited-range values. Andres took an alternative approach; he left #identityHash alone and introduced a new #scaledIdentityHash with the increased range. On this last change, I'd like some feedback on which way the community would like to go. Actually changing #identityHash, as I did, seems to me to be the cleaner answer going forward. Leaving #identityHash alone, as Andres did, has less chance of breaking existing non-core code. Andres and I discussed this issue last night. I'm still in favor of changing #identityHash, though I'm certainly not 100% sure that's the best way to go. I believe that Andres is still in favor of the other approach. I hope he'll contribute to this discussion. > Martin I imagine that I can package the changes :) Great, thanks! Once we figure out which approaches to take, either Andres or I will get you some final code to package. Regards, -Martin

Oct. 29, 2009

Re: [Pharo-project] Linux FFI question: library given function name?

by Schwab,Wilhelm K

Martin, Objdump is a new one; I found readelf, which with the --symbols option seems to do the job, though the names look a little more C++ decorated. That loud thump (by the time it reached your location) you just heard was me slapping my head over missing /lib. Many thanks! Bill -----Original Message----- From: pharo-project-bounces(a)lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Martin McClure Sent: Wednesday, October 28, 2009 3:12 PM To: Pharo-project(a)lists.gforge.inria.fr Subject: Re: [Pharo-project] Linux FFI question: library given function name? Schwab,Wilhelm K wrote: > That seems to work. I found a similar looking script that did not. > So far, it's sending me to a static library that I have yet to > associate with anything that FFI can load, but it's progress. I'm not an expert in this field, but I've gone fishing for similar things before. Here's what I did this time... There's a man page for syslog in sections 2 and 3. The syslog (2) man page (which is for the system call, I think not what you want) references "the C library function syslog()" which I think *is* probably what you're looking for. The fact that it says it's a C library function leads me to believe that it might be in libc, where a lot of this really fundamental stuff hides. So I look in /lib to see what libc*.so files there are, find /lib/libc-2.9.so, and try: objdump -T /lib/libc-2.9.so | grep syslog which prints this: 00000000000ccff0 g DF .text 0000000000000098 GLIBC_2.2.5 syslog 00000000000ccf60 g DF .text 000000000000008b GLIBC_2.4 __syslog_chk 00000000000ccf50 g DF .text 0000000000000010 GLIBC_2.2.5 vsyslog 00000000000cc960 g DF .text 00000000000005e2 GLIBC_2.4 __vsyslog_chk I suspect this is what you're looking for. Using nm on .so files reports that there are no symbols, so it's probably better to use objdump. Regards, -Martin _______________________________________________ Pharo-project mailing list Pharo-project(a)lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Oct. 28, 2009

Re: [Pharo-project] Hashed collection improvements: Some code

by Andres Valloud

I should point out that the changesets I sent implement the scaledIdentityHash approach I just wrote about in the previous email. The mutation methodology is more robust as well (note it can also be used to implement the primIdentityHash approach). Finally, the prime table in the changesets I sent is better. Martin and I need to iron out a small detail to see if we can get rid of some code. Andres. StÃ©phane Ducasse wrote: > Martin > > are these changes related to the graphs you sent? > I will have a look after my hospital check. > Andres? Nicolas? > Any feedback? > Martin I imagine that I can package the changes :) > > > Stef > > On Oct 24, 2009, at 4:01 AM, Martin McClure wrote: > > >> OK, here's a filein that improves Pharo hashed collection >> performance quite a bit. Large collections are much faster, and >> small ones are pretty much the same speed as before. There are >> basically two fairly simple changes; the basic structure and >> algorithms of the collections is unchanged. The changes: >> >> 1. Spread identity hash values. >> 2. Make table sizes prime. >> >> >> File it into PharoCore-1.0-10491rc1.image. It'll take a minute or >> two since it has to rehash the world halfway through. I don't know >> how to make another kind of packaging that can do that, so I'll >> leave that to someone else. >> >> After the filein, there are some test failures, most of which do not >> seem to be *directly* related. I'm hoping someone that knows the >> affected tests can take a look and comment: >> >> >> Unexpectedly pass ObjectFinalizerTests>>#testFinalizationOfEquals >> Not clear why, but this does not seem to be a problem :-) >> >> Fails HostWindowTests>>#testOne >> But this test fails in the core image on Linux; HostWindows do not >> seem to be implemented for Linux. >> >> Error on FontTest>>#testMultistringFont >> Japanese StrikeFonts have nil characterToGlyphMap, >> #createCharacterToGlyphMap answers nil, >> not immediately clear how this is supposed to be initialized for >> fonts >> with codepoints > 255. >> >> PackageInfoTest>>testKernelPackage >> because some method in kernel package for Object is not in Object. >> I've changed #hash, so that's probably it. >> >> Regards, >> >> -Martin >> 'From Pharo1.0rc1 of 19 October 2009 [Latest update: #10491] on 23 >> October 2009 at 5:54:55 pm'! >> >> !ProtoObject methodsFor: 'comparing' stamp: 'MartinMcClure >> 10/22/2009 22:30'! >> identityHashTEMP >> "Answer a SmallInteger whose value is related to the receiver's >> identity. >> This method must not be overridden, except by SmallInteger. >> Primitive. Fails if the receiver is a SmallInteger. Essential. >> See Object documentation whatIsAPrimitive. >> >> Do not override." >> >> ^self primIdentityHash bitShift: 18! ! >> >> >> !Object methodsFor: 'comparing' stamp: 'MartinMcClure 10/22/2009 >> 22:46'! >> hashTEMP >> "Answer a SmallInteger whose value is related to the receiver's >> identity. >> May be overridden, and should be overridden in any classes that >> define = " >> >> ^ self primIdentityHash bitShift: 18! ! >> >> >> !ProtoObject methodsFor: 'comparing' stamp: 'MartinMcClure >> 10/22/2009 21:13'! >> primIdentityHash >> "Answer a SmallInteger whose value is related to the receiver's >> identity. >> This method must not be overridden, except by SmallInteger. >> Primitive. Fails if the receiver is a SmallInteger. Essential. >> See Object documentation whatIsAPrimitive. >> >> Do not override." >> >> <primitive: 75> >> self primitiveFailed! ! >> >> >> !SmallInteger methodsFor: 'comparing' stamp: 'MartinMcClure >> 10/23/2009 17:53'! >> primIdentityHash >> "Senders of primIdentityHash do it because they expect to get an >> answer from 1-4095. >> So they should not send this to SmallIntegers, but should use >> #identityHash" >> >> "^self shouldNotImplement" >> >> "...OK, only here for the sake of FixedIdentitySet, which may not >> need it since it is probably not used for Integers. >> And FixedIdentitySet itself may not be needed now that IdentitySets >> are faster." >> >> ^self! ! >> >> >> !IdentityDictionary methodsFor: 'private' stamp: 'MartinMcClure >> 10/23/2009 14:55'! >> scanFor: anObject >> "Scan the key array for the first slot containing either a nil >> (indicating an empty slot) or an element that matches anObject. >> Answer the index of that slot or zero if no slot is found. This >> method will be overridden in various subclasses that have different >> interpretations for matching elements." >> | finish hash start element | >> finish := array size. >> start := (anObject identityHash \\ finish) + 1. >> >> "Search from (hash mod size) to the end." >> start to: finish do: >> [:index | ((element := array at: index) == nil or: [element key == >> anObject]) >> ifTrue: [^ index ]]. >> >> "Search from 1 to where we started." >> 1 to: start-1 do: >> [:index | ((element := array at: index) == nil or: [element key == >> anObject]) >> ifTrue: [^ index ]]. >> >> ^ 0 "No match AND no empty slot"! ! >> >> >> !IdentitySet methodsFor: 'private' stamp: 'MartinMcClure 10/23/2009 >> 14:57'! >> scanFor: anObject >> "Scan the key array for the first slot containing either a nil >> (indicating an empty slot) or an element that matches anObject. >> Answer the index of that slot or zero if no slot is found. This >> method will be overridden in various subclasses that have different >> interpretations for matching elements." >> | finish hash start element | >> finish := array size. >> start := (anObject identityHash \\ finish) + 1. >> >> "Search from (hash mod size) to the end." >> start to: finish do: >> [:index | ((element := array at: index) == nil or: [element == >> anObject]) >> ifTrue: [^ index ]]. >> >> "Search from 1 to where we started." >> 1 to: start-1 do: >> [:index | ((element := array at: index) == nil or: [element == >> anObject]) >> ifTrue: [^ index ]]. >> >> ^ 0 "No match AND no empty slot"! ! >> >> >> !MethodDictionary methodsFor: 'private' stamp: 'MartinMcClure >> 10/22/2009 21:24'! >> scanFor: anObject >> "Scan the key array for the first slot containing either a nil >> (indicating an empty slot) or an element that matches anObject. >> Answer the index of that slot or zero if no slot is found. This >> method will be overridden in various subclasses that have different >> interpretations for matching elements." >> | element start finish | >> finish := array size. >> start := (anObject primIdentityHash \\ finish) + 1. >> >> "Search from (hash mod size) to the end." >> start to: finish do: >> [:index | ((element := self basicAt: index) == nil or: [element == >> anObject]) >> ifTrue: [^ index ]]. >> >> "Search from 1 to where we started." >> 1 to: start-1 do: >> [:index | ((element := self basicAt: index) == nil or: [element == >> anObject]) >> ifTrue: [^ index ]]. >> >> ^ 0 "No match AND no empty slot"! ! >> >> >> !WeakIdentityKeyDictionary methodsFor: 'private' stamp: >> 'MartinMcClure 10/23/2009 14:54'! >> scanFor: anObject >> "ar 10/21/2000: The method has been copied to this location to >> indicate that whenever #scanFor: changes #scanForNil: must be >> changed in the receiver as well." >> "Scan the key array for the first slot containing either a nil >> (indicating an empty slot) or an element that matches anObject. >> Answer the index of that slot or zero if no slot is found. This >> method will be overridden in various subclasses that have different >> interpretations for matching elements." >> | element start finish hash | >> finish := array size. >> start := (anObject identityHash \\ finish) + 1. >> >> "Search from (hash mod size) to the end." >> start to: finish do: >> [:index | ((element := array at: index) == nil or: [element key == >> anObject]) >> ifTrue: [^ index ]]. >> >> "Search from 1 to where we started." >> 1 to: start-1 do: >> [:index | ((element := array at: index) == nil or: [element key == >> anObject]) >> ifTrue: [^ index ]]. >> >> ^ 0 "No match AND no empty slot"! ! >> >> !WeakIdentityKeyDictionary methodsFor: 'private' stamp: >> 'MartinMcClure 10/23/2009 14:54'! >> scanForNil: anObject >> "Private. Scan the key array for the first slot containing nil >> (indicating an empty slot). Answer the index of that slot." >> | start finish hash | >> finish := array size. >> start := (anObject identityHash \\ array size) + 1. >> >> "Search from (hash mod size) to the end." >> start to: finish do: >> [:index | (array at: index) == nil ifTrue: [^ index ]]. >> >> "Search from 1 to where we started." >> 1 to: start-1 do: >> [:index | (array at: index) == nil ifTrue: [^ index ]]. >> >> ^ 0 "No match AND no empty slot"! ! >> >> >> >> >> >> >> >> "---------------------Do surgery and rehash before >> continuing--------------------------"! >> >> | dict method | >> dict := ProtoObject methodDictionary. >> method := dict at: #identityHashTEMP. >> dict at: #identityHash put: method. >> dict := Object methodDictionary. >> method := dict at: #hashTEMP. >> dict at: #identityHash put: method. >> >> Set rehashAllSets.! >> >> "---------- Life should be... better now :-) >> -----------------------------"! >> >> >> >> !ProtoObject methodsFor: 'comparing' stamp: 'MartinMcClure >> 10/22/2009 22:30'! >> identityHash >> "Answer a SmallInteger whose value is related to the receiver's >> identity. >> This method must not be overridden, except by SmallInteger. >> Primitive. Fails if the receiver is a SmallInteger. Essential. >> See Object documentation whatIsAPrimitive. >> >> Do not override." >> >> ^self primIdentityHash bitShift: 18! ! >> >> >> !Object methodsFor: 'comparing' stamp: 'MartinMcClure 10/22/2009 >> 22:46'! >> hash >> "Answer a SmallInteger whose value is related to the receiver's >> identity. >> May be overridden, and should be overridden in any classes that >> define = " >> >> ^ self primIdentityHash bitShift: 18! ! >> >> >> !FixedIdentitySet methodsFor: 'private' stamp: 'MartinMcClure >> 10/23/2009 14:49'! >> indexOf: anObject >> anObject isNil ifTrue: [self error: 'This class collection cannot >> handle nil as an element']. >> ^ (anObject primIdentityHash bitAnd: self basicSize - 1) + 1! ! >> >> >> !Set class methodsFor: 'sizing' stamp: 'MartinMcClure 10/23/2009 >> 09:44'! >> goodPrimes >> "Answer a sorted array of prime numbers less than one hundred million >> that make good hash table sizes. Should be expanded to more numbers >> if folks >> want to make larger collections. >> Need to check with Andres' book when I get back to work to see if I >> remembered >> it right :-) >> >> Generated with this code: >> >> | prevPrime primes goodPrimes | >> goodPrimes := OrderedCollection new. >> primes := Integer largePrimesUpTo: 100000000. >> goodPrimes add: 5. >> prevPrime := 5. >> primes do: >> [:prime | prime > (prevPrime * 4 // 3) ifTrue: >> [| lowByte | lowByte := prime bitAnd: 16rFF. >> (lowByte > 10 and: [lowByte < 245]) ifTrue: >> [goodPrimes add: prime. >> prevPrime := prime]]]. >> ^goodPrimes asArray printString" >> >> ^#(5 11 17 23 31 43 59 79 107 149 199 269 359 479 641 857 1151 1549 >> 2069 2767 3691 4931 6577 8779 11717 15629 20849 27799 37087 49451 >> 65951 87943 117259 156347 208463 277961 370619 494167 658897 878539 >> 1171393 1561883 2082527 2776727 3702313 4936423 6581909 8775947 >> 11701267 15601723 20802317 27736427 36981911 49309219 65745677 >> 87660917)! >> ]style[(10 311 405 337)f2b,f2,f1,f2! ! >> >> !Set class methodsFor: 'sizing' stamp: 'MartinMcClure 10/23/2009 >> 10:14'! >> goodPrimeAtLeast: lowerLimit >> "Answer the next good prime >= lowerlimit. >> If lowerLimit is larger than the largest known good prime, >> just make it odd." >> >> | primes low mid high prime | >> primes := self goodPrimes. >> low := 1. >> high := primes size. >> lowerLimit > (primes at: high) ifTrue: >> [^lowerLimit even >> ifTrue: [lowerLimit + 1] >> ifFalse: [lowerLimit]]. >> [mid := high - low // 2 + low. >> prime := primes at: mid. >> prime < lowerLimit >> ifTrue: [low := mid] >> ifFalse: [high := mid]. >> high - low <= 1 ifTrue: >> [^primes at: high]. >> prime == lowerLimit ifTrue: >> [^prime]] repeat >> >> ! >> ]style[(28 158 411)f2b,f2,f1! ! >> >> !Set methodsFor: 'private' stamp: 'MartinMcClure 10/23/2009 10:25'! >> growSize >> "Answer what my next higher table size should be" >> ^ self class goodPrimeAtLeast: array size * 2! ! >> >> >> !Set methodsFor: 'private' stamp: 'MartinMcClure 10/23/2009 10:25'! >> grow >> "Grow the elements array and reinsert the old elements" >> | oldElements | >> oldElements := array. >> array := Array new: self growSize. >> tally := 0. >> oldElements do: >> [:each | each == nil ifFalse: [self noCheckAdd: each]]! ! >> >> >> !Set class methodsFor: 'instance creation' stamp: 'MartinMcClure >> 10/23/2009 10:19'! >> sizeFor: nElements >> "Large enough size to hold nElements with some slop (see fullCheck)" >> nElements <= 0 ifTrue: [^ 5]. >> ^ self goodPrimeAtLeast: (nElements+1*4//3)! ! >> >> >> >> !WeakSet methodsFor: 'private' stamp: 'MartinMcClure 10/23/2009 >> 10:26'! >> grow >> "Grow the elements array if needed. >> Since WeakSets just nil their slots, alot of the occupied (in the >> eyes of the set) slots are usually empty. Doubling size if unneeded >> can lead to BAD performance, therefore we see if reassigning the >> <live> elements to a Set of similiar size leads to a sufficiently >> (50% used here) empty set first. >> and reinsert the old elements" >> |oldTally| >> oldTally := tally. >> self growTo: array size. >> oldTally >> 1 < tally ifTrue: [ >> self growTo: self growSize]! ! >> >> >> !MethodPragmaTest methodsFor: 'testing-primitives' stamp: >> 'MartinMcClure 10/23/2009 12:37'! >> testPrimitiveIndexed2 >> "This test useses the #identityHash primitive." >> >> self compile: '<primitive: 75> ^ #idHash' selector: #idHash. >> self assert: self idHash = self primIdentityHash.! ! >> >> >> !SmallInteger reorganize! >> ('arithmetic' * + - / // \\ gcd: quo:) >> ('bit manipulation' bitAnd: bitOr: bitShift: bitXor: hashMultiply >> highBit highBitOfMagnitude lowBit) >> ('comparing' < <= = > >= hash identityHash primIdentityHash ~=) >> ('converting' as31BitSmallInt asFloat) >> ('copying' clone deepCopy shallowCopy veryDeepCopyWith:) >> ('printing' decimalDigitLength destinationBuffer: >> numberOfDigitsInBase: printOn:base: printOn:base:nDigits: >> printString printStringBase: printStringBase:nDigits: threeDigitName) >> ('system primitives' asOop digitAt: digitAt:put: digitLength >> instVarAt: nextInstance nextObject) >> ('testing' even isLarge odd) >> ('private' fromString:radix: highBitOfPositiveReceiver) >> ! >> >> >> !Set class reorganize! >> ('initialization' quickRehashAllSets rehashAllSets) >> ('sizing' goodPrimeAtLeast: goodPrimes) >> ('instance creation' new new: newFrom: sizeFor:) >> ! >> >> ProtoObject removeSelector: #identityHashTEMP! >> Object removeSelector: #hashTEMP! >> >> !ProtoObject reorganize! >> ('apply primitives' tryNamedPrimitive tryNamedPrimitive: >> tryNamedPrimitive:with: tryNamedPrimitive:with:with: >> tryNamedPrimitive:with:with:with: >> tryNamedPrimitive:with:with:with:with: >> tryNamedPrimitive:with:with:with:with:with: >> tryNamedPrimitive:with:with:with:with:with:with: >> tryNamedPrimitive:with:with:with:with:with:with:with: >> tryPrimitive:withArgs:) >> ('closure-prims' privGetInstVar: privRemoteReturnTo: >> privSetInHolder: privSetInstVar:put: privStoreIn:instVar:) >> ('comparing' == identityHash primIdentityHash ~~) >> ('debugging' doOnlyOnce: flag: rearmOneShot withArgs:executeMethod:) >> ('initialize-release' initialize) >> ('method execution' executeMethod: with:executeMethod: >> with:with:executeMethod: with:with:with:executeMethod: >> with:with:with:with:executeMethod:) >> ('objects from disk' rehash) >> ('system primitives' become: cannotInterpret: doesNotUnderstand: >> nextInstance nextObject) >> ('testing' ifNil: ifNil:ifNotNil: ifNotNil: ifNotNil:ifNil: >> isInMemory isNil pointsTo:) >> ! >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > . > >

Oct. 28, 2009

Re: [Pharo-project] Hashed collection changes, the performance graphs

by Andres Valloud

Stephane et al, Martin and I discussed this yesterday night at the PDX Smalltalk user group meeting. Mostly we agree, with perhaps one small exception. In order to make hashed collections simpler and identity hash handling more straightforward, hashed collections should get a small integer range identity hash value. Something between 0 and SmallInteger maxVal, for example. The issue is how to get that. Martin's approach is to change identityHash itself, and provide primIdentityHash to access the existing identityHash implementation (primitive 75). My approach is to leave identityHash as is, and provide a new message called scaledIdentityHash. Note that in both Martin's and my implementation, the new identity hash values are identical. In other words, whereas Martin's changes implement identityHash ^self primIdentityHash bitShift: 18 my changes implement scaledIdentityHash ^self identityHash bitShift: 18 So, really, this is an issue of how to implement the *same* behavior, with the *same* goals. Now, there are advantages and disadvantages for both. I'll try to explain. Martin, please feel free to correct what follows if I miss something. There are two kinds of senders of identityHash. Some (most) do not care whether the answer is well distributed or not. Typically, these are implementors of #hash, for example: SomeObject>>hash ^self someInstanceVariable identityHash If, on one hand, we change identityHash, then the behavior of these hash methods improve because the values they answer would be distributed more uniformly. Nevertheless, precisely because they send identityHash, it could be argued that these hash methods were implemented assuming there would be just a few of these objects in hashed collections. If this turns out not to be the case, then performance could be much better by implementing a better hash method. Some (a few) other senders of identityHash assume that the answer has a particular distribution (in Squeak's/Pharo's case, 0 through 4095). Consequently, the senders may do things like this: SomeObject>>hash ^(self variableA identityHash bitShift: 12) + self variableB identityHash Also, there may be hashed collections which use identityHash to index hash buckets (e.g.: GemStone's own 32 bit object cache for VisualWorks, which uses identityHash to access hash buckets). So, if we change identityHash, these senders which required special care to write would break, requiring them to be written like this: SomeObject>>hash ^(self variableA primIdentityHash bitShift: 12) + self variableB primIdentityHash If, on the other hand, we provide a new scaledIdentityHash method, then all senders of identityHash keep behaving the same way as they do now. Hashed collections, which currently send identityHash, would need to change so that they send scaledIdentityHash instead. Furthermore, any implementor of hash such as SomeObject>>hash ^self someInstanceVariable identityHash would need to be rewritten as SomeObject>>hash ^self someInstanceVariable scaledIdentityHash if hashed collections are expected to contain numerous instances of SomeObject. If these are not rewritten, then they would continue to have the same performance characteristics they have today (which may be adequate to begin with). Clever hacks such as SomeObject>>hash ^(self variableA identityHash bitShift: 12) + self variableB identityHash would also remain undisturbed. Finally, I do not know of any Smalltalk in which identityHash does not answer the actual object header bits for the identity hash. If we change identityHash, then AFAIK Pharo would become the only Smalltalk in which identityHash does not answer consecutive values between 0 and (2^k)-1 (k=12 for Squeak/Pharo, k=14 for VisualWorks 32 bits, k=20 for VisualWorks 64 bits, IIRC k=15 for VA and VisualSmalltalk). Moreover, AFAIK, identityHash has had this behavior for decades. Is it a good idea to change it? I am not so sure on that one. Also, I wouldn't want to break clever hacks that assume identityHash behaves the way it currently does, and I wouldn't want to make Pharo different gratuitously. So that's where we stand. I think I understand where Martin is coming from, and I think Martin understands where I am coming from. One way or the other, we also agree that something should be done to make hashed collections faster in Pharo. Andres. StÃ©phane Ducasse wrote: > Martin > > so what you are saying is that we can gain speed. And that these > changes are beneficial for simple default. > > > >> The test code, a variant of the test from the HashTable SqueakSource >> wiki, is at the bottom of this message. Basically, it adds a bunch >> of instances of Object to a Dictionary and sees how much time it >> takes to look them up again. >> >> From the graphs in the previous message, you can see that >> performance for sizes > 4000 is greatly improved. For size = 10000, >> #at: is 1000 times faster, 2-3 microseconds vs. >2 milliseconds. At >> size=10000, #at:put: is about 200 times faster, ~10 microseconds vs. >> >>> 2 milliseconds, and the large spikes for growing the collection are >>> >> 21 milliseconds vs. > 4 seconds, again a factor of about 200. >> >> Performance for dictionary sizes < 4000 is essentially the same as >> before, so these collections can serve as general-purpose >> collections over a wide range of sizes. I've attached the graphs for >> sizes <4000 to this message so you can see that more clearly than on >> the previous graphs. >> >> These results should hold for any object that inherits #hash from >> Object, in other words uses its identity hash as its equality hash. >> Other objects with better hashes did not have as serious a problem, >> but will probably show increased performance as well due to using >> prime table sizes. >> >> These changes are in Set, so should improve Set's subclasses as >> well. IdentityDictionary, IdentitySet, and WeakIdentityKeyDictionary >> did not have as serious a problem, but should see some improvement. >> MethodDictionaries have been left alone on the assumption that the >> VM depends on the hashing of those. >> > > So where are these changes :) > Should we integrate them? > Andres and other hash freaks what are your point of view? > > >> Since there are still only 4K possible values of identity hash, >> collisions are inevitable in large collections, and the number of >> collisions will grow linearly with collection size. So how well does >> the spread hash / prime table size do at even larger sizes? I ran >> the same test at sizes of one million. As expected, access was quite >> a bit slower than it had been at 10000. Time for #at: was ~250 >> microseconds, and #at:put: was about the same. Note, however, that >> this is still ten times faster than Dictionary previously was at a >> size of 10000; 100 times larger yet 10 times faster. >> >> I had a lot of fun doing this. This is better results than I >> expected, for a fairly minor (though deep) code change. >> > > So where are these changes :) > Should we integrate them? > Andres and other hash freaks what are your point of view? > > > >> | test ord | >> Transcript cr. >> test := Dictionary new. >> [ test size >= 10000] whileFalse: [ >> ord := OrderedCollection new: 100. >> Transcript show: >> [ >> 100 timesRepeat: [ >> test at: ( ord add: Object new ) put: nil ]. >> ] timeToRun asString. >> Transcript tab; >> show: test size asString; >> tab. >> Transcript show: >> [ >> 1000 timesRepeat: [ >> ord do: [ :each | test at: each ] ] >> ] timeToRun asString. >> Transcript tab; show: >> [ >> 1000 timesRepeat: [ >> ord do: [ :each | ] ] >> ] timeToRun asString. >> Transcript cr ] >> >> < >> SmallDictPerformanceGraphs1 >> .png>_______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > . > >

Oct. 28, 2009

Re: [Pharo-project] Shared directories: The bug is quicker than the eye

by David T. Lewis

On any platform that supports the C library fflush(), there is numeric error number ("errno") that is set if the fflush() call does not succeed (type "man fflush" on your Linux or OS X console for details). John pointed out that the #flush method does not provide you with the errno value in the event of a failure, so you cannot directly check the errno value to see if there was a problem. I was showing a way that this could be done using some OSProcess calls rather than the normal #flush. My assumption was that you have some kind of file stream in your code, and that you would do the OSProcess thing instead of #flush just for purposes of debugging the problem. My personal hunch is that you are not likely to see any failure from the #flush calls, even if you are doing the #flush and the data seems to be flushed down the bit bucket. The reason is that the result of the #flush will appear successful as long as the C library fflush() has handed the data over to the operating system. On networked file systems there is a long way to go between the time the data leaves your program (the VM doing fflush()) and the time that something actually lands on the spinning platter. So you can try checking the result of the #flush calls to be sure that they are not failing, but I'm guessing that you won't see any failures. HTH, Dave On Wed, Oct 28, 2009 at 03:10:00PM -0400, Schwab,Wilhelm K wrote: > Dave, > > I'm not a fully trained penguin herder yet, so I have to ask a dumb question: is the idea to connect to the same file that went wrong and get an error condition after the fact, or are you hoping that opening and flushing will cause an error by itself? I suspect the problem is very real, and more "random" than that could hope to uncover. > > Bill > > > -----Original Message----- > From: pharo-project-bounces(a)lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of David T. Lewis > Sent: Sunday, October 25, 2009 3:27 PM > To: Pharo-project(a)lists.gforge.inria.fr; johnmci(a)smalltalkconsulting.com > Subject: Re: [Pharo-project] Shared directories: The bug is quicker than the eye > > I don't know if this helps, but for purposes of debugging here is a way to get at the error code from a flush(). No doubt I wrote the primitive by accident, not noticing that there was already a primitiveFileFlush in the FilePlugin. > > aStream := FileStream readOnlyFileNamed: 'foo.tmp'. > result := OSProcess accessor flushExternalStream: aStream fileID. > errorString := OSProcess accessor primErrorMessageAt: result. > > Dave > > > On Fri, Oct 23, 2009 at 11:57:06AM -0700, John M McIntosh wrote: > > If you invoke the flush primitive then it does this on unix/linux/mac- > > osx/iPhone > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ > > ManPages/man3/fflush.3.html > > & > > http://developer.apple.com/mac/library/documentation/Darwin/Reference/ > > ManPages/man2/write.2.html#//apple_ref/doc/man/2/write > > > > sqInt sqFileFlush(SQFile *f) { > > /* Return the length of the given file. */ > > > > if (!sqFileValid(f)) return interpreterProxy->success(false); > > fflush(getFile(f)); > > return 1; > > } > > > > On windows it does > > http://msdn.microsoft.com/en-us/library/aa364439(VS.85).aspx > > > > sqInt sqFileFlush(SQFile *f) { > > if (!sqFileValid(f)) FAIL(); > > /* note: ignores the return value in case of read-only access */ > > FlushFileBuffers(FILE_HANDLE(f)); > > return 1; > > } > > > > I'll note the api doesn't actually check for errors, give feedback or > > whatever. > > *cough* it could be failing and give you a clue, but you would need to > > resort to FFI to do the file system calls to get the data to > > visualize, or build your own VM with that returns the result. > > > > > > > > On 2009-10-22, at 5:31 PM, Schwab,Wilhelm K wrote: > > > > > Hello all, > > > > > > I'm not sure what to make of this one. I just spent a couple of > > > hours trying to find the "leak" in an algorithm of mine. It was > > > reading roughly 1200 records, claiming to have processed all of > > > them, and yet writing only about 450 rows into an output text file. > > > One clue should have been that the number of output rows was > > > somewhat random; I did not fully appreciate that until I worked > > > around the problme. > > > > > > I tried an explicit #flush - no help. I looked for logic errors and > > > found none. The file was being written to a Windows hosted share > > > mounted by CIFS (which I am learning to view with contempt) from > > > Ubuntu 9.04. You can see where this is going: writing the file > > > locally gave the expected result. > > > > > > Any ideas on how one might further isolate the problem? My trust in > > > Windows is well known<g>; I have never liked shared directories; I > > > _really_ do not like CIFS as compared (reliability wise) to SMBFS; > > > the network between me and the server is in question too (long > > > story). All of that said, file support in Squeak, and hence so far > > > inherited by Pharo, is not the best code I have seen to date, so it > > > is easy to suspect too. Can one argue that since it worked locally, > > > Pharo is not the problem? > > > > > > The little bit that I know of cifs is not encouraging. It sounds as > > > though things moved from an easily killed process into the kernel > > > which shows an almost Windows-like unwillingness to shut down when > > > it cannot see servers. I have found numerous reports of problems > > > copying large files over cifs, and I have enountered them too. > > > > > > Bill > > > > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Oct. 28, 2009

Re: [Pharo-project] Abstract classes

by csrabak＠bol.com.br

Em 28/10/2009 17:52, Lukas Renggli <renggli(a)gmail.com> escreveu: > > >> What would be your isAbstract be useful for? > > To know whether it is wise to instantiate a class or not > That would be a pure guess then. Presumably you will still have to > look at the implementation, at the documentation, at its users or > study the coding conventions of the project to know for sure what to > do with the thing. Yes. Thinking aloud: if we want to insure a class shall not be instantiated, shouldn't we put an error in #new of that class?

Oct. 28, 2009

Re: [Pharo-project] Linux FFI question: library given function name?

by Martin McClure

Schwab,Wilhelm K wrote: > That seems to work. I found a similar looking script that did not. So > far, it's sending me to a static library that I have yet to associate > with anything that FFI can load, but it's progress. I'm not an expert in this field, but I've gone fishing for similar things before. Here's what I did this time... There's a man page for syslog in sections 2 and 3. The syslog (2) man page (which is for the system call, I think not what you want) references "the C library function syslog()" which I think *is* probably what you're looking for. The fact that it says it's a C library function leads me to believe that it might be in libc, where a lot of this really fundamental stuff hides. So I look in /lib to see what libc*.so files there are, find /lib/libc-2.9.so, and try: objdump -T /lib/libc-2.9.so | grep syslog which prints this: 00000000000ccff0 g DF .text 0000000000000098 GLIBC_2.2.5 syslog 00000000000ccf60 g DF .text 000000000000008b GLIBC_2.4 __syslog_chk 00000000000ccf50 g DF .text 0000000000000010 GLIBC_2.2.5 vsyslog 00000000000cc960 g DF .text 00000000000005e2 GLIBC_2.4 __vsyslog_chk I suspect this is what you're looking for. Using nm on .so files reports that there are no symbols, so it's probably better to use objdump. Regards, -Martin

Oct. 28, 2009

Re: [Pharo-project] Making packages shipped with Pharo ready for 1.0

by Adrian Lienhard

Thanks, Nicolas We integrated your fix and now it does not fail anymore. Just have to remove it from expectedFailures. Cheers Adrian On Oct 27, 2009, at 17:58 , Nicolas Cellier wrote: > 2009/10/27 Adrian Lienhard <adi(a)netstyle.ch>: >> Pharo-core tests failing >> ------------------------ >> - FontTest>>testFallback (assumes different font, to be removed) >> - FontTest>>testParagraphFallback (assumes different font, to be >> removed) >> - FontTest>>testResetAfterEmphasized (assumes different font, to be >> removed) >> - StandardSystemFontsTest>>testRestoreDefaultFonts (assumes different >> font, to be removed) >> - TestIndenting>>testNewLineStartsIndentedWhenWrapped (assumes >> different font, to be removed) >> - TestIndenting>>testSetUp (assumes different font, to be removed) >> - LocaleTest>>testIsFontAvailable (some Japanese font checking that >> fails with DejaVu, to be removed) >> - testFinalizationOfEquals (was on my to do list, but works again: >> remove expected failure tag) > > This remind me discussions with Norbert http://bugs.squeak.org/view.php?id=6347 > and also http://lists.squeakfoundation.org/pipermail/squeak-dev/2008-September/13165… > > Nicolas > >> - testObsoleteClasses: again includes classes from MC and others >> (have >> to be checked) >> >> MorphExamplesAndDemos >> --------------------- >> - undeclared references to DescriptionForPartsBin and ScreeningMorph >> >> Cheers, >> Adrian >> >> BTW, Damien, do we know for each package who the maintainer is? Does >> each package have one? >> >> >> _______________________________________________ >> Pharo-project mailing list >> Pharo-project(a)lists.gforge.inria.fr >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > _______________________________________________ > Pharo-project mailing list > Pharo-project(a)lists.gforge.inria.fr > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Oct. 28, 2009

Re: [Pharo-project] Abstract classes

by Lukas Renggli

>> What would be your isAbstract be useful for? > > To know whether it is wise to instantiate a class or not That would be a pure guess then. Presumably you will still have to look at the implementation, at the documentation, at its users or study the coding conventions of the project to know for sure what to do with the thing. Lukas -- Lukas Renggli http://www.lukas-renggli.ch

Oct. 28, 2009