On 07 Nov 2014, at 13:16, phil@highoctane.be wrote:
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
That would mean that you keep 1000s of files open concurrently. Anyway, I just wrote this as a test: '/tmp/pharo-files-test/' asFileReference ensureCreateDirectory in: [ :base | 0 to: 99 do: [ :each | base / each asString writeStreamDo: [ :out | out print: each; crlf; << each asWords; crlf ] ] ]. 100000 timesRepeat: [ | file | file := 99 atRandom asString. '/tmp/pharo-files-test/' asFileReference / file readStreamDo: [ :in | self assert: (in upToEnd beginsWith: file) ] ] And it worked fine.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry. I guess the GC is there to trigger weak reference cleanup. On the other hand, there is always a limited on the number of open files at the OS level.
The main thing is, if you use tons of (open) files, to always close them as fast as possible, and to not let that happen automatically or lose them in the case of exceptions.
On 07 Nov 2014, at 12:42, phil@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil