StandardFileStream>>open:forWrite: serious isse...
Code (Transcript calls of mine) open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f. fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering I have a ton of files to open. And some were slow to return a problem. After digging, I got to the method above. Now at one moment, I get nil fileIDs. What is that retryWithGC: until: thing??? retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value. Huhhhhh, yeah, some lookup going on in a registry... and GCs? For file opens? I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...) I have tested with 100 files and of course it all works since I get the IDs back from the registry. But at one point, it all fails. Registry full? What to do here? TIA Phil
Yes, weird code ;-) The registry seems unlimited, it is just a WeakRegistry. I guess the GC is there to trigger weak reference cleanup. On the other hand, there is always a limited on the number of open files at the OS level. The main thing is, if you use tons of (open) files, to always close them as fast as possible, and to not let that happen automatically or lose them in the case of exceptions.
On 07 Nov 2014, at 12:42, phil@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil
Well, I am not keeping things open... (some local vars to get some easier stepping) measurementForTarget: aTargetId onSection: aSection at: aRegistryKey | item fileRef | item := 'ACQ_', aTargetId, '-', aSection, '_', aRegistryKey,'.txt'. fileRef := self monitoringDirectory / item. ^ [ fileRef readStreamDo: [ :s | s next: Config maxFileCharsToRead ] ] on: Error do: [ :ex | self warn: ('Cannot measure for target {1} on section {2} and registry key {3}' format: { aTargetId asString. aSection asString. aRegistryKey asString }). ex return: '' ] It works for a couple files but at one point, it just doesn't open things. Even if files are closed. When reusing the same files over and over, it works (like readStreamDo: for 10000 times, I get the contents back). I've simplified the call to retryWithGC: ... to avoid the registry completely: retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. ^blockValue But the symptom is the same. Time to go into the filePlugin? Phil On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry. I guess the GC is there to trigger weak reference cleanup. On the other hand, there is always a limited on the number of open files at the OS level.
The main thing is, if you use tons of (open) files, to always close them as fast as possible, and to not let that happen automatically or lose them in the case of exceptions.
On 07 Nov 2014, at 12:42, phil@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil
I've now set ulimit -n 3000 and things are working. Sorry for the noise, but this GC thing is still weird. Phil On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry. I guess the GC is there to trigger weak reference cleanup. On the other hand, there is always a limited on the number of open files at the OS level.
The main thing is, if you use tons of (open) files, to always close them as fast as possible, and to not let that happen automatically or lose them in the case of exceptions.
On 07 Nov 2014, at 12:42, phil@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil
On 07 Nov 2014, at 13:16, phil@highoctane.be wrote:
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
That would mean that you keep 1000s of files open concurrently. Anyway, I just wrote this as a test: '/tmp/pharo-files-test/' asFileReference ensureCreateDirectory in: [ :base | 0 to: 99 do: [ :each | base / each asString writeStreamDo: [ :out | out print: each; crlf; << each asWords; crlf ] ] ]. 100000 timesRepeat: [ | file | file := 99 atRandom asString. '/tmp/pharo-files-test/' asFileReference / file readStreamDo: [ :in | self assert: (in upToEnd beginsWith: file) ] ] And it worked fine.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry. I guess the GC is there to trigger weak reference cleanup. On the other hand, there is always a limited on the number of open files at the OS level.
The main thing is, if you use tons of (open) files, to always close them as fast as possible, and to not let that happen automatically or lose them in the case of exceptions.
On 07 Nov 2014, at 12:42, phil@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil
I did tests like that and they work ok. I am still puzzled on why this happens in my code as only do these readStreamDo: [] and nothing is in parallel. I've been putting more caching in my own code but still, there is something hiding. Phil On Fri, Nov 7, 2014 at 1:27 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 07 Nov 2014, at 13:16, phil@highoctane.be wrote:
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
That would mean that you keep 1000s of files open concurrently.
Anyway, I just wrote this as a test:
'/tmp/pharo-files-test/' asFileReference ensureCreateDirectory in: [ :base | 0 to: 99 do: [ :each | base / each asString writeStreamDo: [ :out | out print: each; crlf; << each asWords; crlf ] ] ].
100000 timesRepeat: [ | file | file := 99 atRandom asString. '/tmp/pharo-files-test/' asFileReference / file readStreamDo: [ :in | self assert: (in upToEnd beginsWith: file) ] ]
And it worked fine.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry. I guess the GC is there to trigger weak reference cleanup. On the other hand, there is always a limited on the number of open files at the OS level.
The main thing is, if you use tons of (open) files, to always close them as fast as possible, and to not let that happen automatically or lose them in the case of exceptions.
On 07 Nov 2014, at 12:42, phil@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil
On 07 Nov 2014, at 13:42, phil@highoctane.be wrote:
I did tests like that and they work ok.
I am still puzzled on why this happens in my code as only do these readStreamDo: [] and nothing is in parallel.
I've been putting more caching in my own code but still, there is something hiding.
Yes, I think so, happy hunting ;-) You could try to hunt for instances and references when the error occurs.
Phil
On Fri, Nov 7, 2014 at 1:27 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 07 Nov 2014, at 13:16, phil@highoctane.be wrote:
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
That would mean that you keep 1000s of files open concurrently.
Anyway, I just wrote this as a test:
'/tmp/pharo-files-test/' asFileReference ensureCreateDirectory in: [ :base | 0 to: 99 do: [ :each | base / each asString writeStreamDo: [ :out | out print: each; crlf; << each asWords; crlf ] ] ].
100000 timesRepeat: [ | file | file := 99 atRandom asString. '/tmp/pharo-files-test/' asFileReference / file readStreamDo: [ :in | self assert: (in upToEnd beginsWith: file) ] ]
And it worked fine.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe <sven@stfx.eu> wrote: Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry. I guess the GC is there to trigger weak reference cleanup. On the other hand, there is always a limited on the number of open files at the OS level.
The main thing is, if you use tons of (open) files, to always close them as fast as possible, and to not let that happen automatically or lose them in the case of exceptions.
On 07 Nov 2014, at 12:42, phil@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil
On 07 Nov 2014, at 12:59 , Sven Van Caekenberghe <sven@stfx.eu> wrote:
Yes, weird code ;-)
Seems to me open:forWrite: would be better off using retryWithGC:until:, which would trigger GC regardless of whether there are registered descriptors for the currently requested handle. That would (hopefully) help against the case where too many handles have been opened (but some are waiting to be finalized), not just the case where you try opening a file in write mode that's already been opened in write mode and discarded. (which is only a restriction on Windows, IIRC) retryWithGC:until:forFileNamed: seems more intended for operations where you are not allowed *any* open handles on the file, such as renames/deletes/moves, etc. Cheers, Henry
Hi Phil, On Fri, Nov 7, 2014 at 3:42 AM, phil@highoctane.be <phil@highoctane.be> wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode "Open the file with the given name. If writeMode is true, allow writing, otherwise open the file in read-only mode." | f | f := fileName asVmPathName. Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] until:[:id| id notNil] forFileNamed: fileName. fileID ifNil: [ ' > Cannot get fileID' logCr. ^ nil]. "allows sender to detect failure" (' > Got fileID: ', fileID printString) logCr. name := fileName. self register. rwmode := writeMode. buffer1 := String new: 1. self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName "Re-implemented to only force GC if a file with the given name exists" | blockValue foundIt | blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. "See if we have a file with the given name" foundIt := self registry keys "hold on strongly for now" anySatisfy:[:file| file name sameAs: fullName]. foundIt ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue := execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue]. Smalltalk garbageCollect. ^execBlock value.
I think its horrible personally but it /should/ work. The idea is that files shouldn't need to be closed explicitly; which is fine; we want to be able to rely on finalization to close them eventually. Hence retryWithGC: is called when an open fails, presumably because it failed because the process is out of file handles, presumably because several as-yet-unfinalized files have yet to be finalized. So running the GC /should/ run finalization. One issue could be that there are bugs meaning that finalization does /not/ happen. For example, Igor could take a look and find out quickly whether that's the case. Another issue could be that the finalization loop is at too low a priority, or has been terminated for some reason. But finalization can not be relied upon to be timely. A better solution is to explicity close files where possible using a form such as forceNewFileNamed:do: et al. Another solution is possible in Spur where we can simplify the finalization by using ephemerons so there are no proxies for files. But that's not relevant immediately. Huhhhhh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is there or not and fail fast, not have all of this thing (debugging kind of nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA Phil
-- best, Eliot
participants (4)
-
Eliot Miranda -
Henrik Johansen -
phil＠highoctane.be -
Sven Van Caekenberghe