On 18 Aug 2016, at 18:35, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Hi Sven!
Yes, this is lot of requests to a public database named GenBank through a public API called "Entrez". I repeated the test today to send you a log of what's happening. The requests are done in batches of 500 uid's because they limit/penalize huge requests. Each uid is human meta-data, I have requested 20,000 samples, and Argentina has one of the worst connections in the world.
I see that you do 10 request concurrently, correct ? In different threads/processes ? I tried the first request that I saw, ZnClient new url: 'http://eutils.ncbi.nlm.nih.gov:80/entrez/eutils/efetch.fcgi?retmode=xml&id=1...'; get; yourself. Which resulted in about 11Mb XML, which you parse and convert I guess ? That would be between 100 and 200 Mb, at least, no ? That already puts a heavy burden on the image/VM. Make sure to properly close and cleanup everything as soon as you can. And all this 40 times (500 x 40 = 20000) ? I think that goes way over the 2Gb limit (which is rather theoretical, it is more like 1.5). Unless you clean up very well afterwards and only keep the minimal useful/needed info in memory.
Another problem I have is the Transcript writes everything once the exception is signaled :( Is there a way to revert to the old behavior where each Transcript show: writes to Transcript in situ?
Do you block the UI thread (while running your main top level expression) ? Try forking.
The ClosedConnection didn't signaled today, but OutOfMemory.
See above.
It would be nice to know how much memory was requested on startup (this is Pharo 4 on Win 8.1), because the "restart the Pharo VM with a larger memory allocation" is useless without any numbers.
Hernán
2016-08-18 9:33 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>: Hi Hernán,
If you do lots of requests (downloads) or if you download large resources (files) and you get some networking exception, all you have to do is catch it, close the connection, maybe wait a couple of seconds and try again. There is (normally) no need for any more magic, unless you found a bug.
ZnClient has some basic support for this, see #numberOfRetries[:] #retryDelay[:] #timeout[:]
Of course, I do not know exactly what you are trying to do, what the other/server end is, what code you have written, what problems you see, what you networking quality is, and so on.
Switching the socket stream implementation is only necessary in very rare cases, it is not recommended that normal users do that.
HTH,
Sven
On 18 Aug 2016, at 06:43, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
Hi,
I have a situation where downloading lot of data results in ConnectionClosed.
httpGet: urlString "Get the response from the given url"
^ [ ... .get: urlString ] on: ZnEntityTooLarge, ConnectionClosed do: [ : ex | ex isResumable ifTrue: [ ex resume ] ifFalse: [ self handleNonResumableConnection: ex ] ].
I read a mail from Sven you could change a global setting in ZnNetworkingUtils as follows:
handleNonResumableConnection: anException
| msg | msg := 'We got a non-resumable exception while connecting to ' , self url host asString , '. Re-try switching ZdcSocketStream to SocketStream?'. (self confirm: msg) ifTrue: [ ZnNetworkingUtils default socketStreamClass: SocketStream. anException retry.]
But I would like to read about better handling, or recommendations. What do you think?
Cheers,
Hernán
<Pharo_OOM.jpg><Zn_Requests.zip>