> On 18 Aug 2016, at 18:35, Hern��n Morales Durand <hernan.morales@gmail.com> wrote:
>
> Hi Sven!
>
> Yes, this is lot of requests to a public database named GenBank through a public API called "Entrez". I repeated the test today to send you a log of what's happening. The requests are done in batches of 500 uid's because they limit/penalize huge requests. Each uid is human meta-data, I have requested 20,000 samples, and Argentina has one of the worst connections in the world.
I see that you do 10 request concurrently, correct ? In different threads/processes ?
I tried the first request that I saw,
ZnClient new url: 'http://eutils.ncbi.nlm.nih.gov:80/entrez/eutils/efetch. '; get; yourself.fcgi?retmode=xml&id=188847, 188840,188835,188827,188833, 188831,5919044,5919041,188829, 188824,188816,188841,188828, 188822,188809,188838,188836, 5919066,5919018,188826,188807, 5919042,188812,5919052, 5919028,188832,188830,188820, 188823,188819,5919056,188784, 5919051,5919010,188814, 5919036,5919011,188815,188794, 5919043,5919009,5919004, 188802,5919040,5919003, 5919006,5919001,5919037, 5919020,188844,188782,188779, 5919067,5919045,5918994, 188783,5919062,5919019, 5918993,188743,188734,5919013, 5919000,5918987,188817, 5919046,5919007,5918996, 5919032,5919029,5919027, 5919021,188719,188795,188785, 188788,188780,5919026,5919024, 5919023,5919022,5919017, 5919002,5919064,5918997, 5919059,5919048,5919005, 188781,5919055,5919047, 5919033,5919030,5919025, 5918992,188739,5919016,188732, 188731,5919035,5918991, 5918990,188834,188774,188763, 5919031,5919012,5918989, 188825,188745,5919015,5918995, 188715,188775,188730,188718, 5919049,188821,188803,5919065, 5919050,5919034,188714, 5918988,188720,5919063, 5919054,5919014,5919008, 188797,5918999,5918998,188717, 5919057,188750,188723,188758, 188769,188752,188751,188724, 188760,188754,5919060,188759, 188757,188756,5919061,188771, 188736,5919038,188761,5919053, 188735,188786,5919058,188818, 188753,188733,188845,188755, 188837,188777,188725,188846, 188744,1002820005,999844743, 941582049,941581974,941581869, 941581732,941581138,941580894, 941580699,850201272,653603877, 653603608,653603509,653603323, 653603248,653602787,827529357, 817037023,807985587,807985574, 807985096,806641362,735659021, 735658976,735658957,735658862, 735658829,735658814,735658794, 735658792,735658766,735658757, 735658751,695118655,695118621, 695118612,695118403,695118088, 695118067,700283821,700282771, 700281917,575524695,550825751, 556709823,556709753,667060998, 568246384,610260256,610260144, 430766974,397187538,397181812, 570436518,374089790,374089754, 374089025,374089020,374088999, 374088998,374088995,549444460, 410062763,472836119,408775754, 408775351,536189361,513791441, 513791357,513791301,513791217, 513790965,513790629,513790321, 513790223,513788781,461681792, 378744435,401666731,401666186, 302377629,302377601,392496866, 392496829,392496762,392496587, 392496297,392496261,392496213, 392496202,392496104,392495779, 392495588,381261925,381261506, 381253498,326200539,339745356, 339744796,339744703,339744578, 339744577,305656401,305656233, 305656149,305656121,305655967, 305655953,305655925,334189684, 320339399,329741913,290758881, 305387591,318039611,289600384, 288806465,288806339,288805758, 224474796,224474781,209916429, 223015240,223015212,223015170, 223015156,223012454,223009682, 205278793,189396910,189175349, 189175200,189175179,189175178, 189175014,189175012,189174852, 171675625,171674995,171674925, 156077831,119034956,154100682, 86450681,78775919,92109049, 48596178,32891159,75905872, 32348701,32348575,32348436, 32348394,61287889,58333751, 18845598,941581228,381273587, 198445276,198445114,198445030, 198444993,116009304,116009141, 1046813254,1046813224, 1046813183,1046813083, 1046813056,1044327069, 1043189398,961420771, 961420766,961420707,961420612, 961420499,961420495,961420492, 961420468,961420465,961420273, 961420254,961420206,961420158, 914343336,1036141119, 1031954968,1025815253, 1025813517,1025813307, 1024312154,961391305, 961391239,961391204,961391190, 961391164,961390978, 1002820131,1002820047, 1002819991,1002819935, 1002819837,1002552665, 1002552627,1002552595, 1002552503,999844882, 999844783,999844777,999844747, 999844727,998453634,998453620, 982278317,941583102,941582715, 941582531,941582178,941582015, 941581822,941581789,941581731, 941581674,941581539,941581502, 941581448,941581294,941581131, 941581007,941580980,941580783, 957761438,821345416,255367289, 255367177,255367163,255366085, 891497898,850203159,850203157, 850202660,850202610,850202608, 850202194,653604148,653604144, 653604143,653604135,653604084, 653604022,653604006,653603906, 653603905,653603896,653603888, 653603850,653603838,653603811, 653603724,653603559,653603399, 653603328,653603288,653603266, 653603234,653603186,653603183, 653603163,653603162,653603154, 653603090,653602987,653602982, 653602979,653602962,653602868, 653602840,653602832,653602802, 807985564,807985482,807985463, 807985332,807985294,807985282, 807985270,807985265,807985240, 807985201,807985176,807985141, 807985134,807985121,807985088, 806642054,806641999,806641444, 806641353,806641250,806641208, 806641180,806641082,806640718, 735659024,735658983,735658901, 735658884,735658841,735658805, 735658747,756762254,666695871, 695118992,695118986,695118959, 695118943,695118900,695118876, 695118874,695118870,695118824, 695118768,695118703,695118692, 695118688,695118661,695118589, 695118438,695118421,695118375, 695118344,695118179,695118173, 695118170&db=nuccore
Which resulted in about 11Mb XML, which you parse and convert I guess ?
That would be between 100 and 200 Mb, at least, no ?
That already puts a heavy burden on the image/VM. Make sure to properly close and cleanup everything as soon as you can.
And all this 40 times (500 x 40 = 20000) ?
I think that goes way over the 2Gb limit (which is rather theoretical, it is more like 1.5).
Unless you clean up very well afterwards and only keep the minimal useful/needed info in memory.
> Another problem I have is the Transcript writes everything once the exception is signaled :(
> Is there a way to revert to the old behavior where each Transcript show: writes to Transcript in situ?
Do you block the UI thread (while running your main top level expression) ? Try forking.
> The ClosedConnection didn't signaled today, but OutOfMemory.
See above.