[Pharo-project] Downloading a file
ZnClient new get: 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/...'. Reports entity too large. How do I save this file to local disk? Thanks. Sean -- View this message in context: http://forum.world.st/Downloading-a-file-tp4586900p4586900.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
On 25 April 2012 17:13, Sean P. DeNigris <sean@clipperadams.com> wrote:
ZnClient new get: 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/...'.
Reports entity too large. How do I save this file to local disk?
this is situation where using stream is preferable. Sven already told how to use it.. i just don't remember how.
Thanks. Sean
-- View this message in context: http://forum.world.st/Downloading-a-file-tp4586900p4586900.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
-- Best regards, Igor Stasenko.
Sean, On 25 Apr 2012, at 17:13, Sean P. DeNigris wrote:
ZnClient new get: 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/...'.
Reports entity too large. How do I save this file to local disk?
One of the key ideas of Zn for me was to offer a Smalltalk codebase where 'The source will be with you, always' would be true, like it should be true in Smalltalk as a whole. Yes, it throws an entity too large exception, to protect you (or your server). This is a feature ! Open the debugger and you'll see how it is using ZnConstants>>#maximumEntitySize, go look there and lo and behold there is a setter to modify this limit. Furthermore, the ZnEntityTooLarge exception is proceedable. Apparently the download is about 40Mb. So it is possible to get to the ByteArray in an Inspector and do whatever you want, like save it manually. But it would be better to do this in a streaming fashion, like Igor said and save it directly to a file. This should work: FileStream newFileNamed: '/tmp/cog.tgz' do: [ :stream | | entity | stream binary. entity := ZnClient new streaming: true; get: 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/...'; entity. entity writeOn: fileStream ] The entity is actually a ZnStreamingEntity (have a look at its #writeOn: method). You can test is on a smaller file FileStream newFileNamed: '/tmp/pharo.png' do: [ :fileStream | | entity | fileStream binary. entity := ZnClient new streaming: true; get: 'http://www.pharo-project.org/images/pharo.png'; entity. entity writeOn: fileStream ] Now, you are right that there should be some convenience method to make that easier. I'll think about it for a bit and let you know if I can come up with something. HTH, Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
Sven Van Caekenberghe wrote
FileStream newFileNamed: '/tmp/cog.tgz' do: [ :stream | | entity | stream binary. entity := ZnClient new streaming: true; get: 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/...'; entity. entity writeOn: fileStream ]
Thanks, Sven! You just beat me to it. The entity part was tricky. I was looking at the result of the #get:, trying to figure out what to do with the ZnLimitedReadStream... The above snippets successfully downloaded the files (I think), but there is something wrong with them. The png doesn't open (mac message about a damaged file) and the .gz doesn't uncompress (it just keeps making differently named .gz files). Sean -- View this message in context: http://forum.world.st/Downloading-a-file-tp4586900p4587681.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
On 25 Apr 2012, at 20:51, Sean P. DeNigris wrote:
The above snippets successfully downloaded the files (I think), but there is something wrong with them. The png doesn't open (mac message about a damaged file) and the .gz doesn't uncompress (it just keeps making differently named .gz files).
That is because the Pharo website is down currently ;-) Note that I used .tgz in the filename, not .tar.gz I'll test some more... Sven
On 25 Apr 2012, at 21:35, Sven Van Caekenberghe wrote:
I'll test some more...
Sean, Sorry, there was indeed something wrong, with my examples, but I couldn't figure it out at first. Things is, you can't say #beOneShot because then the connection is closed after the first succesfull request/response cycle, and with a streaming entity that means that the underlying stream is gone (well closed). It would have been nice if socket stream threw an exception instead of returning zeros ;-) These examples should work: ZnClient new systemPolicy; accept: ZnMimeType imageJpeg; url: 'http://caretaker.wolf359.be/sun-fire-x2100.jpg'; downloadToFileNamed: '/tmp/sun-fire-x2100.jpg'. ZnClient new systemPolicy; accept: ZnMimeType text; url: 'http://zn.stfx.eu/zn/numbers.txt'; downloadToFileNamed: '/tmp/numbers.txt'. Optionally a #close should be sent to the client as well (although garbage collection will close the socket eventually). By using #systemPolicy, success is enforced anyway. And when the pharo website is up again: ZnClient new systemPolicy; accept: ZnMimeType imagePng; url: 'http://www.pharo-project.org/images/pharo.png'; downloadToFileNamed: '/tmp/pharo.png'. Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
Sean P. DeNigris wrote
Sven Van Caekenberghe wrote
entity := ZnClient new streaming: true; get: 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/...';
The above snippets successfully downloaded the files (I think), but there is something wrong with them.
The snippets were fine. The problem was me ;-) Sorry to cause you extra work... your original simpler examples (like the one above) work. I was calling #response instead of #entity, so the saved file included the http headers and made the files unreadable. Thanks for the help. Sean -- View this message in context: http://forum.world.st/Downloading-a-file-tp4586900p4588483.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
On 25 Apr 2012, at 20:15, Sven Van Caekenberghe wrote:
Now, you are right that there should be some convenience method to make that easier. I'll think about it for a bit and let you know if I can come up with something.
With these versions: --- Name: Zinc-HTTP-SvenVanCaekenberghe.262 Author: SvenVanCaekenberghe Time: 25 April 2012, 9:19:35 pm UUID: 483455c8-f370-40e8-8848-036044211929 Ancestors: Zinc-HTTP-SvenVanCaekenberghe.261 added ZnClient>>#downloadToFileNamed: --- Name: Zinc-Tests-SvenVanCaekenberghe.135 Author: SvenVanCaekenberghe Time: 25 April 2012, 9:21:15 pm UUID: 364763a4-b7d7-4d0e-b66b-7e117e9194f3 Ancestors: Zinc-Tests-SvenVanCaekenberghe.134 added test for ZnClient>>#downloadToFileNamed: --- you can now do: ZnClient new url: 'http://www.pharo-project.org/images/pharo.png'; downloadToFileNamed: '/tmp/pharo.png'. or even more correctly: ZnClient new systemPolicy; beOneShot; url: 'http://www.pharo-project.org/images/pharo.png'; downloadToFileNamed: '/tmp/pharo.png'. Sven
Sven Van Caekenberghe wrote
Now, you are right that there should be some convenience method to make that easier. I'll think about it for a bit and let you know if I can come up with something.
What about: download: urlString to: anFSReference "anFSReference can be either: - a directory, where the file will be saved under its original name - otherwise, the file will be saved as the FSReference" | target entity | target := anFSReference isDirectory ifTrue: [ anFSReference / urlString asUrl path last ] ifFalse: [ anFSReference ]. entity := ZnClient new streaming: true; get: urlString; entity. target fileStreamDo: [ :stream | stream binary. entity writeOn: stream ]. This would allow really easy, flexible use like: "Save in this directory as same name" ZnClient download: 'http://www.pharo-project.org/images/pharo.png' to: '/Volumes/Fast/Squeak/Main Working Images/Pharo-1.3' asFile. "Save under new name in this exact place" ZnClient download: 'http://www.pharo-project.org/images/pharo.png' to: '/Volumes/Fast/Squeak/Main Working Images/Pharo-1.3/new_name.png' asFile. -- View this message in context: http://forum.world.st/Downloading-a-file-tp4586900p4588519.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
Sean, On 26 Apr 2012, at 02:23, Sean P. DeNigris wrote:
Sven Van Caekenberghe wrote
Now, you are right that there should be some convenience method to make that easier. I'll think about it for a bit and let you know if I can come up with something.
What about: download: urlString to: anFSReference "anFSReference can be either: - a directory, where the file will be saved under its original name - otherwise, the file will be saved as the FSReference"
| target entity | target := anFSReference isDirectory ifTrue: [ anFSReference / urlString asUrl path last ] ifFalse: [ anFSReference ].
entity := ZnClient new streaming: true; get: urlString; entity. target fileStreamDo: [ :stream | stream binary. entity writeOn: stream ].
This would allow really easy, flexible use like: "Save in this directory as same name" ZnClient download: 'http://www.pharo-project.org/images/pharo.png' to: '/Volumes/Fast/Squeak/Main Working Images/Pharo-1.3' asFile.
"Save under new name in this exact place" ZnClient download: 'http://www.pharo-project.org/images/pharo.png' to: '/Volumes/Fast/Squeak/Main Working Images/Pharo-1.3/new_name.png' asFile.
I implemented ZnClient>>#downloadTo: which accepts both a directory or a filename path string. (Zinc-HTTP-SvenVanCaekenberghe.265 and Zinc-Tests-SvenVanCaekenberghe.136). ZnClient new systemPolicy; accept: ZnMimeType text; url: 'http://zn.stfx.eu/zn/numbers.txt'; downloadTo: '/Users/sven/Desktop'. I temporarily rolled back my use of FileSystem due to http://code.google.com/p/pharo/issues/detail?id=5693 and the fact that I had to hack too much to maintain 1.3 compatibility (you can still see the code in a previous version, Zinc-HTTP-SvenVanCaekenberghe.264). Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
I temporarily rolled back my use of FileSystem due to http://code.google.com/p/pharo/issues/detail?id=5693
this is strange that it works in 1.4 and not in 2.0 because so far we nearly changed anything :) Stef
Stéphane Ducasse wrote
this is strange that it works in 1.4 and not in 2.0 because so far we nearly changed anything :)
It's a vm bug (see http://forum.world.st/Confused-about-isDirectory-td4589441.html), so 1.3, 1.4 and 2.0 should be equally effected...
From issue 5693: 1.4 '/private/tmp' asReference isDirectory. false 2.0 '/private/tmp' asFileReference isDirectory. false
are both wrong results... -- View this message in context: http://forum.world.st/Downloading-a-file-tp4586900p4591149.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
thanks! I prefer bugs that last and do not show up like mushrooms :) On Apr 27, 2012, at 12:14 AM, Sean P. DeNigris wrote:
Stéphane Ducasse wrote
this is strange that it works in 1.4 and not in 2.0 because so far we nearly changed anything :)
It's a vm bug (see http://forum.world.st/Confused-about-isDirectory-td4589441.html), so 1.3, 1.4 and 2.0 should be equally effected...
From issue 5693: 1.4 '/private/tmp' asReference isDirectory. false 2.0 '/private/tmp' asFileReference isDirectory. false
are both wrong results...
-- View this message in context: http://forum.world.st/Downloading-a-file-tp4586900p4591149.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
participants (4)
-
Igor Stasenko -
Sean P. DeNigris -
Stéphane Ducasse -
Sven Van Caekenberghe