- Pharo-dev - lists.pharo.org

[Pharo-project] Command Line AWS S3 Upload/Download Tool using Pharo Smalltalk

by Sven Van Caekenberghe

Command Line AWS S3 Upload/Download Tool using Pharo Smalltalk This is a little story about a tool that I needed and that I finally implemented in Pharo Smalltalk. I think it is quite nice and elegant and might be useful to others too. Also, I think it is important to share little hacks like this to show that what you can do is almost unlimited (as many others have shown in all kinds of projects). This is a bit long, so if you're not interested, please ignore. Amazon S3 is a web service interface to store and retrieve any data from anywhere that is highly scalable, reliable, secure, fast, inexpensive. [ http://aws.amazon.com/s3/ ] [ http://en.wikipedia.org/wiki/Amazon_S3 ] I needed a client to interface with S3 from Linux to store and retrieve archives and backups up to 100s of Mb large. Although I had already written a client for this web service in Pharo some time ago, I thought it would not be good enough for these amounts of data. Furthermore, it had no command line interface. So I Googled and found s3cmd. This installed easily and quickly in Ubuntu. Uploading/downloading some small test files went OK. However, uploading large files gave all kinds of strange errors, even though I was doing this from EC2 directly on the Amazon network. I don't know Python so I was not capable of debugging the tool. But I do know Smalltalk and had the basic code already. [ Zinc-AWS in http://www.squeaksource.com/ZincHTTPComponents ] What was missing was a solution to do streaming uploads and downloads so that these huge files would not be taken into memory all at once and a command line interface. Turns out both were quite easy and elegant to implement. I added the following methods to ZnAWSS3Client: downloadFile: filename fromBucket: bucket "Do a streaming download of the key filename from bucket, creating it as a file with that name in the current directory." | streaming response | streaming := self httpClient streaming. self httpClient streaming: true. response := self at: bucket -> filename. self httpClient streaming: streaming. response isSuccess ifFalse: [ ^ ZnHttpUnsuccessful signal ]. FileStream fileNamed: filename do: [ :stream | response entity writeOn: stream ]. ^ response The #streaming option of the underlying ZnClient is toggled on for this request and reset later to its previous value after the request. The #at: does the actual request, returning a ZnStreamingEntity that is then written to a file. When a streaming entity writes itself to another stream it will actually do a streaming copy (one buffer at a time in a loop) for an efficient download. uploadFile: filename withMd5: md5 inBucket: bucket "Do a streaming upload of the file filename to bucket. When md5 is notNil, use it to validate the ETag of the response." | entry size mimeType fileStream entity response | entry := FileDirectory root directoryEntryFor: filename. size := entry fileSize. mimeType := ZnMimeType forFilenameExtension: (FileDirectory extensionFor: filename). fileStream := FileDirectory root readOnlyFileNamed: filename. mimeType isBinary ifTrue: [ fileStream binary ]. (entity := ZnStreamingEntity type: mimeType length: size) stream: fileStream. self at: bucket -> entry name put: entity. (md5 notNil and: [ (md5 sameAs: self eTag) not ]) ifTrue: [ self error: 'Uploaded ETag does not equal supplied MD5' ]. ^ self httpClient response Here the streaming entity is created based on an existing file, the mime type coming from the extension. Then, #at:put: is used to do the upload. Again, when the streaming entity writes itself to the socket stream it will actually do a streaming copy (one buffer at a time in a loop), for an efficient upload. Amazon S3 delivers MD5 hashes (ETags) that can be used to check the integrity of uploads/downloads. ZnAWSS3Client uses these when its #checkIntegrity option is true using MD5>>#hashMessage:, though not using the streaming option of HashFunction. Since it would be a lot of work to make that possible, I opted for externally supplied MD5 hashes using Linux's md5sum utility. Next, I took a clean Pharo 1.3 image and the standard Pharo built CogVM and loaded the following to create a server image: Gofer new squeaksource: 'XMLSupport'; package: 'ConfigurationOfXMLSupport'; load. (Smalltalk at: #ConfigurationOfXMLSupport) perform: #loadDefault. Gofer new squeaksource: 'ZincHTTPComponents'; package: 'Zinc-HTTP'; package: 'Zinc-AWS'; load. Gofer new squeaksource: 'ADayAtTheBeach'; package: 'NonInteractiveTranscript'; load. The glue code for each tool consists of a Linux shell script and a Smalltalk startup script. This is a tool for personal use, error handling could be a bit better regarding wrong invocations. #!/bin/bash echo S3 Download $* script_home=$(dirname $0) script_home=$(cd $script_home && pwd) image=$script_home/pharo-t3.image script=$script_home/s3-download.st bucket=$1 file=$2 vm=$script_home/bin/CogVM options="-vm-display-null -vm-sound-null" echo $vm $options $image $script $bucket $file $vm $options $image $script $bucket $file NonInteractiveTranscript stdout install. [ | args bucket filename | args := Smalltalk commandLine arguments. bucket := args first. filename := args second. (ZnAWSS3Client new) accessKeyId: 'AAAAXXYYZZ5HGQZ77EQ'; secretAccessKey: 'AAAA+D9n7gabcdefghSfevY/mH6KWsvMAcsjzzzz'; checkIntegrity: false; downloadFile: filename fromBucket: bucket; close. Transcript show: 'OK'; cr; endEntry. ] on: Error do: [ :exception | Smalltalk logError: exception description inContext: exception signalerContext. Transcript show: 'FAILED'; cr; endEntry. ]. SmalltalkImage current quitPrimitive. #!/bin/bash echo S3 Upload $* script_home=$(dirname $0) script_home=$(cd $script_home && pwd) image=$script_home/pharo-t3.image script=$script_home/s3-upload.st bucket=$1 file=$2 file_dir=$(dirname $file) file_name=$(basename $file) file=$(cd $file_dir && pwd)/$file_name vm=$script_home/bin/CogVM options="-vm-display-null -vm-sound-null" compute_md5="md5sum $file" md5=`$compute_md5` echo $vm $options $image $script $bucket $file $md5 $vm $options $image $script $bucket $file $md5 NonInteractiveTranscript stdout install. [ | args bucket filename md5 | args := Smalltalk commandLine arguments. bucket := args first. filename := args second. md5 := args third. (ZnAWSS3Client new) accessKeyId: 'AAAAXXYYZZ5HGQZ77EQ'; secretAccessKey: 'AAAA+D9n7gabcdefghSfevY/mH6KWsvMAcsjzzzz'; checkIntegrity: false; uploadFile: filename withMd5: md5 inBucket: bucket; close. Transcript show: 'OK'; cr; endEntry. ] on: Error do: [ :exception | Smalltalk logError: exception description inContext: exception signalerContext. Transcript show: 'FAILED'; cr; endEntry. ] SmalltalkImage current quitPrimitive. Both tools are very similar: the shell scripts uses the VM to start the right image after massaging some of the arguments. The Smalltalk script retrieves arguments, instanciates and invokes the tool to do its job. NonInteractiveTranscript is used with the #stdout option to write OK or FAILED. In the case of failure a PharoDebug.log is written. #quitPrimitive is used for a fast exit (not marking the changes file). Since then I learned from Igor that #quitPrimitive accepts an exit value so it would be possible to do away with the OK/FAILED writing to stdout and just rely on the exit value for an even better integration with *nix. Uploading/downloading files of 100s of Mb went flawless and was more than fast enough (network IO bound anyway). Repeatably starting/stopping a VM with a 20Mb image is totally acceptable and fast as well. A big thanks to everybody who is working on all the little details all the time to make things like this possible ! Sven

Dec. 9, 2011

Re: [Pharo-project] RB AST Rewriter to eliminate/consoliate temporary variables?

by Lukas Renggli

Hi Stefan, I think there is a lint rule that does that: TempNotRead ... I don't have an image at hand, but I can check tonight if you haven't found it until then. Lukas On 9 December 2011 16:29, Stefan Marr <pharo(a)stefan-marr.de> wrote: > Hi all, > Hi Lukas: > > Do you know if anyone ever implemented a rewriter for the RB AST that reduces the amount of temporary variables if possible? > > I have a naive rewriting step that creates too many temps, but some of them are not strictly necessary. > > However, currently, I run into 'Cannot compile -- stack including temps is too deep' which is, well, ehm, not so nice. > > In case anyone has something in that direction, I would be interested. > > Thanks > Stefan > > -- > Stefan Marr > Software Languages Lab > Vrije Universiteit Brussel > Pleinlaan 2 / B-1050 Brussels / Belgium > http://soft.vub.ac.be/~smarr > Phone: +32 2 629 2974 > Fax: Â +32 2 629 3525 > -- Lukas Renggli www.lukas-renggli.ch

Dec. 9, 2011

Re: [Pharo-project] Sprint friday 16 @ lille

by Sean P. DeNigris

Jealous!!! -- View this message in context: http://forum.world.st/Sprint-friday-16-lille-tp4175752p4177101.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

Dec. 9, 2011

Re: [Pharo-project] How to resurrect an unrepsonsive image?

by Sean P. DeNigris

Norbert Hartl wrote > > Trying to recover the image was not important for me but for pharo. I just > liked to participate in solving a problem that is hard to debug. > Right on, Norbert!! Thank you for doing that. This already happened to me twice and it is a difficult and important problem for us. Sean -- View this message in context: http://forum.world.st/How-to-resurrect-an-unrepsonsive-image-tp4168518p4177… Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

Dec. 9, 2011

[Pharo-project] RB AST Rewriter to eliminate/consoliate temporary variables?

by Stefan Marr

Hi all, Hi Lukas: Do you know if anyone ever implemented a rewriter for the RB AST that reduces the amount of temporary variables if possible? I have a naive rewriting step that creates too many temps, but some of them are not strictly necessary. However, currently, I run into 'Cannot compile -- stack including temps is too deep' which is, well, ehm, not so nice. In case anyone has something in that direction, I would be interested. Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525

Dec. 9, 2011

Re: [Pharo-project] NBOpenGL

by Fernando Olivero

Thanks Henrik for the info, now's its clear to me. Igor, excellent news! i was struggling with NativeBoost, almost have a working version of CGL, but i'm eager to see your implementation. Yes, CGL only provide either fullscreen of offscreen contexts. Cocoa (and Carbon) adds the windowing layer on top of CGL. I'll analyze how to overcome this problem now, cause i want to start rendering with Cairo ASAP! Fernando On Fri, Dec 9, 2011 at 3:06 PM, Igor Stasenko <siguctua(a)gmail.com> wrote: > (resending message due to mailer problem) > > Since it works now only for Windows, Â i didn't thought a lot about > platform-neutral stuff. > Creating and initializing an opengl context is platform-specific, and > it is hard to tell what you may need to create it. > > Regarding opengl on mac: > - i just uploaded the initial snapshot of opengl bindings for mac, see > NBOpenGL-Mac package. > it can create a context using > NBMacContextManager new createContext > > but i am a bit puzzled what to do next, because it looks like CGL > library allows you to create only a fullscreen contexts, > and only Carbon/Cocoa frameworks allowing you to make a windowed > context by attaching context to specific 'drawable'. > So, Fernando (or others), if you know more, take a look what can be done. > Maybe it was a mistake to use CGL interface for it, maybe i should > write bindings using Carbon interface, but > since it promised by apple to be deprecated and removed from system, i > don't know what to do (and Cocoa is an Objective-C, > and making calls to Objective-C code is completely different story). > > On 9 December 2011 11:34, Henrik Sperre Johansen > <henrik.s.johansen(a)veloxit.no> wrote: >> On 09.12.2011 11:27, Henrik Sperre Johansen wrote: >>> >>> >>> That's not specific to offscreen rendering contexts though, (nor >>> pixelformats, another windows artifact) >> >> I should perhaps say; they're not Windows specific as such, but the >> implementation in NBOpenGL is. >> Personally, I'd rather have createContext: taking a crossplatform >> pixel/buffer format as argument, the platform context transcribe that into >> the native object needed as parameter for context creation, and some sort of >> way to check if the context created actually matched your specification. >> Makes the whole "Your context was invalid for those operations" class of >> errors a lot more transparent, which I've found to be the source of trouble >> multiple times at least. >> >> Cheers, >> Henry >> > > > > -- > Best regards, > Igor Stasenko. >

Dec. 9, 2011

[Pharo-project] Exception Handling in the Moose Distribution Image

by Mircea Filip Lungu

Hi guys, This is a bug report I got from one of my students and replicated by myself: In the Moose 4.6 image downloaded from moosetechnology.org, if I execute the following code | c | c := Collection new. c add: 1. the image seems to freeze, although what I would expect is that SubclassResponsibility exception to be raised. This has been observed on Windows and Mac. Do you know about this? Thanks, M. -- Dr. Mircea Lungu Researcher Software Composition Group University of Bern -- Dr. Mircea Lungu Researcher Software Composition Group University of Bern

Dec. 9, 2011

Re: [Pharo-project] GUI Testing Library

by Marcus Denker

On Dec 9, 2011, at 4:08 PM, Tudor Girba wrote: > Btw, I think these should get in the Pharo distribution because they > are just too valuable. > Yes! > Doru > > > On Fri, Dec 9, 2011 at 4:07 PM, Tudor Girba <tudor(a)tudorgirba.com> wrote: >> Thanks a lot, Sean! >> >> It works just great with BDDExtensions and Keymapping-Shortcuts. >> >> Cheers, >> Doru >> >> >> On Thu, Dec 8, 2011 at 3:35 PM, Sean P. DeNigris <sean(a)clipperadams.com> wrote: >>> >>> Tudor Girba-2 wrote >>>> >>>> Would it be possible to simulate keys with modifiers without Keymapping >>>> present? >>>> >>> >>> I think I only used Keymapping for the modifier key syntax, but looking at >>> its repository, the shortcuts are packaged separately. So you could try only >>> loading Keymapping-Shortcuts, which does not hook into the system (so >>> shouldn't break anything) and will add #command, $shift, etc. Let me know if >>> that works for you. >>> >>> Sean >>> >>> -- >>> View this message in context: http://forum.world.st/GUI-Testing-Library-tp4097866p4172952.html >>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com. >>> >> >> >> >> -- >> www.tudorgirba.com >> >> "Every thing has its own flow" > > > > -- > www.tudorgirba.com > > "Every thing has its own flow" > -- Marcus Denker -- http://marcusdenker.de

Dec. 9, 2011

Re: [Pharo-project] GUI Testing Library

by Tudor Girba

Btw, I think these should get in the Pharo distribution because they are just too valuable. Doru On Fri, Dec 9, 2011 at 4:07 PM, Tudor Girba <tudor(a)tudorgirba.com> wrote: > Thanks a lot, Sean! > > It works just great with BDDExtensions and Keymapping-Shortcuts. > > Cheers, > Doru > > > On Thu, Dec 8, 2011 at 3:35 PM, Sean P. DeNigris <sean(a)clipperadams.com> wrote: >> >> Tudor Girba-2 wrote >>> >>> Would it be possible to simulate keys with modifiers without Keymapping >>> present? >>> >> >> I think I only used Keymapping for the modifier key syntax, but looking at >> its repository, the shortcuts are packaged separately. So you could try only >> loading Keymapping-Shortcuts, which does not hook into the system (so >> shouldn't break anything) and will add #command, $shift, etc. Let me know if >> that works for you. >> >> Sean >> >> -- >> View this message in context: http://forum.world.st/GUI-Testing-Library-tp4097866p4172952.html >> Sent from the Pharo Smalltalk mailing list archive at Nabble.com. >> > > > > -- > www.tudorgirba.com > > "Every thing has its own flow" -- www.tudorgirba.com "Every thing has its own flow"

Dec. 9, 2011

Re: [Pharo-project] GUI Testing Library

by Tudor Girba

Thanks a lot, Sean! It works just great with BDDExtensions and Keymapping-Shortcuts. Cheers, Doru On Thu, Dec 8, 2011 at 3:35 PM, Sean P. DeNigris <sean(a)clipperadams.com> wrote: > > Tudor Girba-2 wrote >> >> Would it be possible to simulate keys with modifiers without Keymapping >> present? >> > > I think I only used Keymapping for the modifier key syntax, but looking at > its repository, the shortcuts are packaged separately. So you could try only > loading Keymapping-Shortcuts, which does not hook into the system (so > shouldn't break anything) and will add #command, $shift, etc. Let me know if > that works for you. > > Sean > > -- > View this message in context: http://forum.world.st/GUI-Testing-Library-tp4097866p4172952.html > Sent from the Pharo Smalltalk mailing list archive at Nabble.com. > -- www.tudorgirba.com "Every thing has its own flow"

Dec. 9, 2011