December 2011 - Pharo-dev - lists.pharo.org

[Pharo-project] COTDC Suggestions
by Sean P. DeNigris Dec. 10, 2011

Dec. 10, 2011

This week, we'll try out a previous suggestion (don't remember OP, sorry) to explore a single package, in the hopes that as we learn various classes, our knowledge will be magnified for related classes. Also, if anyone has a particular package or class that they'd really like to see commented, let me know. -- View this message in context: http://forum.world.st/COTDC-Suggestions-tp4115605p4115605.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

3 2

[Pharo-project] Command Line AWS S3 Upload/Download Tool using Pharo Smalltalk
by Sven Van Caekenberghe Dec. 10, 2011

Dec. 10, 2011

Command Line AWS S3 Upload/Download Tool using Pharo Smalltalk This is a little story about a tool that I needed and that I finally implemented in Pharo Smalltalk. I think it is quite nice and elegant and might be useful to others too. Also, I think it is important to share little hacks like this to show that what you can do is almost unlimited (as many others have shown in all kinds of projects). This is a bit long, so if you're not interested, please ignore. Amazon S3 is a web service interface to store and retrieve any data from anywhere that is highly scalable, reliable, secure, fast, inexpensive. [ http://aws.amazon.com/s3/ ] [ http://en.wikipedia.org/wiki/Amazon_S3 ] I needed a client to interface with S3 from Linux to store and retrieve archives and backups up to 100s of Mb large. Although I had already written a client for this web service in Pharo some time ago, I thought it would not be good enough for these amounts of data. Furthermore, it had no command line interface. So I Googled and found s3cmd. This installed easily and quickly in Ubuntu. Uploading/downloading some small test files went OK. However, uploading large files gave all kinds of strange errors, even though I was doing this from EC2 directly on the Amazon network. I don't know Python so I was not capable of debugging the tool. But I do know Smalltalk and had the basic code already. [ Zinc-AWS in http://www.squeaksource.com/ZincHTTPComponents ] What was missing was a solution to do streaming uploads and downloads so that these huge files would not be taken into memory all at once and a command line interface. Turns out both were quite easy and elegant to implement. I added the following methods to ZnAWSS3Client: downloadFile: filename fromBucket: bucket "Do a streaming download of the key filename from bucket, creating it as a file with that name in the current directory." | streaming response | streaming := self httpClient streaming. self httpClient streaming: true. response := self at: bucket -> filename. self httpClient streaming: streaming. response isSuccess ifFalse: [ ^ ZnHttpUnsuccessful signal ]. FileStream fileNamed: filename do: [ :stream | response entity writeOn: stream ]. ^ response The #streaming option of the underlying ZnClient is toggled on for this request and reset later to its previous value after the request. The #at: does the actual request, returning a ZnStreamingEntity that is then written to a file. When a streaming entity writes itself to another stream it will actually do a streaming copy (one buffer at a time in a loop) for an efficient download. uploadFile: filename withMd5: md5 inBucket: bucket "Do a streaming upload of the file filename to bucket. When md5 is notNil, use it to validate the ETag of the response." | entry size mimeType fileStream entity response | entry := FileDirectory root directoryEntryFor: filename. size := entry fileSize. mimeType := ZnMimeType forFilenameExtension: (FileDirectory extensionFor: filename). fileStream := FileDirectory root readOnlyFileNamed: filename. mimeType isBinary ifTrue: [ fileStream binary ]. (entity := ZnStreamingEntity type: mimeType length: size) stream: fileStream. self at: bucket -> entry name put: entity. (md5 notNil and: [ (md5 sameAs: self eTag) not ]) ifTrue: [ self error: 'Uploaded ETag does not equal supplied MD5' ]. ^ self httpClient response Here the streaming entity is created based on an existing file, the mime type coming from the extension. Then, #at:put: is used to do the upload. Again, when the streaming entity writes itself to the socket stream it will actually do a streaming copy (one buffer at a time in a loop), for an efficient upload. Amazon S3 delivers MD5 hashes (ETags) that can be used to check the integrity of uploads/downloads. ZnAWSS3Client uses these when its #checkIntegrity option is true using MD5>>#hashMessage:, though not using the streaming option of HashFunction. Since it would be a lot of work to make that possible, I opted for externally supplied MD5 hashes using Linux's md5sum utility. Next, I took a clean Pharo 1.3 image and the standard Pharo built CogVM and loaded the following to create a server image: Gofer new squeaksource: 'XMLSupport'; package: 'ConfigurationOfXMLSupport'; load. (Smalltalk at: #ConfigurationOfXMLSupport) perform: #loadDefault. Gofer new squeaksource: 'ZincHTTPComponents'; package: 'Zinc-HTTP'; package: 'Zinc-AWS'; load. Gofer new squeaksource: 'ADayAtTheBeach'; package: 'NonInteractiveTranscript'; load. The glue code for each tool consists of a Linux shell script and a Smalltalk startup script. This is a tool for personal use, error handling could be a bit better regarding wrong invocations. #!/bin/bash echo S3 Download $* script_home=$(dirname $0) script_home=$(cd $script_home && pwd) image=$script_home/pharo-t3.image script=$script_home/s3-download.st bucket=$1 file=$2 vm=$script_home/bin/CogVM options="-vm-display-null -vm-sound-null" echo $vm $options $image $script $bucket $file $vm $options $image $script $bucket $file NonInteractiveTranscript stdout install. [ | args bucket filename | args := Smalltalk commandLine arguments. bucket := args first. filename := args second. (ZnAWSS3Client new) accessKeyId: 'AAAAXXYYZZ5HGQZ77EQ'; secretAccessKey: 'AAAA+D9n7gabcdefghSfevY/mH6KWsvMAcsjzzzz'; checkIntegrity: false; downloadFile: filename fromBucket: bucket; close. Transcript show: 'OK'; cr; endEntry. ] on: Error do: [ :exception | Smalltalk logError: exception description inContext: exception signalerContext. Transcript show: 'FAILED'; cr; endEntry. ]. SmalltalkImage current quitPrimitive. #!/bin/bash echo S3 Upload $* script_home=$(dirname $0) script_home=$(cd $script_home && pwd) image=$script_home/pharo-t3.image script=$script_home/s3-upload.st bucket=$1 file=$2 file_dir=$(dirname $file) file_name=$(basename $file) file=$(cd $file_dir && pwd)/$file_name vm=$script_home/bin/CogVM options="-vm-display-null -vm-sound-null" compute_md5="md5sum $file" md5=`$compute_md5` echo $vm $options $image $script $bucket $file $md5 $vm $options $image $script $bucket $file $md5 NonInteractiveTranscript stdout install. [ | args bucket filename md5 | args := Smalltalk commandLine arguments. bucket := args first. filename := args second. md5 := args third. (ZnAWSS3Client new) accessKeyId: 'AAAAXXYYZZ5HGQZ77EQ'; secretAccessKey: 'AAAA+D9n7gabcdefghSfevY/mH6KWsvMAcsjzzzz'; checkIntegrity: false; uploadFile: filename withMd5: md5 inBucket: bucket; close. Transcript show: 'OK'; cr; endEntry. ] on: Error do: [ :exception | Smalltalk logError: exception description inContext: exception signalerContext. Transcript show: 'FAILED'; cr; endEntry. ] SmalltalkImage current quitPrimitive. Both tools are very similar: the shell scripts uses the VM to start the right image after massaging some of the arguments. The Smalltalk script retrieves arguments, instanciates and invokes the tool to do its job. NonInteractiveTranscript is used with the #stdout option to write OK or FAILED. In the case of failure a PharoDebug.log is written. #quitPrimitive is used for a fast exit (not marking the changes file). Since then I learned from Igor that #quitPrimitive accepts an exit value so it would be possible to do away with the OK/FAILED writing to stdout and just rely on the exit value for an even better integration with *nix. Uploading/downloading files of 100s of Mb went flawless and was more than fast enough (network IO bound anyway). Repeatably starting/stopping a VM with a 20Mb image is totally acceptable and fast as well. A big thanks to everybody who is working on all the little details all the time to make things like this possible ! Sven

2 1

[Pharo-project] [ANN] Magma 1.3
by Chris Muller Dec. 9, 2011

Dec. 9, 2011

I am very pleased to announce the immediate availability of Magma 1.3. The following detail the major improvements to 1.3 over 1.2: Serializer Upgrade ==== BlockClosures (Contexts and CompiledMethods too, of course), can now be stored in the database, even with references to inst-vars, self, temps or globals. Split-Ordered Lists ==== Tom Rushworth developed a fantastic Split-Ordered list implementation for Squeak. Working closely with Tom, this package has been extended in Magma to support very large SOL collections, implemented as a MagmaSolHashTable. Thank you Tom! The MagmaSolHashTable is actually a fairly primitive collection, and was used as the basis for two new collections: A MagmaHashTable and a completely new implementation of MagmaDictionary which is much faster than the old one (almost as fast as a preallocated dictionary). Full Monticello Model ==== When using Magma as a Monticello repository the package versions are not zipped but stored directly in the repository as native instances of MCVersions, MCPackages, MCInfos, MCDefinitions, etc. All of these original MC object instances are canonicalized so there is only ONE copy of that MCMethodDefinition from 2002 which is shared rather than copied in every version since. MCPackages are also canonicalized, as is all ancestry! For packages stored in a Magma repository, two new menu options appear on the methods menu and classes menu: browse mc versions browse mc origin The first shows all versions of a particular method or class-definition across all MC versions saved to that repository, in chronological order (no latency). The second opens the MC-Version Inspector in which that version of the method was saved, to see the version notes. This is very useful for quickly knowing WHY changes were made. Note: Currently only Squeak has its IDE tools updated to support this. WriteBarrier Resurrection ==== After two years dormancy, Avi Bryant's original WriteBarrier has been resurrected for use in closure/Cog images. Its original dependency on NewCompiler has been removed. With both the WriteBarrier and Cog, the MagmaBenchmarker reports a new record maximum commit rate -- 456 commits per second !! The WriteBarrier is now on by default, so nothing needs to be done to enjoy the speed up. Proxy Transparency ==== A new extension to the WriteBarrier should now alleviate the need to send #yourself to primitive arguments! Primitives which require arguments are now overridden in the anonymous subclass so that the arguments will automatically be ensured to be fully-reified for the super call. Immutability ==== For performance, TimeSpans, Durations, DateAndTimes, Points, Intervals, Colors are now immutable. Strings are also immutable by default, but now a custom ImmutabilityStrategy may be specified, providing fine control over immutability in case mutable Strings are needed (or if other objects can be made immutable). Platform Independence ==== Magma 1.3 runs on Squeak 4.3 and Pharo 1.3. All of the packages have been renamed in accordance with contemporary package naming conventions employed by most other active Squeak and Pharo projects. Platform-specific behaviors are now in platform-specific packages (named with "-Squeak" or "-Pharo"). The package hierarchy is documented at http://wiki.squeak.org/squeak/6178. More ==== Many minor cleanups, fixes, and usability improvements. This is, by far, the best Magma ever. Please enjoy the new version. - Chris

2 1

Re: [Pharo-project] [squeak-dev] Re: Prune stack serialization
by Mariano Martinez Peck Dec. 9, 2011

Dec. 9, 2011

Hi guys. When we implemented this possibility of pruning the closures when they were clean, one of our crazy tests started to fail. It is not that the test is very important, but I am interested in trying to understand why it fails. The test (simplied) is: | aSortedCollection materialized | instanceVariableForTesting := false. aSortedCollection := SortedCollection sortBlock: [:a :b | instanceVariableForTesting ifTrue: [ a <= b ] ifFalse: [ a >= b ] ]. materialized := self resultOfSerializeAndMaterialize: aSortedCollection. materialized addAll: #(2 3 1). aSortedCollection addAll: #(2 3 1). Well, the thing is that the materialized SortedCollection, it is sortBLock, the closure is quite crazy and the 'instanceVariableForTesting' doesn't have the boolean but instead a #Processor WTF?? So then the #addAll: fails with a MustBeBoolean. Do you think there is something I should check for this closure? I think it may have to be related with the instanceVariable... Just for the record, what we are doing is something like: serializeReferencesOf: aBlockClosure with: aSerialization 1 to: aBlockClosure basicSize do: [ :index | aSerialization encodeReferenceTo: (aBlockClosure basicAt: index) ]. aBlockClosure isClean ifTrue: [ aSerialization stream nextPut: 1. aSerialization encodeReferenceTo: aBlockClosure method. aSerialization encodeReferenceTo: aBlockClosure receiver. ] ifFalse: [ aSerialization stream nextPut: 0. aSerialization encodeReferenceTo: aBlockClosure outerContext. ]. aSerialization encodeReferenceTo: aBlockClosure startpc. aSerialization encodeReferenceTo: aBlockClosure numArgs. ----- materializeReferencesOf: aBlockClosure with: aMaterialization | methodOrContext method context | 1 to: aBlockClosure basicSize do: [ :index | aBlockClosure basicAt: index put: (aMaterialization nextEncodedReference). ]. (aMaterialization stream next = 1) ifTrue: [ " it was a clean BlockClosure, otherwise it should have been 0" method := aMaterialization nextEncodedReference. aBlockClosure outerContext: (MethodContext sender: nil receiver: aMaterialization nextEncodedReference method: method arguments: #() ) startpc: aMaterialization nextEncodedReference numArgs: aMaterialization nextEncodedReference. ] ifFalse: [ context := aMaterialization nextEncodedReference. aBlockClosure outerContext: context startpc: aMaterialization nextEncodedReference numArgs: aMaterialization nextEncodedReference]. ----- Thanks in advance! On Sun, Dec 4, 2011 at 8:33 PM, Mariano Martinez Peck <marianopeck(a)gmail.com > wrote: > Thanks Juan. With the latest version of Eliot, the following are failing: > > > self deny: [ | outerBlockTemp | [ outerBlockTemp printString ] isClean ] > value. > self deny: [ | outerBlockTemp | [ outerBlockTemp :=7 ] isClean ] value. > self deny: [ tempVar + 1 ] isClean. > self deny: [ tempVar := 1 ] isClean. > self deny: [ ClassVar + 1 ] isClean. > self deny: [ ClassVar := 1 ] isClean. > > Thanks > > > On Sat, Dec 3, 2011 at 3:01 AM, Juan Vuletich <juan(a)jvuletich.org> wrote: > >> Eliot Miranda wrote: >> >> >>> >>> On Fri, Dec 2, 20Tha11 at 4:01 PM, Juan Vuletich <juan(a)jvuletich.org<mailto: >>> juan(a)jvuletich.org>> wrote: >>> >>> Eliot Miranda wrote: >>> >>> Hi Juan, >>> >>> ... >>> >>> >>> >>> I can say that it does indeed work. But I'd be really >>> grateful if >>> Eliot took a good look at that code! >>> >>> >>> There is a much better way, using >>> InstructionStream>>**interpreetNextInstructionFor: which >>> abstracts away from the bytecode encoding and allows one to >>> simply use messages. e.g. look at >>> >>> and (SortedCollection sortBlock: [:a :b| a compare: b >>> caseSensitive: false]) sortBlock abstractBytecodes evaluates to >>> >>> an OrderedCollection(#**pushTemporaryVariable: >>> #pushTemporaryVariable: #pushConstant: #send:super:numArgs:) >>> >>> So a check for clean blocks would check for a block's >>> abstractBytecodes not including some particular set of banned >>> messages (which I *think* is pushReceiver >>> pushReceiverVariable: popIntoReceiverVariable: >>> storeIntoReceiverVariable: methodReturnConstant: >>> methodReturnReceiver methodReturnTop). >>> >>> e.g. >>> >>> isClean >>> ^(self abstractBytecodes intersection: #(pushReceiver >>> pushReceiverVariable: popIntoReceiverVariable: >>> storeIntoReceiverVariable: methodReturnConstant: >>> methodReturnReceiver methodReturnTop)) isEmpty >>> >>> >>> Cheers, >>> Juan Vuletich >>> >>> >>> >>> >>> -- best, >>> Eliot >>> >>> >>> Thanks Eliot. However, it looks to me that this needs more tweaking. >>> >>> <blush>Damn right :) I've done zero testing. It was just an >>> example</blush> >>> >>> For instance, your method answers true for >>> [ xx size ] isClean >>> and >>> [ ^ nil ] isClean >>> while mine answers false for both. Maybe we could serialize non >>> local returns using the Compiler (although we won't be able to >>> return anywhere), but external variable acess is a problem. >>> >>> Will keep playing with this for a while... >>> >>> >>> Can you (can I??) write some tests? What's xx in the above? Ah! [^nil] >>> isn't clean in my code since it should be pc <= end, not pc < end!! Try >>> the attached... >>> >> >> This is a start: >> >> testIsClean >> " >> ClosureTests new testIsClean >> " >> | tempVar | >> tempVar := 1. >> self assert: [ 3 + 4 ] isClean. >> self assert: [ :a | a * 2 ] isClean. >> self assert: [ Smalltalk size ] isClean. >> self assert: [ :blockArg | blockArg printString ] isClean. >> self assert: [ | blockTemp | blockTemp printString ] isClean. >> self assert: [ | blockTemp | blockTemp :=7 ] isClean. >> self deny: [ | outerBlockTemp | [ outerBlockTemp printString ] isClean >> ] value. >> self deny: [ | outerBlockTemp | [ outerBlockTemp :=7 ] isClean ] value. >> self deny: [ tempVar + 1 ] isClean. >> self deny: [ tempVar := 1 ] isClean. >> self deny: [ ivar + 1 ] isClean. >> self deny: [ ivar := 1 ] isClean. >> self deny: [ ^ true ] isClean. >> self deny: [ self printString ] isClean. >> self deny: [ ^ self ] isClean. >> self deny: [ ClassVar + 1 ] isClean. >> self deny: [ ClassVar := 1 ] isClean. >> >> This test passes with my implementation. >> >> >> >>> >>> Cheers, >>> Juan Vuletich >>> >>> >>> >>> >>> -- >>> best, >>> Eliot >>> >>> >> Cheers, >> Juan Vuletich >> >> > > > -- > Mariano > http://marianopeck.wordpress.com > > -- Mariano http://marianopeck.wordpress.com

1 0

[Pharo-project] MenuMorph instances
by Sean P. DeNigris Dec. 9, 2011

Dec. 9, 2011

"MenuMorph allInstances" returns an array of 27 in my personal 1.3 image, and 5 in a fresh 1.3 rc. Are they cached? orphaned? -- View this message in context: http://forum.world.st/MenuMorph-instances-tp4178039p4178039.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

2 1

[Pharo-project] [COTDC] 94 - AnnouncementSet
by sean＠clipperadams.com Dec. 9, 2011

Dec. 9, 2011

93 classes done 802 classes remaining Continuing in Announcements... Today: AnnouncementSet Comment Of The Day Contest - One Day One Comment Rules: #1: Each day a not commented class is elected. Each day the best comment will be integrated with name of the author(s). #2: If you cannot comment it, deprecate it. Results: http://code.google.com/p/pharo/wiki/CommentOfTheDayContest p.s. this message was sent from a pharo image. Did you know you could easily do that?!

1 0

[Pharo-project] Exception Handling in the Moose Distribution Image
by Mircea Filip Lungu Dec. 9, 2011

Dec. 9, 2011

Hi guys, This is a bug report I got from one of my students and replicated by myself: In the Moose 4.6 image downloaded from moosetechnology.org, if I execute the following code | c | c := Collection new. c add: 1. the image seems to freeze, although what I would expect is that SubclassResponsibility exception to be raised. This has been observed on Windows and Mac. Do you know about this? Thanks, M. -- Dr. Mircea Lungu Researcher Software Composition Group University of Bern -- Dr. Mircea Lungu Researcher Software Composition Group University of Bern

2 1

[Pharo-project] RB AST Rewriter to eliminate/consoliate temporary variables?
by Stefan Marr Dec. 9, 2011

Dec. 9, 2011

Hi all, Hi Lukas: Do you know if anyone ever implemented a rewriter for the RB AST that reduces the amount of temporary variables if possible? I have a naive rewriting step that creates too many temps, but some of them are not strictly necessary. However, currently, I run into 'Cannot compile -- stack including temps is too deep' which is, well, ehm, not so nice. In case anyone has something in that direction, I would be interested. Thanks Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525

2 5

Re: [Pharo-project] Exception Handling in the Moose Distribution Image
by Marcus Denker Dec. 9, 2011

Dec. 9, 2011

On Dec 9, 2011, at 4:25 PM, Mircea Filip Lungu wrote: > Hi guys, > > This is a bug report I got from one of my students and replicated by myself: > > In the Moose 4.6 image downloaded from moosetechnology.org, if I > execute the following code > > | c | > c := Collection new. > c add: 1. > > the image seems to freeze, although what I would expect is that > SubclassResponsibility > exception to be raised. This has been observed on Windows and Mac. > > Do you know about this? > This has been fixed in 1.4 last week... as a workaround, I think filing in this should work: -- Marcus Denker -- http://marcusdenker.de

1 0

Re: [Pharo-project] NBOpenGL
by Fernando Olivero Dec. 9, 2011

Dec. 9, 2011

Thanks Henrik for the info, now's its clear to me. Igor, excellent news! i was struggling with NativeBoost, almost have a working version of CGL, but i'm eager to see your implementation. Yes, CGL only provide either fullscreen of offscreen contexts. Cocoa (and Carbon) adds the windowing layer on top of CGL. I'll analyze how to overcome this problem now, cause i want to start rendering with Cairo ASAP! Fernando On Fri, Dec 9, 2011 at 3:06 PM, Igor Stasenko <siguctua(a)gmail.com> wrote: > (resending message due to mailer problem) > > Since it works now only for Windows, Â i didn't thought a lot about > platform-neutral stuff. > Creating and initializing an opengl context is platform-specific, and > it is hard to tell what you may need to create it. > > Regarding opengl on mac: > - i just uploaded the initial snapshot of opengl bindings for mac, see > NBOpenGL-Mac package. > it can create a context using > NBMacContextManager new createContext > > but i am a bit puzzled what to do next, because it looks like CGL > library allows you to create only a fullscreen contexts, > and only Carbon/Cocoa frameworks allowing you to make a windowed > context by attaching context to specific 'drawable'. > So, Fernando (or others), if you know more, take a look what can be done. > Maybe it was a mistake to use CGL interface for it, maybe i should > write bindings using Carbon interface, but > since it promised by apple to be deprecated and removed from system, i > don't know what to do (and Cocoa is an Objective-C, > and making calls to Objective-C code is completely different story). > > On 9 December 2011 11:34, Henrik Sperre Johansen > <henrik.s.johansen(a)veloxit.no> wrote: >> On 09.12.2011 11:27, Henrik Sperre Johansen wrote: >>> >>> >>> That's not specific to offscreen rendering contexts though, (nor >>> pixelformats, another windows artifact) >> >> I should perhaps say; they're not Windows specific as such, but the >> implementation in NBOpenGL is. >> Personally, I'd rather have createContext: taking a crossplatform >> pixel/buffer format as argument, the platform context transcribe that into >> the native object needed as parameter for context creation, and some sort of >> way to check if the context created actually matched your specification. >> Makes the whole "Your context was invalid for those operations" class of >> errors a lot more transparent, which I've found to be the source of trouble >> multiple times at least. >> >> Cheers, >> Henry >> > > > > -- > Best regards, > Igor Stasenko. >

1 0