December 2015 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] SecureSession frame types

by Robert Withers

Irked and confused how to share updatable documents through dropbox. It didn't help that I had logged off the daemon. I reordered the 64bit header. Here's an updated link. If the doc changes in the future, I hope this link continues to work. https://www.dropbox.com/s/zj2i1votg432b2p/FrameTypes.pptx?dl=0 regards, robert On 12/20/2015 05:31 AM, Robert Withers wrote: > Excuse me for failing to complete this email./ I wanted to ask for > feedback on these structures, especially the header specification word > structure. So these fields make sense? > > thank you, > robert > > On 12/20/2015 05:27 AM, Robert Withers wrote: >> I wrote up the various SecureSession frame types in presentation >> document, which you can get here: >> https://www.dropbox.com/s/3xfa5oqs8ihj88v/FrameTypes.pptx?dl=0 >> >> These are not yet implemented, but it is progressing. Ne quid nemis. >> >> thanks, > -- . .. .. ^,^ robert

Dec. 20, 2015

Re: [Pharo-dev] [Vm-dev] VM Maker: VMMaker.oscog-eem.1609.mcz

by Tudor Girba

Merry Christmas to you, too :)! Doru > On Dec 17, 2015, at 8:12 PM, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > > Hi All, > > the real Cog 64-bit Spur x64 VM just evaluated 3+4 correctly on Mac OS X: > > <Screen Shot 2015-12-17 at 11.10.14.png> > â > > Early days yet. This is a debug VM. The optimised VM does not yet display the prompt. But we can be confident that a 64-bit JIT for Spur will be available some time in January. > â > > On Thu, Dec 17, 2015 at 10:42 AM, <commits(a)source.squeak.org> wrote: > > Eliot Miranda uploaded a new version of VMMaker to project VM Maker: > http://source.squeak.org/VMMaker/VMMaker.oscog-eem.1609.mcz > > ==================== Summary ==================== > > Name: VMMaker.oscog-eem.1609 > Author: eem > Time: 17 December 2015, 10:41:49.192 am > UUID: 771cdc25-5d27-4818-8943-2ba5c0e31791 > Ancestors: VMMaker.oscog-eem.1608 > > Cogit: Fix the (arguably bogus) register save/restore code for safe trampolines, including the ceScheduleScavenge call. > > =============== Diff against VMMaker.oscog-eem.1608 =============== > > Item was added: > + ----- Method: CogARMCompiler>>genSaveRegForCCall (in category 'abi') ----- > + genSaveRegForCCall > + "Save the general purpose registers for a call into the C run-time from a trampoline." > + "Save none, because the ARM ABI only defines callee saved registers, no caller-saved regs." > + "cogit gen: STMFD operand: 16r7F"! > > Item was removed: > - ----- Method: CogARMCompiler>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - "Save none, because the ARM ABI only defines callee saved registers, no caller-saved regs." > - "cogit gen: STMFD operand: 16r7F"! > > Item was added: > + ----- Method: CogARMCompiler>>genSaveRegsForCCall (in category 'abi') ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a trampoline call." > + "Save none, because the ARM ABI only defines callee saved registers, no caller-saved regs." > + "cogit gen: STMFD operand: 16r7F" > + self flag: 'this will change with Sista when we hope to be able to allocate arbitrary registers'! > > Item was removed: > - ----- Method: CogAbstractInstruction>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - self subclassResponsibility! > > Item was changed: > ----- Method: CogIA32Compiler>>genRestoreRegsExcept: (in category 'abi') ----- > genRestoreRegsExcept: abstractReg > | realReg | > realReg := self concreteRegister: abstractReg. > self assert: (EDI > EAX and: [EDI - EAX + 1 = 6]). > EAX to: EDI do: > [:reg| > + (reg between: ESP and: EBP) ifFalse: > + [realReg = reg > + ifTrue: [cogit AddCq: 4 R: ESP] > + ifFalse: [cogit PopR: reg]]]. > - realReg = reg ifTrue: [cogit AddCq: 4 R: ESP] ifFalse: [cogit PopR: reg]]. > ^0! > > Item was removed: > - ----- Method: CogIA32Compiler>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - > - self assert: (EDI > EAX and: [EDI - EAX + 1 = 8]). > - EDI to: EAX by: -1 do: > - [:reg| > - (reg between: ESP and: EBP) ifFalse: > - [cogit PushR: reg]]. > - ^0! > > Item was added: > + ----- Method: CogIA32Compiler>>genSaveRegsForCCall (in category 'abi') ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a call into the C run-time from a trampoline." > + > + self assert: (EDI > EAX and: [EDI - EAX + 1 = 8]). > + EDI to: EAX by: -1 do: > + [:reg| > + (reg between: ESP and: EBP) ifFalse: > + [cogit PushR: reg]]. > + ^0! > > Item was removed: > - ----- Method: CogMIPSELCompiler>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "This method is poorly named. Is this for a Smalltalk -> C call or C -> Smalltalk call? > - If the former we don't need to do anything because all of the abstract registers are > - allocated to C preserved registers." > - self flag: #bogus.! > > Item was added: > + ----- Method: CogMIPSELCompiler>>genSaveRegsForCCall (in category 'abi') ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a call into the C run-time from a trampoline. > + We don't need to do anything because all of the abstract registers are > + allocated to C preserved registers." > + self flag: 'this will change with Sista when we hope to be able to allocate arbitrary registers'! > > Item was changed: > ----- Method: CogX64Compiler>>genRestoreRegs (in category 'abi') ----- > genRestoreRegs > "Restore the general purpose registers for a trampoline call. > c.f. genSaveRegisters" > + RAX to: R15 do: > + [:reg| > + (reg between: RSP and: RBP) ifFalse: > + [cogit PopR: reg]]. > - cogit > - PopR: RAX; > - PopR: RBX; > - PopR: RCX; > - PopR: RDX; > - PopR: RSI; > - PopR: RDI; > - PopR: R8; > - PopR: R9; > - PopR: R10; > - PopR: R11; > - PopR: R12; > - PopR: R13; > - PopR: R14; > - PopR: R15. > ^0! > > Item was changed: > ----- Method: CogX64Compiler>>genRestoreRegsExcept: (in category 'abi') ----- > genRestoreRegsExcept: abstractReg > | realReg | > realReg := self concreteRegister: abstractReg. > self assert: (R15 > RAX and: [R15 - RAX + 1 = 16]). > RAX to: R15 do: > [:reg| > + (reg between: RSP and: RBP) ifFalse: > + [realReg = reg > + ifTrue: [cogit AddCq: 8 R: RSP] > + ifFalse: [cogit PopR: reg]]]. > - realReg = reg ifTrue: [cogit AddCq: 4 R: RSP] ifFalse: [cogit PopR: reg]]. > ^0! > > Item was removed: > - ----- Method: CogX64Compiler>>genSaveRegisters (in category 'abi') ----- > - genSaveRegisters > - "Save the general purpose registers for a trampoline call." > - > - self assert: (R15 > RAX and: [R15 - RAX + 1 = 16]). > - R15 to: RAX by: -1 do: [:reg| cogit PushR: reg]. > - ^0! > > Item was added: > + ----- Method: CogX64Compiler>>genSaveRegsForCCall (in category 'abi') ----- > + genSaveRegsForCCall > + "Save the general purpose registers for a trampoline call." > + > + self assert: (R15 > RAX and: [R15 - RAX + 1 = 16]). > + R15 to: RAX by: -1 do: > + [:reg| > + (reg between: RSP and: RBP) ifFalse: > + [cogit PushR: reg]]. > + ^0! > > Item was changed: > ----- Method: Cogit>>compileCallFor:numArgs:arg:arg:arg:arg:resultReg:saveRegs: (in category 'initialization') ----- > compileCallFor: aRoutine numArgs: numArgs arg: regOrConst0 arg: regOrConst1 arg: regOrConst2 arg: regOrConst3 resultReg: resultRegOrNone saveRegs: saveRegs > "Generate a call to aRoutine with up to 4 arguments. If resultRegOrNone is not > NoReg assign the C result to resultRegOrNone. If saveRegs, save all registers. > Hack: a negative arg value indicates an abstract register, a non-negative value > indicates a constant." > <var: #aRoutine type: #'void *'> > <inline: false> > cStackAlignment > objectMemory wordSize ifTrue: > [backEnd > genAlignCStackSavingRegisters: saveRegs > numArgs: numArgs > wordAlignment: cStackAlignment / objectMemory wordSize]. > saveRegs ifTrue: > + [backEnd genSaveRegsForCCall]. > - [backEnd genSaveRegisters]. > backEnd genMarshallNArgs: numArgs arg: regOrConst0 arg: regOrConst1 arg: regOrConst2 arg: regOrConst3. > self CallFullRT: (self cCode: [aRoutine asUnsignedInteger] > inSmalltalk: [self simulatedTrampolineFor: aRoutine]). > resultRegOrNone ~= NoReg ifTrue: > [backEnd genWriteCResultIntoReg: resultRegOrNone]. > saveRegs ifTrue: > [numArgs > 0 ifTrue: > [backEnd genRemoveNArgsFromStack: numArgs]. > resultRegOrNone ~= NoReg > ifTrue: [backEnd genRestoreRegsExcept: resultRegOrNone] > ifFalse: [backEnd genRestoreRegs]]! > > > > > -- > _,,,^..^,,,_ > best, Eliot -- www.tudorgirba.com www.feenk.com "We cannot reach the flow of things unless we let go."

Dec. 20, 2015

Re: [Pharo-dev] [Vm-dev] VM Maker: VMMaker.oscog-eem.1609.mcz

by stepharo

Super !!! Eliot what will be 1 class maxval then? Stef Le 17/12/15 20:12, Eliot Miranda a Ã©crit : > Hi All, > > the real Cog 64-bit Spur x64 VM just evaluated 3+4 correctly on Mac > OS X: >

Dec. 20, 2015

Re: [Pharo-dev] SecureSession frame types

by Robert Withers

Excuse me for failing to complete this email./ I wanted to ask for feedback on these structures, especially the header specification word structure. So these fields make sense? thank you, robert On 12/20/2015 05:27 AM, Robert Withers wrote: > I wrote up the various SecureSession frame types in presentation > document, which you can get here: > https://www.dropbox.com/s/3xfa5oqs8ihj88v/FrameTypes.pptx?dl=0 > > These are not yet implemented, but it is progressing. Ne quid nemis. > > thanks, -- . .. .. ^,^ robert

Dec. 20, 2015

SecureSession frame types

by Robert Withers

I wrote up the various SecureSession frame types in presentation document, which you can get here: https://www.dropbox.com/s/3xfa5oqs8ihj88v/FrameTypes.pptx?dl=0 These are not yet implemented, but it is progressing. Ne quid nemis. thanks, -- . .. .. ^,^ robert

Dec. 20, 2015

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

by Dimitris Chloupis

not really because as I said UnixProcess already has an implementation for doing this, you dont need any of those protocols/libraries you mentioned, it does not even need Fuel. It already works. And the reason why I mentioned Python multiprocessing module is because multiprocessing does not only implement a specific way of communication between processes its also handle concurrency issues like data synchronization and much the of messy problems that come with concurrency. And this is the are of improvement, its not what kind of format the communication is , as you said it should be up to yourself how to format your date, maybe you want json, maybe fuel, maybe you have your object and classes. The real focus here is true concurrency issues on may experience because you will be having two image running alongside with no guarantess when one finish on action and framework to synchronize data. Obviously one can implement his own but I think Pharo would greatly benefit to have an establish default framework , even a small one for OS-Process concurency. On Sun, Dec 20, 2015 at 11:09 AM Pierce Ng <pierce(a)samadhiweb.com> wrote: > On Sun, Dec 20, 2015 at 08:56:43AM +0000, Dimitris Chloupis wrote: > > Fuel I assume enters here the equation as a data exchange format , the > > problem I have with fuel is that its not backward compatible which for me > > Once you have a set of OS processes running Pharo how they talk to each > other > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, > JSONRPC, > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc. > > Pierce > >

Dec. 20, 2015

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

by Pierce Ng

On Sun, Dec 20, 2015 at 08:56:43AM +0000, Dimitris Chloupis wrote: > Fuel I assume enters here the equation as a data exchange format , the > problem I have with fuel is that its not backward compatible which for me Once you have a set of OS processes running Pharo how they talk to each other is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, JSONRPC, msgpack, coordinate via Redis/SQLite/PostgreSQL, etc. Pierce

Dec. 20, 2015

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

by Dimitris Chloupis

I mentions this because in OSProcess-Unix there is UnixProcess class, at the class side there are several examples that launch a separate instance of Squeak/Pharo and make the two communicate with each other. In the instance of the class there are methods like forkHeadlessSqueak , forkHeadlessSqueakAndDo: etc. So the functionality is there but since Mariano said he will be working to improve OSProcess I thought that would a very useful area to work on. Since not only will allows running on multiple cores but even to lift any memory limits. Also I think this is a nice way to make the pharo enviroment which I mean the running vm with an image and satelite files not only a collection of objects but also a object itself that can communicate with other pharo enviroment instances (Pharo running processes) . Fuel I assume enters here the equation as a data exchange format , the problem I have with fuel is that its not backward compatible which for me is a very big deal especially when we talk about threading and processes which is a pretty fundamental feature. We need something that will be stable but also code orientated that will remain backward compatible because the last thing you want is initiating multiple processes that cant understand each other because they happen to use diffirent image versions that use diffirent verrsions of Fuel or what else. On Sun, Dec 20, 2015 at 1:46 AM David T. Lewis <lewis(a)mail.msen.com> wrote: > Mariano, > > I am really happy to see this and I look forword to helping you in > any way I can. > > Dimitris, > > It is really interesting that you should suggest this. I think it may > be a bit off topic for Mariano's initial goals, but Mariano is the > driving force behind Fuel, which is the serializer that I use for > RemoteTask in OSProcess/CommandShell. This attempts to do much of > what you suggest, using Fuel to return the results of processing > executed in a forked image (in the unix sense of "forked", i.e. > extremely memory efficient). > > So maybe if I can help Mariano with his project (which of course I > plan to do anyway), then he might be able to help me turn RemoteTask > into something reliable enough for general purpose use. > > Another nice connection here is that Eliot's Spur image model will > make #forkSqueak (the primitive behind RemoteTask) more efficient > in cases where the cooperating images do a lot of processing, because > memory changes are more localized in the Spur memory model. In principle > it should be possible to run very large object memories in 64-bit Spur, > while forking cooperating images that consume very little additional > real memory. > > Dave > > > On Sat, Dec 19, 2015 at 01:33:07PM +0000, Dimitris Chloupis wrote: > > I am very excited for such a project. > > > > I hope you manage to fix the stdout problem on macos. > > > > One thing I would like to see is an expansion of the ability of OSProcess > > to launch multiple instances of pharo and make them communicate with each > > other. Thats a way to have pharo running on multiple cores. Maybe > establish > > a framework of communication of images that will be based on the existing > > design of the fork mechanism in Pharo. > > > > Python is doing something similar with multiprocessing module which > > launches multiple processes of cpython executable and makes them > > communication with each other through a similar api to its threading > > module. This way python takes advantage of multiple cores. Python threads > > like pharo are green threads and pharo like python has a GIL like > > mechanism. > > > > Taking into account that current computers have at least 4 cores, its > not a > > bad idea to speed up pharo like this at least 400% ;) > > > > On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck < > marianopeck(a)gmail.com> > > wrote: > > > > > Dear all, > > > > > > I am tremendously happy to announce you that Pharo Consortium will > sponsor > > > yet another development effort. In this particular case, it's my honor > to > > > carry on such an effort and I will be developing and improving a few > things > > > and ideas. All the contract and paperwork has already been done so it's > > > time to start working. > > > > > > Regarding the developments itself, we discussed about these topics: > > > > > > 1) Experiment with a simpler yet more limited OSProcess alternative to > > > execute OS commands. > > > 2) Improving FileSystem in order to better deal with some POSIX stuff > like > > > symbolic links, unix file permissions, etc etc. > > > > > > I have already started with 1). OSProcess is super complete and it > > > involves some packages (OSProcess / CommandShell) as well as some VM > > > plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of > features > > > (like forking the running image) but we would like to focus only in > > > executing OS commands. In addition, OSProcess dates from quite some > years > > > ago when many of the current infrastructure features did not exist yet. > > > > > > So the idea is to think a simpler alternative to execute OS commands, > > > using new features such as threaded FFI (for example, for reading async > > > from pipes), pinned objects, etc. We want to use FFI as much as > possible > > > rather than VM plugins. > > > > > > From the image side, we are thinking about an API which would look > like a > > > builder-like API (FileSystem, XStream, etc). > > > > > > We have been discussing with David Lewis (OSProcess author) as well as > > > with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane > Ducasse, etc > > > in order to agree in a project that would be worth doing (otherwise we > > > would continue using existing tools). And they have all been very kind > with > > > me providing positive discussions. > > > > > > I will soon do a survey to measure the most common use cases of > OSProcess > > > and related tools. > > > > > > If you have any thought, question, feedback, or whatever, please let us > > > know. > > > > > > Finally, note that I will be doing this project together with another > > > client's project (Quuve) and my life does not have much free time these > > > days...so please be patient!!! > > > > > > Best, > > > > > > -- > > > Mariano > > > http://marianopeck.wordpress.com > > > > >

Dec. 20, 2015

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

by David T. Lewis

Mariano, I am really happy to see this and I look forword to helping you in any way I can. Dimitris, It is really interesting that you should suggest this. I think it may be a bit off topic for Mariano's initial goals, but Mariano is the driving force behind Fuel, which is the serializer that I use for RemoteTask in OSProcess/CommandShell. This attempts to do much of what you suggest, using Fuel to return the results of processing executed in a forked image (in the unix sense of "forked", i.e. extremely memory efficient). So maybe if I can help Mariano with his project (which of course I plan to do anyway), then he might be able to help me turn RemoteTask into something reliable enough for general purpose use. Another nice connection here is that Eliot's Spur image model will make #forkSqueak (the primitive behind RemoteTask) more efficient in cases where the cooperating images do a lot of processing, because memory changes are more localized in the Spur memory model. In principle it should be possible to run very large object memories in 64-bit Spur, while forking cooperating images that consume very little additional real memory. Dave On Sat, Dec 19, 2015 at 01:33:07PM +0000, Dimitris Chloupis wrote: > I am very excited for such a project. > > I hope you manage to fix the stdout problem on macos. > > One thing I would like to see is an expansion of the ability of OSProcess > to launch multiple instances of pharo and make them communicate with each > other. Thats a way to have pharo running on multiple cores. Maybe establish > a framework of communication of images that will be based on the existing > design of the fork mechanism in Pharo. > > Python is doing something similar with multiprocessing module which > launches multiple processes of cpython executable and makes them > communication with each other through a similar api to its threading > module. This way python takes advantage of multiple cores. Python threads > like pharo are green threads and pharo like python has a GIL like > mechanism. > > Taking into account that current computers have at least 4 cores, its not a > bad idea to speed up pharo like this at least 400% ;) > > On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck <marianopeck(a)gmail.com> > wrote: > > > Dear all, > > > > I am tremendously happy to announce you that Pharo Consortium will sponsor > > yet another development effort. In this particular case, it's my honor to > > carry on such an effort and I will be developing and improving a few things > > and ideas. All the contract and paperwork has already been done so it's > > time to start working. > > > > Regarding the developments itself, we discussed about these topics: > > > > 1) Experiment with a simpler yet more limited OSProcess alternative to > > execute OS commands. > > 2) Improving FileSystem in order to better deal with some POSIX stuff like > > symbolic links, unix file permissions, etc etc. > > > > I have already started with 1). OSProcess is super complete and it > > involves some packages (OSProcess / CommandShell) as well as some VM > > plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of features > > (like forking the running image) but we would like to focus only in > > executing OS commands. In addition, OSProcess dates from quite some years > > ago when many of the current infrastructure features did not exist yet. > > > > So the idea is to think a simpler alternative to execute OS commands, > > using new features such as threaded FFI (for example, for reading async > > from pipes), pinned objects, etc. We want to use FFI as much as possible > > rather than VM plugins. > > > > From the image side, we are thinking about an API which would look like a > > builder-like API (FileSystem, XStream, etc). > > > > We have been discussing with David Lewis (OSProcess author) as well as > > with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane Ducasse, etc > > in order to agree in a project that would be worth doing (otherwise we > > would continue using existing tools). And they have all been very kind with > > me providing positive discussions. > > > > I will soon do a survey to measure the most common use cases of OSProcess > > and related tools. > > > > If you have any thought, question, feedback, or whatever, please let us > > know. > > > > Finally, note that I will be doing this project together with another > > client's project (Quuve) and my life does not have much free time these > > days...so please be patient!!! > > > > Best, > > > > -- > > Mariano > > http://marianopeck.wordpress.com > >

Dec. 19, 2015

Re: [Pharo-dev] [ANN] Pharo Consortium Sponsored Development Effort

by Torsten Bergmann

Dec. 19, 2015