- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Fun with AST evaluation

by ClÃ©ment Bera

Starting from here you can build a full AST interpreter that mutates a context instance that you create by interpreting MessageNode... On Tue, Apr 17, 2018 at 5:45 PM, Marcus Denker <marcus.denker(a)inria.fr> wrote: > Hi, > > This is fun experiment: > > Every RBValueNode subclass evaluates to something. So why not implement a > #evaluate method? > > This is especially fun to do for a specific context, as this means that > you could evaluate every RBVariableNode to see the value. > > So lets implement a method in RBValueNode: > > evaluateWithContext: aContext > > | methodNode | > > If we want to compile a method for it, we need to wrap it into a return > and a method. Like when evaluating in the debugger, we create a #DoItIn: > method that > will get the context as a parameter to read the values from (temps from > the context, ivars from context receiver). > > This works like this: > > methodNode := RBMethodNode > selector: #DoItIn: > arguments: { RBVariableNode named: 'ThisContext' } > body: (RBReturnNode value: self) asSequenceNode. > > to read temps from the context parameter, we use the rewrite method that > is used by doits already: > methodNode rewriteTempsForContext: aContext. > > and now we can execute: > > ^methodNode generate valueWithReceiver: aContext receiver arguments: > {aContext}. > > For testing, I put a #haltOnce into Rectangle>>#area and then use the > inspector on the context to evaluate a AST node that is a temporary > variable. > The screenshot shows executing > > self method ast assignmentNodes first variable evaluateWithContext: self > > and it returns correctly 104: > > > > > Of course there are simpler ways to read variables by name from a context > (e.g. we added #lookupSymbol: ), but #evaluateWithContext: can be called on > other Nodes, too. > E.g. just a #evaluate could be nice for creating Blocks programmatically > (instead of calling the compiler on a Pretty-Print result like > #createBlockFor: in the ParseTreeSearcher). > > A version that does not take context and receiver into account is this: > > evaluate > | methodNode | > methodNode := RBMethodNode > selector: #DoIt > body: (RBReturnNode value: self) asSequenceNode. > ^methodNode generate valueWithReceiver: nil arguments: #(). > > > and then you can do things like > > blockNode: = RBBlockNode > body: (RBLiteralNode value: 5) asSequenceNode > > blockNode evaluate > > to create blocks. > > > Marcus > > -- ClÃ©ment BÃ©ra https://clementbera.github.io/ https://clementbera.wordpress.com/

April 17, 2018

Fun with AST evaluation

by Marcus Denker

Hi, This is fun experiment: Every RBValueNode subclass evaluates to something. So why not implement a #evaluate method? This is especially fun to do for a specific context, as this means that you could evaluate every RBVariableNode to see the value. So lets implement a method in RBValueNode: evaluateWithContext: aContext | methodNode | If we want to compile a method for it, we need to wrap it into a return and a method. Like when evaluating in the debugger, we create a #DoItIn: method that will get the context as a parameter to read the values from (temps from the context, ivars from context receiver). This works like this: methodNode := RBMethodNode selector: #DoItIn: arguments: { RBVariableNode named: 'ThisContext' } body: (RBReturnNode value: self) asSequenceNode. to read temps from the context parameter, we use the rewrite method that is used by doits already: methodNode rewriteTempsForContext: aContext. and now we can execute: ^methodNode generate valueWithReceiver: aContext receiver arguments: {aContext}. For testing, I put a #haltOnce into Rectangle>>#area and then use the inspector on the context to evaluate a AST node that is a temporary variable. The screenshot shows executing self method ast assignmentNodes first variable evaluateWithContext: self and it returns correctly 104: Of course there are simpler ways to read variables by name from a context (e.g. we added #lookupSymbol: ), but #evaluateWithContext: can be called on other Nodes, too. E.g. just a #evaluate could be nice for creating Blocks programmatically (instead of calling the compiler on a Pretty-Print result like #createBlockFor: in the ParseTreeSearcher). A version that does not take context and receiver into account is this: evaluate | methodNode | methodNode := RBMethodNode selector: #DoIt body: (RBReturnNode value: self) asSequenceNode. ^methodNode generate valueWithReceiver: nil arguments: #(). and then you can do things like blockNode: = RBBlockNode body: (RBLiteralNode value: 5) asSequenceNode blockNode evaluate to create blocks. Marcus

April 17, 2018

[Pharo 7.0-dev] Build #776: 21722-Small-Cleanup-Syntax-highlighting-remove-a-reference-to-SHParserST80

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #776 was: SUCCESS. The Pull Request #1210 was integrated: "21722-Small-Cleanup-Syntax-highlighting-remove-a-reference-to-SHParserST80" Pull request url: https://github.com/pharo-project/pharo/pull/1210 Issue Url: https://pharo.fogbugz.com/f/cases/21722 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

April 17, 2018

Re: [Pharo-dev] Pavel's ChangeLog week of 2018-02-05

by Thierry Goubier

2018-04-17 14:28 GMT+02:00 ClÃ©ment Bera <bera.clement(a)gmail.com>: > > > On Tue, Apr 17, 2018 at 2:08 PM, Thierry Goubier <thierry.goubier(a)gmail.com> > wrote: >> >> 2018-04-17 14:03 GMT+02:00 ClÃ©ment Bera <bera.clement(a)gmail.com>: >> > Hi Pavel, >> > >> > I'm looking at PosixSharedMemory again since I have to write a Master >> > student proposal and I think this could be a good topic. I'm not really >> > expert on SharedMemory, so I'm going to share what I have in mind, we >> > can >> > discuss about it, you could also co-supervise the student indirectly >> > (though >> > officially the supervisor has to be VUB staff). >> > >> > Implementation-wise, I was thinking what could be interesting are: >> > - Making PosixSharedMemory compatible with TaskIt so from you image you >> > can >> > create a SharedMemory buffer, spawn another image+VM and attach it to >> > the >> > shared memory to have multiple threads working on the shared section. >> > - Add the #at:if:put: primitive, which write into the shared memory >> > using >> > compare and swap instruction for efficient thread-safe access. >> > - Add on SharedMemory all the primitives to read/write native types to >> > the >> > buffer (int64, double, etc.) with CAS and non CAS instructions. >> > - Maybe add APIs to read/write objects through Fuel to pass them by >> > copy, >> > though this looks difficult in some cases. >> > - Implement a lock system or a semaphore system on top of the CAS >> > - implement lock-free and lock-full algorithm using CAS and non CAS >> > instructions (I think a first try would be parallelSort on a 1Gb buffer >> > of >> > int32 with 4 native threads) >> > >> > What do you think ? Do you have ideas ? Are you interested ? Do you >> > thing >> > having a student on this would be nice ? >> > >> > The master thesis proposal has to include a research question. I am not >> > sure >> > what other languages do regarding shared memory.It's not clear so far >> > what >> > the research question is. >> >> I can forward that to a researcher here working on distributed shared >> memory. >> >> Research question can be: >> - heterogeneity (x86 + pi at the same time) >> - load balancing between competing images on heterogeneous hardware >> ... >> - Object migration (pointer forwarding). Probably not, state of the >> art is very advanced on that which means a costly implementation to >> reach parity. >> > > SharedMemory on heterogenous hardware ? Do you mean you need to physically > plug the memory into a Raspberry Pie and an x86 computer ? Or you mean > exporting the RAM of one hardware as NFS to the others ? I was just thinking > sharing the memory between multiple pairs of image+VM on the same machine to > be able to run some multi-threaded algorithm on the shared buffer. I know > it's not much but we need to start somewhere. No, it's having a memory abstraction (memory chunks) handled by a server on a host (x86 or ARM), sending them over MPI to clients (x86 or ARM), each client accessing it through the OS shared memory, releasing it when done so that other tasks can work on it (distributed pipeline). > The student would have 6 months, including 1 month to write the thesis so it > cannot be too heavy. Something that we can re-use with a minor research > contribution would be nice. Given the state of the art in the field (20 years + of shared memory + distributed shared memory + distributed object store and migration already done), the only easy one that I know of is heterogeneity. And even there, doing something worthy of a paper is hard. Thierry >> Thierry >> >> > >> > Best, >> > >> > On Mon, Feb 12, 2018 at 10:33 AM, Pavel Krivanek >> > <pavel.krivanek(a)gmail.com> >> > wrote: >> >> >> >> Hi, >> >> >> >> among other less interesting things, I spent some time on existing >> >> PosixSharedMemory project. It is a UFFI binding for the LibC methods >> >> that provide support for the memory allocation between several >> >> separate processes. I significantly improved the performance by >> >> implementing the block access. Writing of 10MB byte array takes about >> >> 1 millisecond, reading of it from other image took me about 4 >> >> milliseconds. While serialization with Fuel is very fast, it opens >> >> interesting possibilities. >> >> To have a shared memory without synchronization tools is not very >> >> useful so I wrote a basic UFFI interface for the POSIX named >> >> semaphores. They are quite easy to use and work nicely with Pharo. The >> >> VM can all wait on the semaphore or it can check the status of it >> >> periodically in an image thread. It has two small disadvantages. It >> >> requires to dynamically link the next library (pthread) and they must >> >> be cleaned manually. I plan to look at System V alternative in future. >> >> Now we should write a nice framework for inter-image communication on >> >> top of it or/and adopt Seamless for it ;-) >> >> >> >> Cheers, >> >> -- Pavel >> >> >> > >> > >> > >> > -- >> > ClÃ©ment BÃ©ra >> > https://clementbera.github.io/ >> > https://clementbera.wordpress.com/ >> > > > > -- > ClÃ©ment BÃ©ra > https://clementbera.github.io/ > https://clementbera.wordpress.com/

April 17, 2018

Re: [Pharo-dev] Pavel's ChangeLog week of 2018-02-05

by ClÃ©ment Bera

On Tue, Apr 17, 2018 at 2:08 PM, Thierry Goubier <thierry.goubier(a)gmail.com> wrote: > 2018-04-17 14:03 GMT+02:00 ClÃ©ment Bera <bera.clement(a)gmail.com>: > > Hi Pavel, > > > > I'm looking at PosixSharedMemory again since I have to write a Master > > student proposal and I think this could be a good topic. I'm not really > > expert on SharedMemory, so I'm going to share what I have in mind, we can > > discuss about it, you could also co-supervise the student indirectly > (though > > officially the supervisor has to be VUB staff). > > > > Implementation-wise, I was thinking what could be interesting are: > > - Making PosixSharedMemory compatible with TaskIt so from you image you > can > > create a SharedMemory buffer, spawn another image+VM and attach it to the > > shared memory to have multiple threads working on the shared section. > > - Add the #at:if:put: primitive, which write into the shared memory using > > compare and swap instruction for efficient thread-safe access. > > - Add on SharedMemory all the primitives to read/write native types to > the > > buffer (int64, double, etc.) with CAS and non CAS instructions. > > - Maybe add APIs to read/write objects through Fuel to pass them by copy, > > though this looks difficult in some cases. > > - Implement a lock system or a semaphore system on top of the CAS > > - implement lock-free and lock-full algorithm using CAS and non CAS > > instructions (I think a first try would be parallelSort on a 1Gb buffer > of > > int32 with 4 native threads) > > > > What do you think ? Do you have ideas ? Are you interested ? Do you thing > > having a student on this would be nice ? > > > > The master thesis proposal has to include a research question. I am not > sure > > what other languages do regarding shared memory.It's not clear so far > what > > the research question is. > > I can forward that to a researcher here working on distributed shared > memory. > > Research question can be: > - heterogeneity (x86 + pi at the same time) > - load balancing between competing images on heterogeneous hardware > ... > - Object migration (pointer forwarding). Probably not, state of the > art is very advanced on that which means a costly implementation to > reach parity. > > SharedMemory on heterogenous hardware ? Do you mean you need to physically plug the memory into a Raspberry Pie and an x86 computer ? Or you mean exporting the RAM of one hardware as NFS to the others ? I was just thinking sharing the memory between multiple pairs of image+VM on the same machine to be able to run some multi-threaded algorithm on the shared buffer. I know it's not much but we need to start somewhere. The student would have 6 months, including 1 month to write the thesis so it cannot be too heavy. Something that we can re-use with a minor research contribution would be nice. Thierry > > > > > Best, > > > > On Mon, Feb 12, 2018 at 10:33 AM, Pavel Krivanek < > pavel.krivanek(a)gmail.com> > > wrote: > >> > >> Hi, > >> > >> among other less interesting things, I spent some time on existing > >> PosixSharedMemory project. It is a UFFI binding for the LibC methods > >> that provide support for the memory allocation between several > >> separate processes. I significantly improved the performance by > >> implementing the block access. Writing of 10MB byte array takes about > >> 1 millisecond, reading of it from other image took me about 4 > >> milliseconds. While serialization with Fuel is very fast, it opens > >> interesting possibilities. > >> To have a shared memory without synchronization tools is not very > >> useful so I wrote a basic UFFI interface for the POSIX named > >> semaphores. They are quite easy to use and work nicely with Pharo. The > >> VM can all wait on the semaphore or it can check the status of it > >> periodically in an image thread. It has two small disadvantages. It > >> requires to dynamically link the next library (pthread) and they must > >> be cleaned manually. I plan to look at System V alternative in future. > >> Now we should write a nice framework for inter-image communication on > >> top of it or/and adopt Seamless for it ;-) > >> > >> Cheers, > >> -- Pavel > >> > > > > > > > > -- > > ClÃ©ment BÃ©ra > > https://clementbera.github.io/ > > https://clementbera.wordpress.com/ > > -- ClÃ©ment BÃ©ra https://clementbera.github.io/ https://clementbera.wordpress.com/

April 17, 2018

Re: [Pharo-dev] Pavel's ChangeLog week of 2018-02-05

by Thierry Goubier

2018-04-17 14:03 GMT+02:00 ClÃ©ment Bera <bera.clement(a)gmail.com>: > Hi Pavel, > > I'm looking at PosixSharedMemory again since I have to write a Master > student proposal and I think this could be a good topic. I'm not really > expert on SharedMemory, so I'm going to share what I have in mind, we can > discuss about it, you could also co-supervise the student indirectly (though > officially the supervisor has to be VUB staff). > > Implementation-wise, I was thinking what could be interesting are: > - Making PosixSharedMemory compatible with TaskIt so from you image you can > create a SharedMemory buffer, spawn another image+VM and attach it to the > shared memory to have multiple threads working on the shared section. > - Add the #at:if:put: primitive, which write into the shared memory using > compare and swap instruction for efficient thread-safe access. > - Add on SharedMemory all the primitives to read/write native types to the > buffer (int64, double, etc.) with CAS and non CAS instructions. > - Maybe add APIs to read/write objects through Fuel to pass them by copy, > though this looks difficult in some cases. > - Implement a lock system or a semaphore system on top of the CAS > - implement lock-free and lock-full algorithm using CAS and non CAS > instructions (I think a first try would be parallelSort on a 1Gb buffer of > int32 with 4 native threads) > > What do you think ? Do you have ideas ? Are you interested ? Do you thing > having a student on this would be nice ? > > The master thesis proposal has to include a research question. I am not sure > what other languages do regarding shared memory.It's not clear so far what > the research question is. I can forward that to a researcher here working on distributed shared memory. Research question can be: - heterogeneity (x86 + pi at the same time) - load balancing between competing images on heterogeneous hardware ... - Object migration (pointer forwarding). Probably not, state of the art is very advanced on that which means a costly implementation to reach parity. Thierry > > Best, > > On Mon, Feb 12, 2018 at 10:33 AM, Pavel Krivanek <pavel.krivanek(a)gmail.com> > wrote: >> >> Hi, >> >> among other less interesting things, I spent some time on existing >> PosixSharedMemory project. It is a UFFI binding for the LibC methods >> that provide support for the memory allocation between several >> separate processes. I significantly improved the performance by >> implementing the block access. Writing of 10MB byte array takes about >> 1 millisecond, reading of it from other image took me about 4 >> milliseconds. While serialization with Fuel is very fast, it opens >> interesting possibilities. >> To have a shared memory without synchronization tools is not very >> useful so I wrote a basic UFFI interface for the POSIX named >> semaphores. They are quite easy to use and work nicely with Pharo. The >> VM can all wait on the semaphore or it can check the status of it >> periodically in an image thread. It has two small disadvantages. It >> requires to dynamically link the next library (pthread) and they must >> be cleaned manually. I plan to look at System V alternative in future. >> Now we should write a nice framework for inter-image communication on >> top of it or/and adopt Seamless for it ;-) >> >> Cheers, >> -- Pavel >> > > > > -- > ClÃ©ment BÃ©ra > https://clementbera.github.io/ > https://clementbera.wordpress.com/

April 17, 2018

Re: [Pharo-dev] Pavel's ChangeLog week of 2018-02-05

by ClÃ©ment Bera

Hi Pavel, I'm looking at PosixSharedMemory again since I have to write a Master student proposal and I think this could be a good topic. I'm not really expert on SharedMemory, so I'm going to share what I have in mind, we can discuss about it, you could also co-supervise the student indirectly (though officially the supervisor has to be VUB staff). Implementation-wise, I was thinking what could be interesting are: - Making PosixSharedMemory compatible with TaskIt so from you image you can create a SharedMemory buffer, spawn another image+VM and attach it to the shared memory to have multiple threads working on the shared section. - Add the #at:if:put: primitive, which write into the shared memory using compare and swap instruction for efficient thread-safe access. - Add on SharedMemory all the primitives to read/write native types to the buffer (int64, double, etc.) with CAS and non CAS instructions. - Maybe add APIs to read/write objects through Fuel to pass them by copy, though this looks difficult in some cases. - Implement a lock system or a semaphore system on top of the CAS - implement lock-free and lock-full algorithm using CAS and non CAS instructions (I think a first try would be parallelSort on a 1Gb buffer of int32 with 4 native threads) What do you think ? Do you have ideas ? Are you interested ? Do you thing having a student on this would be nice ? The master thesis proposal has to include a research question. I am not sure what other languages do regarding shared memory.It's not clear so far what the research question is. Best, On Mon, Feb 12, 2018 at 10:33 AM, Pavel Krivanek <pavel.krivanek(a)gmail.com> wrote: > Hi, > > among other less interesting things, I spent some time on existing > PosixSharedMemory project. It is a UFFI binding for the LibC methods > that provide support for the memory allocation between several > separate processes. I significantly improved the performance by > implementing the block access. Writing of 10MB byte array takes about > 1 millisecond, reading of it from other image took me about 4 > milliseconds. While serialization with Fuel is very fast, it opens > interesting possibilities. > To have a shared memory without synchronization tools is not very > useful so I wrote a basic UFFI interface for the POSIX named > semaphores. They are quite easy to use and work nicely with Pharo. The > VM can all wait on the semaphore or it can check the status of it > periodically in an image thread. It has two small disadvantages. It > requires to dynamically link the next library (pthread) and they must > be cleaned manually. I plan to look at System V alternative in future. > Now we should write a nice framework for inter-image communication on > top of it or/and adopt Seamless for it ;-) > > Cheers, > -- Pavel > > -- ClÃ©ment BÃ©ra https://clementbera.github.io/ https://clementbera.wordpress.com/

April 17, 2018

Re: [Pharo-dev] [Pharo 7.0-dev] Build #775: Revert "21429-Watchpoints-should-be-cleaned-when-system-changed"

by Marcus Denker

It is nice how easy a faulty merge can be reverted nowâ¦ > On 17 Apr 2018, at 13:33, ci-pharo-ci-jenkins2(a)inria.fr wrote: > > There is a new Pharo build available! > > The status of the build #775 was: SUCCESS. > > The Pull Request #1209 was integrated: "Revert "21429-Watchpoints-should-be-cleaned-when-system-changed"" > Pull request url: https://github.com/pharo-project/pharo/pull/1209 > > Issue Url: https://pharo.fogbugz.com/f/cases/revert > Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

April 17, 2018

[Pharo 7.0-dev] Build #775: Revert "21429-Watchpoints-should-be-cleaned-when-system-changed"

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #775 was: SUCCESS. The Pull Request #1209 was integrated: "Revert "21429-Watchpoints-should-be-cleaned-when-system-changed"" Pull request url: https://github.com/pharo-project/pharo/pull/1209 Issue Url: https://pharo.fogbugz.com/f/cases/revert Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

April 17, 2018

[Pharo 7.0-dev] Build #774: 21429-Watchpoints-should-be-cleaned-when-system-changed

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #774 was: FAILURE. The Pull Request #1162 was integrated: "21429-Watchpoints-should-be-cleaned-when-system-changed" Pull request url: https://github.com/pharo-project/pharo/pull/1162 Issue Url: https://pharo.fogbugz.com/f/cases/21429 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

April 17, 2018