[Pharo-project] Opal Decompiler status
I see that the OCFakeDecompiler which uses the old and horrible (*) Decompiler is still in use. Though, IRBytecodeDecompiler seems already quite advanced. Could we have a primer on what remains to do on the Opal front? (*) just look number of inst vars, count how many times their state change, and measure average method length for an approximate definition of horrible
On Apr 27, 2013, at 5:26 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
I see that the OCFakeDecompiler which uses the old and horrible (*) Decompiler is still in use. Though, IRBytecodeDecompiler seems already quite advanced.
The idea actually is to not have a decompiler BC->AST. We have one BC->IR for manipulating things on the byte code level (e.g. he new class builder will use it). BC->IR->BC is working and tested after every commit. But on the AST level, idea is to have a higher level representation of code always available. (compressed ASTs in the long term, compressed source as the intermediate solution). Yes, I know that people will be highly skeptical about this. Up to the point that I even don't expect to convince anyone other than by actually showing a working version⦠This goes hand-in-hand with the "one file" image: just one .pharo image, no .sources, no .changes. (Which real conservative people will not like, either). We need to see how fast this progresses, if we see that it will not be ready for 3.0, we should add a IR->AST decompiler, but I hope the alternative will show that it is not necessary.
Could we have a primer on what remains to do on the Opal front?
-> debug BC-IR->AST->text mapping more (lots of it working already) -> debug temp variable lookup in optimized blocks in the debugger (almost there, too) -> Go through all the test for the old Compiler and port them to the new. -> Better compiler API and refactor the whole system to use it (but keep a backward compatible API at least to some extend) Marcus
On Apr 27, 2013, at 5:59 PM, Marcus Denker <marcus.denker@inria.fr> wrote:.
Could we have a primer on what remains to do on the Opal front?
-> debug BC-IR->AST->text mapping more (lots of it working already) -> debug temp variable lookup in optimized blocks in the debugger (almost there, too) -> Go through all the test for the old Compiler and port them to the new. -> Better compiler API and refactor the whole system to use it (but keep a backward compatible API at least to some extend)
All the TODOs are in the bug tracker, Project "Compiler (Opal)". There are right now 29 Issues. Marcus
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)? 2013/4/27 Marcus Denker <marcus.denker@inria.fr>
On Apr 27, 2013, at 5:59 PM, Marcus Denker <marcus.denker@inria.fr> wrote:.
Could we have a primer on what remains to do on the Opal front?
-> debug BC-IR->AST->text mapping more (lots of it working already) -> debug temp variable lookup in optimized blocks in the debugger
(almost there, too)
-> Go through all the test for the old Compiler and port them to the new. -> Better compiler API and refactor the whole system to use it (but keep a backward compatible API at least to some extend)
All the TODOs are in the bug tracker, Project "Compiler (Opal)".
There are right now 29 Issues.
Marcus
On Apr 27, 2013, at 6:42 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)?
For the Debugger, everything stays as it is (until Pharo 4 or so)⦠if you look at it, the current Debugger never decompiles if there is source available. It *compiles* to get -> the mapping BC -> text -> the information how temporary variables in the byte code are actually related to temps in the source. (this is very complex, temps can be e.g. stored on the heap if they are written to in a closure, and when they are just read they have a different offset in each closure they are in. Conversely, some temps in the byte code are used to store the array that hold the variables that are on the heapâ¦) The old compiler recorded mappings while compiling that where encoded⦠quite complex, at least for my tiny brain. So the new just keeps the AST annotated with all the needed infos, this is much easier to debug (and we do have the memory these days, and as we cache the AST, it's even fast). So the debugger needs the compiler. The decompiler now exists just to make text so that we can call the compiler in the case there is no .sources. The debugger *always* compiles to get the mappings, as soon as there is source code the decompiler will never be used. (and even if the decompiler is used, the compiler is called right after on it's results so one can record the mappings). So if you make sure there is always source-code (or a representation with the right amount of meta data), you don't need the decompiler. So at the start it will be the source code. And yes, this takes memory. But we are in 2013 and if we have one thing than it's memory. (the $25 Raspi comes with 256MB, the $35 with 512MBâ¦). We could have shipped 2.0 with 8MB of useless Monticello meta data and nobody would have even realized it. (like we have now megabytes of fonts in the imageâ¦). Yet the source is special⦠I really wonder why. (And yes, there should be solutions to not need unused data in main memory and solutions to share across multiple images data that is the same⦠but for all kinds of stuff, not just source code). Marcus
Right, if you embed source along with CompiledMethod (to handle the case when the CompiledMethod has been replaced but one of its block of code is still used), then you don't need Decompiler in the first place. Nonetheless, I would remove the .sources but not the .changes. The .changes is not only a source code repository (that feature can go away if you want), it's before all a change log, an insurance to retrieve some change when everything else failed (image crashed or worse corrupted image won't restart). You can make the insurance optional if you want, but please continue to provide an equivalent service. A change file is the most simple thing that could possibly work IMO. But this is another thread. Concerning the temp mapping, I agree, we should not compete with Eliot, he's very tough at tedious tasks, our only weapon is lazyness, lazyness often leads to efficiency ;) I'm curious to check how you handled this complex part with AST when I'll have more time to dig. 2013/4/27 Marcus Denker <marcus.denker@inria.fr>
On Apr 27, 2013, at 6:42 PM, Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote:
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)?
For the Debugger, everything stays as it is (until Pharo 4 or so)⦠if you look at it, the current Debugger never decompiles if there is source available. It *compiles* to get -> the mapping BC -> text -> the information how temporary variables in the byte code are actually related to temps in the source. (this is very complex, temps can be e.g. stored on the heap if they are written to in a closure, and when they are just read they have a different offset in each closure they are in. Conversely, some temps in the byte code are used to store the array that hold the variables that are on the heapâ¦)
The old compiler recorded mappings while compiling that where encoded⦠quite complex, at least for my tiny brain.
So the new just keeps the AST annotated with all the needed infos, this is much easier to debug (and we do have the memory these days, and as we cache the AST, it's even fast).
So the debugger needs the compiler. The decompiler now exists just to make text so that we can call the compiler in the case there is no .sources. The debugger *always* compiles to get the mappings, as soon as there is source code the decompiler will never be used. (and even if the decompiler is used, the compiler is called right after on it's results so one can record the mappings).
So if you make sure there is always source-code (or a representation with the right amount of meta data), you don't need the decompiler.
So at the start it will be the source code. And yes, this takes memory. But we are in 2013 and if we have one thing than it's memory. (the $25 Raspi comes with 256MB, the $35 with 512MBâ¦).
We could have shipped 2.0 with 8MB of useless Monticello meta data and nobody would have even realized it. (like we have now megabytes of fonts in the imageâ¦). Yet the source is special⦠I really wonder why.
(And yes, there should be solutions to not need unused data in main memory and solutions to share across multiple images data that is the same⦠but for all kinds of stuff, not just source code).
Marcus
On Apr 27, 2013, at 8:01 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Right, if you embed source along with CompiledMethod (to handle the case when the CompiledMethod has been replaced but one of its block of code is still used), then you don't need Decompiler in the first place.
Nonetheless, I would remove the .sources but not the .changes. The .changes is not only a source code repository (that feature can go away if you want), it's before all a change log, an insurance to retrieve some change when everything else failed (image crashed or worse corrupted image won't restart). You can make the insurance optional if you want, but please continue to provide an equivalent service. A change file is the most simple thing that could possibly work IMO. But this is another thread.
Yes, we have a solution for that⦠the thing is that the .changes has 3 roles. In the 1978, it was very clever to design one extremely memory efficient mechanism to kill all three: 1) keep a history of all the accepted method sources in the image (see "versions") 2) store the code on disk efficiently with no duplications (They counted those extremely expensive disks in the one-diget Megabytes back then). 3) have transaction log for the source code in case of image crash. Now today, we can solve all three using three different methods: 1) we will just keep the accepted methods in memory until you commit to monticello/git (in case of git one can even commit all the intermediates, for MC one would just commit the latest). after the in-image history is pruned (like it is now eventually anyway when you start with a fresh image). 2) We don't need to store the *current* version on disk. Compressed in memory. Yes, that is what progress brings⦠3) Disk space is so cheap it is not even funny. -> we will just have a dedicated log. And we can even store Objects⦠And then we can leverage the extreme simplicity of the resulting scheme.. simplicity is an amazing thing. (Or the contrary: I think people understate the effects of complexity). Marcus
But I would like that we finish first all the other cool tasks for 3.0 before opening something new. Finishing is difficult so let us focus on it. Stef On Apr 27, 2013, at 8:13 PM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Apr 27, 2013, at 8:01 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Right, if you embed source along with CompiledMethod (to handle the case when the CompiledMethod has been replaced but one of its block of code is still used), then you don't need Decompiler in the first place.
Nonetheless, I would remove the .sources but not the .changes. The .changes is not only a source code repository (that feature can go away if you want), it's before all a change log, an insurance to retrieve some change when everything else failed (image crashed or worse corrupted image won't restart). You can make the insurance optional if you want, but please continue to provide an equivalent service. A change file is the most simple thing that could possibly work IMO. But this is another thread.
Yes, we have a solution for thatâ¦
the thing is that the .changes has 3 roles. In the 1978, it was very clever to design one extremely memory efficient mechanism to kill all three:
1) keep a history of all the accepted method sources in the image (see "versions") 2) store the code on disk efficiently with no duplications (They counted those extremely expensive disks in the one-diget Megabytes back then). 3) have transaction log for the source code in case of image crash.
Now today, we can solve all three using three different methods:
1) we will just keep the accepted methods in memory until you commit to monticello/git (in case of git one can even commit all the intermediates, for MC one would just commit the latest). after the in-image history is pruned (like it is now eventually anyway when you start with a fresh image). 2) We don't need to store the *current* version on disk. Compressed in memory. Yes, that is what progress brings⦠3) Disk space is so cheap it is not even funny. -> we will just have a dedicated log. And we can even store Objectsâ¦
And then we can leverage the extreme simplicity of the resulting scheme.. simplicity is an amazing thing. (Or the contrary: I think people understate the effects of complexity).
Marcus
On Apr 27, 2013, at 8:00 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Right, if you embed source along with CompiledMethod (to handle the case when the CompiledMethod has been replaced but one of its block of code is still used), then you don't need Decompiler in the first place.
Nonetheless, I would remove the .sources but not the .changes. The .changes is not only a source code repository (that feature can go away if you want), it's before all a change log, an insurance to retrieve some change when everything else failed (image crashed or worse corrupted image won't restart). You can make the insurance optional if you want, but please continue to provide an equivalent service. A change file is the most simple thing that could possibly work IMO. But this is another thread.
I agree, we have been discussing (and I do not like much the idea to have the changes inside the image as objects because now we have a nice literal object parser/writer and making sure that we can never lose source code is cool). In any case we are working on a new change format recording more high-level entity (such refactoring). We looked again at the model that veronica did, and Ezequiel and we looked again at DeltaStreams. We should digest all that and come up with a new alternate changes model file. I was about to contact goran to see if he wants to participate/ read a paper but we should write the paper first :)
Concerning the temp mapping, I agree, we should not compete with Eliot, he's very tough at tedious tasks, our only weapon is lazyness, lazyness often leads to efficiency ;) I'm curious to check how you handled this complex part with AST when I'll have more time to dig.
2013/4/27 Marcus Denker <marcus.denker@inria.fr>
On Apr 27, 2013, at 6:42 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)?
For the Debugger, everything stays as it is (until Pharo 4 or so)⦠if you look at it, the current Debugger never decompiles if there is source available. It *compiles* to get -> the mapping BC -> text -> the information how temporary variables in the byte code are actually related to temps in the source. (this is very complex, temps can be e.g. stored on the heap if they are written to in a closure, and when they are just read they have a different offset in each closure they are in. Conversely, some temps in the byte code are used to store the array that hold the variables that are on the heapâ¦)
The old compiler recorded mappings while compiling that where encoded⦠quite complex, at least for my tiny brain.
So the new just keeps the AST annotated with all the needed infos, this is much easier to debug (and we do have the memory these days, and as we cache the AST, it's even fast).
So the debugger needs the compiler. The decompiler now exists just to make text so that we can call the compiler in the case there is no .sources. The debugger *always* compiles to get the mappings, as soon as there is source code the decompiler will never be used. (and even if the decompiler is used, the compiler is called right after on it's results so one can record the mappings).
So if you make sure there is always source-code (or a representation with the right amount of meta data), you don't need the decompiler.
So at the start it will be the source code. And yes, this takes memory. But we are in 2013 and if we have one thing than it's memory. (the $25 Raspi comes with 256MB, the $35 with 512MBâ¦).
We could have shipped 2.0 with 8MB of useless Monticello meta data and nobody would have even realized it. (like we have now megabytes of fonts in the imageâ¦). Yet the source is special⦠I really wonder why.
(And yes, there should be solutions to not need unused data in main memory and solutions to share across multiple images data that is the same⦠but for all kinds of stuff, not just source code).
Marcus
But we are in 2013 and if we have one thing than it's memory.
I wouldn't oppose to having the sources (or the AST or whatever ;-) in [object-]memory, although retaining the option of _also_ still writing all the changes into an external changelog (in case of VM, or the less rare case, Windows crashes). But no, we don't have "enough memory" (although now is perhaps the time to announce that "64GB ought to be enough for anybody" ;-))).
(the $25 Raspi comes with 256MB, the $35 with 512MB.)
Incidentally, ~512 MB is the max you can get on Windows (otherwise crash after an ugly ~ 30 sec freeze; I checked sometime around the 2.0 release), and this is not enough for some applications. The Moose people complained a few years back (unable to, on Windows, load a large but otherwise fine image saved on Mac), the only solution offered was "change a #DEFINE or makefile or something like that and rebuild the VM, then pray it works". Currently I avoid using Pharo in case I even suspect the app might come anywhere near that limit (to avoid nasty surprises / hasty rewrites later). Only problems I've heard of concerning DLL's mapped at "weird" offsets in address space (this was cited as reason for the memory limit) were related to trojan horses / stealth viruses tampering with system, and anti-virus / anti-malware SW doing basically the same ;-))) Or something getting loaded really early, geting the DLL - in this case it was kernel.dll - mapped at unusual offset which then got re-used for all other applications, but this was mentioned in context where in the end they were unable to allocate sufficiently large block of contiguous memory because something was siting in the middle of address space - and i _think_ think it was concerning LuaJIT. Would it be possible to prod Jenkins to produce (additional) windows builds with the limit lifted (or set to 2 GB; optionally perhaps a 1GB build) ? So interested testers could try them - as the problem was reported as only manifesting on some comps., while others ran completely fine. I think somebody mentioned that thanks to Jenkins it is/will be a childs play to add/clone another build(s). IMO it would be great to do some larger-scale testing whether this problem still manifests itself and if so whether it could be worked around, or traced to whatever fucks windows up so they load DLL into the middle of the address space ... And lot more people would participate if "build your own VM" would be an optional step (I know, it's not that hard, and setting up mingw or msvc is not that hard either, but it takes a few hours to get it all up & running). I've "played with" (used to solve actual problems / write applications which are used in production, i just got to choose the language ...) a LOT of prog. langs, and I don't remember this problem mentioned elsewhere. For example SBCL, Smalltalk/X, Factor, SWI Prolog, Haskell (a few very different - on the inside as much as on the outside - examples) are able to utilise the whole 2GB (if your windows are configured to the standard 2GB/2GB user/kernel space split). Even that abomination that starts with letter J and is touted as The_Only_VM_and_Language is able to utilize ~1.5 GB on a 32bit system. I'd be able & willing to test on XP/32 and Win7/32+64. I can even test on ~300 machines, although there are only a few hardware configurations, and all are installed according to the same template and mainly use the same software and antivirus. So it's probably not nearly as interesting as 30 totally random machines ... Anybody else thinks this sounds like a good idea ? -----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Marcus Denker Sent: Saturday, April 27, 2013 7:08 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] Opal Decompiler status On Apr 27, 2013, at 6:42 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)?
For the Debugger, everything stays as it is (until Pharo 4 or so). if you look at it, the current Debugger never decompiles if there is source available. It *compiles* to get -> the mapping BC -> text -> the information how temporary variables in the byte code are actually related to temps in the source. (this is very complex, temps can be e.g. stored on the heap if they are written to in a closure, and when they are just read they have a different offset in each closure they are in. Conversely, some temps in the byte code are used to store the array that hold the variables that are on the heap.) The old compiler recorded mappings while compiling that where encoded. quite complex, at least for my tiny brain. So the new just keeps the AST annotated with all the needed infos, this is much easier to debug (and we do have the memory these days, and as we cache the AST, it's even fast). So the debugger needs the compiler. The decompiler now exists just to make text so that we can call the compiler in the case there is no .sources. The debugger *always* compiles to get the mappings, as soon as there is source code the decompiler will never be used. (and even if the decompiler is used, the compiler is called right after on it's results so one can record the mappings). So if you make sure there is always source-code (or a representation with the right amount of meta data), you don't need the decompiler. So at the start it will be the source code. And yes, this takes memory. But we are in 2013 and if we have one thing than it's memory. (the $25 Raspi comes with 256MB, the $35 with 512MB.). We could have shipped 2.0 with 8MB of useless Monticello meta data and nobody would have even realized it. (like we have now megabytes of fonts in the image.). Yet the source is special. I really wonder why. (And yes, there should be solutions to not need unused data in main memory and solutions to share across multiple images data that is the same. but for all kinds of stuff, not just source code). Marcus
On 30 April 2013 17:45, <Miloslav.Raus@cuzk.cz> wrote:
But we are in 2013 and if we have one thing than it's memory.
I wouldn't oppose to having the sources (or the AST or whatever ;-) in [object-]memory, although retaining the option of _also_ still writing all the changes into an external changelog (in case of VM, or the less rare case, Windows crashes).
But no, we don't have "enough memory" (although now is perhaps the time to announce that "64GB ought to be enough for anybody" ;-))).
(the $25 Raspi comes with 256MB, the $35 with 512MB.)
Incidentally, ~512 MB is the max you can get on Windows (otherwise crash after an ugly ~ 30 sec freeze; I checked sometime around the 2.0 release), and this is not enough for some applications.
The Moose people complained a few years back (unable to, on Windows, load a large but otherwise fine image saved on Mac), the only solution offered was "change a #DEFINE or makefile or something like that and rebuild the VM, then pray it works".
Currently I avoid using Pharo in case I even suspect the app might come anywhere near that limit (to avoid nasty surprises / hasty rewrites later).
Only problems I've heard of concerning DLL's mapped at "weird" offsets in address space (this was cited as reason for the memory limit) were related to trojan horses / stealth viruses tampering with system, and anti-virus / anti-malware SW doing basically the same ;-))) Or something getting loaded really early, geting the DLL - in this case it was kernel.dll - mapped at unusual offset which then got re-used for all other applications, but this was mentioned in context where in the end they were unable to allocate sufficiently large block of contiguous memory because something was siting in the middle of address space - and i _think_ think it was concerning LuaJIT.
Well, it is not just about mapping DLL at random place, causing address space fragmentation (AFAIK DLL's are always mapped to higher 2GB address range). The real reason for putting hard limit of 512 MB, was to leave rest of space for those DLLs to use. Of course, it depends what is most suitable limit for one or another project. Currently, we are living with limit which is most suitable for Croquet project (don't ask me what DLLs it uses, that require lots of free system memory space, but i can imagine it requires a lot because of 3D , textures , sound etc). So, you see, it is a double-edged sword: if you increase reserved address space for object memory, you inevitably leaving less for all of DLL's your project uses. So, <<Currently you avoid using Pharo in case you even suspect the app might come anywhere near that limit (to avoid nasty surprises / hasty rewrites later)>> but if you change that limit , you will still <<avoid using Pharo in case when external DLLs you using require lots of system memory (to avoid nasty surprises / hasty rewrites later)>>. Result is the same => avoid using Pharo. Over. :) Would it be possible to prod Jenkins to produce (additional) windows builds
with the limit lifted (or set to 2 GB; optionally perhaps a 1GB build) ? So interested testers could try them - as the problem was reported as only manifesting on some comps., while others ran completely fine. I think somebody mentioned that thanks to Jenkins it is/will be a childs play to add/clone another build(s).
I think better would be to parametrize this limit, so you can specify from command line your preferable limit. Doing substantial changes in object memory model (to support fragmented object memory), i think not worth the effort, because in next 5 years, everything will be 64bit anyways (including Pharo VM), and this will be not an issue for the rest of our lives (i hope ;).
IMO it would be great to do some larger-scale testing whether this problem still manifests itself and if so whether it could be worked around, or traced to whatever fucks windows up so they load DLL into the middle of the address space ... And lot more people would participate if "build your own VM" would be an optional step (I know, it's not that hard, and setting up mingw or msvc is not that hard either, but it takes a few hours to get it all up & running).
I've "played with" (used to solve actual problems / write applications which are used in production, i just got to choose the language ...) a LOT of prog. langs, and I don't remember this problem mentioned elsewhere.
I can reflect this statement to you: a LOT of prog. langs having problems, which i don't remember ever mentioned for Pharo.
In other words: every language has own problems.
For example SBCL, Smalltalk/X, Factor, SWI Prolog, Haskell (a few very different - on the inside as much as on the outside - examples) are able to utilise the whole 2GB (if your windows are configured to the standard 2GB/2GB user/kernel space split). Even that abomination that starts with letter J and is touted as The_Only_VM_and_Language is able to utilize ~1.5 GB on a 32bit system.
I'd be able & willing to test on XP/32 and Win7/32+64. I can even test on ~300 machines, although there are only a few hardware configurations, and all are installed according to the same template and mainly use the same software and antivirus. So it's probably not nearly as interesting as 30 totally random machines ...
Anybody else thinks this sounds like a good idea ?
-----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto: pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Marcus Denker Sent: Saturday, April 27, 2013 7:08 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] Opal Decompiler status
On Apr 27, 2013, at 6:42 PM, Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote:
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)?
For the Debugger, everything stays as it is (until Pharo 4 or so). if you look at it, the current Debugger never decompiles if there is source available. It *compiles* to get -> the mapping BC -> text -> the information how temporary variables in the byte code are actually related to temps in the source. (this is very complex, temps can be e.g. stored on the heap if they are written to in a closure, and when they are just read they have a different offset in each closure they are in. Conversely, some temps in the byte code are used to store the array that hold the variables that are on the heap.)
The old compiler recorded mappings while compiling that where encoded. quite complex, at least for my tiny brain.
So the new just keeps the AST annotated with all the needed infos, this is much easier to debug (and we do have the memory these days, and as we cache the AST, it's even fast).
So the debugger needs the compiler. The decompiler now exists just to make text so that we can call the compiler in the case there is no .sources. The debugger *always* compiles to get the mappings, as soon as there is source code the decompiler will never be used. (and even if the decompiler is used, the compiler is called right after on it's results so one can record the mappings).
So if you make sure there is always source-code (or a representation with the right amount of meta data), you don't need the decompiler.
So at the start it will be the source code. And yes, this takes memory. But we are in 2013 and if we have one thing than it's memory. (the $25 Raspi comes with 256MB, the $35 with 512MB.).
We could have shipped 2.0 with 8MB of useless Monticello meta data and nobody would have even realized it. (like we have now megabytes of fonts in the image.). Yet the source is special. I really wonder why.
(And yes, there should be solutions to not need unused data in main memory and solutions to share across multiple images data that is the same. but for all kinds of stuff, not just source code).
Marcus
-- Best regards, Igor Stasenko.
Hi I think that you have some points :) Indeed for Moose we should pay attention because we often deploy on windows.
I wouldn't oppose to having the sources (or the AST or whatever ;-) in [object-]memory, although retaining the option of _also_ still writing all the changes into an external changelog (in case of VM, or the less rare case, Windows crashes).
But no, we don't have "enough memory" (although now is perhaps the time to announce that "64GB ought to be enough for anybody" ;-))).
(the $25 Raspi comes with 256MB, the $35 with 512MB.)
Incidentally, ~512 MB is the max you can get on Windows (otherwise crash after an ugly ~ 30 sec freeze; I checked sometime around the 2.0 release), and this is not enough for some applications.
The Moose people complained a few years back (unable to, on Windows, load a large but otherwise fine image saved on Mac), the only solution offered was "change a #DEFINE or makefile or something like that and rebuild the VM, then pray it works".
Currently I avoid using Pharo in case I even suspect the app might come anywhere near that limit (to avoid nasty surprises / hasty rewrites later).
Only problems I've heard of concerning DLL's mapped at "weird" offsets in address space (this was cited as reason for the memory limit) were related to trojan horses / stealth viruses tampering with system, and anti-virus / anti-malware SW doing basically the same ;-))) Or something getting loaded really early, geting the DLL - in this case it was kernel.dll - mapped at unusual offset which then got re-used for all other applications, but this was mentioned in context where in the end they were unable to allocate sufficiently large block of contiguous memory because something was siting in the middle of address space - and i _think_ think it was concerning LuaJIT.
Would it be possible to prod Jenkins to produce (additional) windows builds with the limit lifted (or set to 2 GB; optionally perhaps a 1GB build) ? So interested testers could try them - as the problem was reported as only manifesting on some comps., while others ran completely fine. I think somebody mentioned that thanks to Jenkins it is/will be a childs play to add/clone another build(s). yes this is a good idea.
IMO it would be great to do some larger-scale testing whether this problem still manifests itself and if so whether it could be worked around, or traced to whatever fucks windows up so they load DLL into the middle of the address space ... And lot more people would participate if "build your own VM" would be an optional step (I know, it's not that hard, and setting up mingw or msvc is not that hard either, but it takes a few hours to get it all up & running).
I've "played with" (used to solve actual problems / write applications which are used in production, i just got to choose the language ...) a LOT of prog. langs, and I don't remember this problem mentioned elsewhere.
Clearly we pay the lag of not moving.
For example SBCL, Smalltalk/X, Factor, SWI Prolog, Haskell (a few very different - on the inside as much as on the outside - examples) are able to utilise the whole 2GB (if your windows are configured to the standard 2GB/2GB user/kernel space split). Even that abomination that starts with letter J and is touted as The_Only_VM_and_Language is able to utilize ~1.5 GB on a 32bit system.
Yes they probably put also lot of money on the table.
I'd be able & willing to test on XP/32 and Win7/32+64. I can even test on ~300 machines, although there are only a few hardware configurations, and all are installed according to the same template and mainly use the same software and antivirus. So it's probably not nearly as interesting as 30 totally random machines ...
Anybody else thinks this sounds like a good idea ?
Yes
-----Original Message----- From: pharo-project-bounces@lists.gforge.inria.fr [mailto:pharo-project-bounces@lists.gforge.inria.fr] On Behalf Of Marcus Denker Sent: Saturday, April 27, 2013 7:08 PM To: Pharo-project@lists.gforge.inria.fr Subject: Re: [Pharo-project] Opal Decompiler status
On Apr 27, 2013, at 6:42 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)?
For the Debugger, everything stays as it is (until Pharo 4 or so). if you look at it, the current Debugger never decompiles if there is source available. It *compiles* to get -> the mapping BC -> text -> the information how temporary variables in the byte code are actually related to temps in the source. (this is very complex, temps can be e.g. stored on the heap if they are written to in a closure, and when they are just read they have a different offset in each closure they are in. Conversely, some temps in the byte code are used to store the array that hold the variables that are on the heap.)
The old compiler recorded mappings while compiling that where encoded. quite complex, at least for my tiny brain.
So the new just keeps the AST annotated with all the needed infos, this is much easier to debug (and we do have the memory these days, and as we cache the AST, it's even fast).
So the debugger needs the compiler. The decompiler now exists just to make text so that we can call the compiler in the case there is no .sources. The debugger *always* compiles to get the mappings, as soon as there is source code the decompiler will never be used. (and even if the decompiler is used, the compiler is called right after on it's results so one can record the mappings).
So if you make sure there is always source-code (or a representation with the right amount of meta data), you don't need the decompiler.
So at the start it will be the source code. And yes, this takes memory. But we are in 2013 and if we have one thing than it's memory. (the $25 Raspi comes with 256MB, the $35 with 512MB.).
We could have shipped 2.0 with 8MB of useless Monticello meta data and nobody would have even realized it. (like we have now megabytes of fonts in the image.). Yet the source is special. I really wonder why.
(And yes, there should be solutions to not need unused data in main memory and solutions to share across multiple images data that is the same. but for all kinds of stuff, not just source code).
Marcus
Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot.
Me too :) But marcus knows it since the end of his PhD :D. This is why I really want to see AST compression.
My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)?
2013/4/27 Marcus Denker <marcus.denker@inria.fr>
On Apr 27, 2013, at 5:59 PM, Marcus Denker <marcus.denker@inria.fr> wrote:.
Could we have a primer on what remains to do on the Opal front?
-> debug BC-IR->AST->text mapping more (lots of it working already) -> debug temp variable lookup in optimized blocks in the debugger (almost there, too) -> Go through all the test for the old Compiler and port them to the new. -> Better compiler API and refactor the whole system to use it (but keep a backward compatible API at least to some extend)
All the TODOs are in the bug tracker, Project "Compiler (Opal)".
There are right now 29 Issues.
Marcus
participants (5)
-
Igor Stasenko -
Marcus Denker -
Miloslav.Raus＠cuzk.cz -
Nicolas Cellier -
stephane ducasse