April 2013 - Pharo-dev - lists.pharo.org

Re: [Pharo-project] Trying to understand DateAndTime

by stephane ducasse

On Apr 27, 2013, at 6:33 PM, Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> wrote: > Since DateAndTime comment is out of date (sic !) and current implementation seems not free of bug, here is an effort to summarize and understand the whole thing, let's call this a review. Thanks this is a great initiative! > Please tell me where my interpretations are wrong. > > 1) The DateAndTime is stored internally as > - a point in UTC time (julianDayNumber, seconds, nanos) > - an offset from UTC (a Duration) denoting the local time zone > > 2) The ticks method returns the UTC part {julianDayNumber. seconds. nanos} > > 3) DateAndTime print themselves in local time using the offset > > 4) other methods (with some exceptions) return the local dateAndTime parts (year month day hours minutes seconds) > > Exceptions are: > - secondsSinceMidnight (which should be renamed secondsSinceMidnightUTC IMO, even if the method is classified private) > - julianDayNumberUTC as the name tells > > So far so good (but secondsSinceMidnight which is error prone) > > What I find strange: > - #hash uses the offset... Why ??? > > d1 := DateAndTime now. > d2 := d1 offset: d1 offset + 1 hours. > { > d1 = d2. > d1 hash = d2 hash. > } > gives #(true false), a clue? > > - #< compares julianDayNumber (the UTC inst. var.) with otherDate julianDayNumber (with otherDate local offset) which seems wrong, it should better use julianDayNumberUTC or (normalized) ticks. > > - #= could be simplified and accelerated if using #hasEqualTicks: (if the ticks are correctly normalized though which would be the case since #ticks:offset: does, unfortunately #setJdn:seconds:nanos:offset: does not). > > Some senders of #setJdn:seconds:nanos:offset: don't normalize the day/seconds/nanos, though it should be their responsibility (DateAndTime todayAtMilliSeconds: xxx) (DateAndTime todayAtNanoSeconds: xxx), not counting year:month:day:hour:minute:second:nanoSecond:offset: which does not either (many senders...). > Maybe we should better let the normalization responsibility to #setJdn:seconds:nanos:offset:. > > It remain to analyze the conversion protocol (asYear etc...) which seems to use the local time year, but I don't understand the whole Timespan thing right now (why DateAndTime now asYear starts now, and not on january 1st?). > > Nicolas > >

April 27, 2013

Re: [Pharo-project] [Ann] TxText. New text editor done!

by stephane ducasse

Hi denis Alain is gone on vacation but it told me that if you code scale he will throw away what he did :) May be the decorators that alain wrote can be merged. In any case this is excellent to get a better system. Stef On Apr 27, 2013, at 5:17 PM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > I should say it is not difficult to do with current design. I were > think about it and current layout building logic requires little > changes, maybe just one spesific method. > > But there is cases where any text change required rebuilding full > layout. Fo example, single line text with wrapping mode. > > > 2013/4/27, Igor Stasenko <siguctua(a)gmail.com>: >> Very impressive progress, Denis! :) >> >> I going to finish low-level stuff in Athens (path approximation / >> point location) >> and then go straight to text rendering with existing Text. >> Next stage will be to take TxText model and provide rendering for it >> with Athens, >> as well as review the code. >> >> About text editing and re-layout. >> It should be possible that any local update (which affecting only >> single character span or multiple adjusted character spans will >> require only local recalculations of text layout). Otherwise there is >> no reason why in our model we don't keep whole text in single big >> buffer (like Text does). >> Splitting text on multiple spans was the reason to make local layout >> updates possible. >> But i know it is not that easy to do :) >> >> >> -- >> Best regards, >> Igor Stasenko. >> >> >

April 27, 2013

Re: [Pharo-project] Opal Decompiler status

by Marcus Denker

On Apr 27, 2013, at 8:01 PM, Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> wrote: > Right, if you embed source along with CompiledMethod (to handle the case when the CompiledMethod has been replaced but one of its block of code is still used), then you don't need Decompiler in the first place. > > Nonetheless, I would remove the .sources but not the .changes. The .changes is not only a source code repository (that feature can go away if you want), it's before all a change log, an insurance to retrieve some change when everything else failed (image crashed or worse corrupted image won't restart). You can make the insurance optional if you want, but please continue to provide an equivalent service. A change file is the most simple thing that could possibly work IMO. But this is another thread. > Yes, we have a solution for thatâ¦ the thing is that the .changes has 3 roles. In the 1978, it was very clever to design one extremely memory efficient mechanism to kill all three: 1) keep a history of all the accepted method sources in the image (see "versions") 2) store the code on disk efficiently with no duplications (They counted those extremely expensive disks in the one-diget Megabytes back then). 3) have transaction log for the source code in case of image crash. Now today, we can solve all three using three different methods: 1) we will just keep the accepted methods in memory until you commit to monticello/git (in case of git one can even commit all the intermediates, for MC one would just commit the latest). after the in-image history is pruned (like it is now eventually anyway when you start with a fresh image). 2) We don't need to store the *current* version on disk. Compressed in memory. Yes, that is what progress bringsâ¦ 3) Disk space is so cheap it is not even funny. -> we will just have a dedicated log. And we can even store Objectsâ¦ And then we can leverage the extreme simplicity of the resulting scheme.. simplicity is an amazing thing. (Or the contrary: I think people understate the effects of complexity). Marcus

April 27, 2013

Re: [Pharo-project] Opal Decompiler status

by Nicolas Cellier

Right, if you embed source along with CompiledMethod (to handle the case when the CompiledMethod has been replaced but one of its block of code is still used), then you don't need Decompiler in the first place. Nonetheless, I would remove the .sources but not the .changes. The .changes is not only a source code repository (that feature can go away if you want), it's before all a change log, an insurance to retrieve some change when everything else failed (image crashed or worse corrupted image won't restart). You can make the insurance optional if you want, but please continue to provide an equivalent service. A change file is the most simple thing that could possibly work IMO. But this is another thread. Concerning the temp mapping, I agree, we should not compete with Eliot, he's very tough at tedious tasks, our only weapon is lazyness, lazyness often leads to efficiency ;) I'm curious to check how you handled this complex part with AST when I'll have more time to dig. 2013/4/27 Marcus Denker <marcus.denker(a)inria.fr> > > On Apr 27, 2013, at 6:42 PM, Nicolas Cellier < > nicolas.cellier.aka.nice(a)gmail.com> wrote: > > > Thanks Marcus. > > I'm among the skepticals concerning the AST, but I'd like to be proven > wrong, because AST level would certainly simplify things a lot. > > My main reserve is that interruption can occur at BC level, so for > debugging purpose, what are your plans? > > Will you move the PC at some AST node boundary (if it is possible given > side effects of some BC)? > > > For the Debugger, everything stays as it is (until Pharo 4 or so)â¦ if you > look at it, the current Debugger never decompiles if there is source > available. > It *compiles* to get > -> the mapping BC -> text > -> the information how temporary variables in the byte code are > actually related to temps in the source. > (this is very complex, temps can be e.g. stored on the heap if > they are written to in a closure, and when they are just > read they have a different offset in each closure they are > in. Conversely, some temps in the byte code are used > to store the array that hold the variables that are on the > heapâ¦) > > The old compiler recorded mappings while compiling that where encodedâ¦ > quite complex, at least for my tiny brain. > > So the new just keeps the AST annotated with all the needed infos, this is > much easier to debug (and we do have the > memory these days, and as we cache the AST, it's even fast). > > So the debugger needs the compiler. The decompiler now exists just to make > text so that we can call the compiler > in the case there is no .sources. The debugger *always* compiles to get > the mappings, as soon as there is source code > the decompiler will never be used. (and even if the decompiler is used, > the compiler is called right after on it's results so > one can record the mappings). > > So if you make sure there is always source-code (or a representation with > the right amount of meta data), you don't need the decompiler. > > So at the start it will be the source code. And yes, this takes memory. > But we are in 2013 and if we have one thing than it's memory. > (the $25 Raspi comes with 256MB, the $35 with 512MBâ¦). > > We could have shipped 2.0 with 8MB of useless Monticello meta data and > nobody would have even realized it. (like we > have now megabytes of fonts in the imageâ¦). Yet the source is specialâ¦ I > really wonder why. > > (And yes, there should be solutions to not need unused data in main memory > and solutions to share across multiple images > data that is the sameâ¦ but for all kinds of stuff, not just source code). > > Marcus >

April 27, 2013

Re: [Pharo-project] Opal Decompiler status

by Marcus Denker

On Apr 27, 2013, at 6:42 PM, Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> wrote: > Thanks Marcus. > I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. > My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? > Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)? > For the Debugger, everything stays as it is (until Pharo 4 or so)â¦ if you look at it, the current Debugger never decompiles if there is source available. It *compiles* to get -> the mapping BC -> text -> the information how temporary variables in the byte code are actually related to temps in the source. (this is very complex, temps can be e.g. stored on the heap if they are written to in a closure, and when they are just read they have a different offset in each closure they are in. Conversely, some temps in the byte code are used to store the array that hold the variables that are on the heapâ¦) The old compiler recorded mappings while compiling that where encodedâ¦ quite complex, at least for my tiny brain. So the new just keeps the AST annotated with all the needed infos, this is much easier to debug (and we do have the memory these days, and as we cache the AST, it's even fast). So the debugger needs the compiler. The decompiler now exists just to make text so that we can call the compiler in the case there is no .sources. The debugger *always* compiles to get the mappings, as soon as there is source code the decompiler will never be used. (and even if the decompiler is used, the compiler is called right after on it's results so one can record the mappings). So if you make sure there is always source-code (or a representation with the right amount of meta data), you don't need the decompiler. So at the start it will be the source code. And yes, this takes memory. But we are in 2013 and if we have one thing than it's memory. (the $25 Raspi comes with 256MB, the $35 with 512MBâ¦). We could have shipped 2.0 with 8MB of useless Monticello meta data and nobody would have even realized it. (like we have now megabytes of fonts in the imageâ¦). Yet the source is specialâ¦ I really wonder why. (And yes, there should be solutions to not need unused data in main memory and solutions to share across multiple images data that is the sameâ¦ but for all kinds of stuff, not just source code). Marcus

April 27, 2013

Re: [Pharo-project] Opal Decompiler status

by Nicolas Cellier

Thanks Marcus. I'm among the skepticals concerning the AST, but I'd like to be proven wrong, because AST level would certainly simplify things a lot. My main reserve is that interruption can occur at BC level, so for debugging purpose, what are your plans? Will you move the PC at some AST node boundary (if it is possible given side effects of some BC)? 2013/4/27 Marcus Denker <marcus.denker(a)inria.fr> > > On Apr 27, 2013, at 5:59 PM, Marcus Denker <marcus.denker(a)inria.fr> > wrote:. > > > >> Could we have a primer on what remains to do on the Opal front? > >> > > > > > > -> debug BC-IR->AST->text mapping more (lots of it working already) > > -> debug temp variable lookup in optimized blocks in the debugger > (almost there, too) > > -> Go through all the test for the old Compiler and port them to the new. > > -> Better compiler API and refactor the whole system to use it > > (but keep a backward compatible API at least to some extend) > > > All the TODOs are in the bug tracker, Project "Compiler (Opal)". > > There are right now 29 Issues. > > Marcus >

April 27, 2013

[Pharo-project] Trying to understand DateAndTime

by Nicolas Cellier

Since DateAndTime comment is out of date (sic !) and current implementation seems not free of bug, here is an effort to summarize and understand the whole thing, let's call this a review. Please tell me where my interpretations are wrong. 1) The DateAndTime is stored internally as - a point in UTC time (julianDayNumber, seconds, nanos) - an offset from UTC (a Duration) denoting the local time zone 2) The ticks method returns the UTC part {julianDayNumber. seconds. nanos} 3) DateAndTime print themselves in local time using the offset 4) other methods (with some exceptions) return the local dateAndTime parts (year month day hours minutes seconds) Exceptions are: - secondsSinceMidnight (which should be renamed secondsSinceMidnightUTC IMO, even if the method is classified private) - julianDayNumberUTC as the name tells So far so good (but secondsSinceMidnight which is error prone) What I find strange: - #hash uses the offset... Why ??? d1 := DateAndTime now. d2 := d1 offset: d1 offset + 1 hours. { d1 = d2. d1 hash = d2 hash. } gives #(true false), a clue? - #< compares julianDayNumber (the UTC inst. var.) with otherDate julianDayNumber (with otherDate local offset) which seems wrong, it should better use julianDayNumberUTC or (normalized) ticks. - #= could be simplified and accelerated if using #hasEqualTicks: (if the ticks are correctly normalized though which would be the case since #ticks:offset: does, unfortunately #setJdn:seconds:nanos:offset: does not). Some senders of #setJdn:seconds:nanos:offset: don't normalize the day/seconds/nanos, though it should be their responsibility (DateAndTime todayAtMilliSeconds: xxx) (DateAndTime todayAtNanoSeconds: xxx), not counting year:month:day:hour:minute:second:nanoSecond:offset: which does not either (many senders...). Maybe we should better let the normalization responsibility to #setJdn:seconds:nanos:offset:. It remain to analyze the conversion protocol (asYear etc...) which seems to use the local time year, but I don't understand the whole Timespan thing right now (why DateAndTime now asYear starts now, and not on january 1st?). Nicolas

April 27, 2013

Re: [Pharo-project] Opal Decompiler status

by Marcus Denker

On Apr 27, 2013, at 5:59 PM, Marcus Denker <marcus.denker(a)inria.fr> wrote:. > >> Could we have a primer on what remains to do on the Opal front? >> > > > -> debug BC-IR->AST->text mapping more (lots of it working already) > -> debug temp variable lookup in optimized blocks in the debugger (almost there, too) > -> Go through all the test for the old Compiler and port them to the new. > -> Better compiler API and refactor the whole system to use it > (but keep a backward compatible API at least to some extend) All the TODOs are in the bug tracker, Project "Compiler (Opal)". There are right now 29 Issues. Marcus

April 27, 2013

Re: [Pharo-project] Opal Decompiler status

by Marcus Denker

On Apr 27, 2013, at 5:26 PM, Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> wrote: > I see that the OCFakeDecompiler which uses the old and horrible (*) Decompiler is still in use. > Though, IRBytecodeDecompiler seems already quite advanced. The idea actually is to not have a decompiler BC->AST. We have one BC->IR for manipulating things on the byte code level (e.g. he new class builder will use it). BC->IR->BC is working and tested after every commit. But on the AST level, idea is to have a higher level representation of code always available. (compressed ASTs in the long term, compressed source as the intermediate solution). Yes, I know that people will be highly skeptical about this. Up to the point that I even don't expect to convince anyone other than by actually showing a working versionâ¦ This goes hand-in-hand with the "one file" image: just one .pharo image, no .sources, no .changes. (Which real conservative people will not like, either). We need to see how fast this progresses, if we see that it will not be ready for 3.0, we should add a IR->AST decompiler, but I hope the alternative will show that it is not necessary. > Could we have a primer on what remains to do on the Opal front? > -> debug BC-IR->AST->text mapping more (lots of it working already) -> debug temp variable lookup in optimized blocks in the debugger (almost there, too) -> Go through all the test for the old Compiler and port them to the new. -> Better compiler API and refactor the whole system to use it (but keep a backward compatible API at least to some extend) Marcus

April 27, 2013

[Pharo-project] Opal Decompiler status

by Nicolas Cellier

I see that the OCFakeDecompiler which uses the old and horrible (*) Decompiler is still in use. Though, IRBytecodeDecompiler seems already quite advanced. Could we have a primer on what remains to do on the Opal front? (*) just look number of inst vars, count how many times their state change, and measure average method length for an approximate definition of horrible

April 27, 2013