- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] DebugSession>>activePC:
by Eliot Miranda Jan. 21, 2019

Jan. 21, 2019

Hi Marcus, On Fri, Jan 18, 2019 at 5:15 AM Marcus Denker via Pharo-dev < pharo-dev(a)lists.pharo.org> wrote: > > > On 11 Jan 2019, at 20:28, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > > > > Hi Thomas, > > > > forgive me, my first response was too terse. Having thought about it > in the shower it becomes clear :-) > > > >> On Jan 11, 2019, at 6:49 AM, Thomas Dupriez < > tdupriez(a)ens-paris-saclay.fr> wrote: > >> > >> Hi, > >> > >> Yes, my question was just of the form: "Hey there's this method in > DebugSession. What is it doing? What's the intention behind it? Does > someone know?". There was no hidden agenda behind it. > >> > >> @Eliot > >> > >> After taking another look at this method, there's something I don't > understand: > >> > >> activePC: aContext > >> ^ (self isLatestContext: aContext) > >> ifTrue: [ interruptedContext pc ] > >> ifFalse: [ self previousPC: aContext ] > >> > >> isLatestContext: checks whether its argument is the suspended context > (the context at the top of the stack of the interrupted process). And if > that's true, activePC: returns the pc of **interruptedContext**, not of the > suspended context. These two contexts are different when the debugger opens > on an exception, so this method is potentially returning a pc for another > context than its argument... > >> > >> Another question I have to improve the comment for this method is: > what's the high-level meaning of this concept of "activePC". You gave the > formal definition, but what's the point of defining this so to speak? What > makes this concept interesting enough to warrant defining it and giving it > a name? > > > > There are two âmodesâ where a pc us mapped to a source range. One is > when stepping a context in the debugger (the context is on top and is > actively executing bytecodes). Here the debugger stops immediately before > a send or assignment or return, so that for sends we can do into or over, > or for assignments or returns check stack top to see what will be assigned > or returned. In this mode we want the pc of the send, assign or return to > map to the source range for the send, or the expression being assigned or > returned. Since this is the âcommon caseâ, and since this is the only > choice that makes sense for assignments ta and returns, the bytecode > compiler constructs itâs pc to source range map in terms of the pc of the > first byte if the send, assign or return bytecode. > > > > The second âmodeâ is when selecting a context below the top context. > The pc for any context below the top context will be the return pc for a > send, because the send has already happened. The compiler could choose to > map this pc to the send, but it would not match what works for the common > case. Another choice would appear be to have two map entries, one for the > send and one for the return pc, both mapping to the source range. But this > wouldnât work because the result of a send might be assigned or returned > and so there is a potential conflict. I stead the reasonable solution is > to select the previous pc for contexts below the top of context, which will > be the pc for the start of the send bytecode. > > > > > I checked with Thomas > > -> for source mapping, we use the API of the method map. The map does the > âget the mapping for the instruction beforeâ, it just needs to be told that > we ask the range for an active context: > > #rangeForPC:contextIsActiveContext: > > it is called > > ^aContext debuggerMap > rangeForPC: aContext pc > contextIsActiveContext: (self isLatestContext: aContext) ] > > So the logic was move from the debugger to the Map. (I think this is even > your design?), and thus the logic inside the debugger is not needed > anymore. > "Design" is giving my code a little too much respect. I was desperately trying to get something to work to be able to deploy Cog with the new closure model. I happily admit that DebuggerMethodMap in Squeak is ugly code. It had to be extended recently to handle full blocks. But it would be great to rewrite it. I dream of a harmonisation of Squeak/Pharo/Cuis execution classes such that we have the same Context, CompiledCode, CompiledBlock, CompiledMethod, debuggerMap and BytecodeEncoder (which is effectively the back end of the compiler that generates bytecode, and the interface to the debugger when bytecode is analyses or executed in the debugger), which would make my life easier maintaining the VM and execution classes, especially as we introduce Sista. I think Opal as a separate compiler is great; the work you've done with mustBeBoleanMagic is superb. At the same time the Squeak compiler is fine for its job and I still find it easier to understand than Opal (probably because I'm not spending enough time with it). One major reason for the harmonization is bug fixes. Right now I use SistaV1 and 64-bits as my default work environment, in a 64-bit Squeak image that was compiled with V3PlusClosures, so that half the code base (the base image) is the old bytecode set, but all of VMMaker is the new bytecode set & full blocks. So far in the past few months I've fixed a few issues with pc mapping, debugging in the different bytecode sets, searching for literals with full blocks, fixing Slang for full blocks, etc. And I'd like these bug fixes to make it into Pharo too. If this speaks to you Marcus, maybe we can try and figure out a path that doesn't ruffle too many feathers. -> For the question why the AST node of the Block has no simple method to > check if it is an âisOptimizedâ block (e.g. in an ifTrue:): Yes, that might > be nice. The reason why is is not there is that the compiler internally > using OCOptimizedBlockScope and a check on the AST level was never needed. > I would have added it as soon as it would be needed (quite easy to do). > OK. isOptimized is something I use a lot when analyzing parse trees in looking at the relationship between bytecode and hit code. For example it is key to work I;'m doing right now on register allocation for Sista. On the AST level there is already a check, though, if a *send* is > optimized, which is used for code generation (#isInlined). > The Squeak versions here are MessageNode>>isOptimized, isOptimizedLoop & isOptimizedWhileLoop The question why we did not add more compatibility layers to the old AST is > that is is quite different.. so even with some methods here and there 99% > of clients need changes, so I did not even investigate it too much. With > the idea that if we end up in a situation where we need just a method, we > can just add it as needed. > Reasonable. I simply have to maintain compatibility for the two parse trees. > In general, I have no srtong feelings about the details of the > implementation of Opal. Itâs just what was possible, with the resources and > the understanding that I had at the point it was done. There are for sure > even mistakes. I always make mistakes. > I can join you in all of these for the Squeak compiler. It was not possible to have someone to pair with when modifying the Squeak compiler to conform to my closure design; I was in a tiny startup and had to do all execution technology work. And I always make mistakes too :-) > In addition, it is based on a prior compiler which was done for a > different closure model whose Design-choices influenced it too much I > think, sometimes good, sometimes not. > > But I like the âhigh levelâ: using a shared AST between the compiler and > the tools *and* having a mapping BC -> AST -> Text. > Yes, this is great. > Of course I understand that the choice to use the RB AST for the compiler > is not a âtraditionalâ one.. but it turned out to work very well *and* it > brings some amazing power, as we have now not only a mapping bc->text > offset, but a mapping bc->AST, too. This e.g. (just a a simple example) > makes the magic of the runtime transforming deprecations possible. See > #transform on class Deprecation, the #sourceNodeExecuted: > > transform > | node rewriteRule aMethod | > self shouldTransform ifFalse: [ ^ self ]. > self rewriterClass ifNil:[ ^ self signal ]. > aMethod := self contextOfSender method. > aMethod isDoIt ifTrue:[^ self]. "no need to transform doits" > node := self contextOfSender sourceNodeExecuted. > rewriteRule := self rewriterClass new > replace: rule key with: rule value. > (rewriteRule executeTree: node) > ifFalse: [ ^ self ]. > node replaceWith: rewriteRule tree. > Author > useAuthor: 'AutoDeprecationRefactoring' > during: [aMethod origin compile: aMethod ast formattedCode > classified: aMethod protocol]. > +10 > Marcus _,,,^..^,,,_ best, Eliot

1 0

Re: [Pharo-dev] Better management of encoding of environment variables
by Guillermo Polito Jan. 19, 2019

Jan. 19, 2019

On Fri, Jan 18, 2019 at 1:48 PM Ben Coman via Pharo-dev < pharo-dev(a)lists.pharo.org> wrote: > > > > > On Wed, 16 Jan 2019 at 18:37, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > >> Still, one of the conclusions of previous discussions about the encoding >> of environment variables was/is that there is no single correct solution. >> OS's are not consistent in how the encoding is done in all (historical) >> contexts (like sometimes, > > > >> 1 env var defines the encoding to use for others, > > > ouch. That one point nearly made my retract my comment next paragraph, > but is there much more complexity? > or just a case of utf8<==>appSpecificEncoding rather than > ascii<==>appSpecificEncoding ? > It's not muuuuch more complex. The problem is that usually the bugs that arise from wrongly managing such conversions can be super obscure. > Sorry if I'm rehashing past discussion (do you have a link?), but > considering... > * 92% of web pages are UTF8 encoded[1] such that pragmatically UTF8 *is* > the standard for text > * Strings so pervasive in a system > ...would there be an overall benefit to adopt UTF8 as the encoding for > Strings > consistently provided across the cross-platform vm interface? > (i.e. fixing platforms that don't comply to the standard due to their > historical baggage) > > And I found it interesting Microsoft are making some moves towards UTF8 > [2]... > "With insider build 17035 and the April 2018 update (nominal build 17134) > for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" > checkbox appeared for setting the locale code page to UTF-8.[a] This allows > for calling "narrow" functions, including fopen and SetWindowTextA, with > UTF-8 strings. " > > The approach vm-side could be similar to Section 10 How to do text on > Windows [3] > with the philosophy of "performing the [conversions] as close to API calls > as possible, > and never holding the [converted] data." > > [1] > https://w3techs.com/technologies/history_overview/character_encoding/ms/y > [2] https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows > [3] http://utf8everywhere.org/ > > > different applications do different things, and other such nice stuff), >> and certainly not across platforms. >> >> So this is really complex. >> >> Do we want to hide this in some obscure VM C code that very few people >> can see, read, let alone help with ? >> >> The image side is perfectly capable of dealing with platform differences >> in a clean/clear way, and at least we can then use the full power of our >> language and our tools. >> > > Big question... Do we currently have primitives of the same name returning > different encodings on different platforms? I presume that would be > awkward. > If the image is handle encoding differences, should separate primitives be > used? e.g. utf8GetEnv & utf16getEnv > > Could I get some feedback on [4] saying... **The Single Most Important > Fact About Encodings** > If you completely forget everything I just explained, please remember one > extremely important fact. > It does not make sense to have a string without knowing what encoding it > uses. " > > And so... does our String nowadays require an 'encoding' instance variable > such that this is *always* associated? > This might remove any need for separate utf8GetEnv & utf16getEnv (if that > was even a reasonable idea). > I think that will just overcomplicate things. Right now, all Strings in Pharo are unicode strings. Characters are represented with their corresponding unicode codepoint. If all characters in a string have codepoints < 256 then they are just stored in a bytestring. Otherwise they are WideStrings. I think assuming a single representation for strings, and then encode when interacting with external apps/APIs is MUCH simpler. > cheers -ben > > [4] > https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-softwa… > > > >> > On 16 Jan 2019, at 10:59, Guillermo Polito <guillermopolito(a)gmail.com> >> wrote: >> > >> > Hi Nicolas, >> > >> > On Wed, Jan 16, 2019 at 10:25 AM Nicolas Cellier < >> nicolas.cellier.aka.nice(a)gmail.com> wrote: >> > IMO, windows VM (and plugins) should do the UCS2 -> UTF8 conversion >> because the purpose of a VM is to provide an OS independant faÃ§ade. >> > I made progress recently in this area, but we should finish the >> job/test/consolidate. >> > >> > I'm following your changes for windows from the shadows and I think >> they are awesome :). >> > >> > If someone bypass the VM and use direct windows API thru FFI, then he >> takes the responsibility, but uniformity doesn't hurt. >> > >> > So far we are using FFI for this, as you say we create first >> Win32WideStrings from utf8 strings and then we use ffi calls to the *W >> functions. >> > I don't think we can make it for Pharo7.0.0. The cycle to build, do >> some acceptance tests, and then bless a new VM as stable is far too long >> for our inminent release :). >> > >> > But this could be for a 7.1.0, and if you like I can surely give a hand >> on this. >> > >> > Guille >> >> >> -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

5 6

Re: [Pharo-dev] Better management of encoding of environment variables
by Guillermo Polito Jan. 19, 2019

Jan. 19, 2019

On Fri, Jan 18, 2019 at 1:58 AM David T. Lewis via Pharo-dev < pharo-dev(a)lists.pharo.org> wrote: > On Thu, Jan 17, 2019 at 04:57:18PM +0100, Sven Van Caekenberghe wrote: > > > > > On 16 Jan 2019, at 23:23, Eliot Miranda <eliot.miranda(a)gmail.com> > wrote: > > > > > > On Wed, Jan 16, 2019 at 2:37 AM Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > > > > > > The image side is perfectly capable of dealing with platform > differences > > > in a clean/clear way, and at least we can then use the full power of > our > > > language and our tools. > > > > > Agreed. +1 At the same time I think it is very important that we don't reply > > on the FFI for environment variable access. This is a basic > cross-platform > > facility. So I would like to see the environment accessed through > primitives, > > but have the image place interpretation on the result of the > primitive(s), > > and have the primitive(s) answer a raw result, just a sequence of > uninterpreted > > bytes. > Having looked at it not so long ago, I'll add my 2cts. Environment access is a very particular scenario. We have in Pharo many startup actions that directly or indirectly (FileLocator home?) require environment variable access, and thus we have to be really careful and picky to make sure that they all work, dependencies are installed in the right order and so on... In Pharo6 this was specially difficult because FFI was dynamically compiling methods, => which required access to argument names, => which required access to the sources files, => which required access to the env vars (because in Pharo the source/changes files are looked up in other directories than the image/vm ones) => which loops :) In Pharo7 argument names in FFI calls are embedded in the method meta-data so all that is avoided. Still I'd agree that moving this support to a primitive would make it less fragile. I'd apply the same to getting/setting the working directory. > > > OK, I can understand that ENV VAR access is more fundamental than FFI > > (although FFI is already essential for Pharo, also during startup). > > > > > VisualWorks takes this approach and provides a class UninterpretedBytes > > > that the VM is aware of. That's always seemed like an ugly name and > > > overkill to me. I would just use ByteArray and provide image level > > > conversion from ByteArray to String, which is what I believe we have > anyway. > > > > Right, bytes are always uninterpreted, else they would be something else. > > We got ByteArray>>#decodedWith: and ByteArray>>#utf8Decoded and our > ByteArray > > inspector decodes automatically if it can. > > > > Hi Sven, > > I am the author of the getenv primitives, and I am also sadly uninformed > about matters of character sets and strings in a multilingual environment. > > The primitives answer environment variable variable values as ByteString > rather than ByteArray. This made sense to me at the time that I wrote it, > because ByteString is easy to display in an inspector, and because it is > easily converted to ByteArray. > > For an American English speaker this seems like a good choice, but I > wonder now if it is a bad decision. Well, as soon as you want to manage some internationalisation, indeed it is. But also it is a source of bugs, because assuming ascii is not right for english either. Most platforms will assume utf8 by default, and it's not quite the same for many symbols :). For example, Character allByteCharacters size. => 256 Character allByteCharacters utf8Encoded size. 384 Character allByteCharacters select: [ :c | c asString utf8Encoded size > 1 ]. 'â¬ âÆââ¦â â¡Ëâ°Å â¹Å Å½ âââââ¢ââËâ¢Å¡âºÅ Å¾Å¸ Â¡Â¢Â£Â¤Â¥Â¦Â§Â¨Â©ÂªÂ«Â¬ÂÂ®Â¯Â°Â±Â²Â³Â´ÂµÂ¶Â·Â¸Â¹ÂºÂ»Â¼Â½Â¾Â¿ÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃÃ Ã¡Ã¢Ã£Ã¤Ã¥Ã¦Ã§Ã¨Ã©ÃªÃ«Ã¬ÃÃ®Ã¯Ã°Ã±Ã²Ã³Ã´ÃµÃ¶Ã·Ã¸Ã¹ÃºÃ»Ã¼Ã½Ã¾Ã¿' Of course, many of those characters may not be used in the day-to-day of many people, but as soon as we find one of those (I'm thinking about the not super strange case of a database storing names :)). Also think about the poor windows users (like myself since 2 weeks ago), that have to think about utf16! BTW, I hope I'm not breaking anybody's mail client by pasting strange characters here :D (and if so, you may want suggest them to review how they manage encoding :)) After all, it is also trivially easy > to convert a ByteArray to ByteString for display in the image. > Yes, but it's sometimes difficult to find such places, as there are many primitives spread in a lot of places doing the wrong thing, which is a source of bugs... I'd like to fix it from the root, the question is how to do it without breaking ^^. In Pharo we are doing at many places, self primitiveXXX asByteArray utf8Decoded So making the primitives return ByteArray instances instead of ByteString should be safe enough :). But this is in my opinion clearly a hack instead of fixing the real problem, and we have to be careful to guard such patterns with comments everywhere explaining why the bytearray conversion is really needed there... > Would it be helpful to have getenv primitives that answer ByteArray > instead, and to let all conversion (including in OSProcess) be done in > the image? > Well, personally I would like that getenv/setenv and getcwd setcwd support are not in a plugin but as a basic service provided by the vm. Cheers, Guille > > Thanks, > Dave > > > -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

3 2

literate programming
by Nicolai Hess Jan. 18, 2019

Jan. 18, 2019

Physically Based Rendering an interesting book, using literate programming, as free book: http://www.pbr-book.org/

1 0

Re: [Pharo-dev] [ANN] New stable VM version.
by Esteban Lorenzano Jan. 18, 2019

Jan. 18, 2019

> On 18 Jan 2019, at 15:57, Sean P. DeNigris via Pharo-dev <pharo-dev(a)lists.pharo.org> wrote: > > > From: "Sean P. DeNigris" <sean(a)clipperadams.com> > Subject: Re: [ANN] New stable VM version. > Date: 18 January 2019 at 15:57:12 CET > To: pharo-dev(a)lists.pharo.org > > > EstebanLM wrote >> Enjoy > > Thanks! Which image versions does this affect? I want to clear my Launcher > cachesâ¦ Pharo 7.0 and Pharo 8.0 images. > > > > ----- > Cheers, > Sean > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html > > >

1 0

[ Save The Date ] Pharo Days 2019: Thursday April 4 & Friday April 5 @ Lille, FR
by Sven Van Caekenberghe Jan. 18, 2019

Jan. 18, 2019

[ Save The Date ] Pharo Days 2019: Thursday April 4 & Friday April 5 @ Lille, FR Dear members of the Pharo community, We are happy to announce that we will be organising Pharo Days this year, in Lille, France. This will be a two day event: Thursday April 4 & Friday April 5. The main venue will be the Amphitheatre of INRIA Lille - Nord Europe. Each day will consist of a number of short 20 to 30 minutes tech talks in the morning, with a more free format in the afternoon: general hacking space, pair programming, demos, side meetings, tutorials, coding sprints, Q&A's, shows us your projects - real life human interaction. Of course, there will be social events as well. Please join us to make this another successful edition, after Annecy (FR) 2011, Lille 2012, Bern (CH) & Lille 2013, Lille 2015, Namur (BE) 2016 and Lille 2017. Mark your calendars, we will provide more details when they become available. The Pharo Board, Association & Consortium http://pharo.org http://association.pharo.org http://consortium.pharo.org

1 0

[ANN] New stable VM version.
by Esteban Lorenzano Jan. 18, 2019

Jan. 18, 2019

Hi, Yes, I needed to promote a new stable version because last weekâs was having a bug (which is not fixed). This one is based on the build: 201901172323-5a38b34 Enjoy (and please report problems) Esteban

4 3

Re: [Pharo-dev] DebugSession>>activePC:
by ducasse Jan. 18, 2019

Jan. 18, 2019

>> >> On my TODO is to make it stand-alone and provide is as a âcompatibility transformâ, too. > > I have to dig because I remember that I went over all the deprecation in Pharo 60 and started to look at the ones that I > could âtransformifiedâ so that we get a nice package that rewrite more :) > > >> So we can add it to methods that we want to keep for compatibility, but they will nevertheless transform the code automatically. >> (this then might be disabled in production to not transform) > > Yes I like that I will look for my code. Apparently I published under MCSmalltalkhubRepository owner: 'PharoExtras' project: âMigrator' Several packages MigratorPharo60 contains what I did for Pharo60 probably Migrator contains some of the Pharo70 Now I would like to understand how to organise it. I have the impression that we should keep the one that can be transformed. I do not think that we should go to Pharo50 because it is too old. Stef

1 0

[Pharo 7.0] Build #131: 22887-TonelWritercreateDefaultOrganizationFrom-can-return-string
by ci-pharo-ci-jenkins2＠inria.fr Jan. 18, 2019

Jan. 18, 2019

There is a new Pharo build available! The status of the build #131 was: SUCCESS. The Pull Request #2267 was integrated: "22887-TonelWritercreateDefaultOrganizationFrom-can-return-string" Pull request url: https://github.com/pharo-project/pharo/pull/2267 Issue Url: https://pharo.fogbugz.com/f/cases/22887 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

1 0

Re: [Pharo-dev] DebugSession>>activePC:
by Eliot Miranda Jan. 18, 2019

Jan. 18, 2019

Hi Thomas, > On Jan 10, 2019, at 2:24 AM, Thomas Dupriez via Pharo-dev <pharo-dev(a)lists.pharo.org> wrote: > > <mime-attachment> in a stack of contexts the active pc is different for the top context. For other than the top context, a contextâs pc will be pointing after the send that created the context above it, so to find the pc of the send one finds the previous pc. For the top context its pc is the active pc. Typically the debugger is invoked in two different modes, interruption or exception. When interrupted, a process is stopped at the next suspension point (method entry or backward branch) and the top context in the process is the context to be displayed in the debugger. When an exception occurs the exception search machinery will find the signaling context, the context that raised the exception, which will be below the search machinery and the debugger invocation above that. The active pc of the signaling context will be the of for the send of digbsl et al. So the distinction is important and the utility method is probably useful. Do you want to remove the method simply because there are no senders in the image? If so, this is indicative of a serious problem with the Pharo development process. In the summer I ported VMMaker.oscog to Pharo 6. Now as feenk try and build a VMMaker.oscog image on Pharo 7, the system is broken, in part because of depreciations and in part because useful methods (isOptimisedBlock (isOptimizedBlock?) in the Opal compiler) have been removed. Just because a method is not in the image does not imply it is not in use. It simply means that it is not in use in the base image. As the system gets modularised this issue will only increase. There are lots of collection methods that exist as a library that are not used in the base image and removing them would clearly damage the library for users. This is the case for lots of so-called system code. There are users out there, like those of us in the vm team, who rely on such system code, and it is extremely unsettling and frustrating to have that system code change all the time. If Pharo is to be a useful platform to the vm team it has to be more stable.

5 9