April 2020 - Pharo-dev - lists.pharo.org

[Pharo-users] Label truncation in new Pharo 8.0 64-bit (stable)

by Shaping

(Third try.) Can someone answer the questions below the screenshot showing the truncated labels? They are repeated here: 1) Can anyone suggest a fix (a code fix) for the truncated labels shown below in the screenshot of the Setting window? This happens in the Git client too? It happens when a fixed-width font is chosen as the default font. This is a design flaw in the GUIs. Cycling Settings (or any other Windows showing the same truncated-labels problem) isnât a fix, even if the window renders correctly the second time. The damaged area of the window is still not being invalidated correctly and then redrawn, or it is, and then is partially overdraw by the data fields to the right of the labels. Changing the font to any fixed-width font causes this problem, even without changing the default font size (10). You can see this by expanding all nodes in Settings, and paging down through them. The longest labels will be incompletely rendered. A possible solution is to use a 2-column layout, where the left column displays right-justified labels and the right one displays the matching left-justified data, checkbox, dropdown, etc. The left label-column could be auto-sized on opening after the longest labelâs size is determined. I mention this problem many months ago, in the midst of another failed port from VW, and presented the same screenshots. Stef made some comments about how this should not be happening, and seemed to be unhappy with the situation. I thought therefore that there would be some resolve to fix the issue broadly throughout the dev environ, but I donât know what came of that. I figured the fix was waiting for Spec2, which is now here. Is there a Spec2 expert here who can suggest how to use the new framework to fix the problem? 2) Will all Pharo tools be rewritten in Spec2, and does its formatting scheme solve the label-formatting problem, as font size is changed? 3) Why donât we opt for easy elastic/zoomable GUIs by implementing Pharo as a web app? Has this been discussed? Willow is looking very good. Is rendering speed still a problem for a complex web GUI served by localhost? For the localhost case, the speed difference seems to be approaching 0. Shaping From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of shaping(a)uurda.org Sent: Sunday, 19 April, 2020 09:32 To: Pharo Development List <pharo-dev(a)lists.pharo.org>; pharo-users(a)lists.pharo.org Subject: Re: [Pharo-users] [Pharo-dev] FW: Label truncation in new Pharo 8.0 64-bit (stable) On 2020-04-19 03:01, Sven Van Caekenberghe wrote: Works for me (this font is larger than the default one). macOS, Pharo 9 64bit The screenshot below is taken on Windows 10 Pro. This time I did not change the theme. I did also update the fonts listed in the dialog before choosing FiraCode. The labels are still truncated. This is another new Pharo 8 64-bit (stable) with only the font changed. Why would formatting not work on Windows 10? Can any Windows 10 Pharo users verify this problem? On 19 Apr 2020, at 06:05, Shaping <shaping(a)uurda.org <mailto:shaping@uurda.org> > wrote: Can anyone suggest a fix for the truncated labels shown below in the screenshot of the Setting window? This happens in the Git client too. Will Pharo tools be rewritten in Spec2, and does its formatting scheme solve generally the text-formatting problem, as font size is changed? Why donât we opt for easy elastic rendering of GUIs by implementing Pharo as a web app? Has this been discussed? Willow is looking very good. Shaping From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org> ] On Behalf Of Shaping Sent: Saturday, 18 April, 2020 04:16 To: 'Any question about pharo is welcome' <pharo-users(a)lists.pharo.org <mailto:pharo-users@lists.pharo.org> >; 'Pharo Development List' <pharo-dev(a)lists.pharo.org <mailto:pharo-dev@lists.pharo.org> > Subject: [Pharo-users] Label truncation in new Pharo 8.0 64-bit (stable) Hi everyone. Below is a Pharo 8.0 64-bit (stable) image (installed with the new Launcher 2.0) with two changes: -Light theme selected. -FiraCode 11 chosen as the default font. <image001.png> Truncated labels in the GUI on font change has always been a problem. Am I the only one experiencing this? Shaping From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org <mailto:pharo-users-bounces@lists.pharo.org> ] On Behalf Of Sanjay Minni Sent: Friday, 17 April, 2020 11:05 To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org <mailto:pharo-users@lists.pharo.org> > Subject: Re: [Pharo-users] issues in setting up a new project on Github/Iceberg Thanks tried just https and this worked too I suppose once cloned it does not really matter whether it was done using "clone from github.com" or "clone remote repository" --- Sanjay Minni +91-9900-902902 On Fri, 17 Apr 2020 at 19:43, Guillermo Polito <guillermopolito(a)gmail.com <mailto:guillermopolito@gmail.com> > wrote: Hi, El 17 abr 2020, a las 15:30, Sanjay Minni <sm(a)planage.com <mailto:sm@planage.com> > escribiÃ³: Hi Guillermo, Ok then given your comments - right now got it working as such - initially created as private in Github - changed to public - created clone from github - (did a repair repository (dont know why) and created "src" and designated it as source folder) - did a commit and push - changed repository back to private seems to work can you give an example on how the "Remote URL" field is to be filled up - note its not taking HTTP as i am not sure about the syntax, assume SSH, Github, MyName, MyRepository, will try that as well you should be able to use the URLs provided by the clone buttons in Github/gitlab/bitbucket, both https and ssh. <image004.png> note Just for Info to other users: I am using Windows 10, it seems to have SSH tools inbuilt and was able to use ssh (with passphrase) without hiccup, by putting my key files path in pharo setting-> ...-> use custom SSH keys, and uploading the .pub key in Github :) Thanks --- Sanjay Minni +91-9900-902902 On Fri, 17 Apr 2020 at 18:36, Guillermo Polito <guillermopolito(a)gmail.com <mailto:guillermopolito@gmail.com> > wrote: Hi Sanjay, This is a bug in Iceberg github integration that tries to access the repository to get metadata using Github REST API using anonymous access apparently. And since your project is private, this fails. As a workaroung, I suggest you to clone using the last option that requires an url only: <PastedGraphic-3.png> This option will not use the Github API, so you should find no problems. Iâm planning doing an Iceberg sprint and fix many issues in the coming weeks. (Also if somebody wants to join, this is an open call :)) Guille El 17 abr 2020, a las 12:49, Sanjay Minni <sm(a)planage.com <mailto:sm@planage.com> > escribiÃ³: Using Pharo 8 64 bit on Windows 10 I am trying to setup a new project on Github using Iceberg following the document "Manage your code with Iceberg dtd 25 March 2019) After creating the project in github when I try to clone from Github as (I am at doc item1.4) The system throws an error and the project does not appear in iceberg although the files are downloaded. This happen in both cases whether I use HTTP or SSH What is going wrong ? also: assuming the clone is made, as per next item in doc 1.5, the directory "src" is created after the clone is made. is that correct ?... then how will the packages sources be mapped to ..\src <http://forum.world.st/file/t368721/1-GettingFirstEmptyProjFromGit.jpg> <http://forum.world.st/file/t368721/0-GithubProjectCreation.jpg> ----- cheers, Sanjay -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

April 21, 2020

[Pharo 9.0] Build #247: FAILURE

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #247 was: FAILURE. Could not extract further issue information from commit message: Delete stale.yml Remove stale Bot. People do not like it Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

April 21, 2020

Re: [Pharo-dev] [Vm-dev] Pony for Pharo VM

by Shaping

Yeah, the search probably works for me because Iâm logged in as a member. Try this: https://ponylang.zulipchat.com/ and search for âlzipâ at the top. This should take you to bits of the thread by Rocco. In those bits you should find the link to his YT vid showing the actor network with message flows: https://www.youtube.com/watch?v=BslZY0D_xAg Shaping From: Robert [mailto:robert.withers@pm.me] Sent: Tuesday, 21 April, 2020 02:38 To: Shaping <shaping(a)uurda.org>; 'Open Smalltalk Virtual Machine Development Discussion' <vm-dev(a)lists.squeakfoundation.org>; 'Pharo Development List' <pharo-dev(a)lists.pharo.org> Subject: Re: [Vm-dev] [Pharo-dev] Pony for Pharo VM Your link does not work. https://ponylang.zulipchat.com/#narrow/search/lzip On 4/21/20 2:05 AM, Shaping wrote: The Pony compiler and runtime need to be studied. What better way than to bring the Pony compiler into Squeak? Build a Pony runtime inside Squeak, with the vm simulator. Build a VM. Then people will learn Pony and it would be great! Yes, that is one way. Then we can simulate the new collector with Smalltalk in the usual way, whilst also integrating ref-caps and dynamic types (the main challenge). We already know that Orca works in Pony (in high-performance productionânot an experiment or toy). Still there will be bugs and perhaps room for improvements. Smalltalk simulation would help greatly there. The simulated Pony-Orca (the term used in the Orca paper) or simulated Smalltalk-Orca, if we can tag classes with ref-caps and keep Orca working, will run even more slowly in simulation-mode with all that message-passing added to the mix. The cost of message passing reduces down when using the CogVM JIT. It is indeed somewhat slower when running in the simulator. I think the objective should be to run the Pony bytecodes Pony is a language, compiler and runtime. The compiler converts Pony source to machine code. on the jitting CogVM. This VM allows you to install your own BytecodeEncoderSet. Note that I was definitely promoting a solution of running Pony on the CogVM, not Orca. Pony is not a VM, either--no bytes codes. We would be studying Orca structure in the Pony C/C++, how that fits with the ref-caps, and then determine how to write something similar in the VM or work Smalltalk dynamic types into the existing Pony C/C++ (not nearly as fun, probably). Iâm starting to study the Pharo VM. Can someone suggest what to read. I see what appears to be outdated VM-related material. Iâm not sure what to study (besides the source code) and what to ignore. Iâm especially interested to know what not to read. I would suggest sticking to Squeak, instead of Pharo, as that is where the VM is designed & developed. Okay. How do Pharoâs and Squeakâs VMs differ? I thought OpenSmalltalkVM was the common VM. I also read something recently from Eliot that seemed to indicate a fork. I thought Pharo had the new tools, like GT, but Iâm not sure. I donât follow Squeak anymore. Here's a couple of interesting blogs covering the CogVM [1][2] regarding VM documentation. The problem is easy to understand. It reduces to StW GCing in a large heap and how to make instead may small, well-managed heaps, one per actor. Orca does that already and demonstrates very high performance. Thatâs what the Orca paper is about. The CogVM has a single heap, divided into "segments" I believe they are called, to dynamically grow to gain new heap space. Yeahâno, it wonât work. Sympathies. Empathies. https://ponylang.zulipchat.com/#narrow/search/lzip Read the thread above and watch the video to sharper your imagination and mental model, somewhat, for how real object-oriented programs work at run-time. The video details are fuzzy, but you can get a good feel for message flow. This should have happened first in Smalltalk. The performance of the GC in the CogVM is demonstrated with this profiling result running all Cryptography tests. Load Cryptography with this script, open the Test Runner select Cryptography tests and click 'Run Profiled': Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'. Here are the profiling results. - 12467 tallies, 12696 msec. **Leaves** 13.8% {1752ms} RGSixtyFourBitRegister64>>loadFrom: 8.7% {1099ms} RGSixtyFourBitRegister64>>bitXor: 7.2% {911ms} RGSixtyFourBitRegister64>>+= 6.0% {763ms} SHA256Inlined64>>processBuffer 5.9% {751ms} RGThirtyTwoBitRegister64>>loadFrom: 4.2% {535ms} RGThirtyTwoBitRegister64>>+= 3.9% {496ms} Random>>nextBytes:into:startingAt: 3.5% {450ms} RGThirtyTwoBitRegister64>>bitXor: 3.4% {429ms} LargePositiveInteger(Integer)>>bitShift: 3.3% {413ms} [] SystemProgressMorph(Morph)>>updateDropShadowCache 3.0% {382ms} RGSixtyFourBitRegister64>>leftRotateBy: 2.2% {280ms} RGThirtyTwoBitRegister64>>leftRotateBy: 1.6% {201ms} Random>>generateStates 1.5% {188ms} SHA512p256(SHA512)>>processBuffer 1.5% {184ms} SHA256Test(TestCase)>>timeout:after: 1.4% {179ms} SHA1Inlined64>>processBuffer 1.4% {173ms} RGSixtyFourBitRegister64>>bitAnd: **Memory** old -16,777,216 bytes young +18,039,800 bytes used +1,262,584 bytes free -18,039,800 bytes **GCs** full 1 totalling 86 ms (0.68% uptime), avg 86 ms incr 307 totalling 81 ms (0.6% uptime), avg 0.3 ms tenures 7,249 (avg 0 GCs/tenure) root table 0 overflows As shown, 1 full GC occurred in 86 ms Not acceptable. Too long. and 307 incremental GCs occurred for a total of 81 ms. All of this GC activity occurred within a profile run lasting 12.7 seconds. The total GC time is just 1.31% of the total time. Very fast. Not acceptable. Too long. And, worse, it wonât scale. The problem is not the percentage; itâs the big delays amidst other domain-specific computation. These times must be much smaller and spread out across many pauses during domain-specific computations. No serious real-time apps can be made in this case. I suggest studying the Pony and Orca material, if the video and accompanying explanation donât clarify Pony-Orca speed and scale. The solution for Smalltalk is more complicated, and will involve a concurrent collector. The best one I can find now is Orca. If you know a better one, please share your facts. As different event loops on different cores will use the same externalizing remote interface This idea is not clear. Is there a description of it? So I gather that the Orca/Pony solution does not treat inter-actor messages, within the same process to be remote calls? Why would the idea of âremoteâ enter here? The execution scope is an OS process. Pony actors run on their respective threads in one OS process. Message passing is zero-copy; all âpassingâ is done by reference. No data is actually copied. The scheduler interleaves all threads needing to share a core if there are more actors than cores. Switching time for actor threads, in that case, is 5 to 15 ns. This was mentioned before. Opportunistic work stealing happens. That means that all the cores stay as busy as possible if there is any work at all left to do. All of this happens by design without intervention or thought from the programmer. You can read about this in the links given earlier. I suggest we copy the design for Smalltalk. If each core has a separate thread and thus a separate event loop, it makes sense to have references to actors in other event loops as a remote actor. Thus the parallelism is well defined. to reach other event loops, we do not need a runtime that can run on all of those cores. We just need to start the minimal image on the CogVM with remote capabilities Pony doesnât yet have machine-node remoteness. The networked version is being planned, but is a ways off still. By remote, do you mean: another machine or another OS/CogVM process on the same machine? Yes, I mean both. I also mean between two event loops within the same process, different threads. I think the Pony runtime is still creating by default just one OS process per app and as many threads as needed, with each actor having only one thread of execution by definition of what an actor is (single-threaded, very simple, very small). A scheduler keeps all cores busy, running and interleaving all the current actor threads. Message tracing maintains ref counts. A cycle-detector keep things tidy. Do Squeak and Pharo have those abilities? to share workload. With Pony-Orca, sharing of the workload doesnât need to be managed by the programmer. When I said sharing of workload is a primary challenge, I do not mean explicitly managing concurrency, the event loop ensures that concurrency safety. I meant that the design of a parallelized application into concurrent actors is the challenge, If you can write a state-machine with actors that each do one very simple, preferably reusable thing in response to received async messages, then itâs not a challenge. We do have to learn how to do it. Itâs not what most of us are used to. Pony is a good tool for practicing, even if the syntax is not interesting. Still, as mentioned, we should make tools to help with that state-machine construction. That comes later, but it must happen. Pony has Actors. It also has Classes. The actors have behaviours. Think of these as async methods. Smalltalk would need new syntax for Actors, behaviours, and the ref-caps that type the objects. Doing this last bit well is the task that concerns me most. that exists for Smalltalk capabilities and Pony capabilities. In fact, instead of talking about actors, concurrency & parallel applications, I prefer to speak of a capabilities model, inherently on an event loop which is the foal point for safe concurrency. I suggest a study of the Pony scheduler. There are actors, mailboxes, message queues, and the scheduler, mainly. You donât need to be concerned about safety. Itâs been handled for you by the runtime and ref-caps. Thatâs one of basic reasons for the existence of Pony-Orca. The Pony-Orca dev writes his actors, and they run automatically in load-balance, via the actor-thread scheduler and work-stealing, when possible, on all the cores. Making Smalltalk work with Orca is, at this early stage, about understanding how Orca works (study the C++ and program in Pony) and how to implement it, if possible, in a Smalltalk simulator. Concerning Orca in particular, if you notice at end of the paper, they tested Orca against Erlang VM, C4, and G1, and it performed much better than all. I suppose it should be measured against the CogVM, to know for sure is the single large heap is a performance bottleneck as compared to Pony/Orca performance with tiny per-actor heaps. I donât have time for Pony programming these days--I canât even read about these days. Go ahead if you wish. Your time is better spent in other ways, though. The speed and scale advantages of Orca over the big-heap approach have been demonstrated. That was done some time ago. Read the paper by Clebsch and friends for details. Read Wallaroo Labâs field-experience whilst preparing to use Pony. Or better, learn to write a Pony program. If your resources donât allow that, chat with Rocco Bowling (link above). Everyone on Pony Zulip is very helpful and super-enthusiastic about Ponyâand it doesnât even have its own debugger the last time I checked. The tooling is poor, and people still love this thing. Odd. The biggest challenge, I think you would agree is the system/application design that provides the opportunities to take advantage of parallelism. It kinda fits the microservices arch. So, we would run 64 instances of squeak to take the multicore to town. No, thatâs much slower. Squeak/Pharo still has the basic threading handicap: a single large heap. In my proposal, with 64 separate squeak processes running across 64 cores, there will be 64 heaps, That would be too few actors, in general. We are not thinking on the same scale for speed and actor-count. Expect actor counts to scale into the thousands or tens of thousands. There are about 100 in the app above. 1 per process. There will be a finite number of Capability actors in each event loop. This finite set of actors within one event loop will be GC-able by the global collector, full & incremental. As all inter-event loop interaction occurs through remote message passing, the differences between inter-vat (a vat is the event loop) communication within one process (create two local Vats), inter-vat communication between event-loops in different processes on the same machine and inter-vat communication between event-loops in different processes on different machines are all modeled exactly the same: remote event loops. Hereâs the gist of the problem again: the big heap will not work and must go away, if we are to have extreme speed and a generalized multithreading programming solution. I am not convinced of this. You must read of othersâ measurements, or write your own programs, and do the tests to get those measurements. Read about the measurements made in the academic paper I cited. Thatâs the easy way. You can also read the one from Sebastian Blessing from 2013: https://www.ponylang.io/media/papers/a_string_of_ponies.pdf My current understanding is that Pony-Orca (or Smalltalk-Orca) starts one OS process, and then spawns threads, as new actors begin working. You donât need to do anything special as a programmer to make that happen. You just write the actors, keep them small, use the ref-caps correctly so that the program compiles (the ref-caps must also be applied to Smalltalk classes), and organize your synchronous code into classes, as usual. Functions run synchronous code. Behaviours run asynchronous code. My point was "writing the actors" and "organizing your synchronous code into classes" are challenging in the sense of choosing what is asynchronous and what is synchronous. Yup, but only for a while. Then you get used to it, and canât imagine anything different, like not having a big heap. The parallel design space holds primacy. No, strictly, the state-machine design does. The parallelization is done for you. Youâre not parallelizing anything. Thatâs not your job. (What a relief, yes?) Youâre an application programmer. Youâre writing a state-machine for your app, and distributing its work across specialized actors, which you code and whose async messages to each other change object data slots (wherever they happen to liveâwhich need not concern you), and thus change the state of the state-machine you designed. You canât use the multicore hardware you already own or the goodness in the Orca and ref-cap design if you canât write a state-machine, and use actors, or donât have a tool to help you do that. Most of us will want to use such a tool even if we are fluent at state-machine design. This doesnât even exist in Pony. Itâs very raw over there, but you get used to the patterns, as with any new strategy. Still I want a tool. Donât you? Two tasks: 1) build tools to help us make state-machines in a reliable pleasant way, so that we feel compelled and happy to do it; and 2) implement Pony-style scheduling, ref-caps, and Orca memory management work in Smalltalk. The issue is not whether to use Pony. I donât like Pony, the language; itâs okay, even very good, but itâs not Smalltalk. I like Smalltalk, who concurrency model is painfully lame. Squeak concurrency model. Installer ss project: 'Cryptography'; install: 'CapabilitiesLocal' What abilities does the above install give Squeak? This installs a local only (no remote capabilites) capabilities model that attempts to implement the following in Squeak, the E-Rights capabilities model. [3] This also ensures inter-actor concurrency safety. So your use of Pony is purely to access the Orca vm? Orca is not a VM; itâs a garbage collection protocol for actor-based systems. I suggest using Pony-Orca to learn how Orca works, and then replace the Pony part of Pony-Orca with Smalltalk (dynamic typing), keeping the ref-caps (because they provide the guarantees). I realize that this is a big undertaking. Or: write a new implementation of Orca in Smalltalk for the VM. This is currently second choice, but that could change. I think you will find the CogVM quite interesting and performant. --Not with its current architecture. If the CogVM is not able to: 1) dynamically schedule unlimited actor-threads on all cores Why not separate actor event-loop processes on each core, communicating remotely? [4][5] --Because it will continue the current Smalltalk-concurrency lameness. Itâs a patch. And still it will not allow the system to scale. The concurrency problem has been solved nearly optimally and at high resolution in the current Pony-Orca. Thereâs room for improvement, but itâs already in a completely different performance league compared to any big-heap Smalltalk. If Iâm to work hard on an implementation of this design for Smalltalk, I need a much greater speed-up and scaling ability than what these patches give. 2) automatically load-balance Use of mobility with actors would allow for automated rebalancing. Speed hit. Too slow/wasteful. Moving an actor isnât needed if the each has its own heap. 3) support actor-based programs innately With this code, asynchronous computation of "number eventual * 100" occurs in an event loop and resolves the promise [:number | number eventual * 100] value: 0.03 "returning an unresolved promise until the async computation completes and resolves the promise" Promises and notifications are fine. Both happen in Pony-Orca. But the promises donât fix the big performance problems. Am I wrong to state that this model allows innate support to actors? Or were you somehow stating that the VM would need innate support? Why does the VM have to know? Itâs not enough. We still have the big pauses from GCs in a large heap. 4) guarantee no data-races The issue to observe is whether computations are long running and livelock the event loop from handling other activations. This is a shared issue, as Pony/Orca are also susceptible to this. Yes, and a dedicated cycle-detecting actor watches for this in Pony-Orca. E-right's event loops ensure no data races, as long as actor objects are not accessible from more than one event-loop. Speed hit. No blocking and no write barriers exist in Pony-Orca. You canât wait. If you need to âwait,â you set a timer and respond to the event when the timer fires. Imagine a cloud based compute engine, processing Cassandra events that uses inter-machine actors to process the massively parallel Cassandra database. Inter-thread communication is not sufficient as there are hundreds of separate nodes. Yes; I didnât claim otherwise. The networked version is coming. See above. My point is that the âremoteâ characterization is not needed. Itâs not helping us describe and understand. Design wise, it makes much sense to treat inter-thread, inter-process and inter-machine concurrency as the same remote interface. No new design is needed for concurrency and interfacing. There is much to implement, however. The design is already done, modulo the not-yet-present network extension. Interfacing between actors is always by async messaging. Messaging will work as transparently as possible in the networked version across machine nodes. The issue is how most efficiently to use Orca, which happens to be working in Pony. Pony is in production in two internal, speed-demanding, banking apps and in Wallaroo Labsâ high-rate streaming product. Pony is a convenient way to study and use a working implementation of Orca. Ergo, use Pony, even if we only study it as a good example of how to use Orca. Some tweaks (probably a lot of them) could allow use of dynamic types. We could roll our own implementation of Orca for the current Pharo VM, but that seems like more work than tweaking a working Pony compiler and runtime. Iâm not sure about that. You know the VM better than I. (I was beginning my study of the Pharo/OpenSmalltalkVM when I found Pony.) Sounds like you might regret your choice and took the wrong path. I donât see how you form that conclusion. Iâve not chosen yet. You stated you are not thrilled with using Pony. I donât like the Pony language syntax. I donât like anything that looks like Algo-60. Pony is a language, compiler and runtime implementing Orca. The other stuff is good. And Iâve not had much time to use it; I suspect I could like it more. [â¦] If most of what Squeak/Pharo offers is pleasant/productive VM simulation, much work still remains to achieve even a basic actor system and collector, but the writing of VM code in Smalltalk and compiling it to C may be much more productive than writing C++. The C++ for the Pony compiler and runtime, however, already compiles and works well. Thus, starting the work in C++ is somewhat tempting. Can someone explain the limits of how the VM simulator can be used? How much VM core C is not a part of what can be compiled from Smalltalk? Can all VM C code be compiled from Smalltalk? Can someone answer the above question? [1] Cog Blog - http://www.mirandabanda.org/cogblog/ [2] Smalltalk, Tips 'n Tricks - https://clementbera.wordpress.com/ [3] Capability Computation - http://erights.org/elib/capability/index.html [4] Concurrency (Event Loops) - http://erights.org/elib/concurrency/index.html [5] Distributed Programming - http://erights.org/elib/distrib/index.html Shaping -- Kindly, Robert

April 21, 2020

[Pharo 9.0] Build #246: Fix Process>#isSuspended

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #246 was: FAILURE. The Pull Request #6197 was integrated: "Fix Process>#isSuspended" Pull request url: https://github.com/pharo-project/pharo/pull/6197 Issue Url: https://github.com/pharo-project/pharo/issues/isSuspendedProcess Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

April 21, 2020

[Mm10s] 2020-04-20

by tesonep＠gmail.com

Hi, a little resume of my tasks last week. - Monday Holiday - VM tasks documentation - Reviewing PRs - Lowcode Sync - Read Lowcode Code - JIT Testing - Idle VM: FIxing problems with semaphores and Threads (they had a lock-free implementation but it was not robust enough). - Idle VM: Fixing AIO interrupt error (The AIO poll has to be aware of external semaphores to signal) - Idle VM: Adding SDL support to run in the main thread (Fully implementing the SDL binding in UFFI, so we can configure it). - Idle VM: Fixing callbacks problems [W/Esteban] (The callback queue was using a WeakSet that was concurrently accessed and modified, also it is growing and shrinking all the time). - Idle VM: Errors debugging in callbacks [W/Esteban] [WIP] - Task-it Problem [WIP]: Some contexts are kept for debugging, producing leaks. - Idle VM: Adding an AIO implementation using KQueue for OSX. Select is not a good player with FIFO and Pipes in OSX, in Linux goes well. This week (starting 2020-04-20), just the tip of the iceberg. - Large Images: Checking Remaining issues - Task-it Problem - Pharo Status presentation - Pharo Status Techtalk. - [VM] Fixing Tests in Idle. - Reviewing PR & Issues - [VM] Writing tests for JIT generation -- Pablo Tesone. tesonep(a)gmail.com

April 21, 2020

Re: [Pharo-dev] [Vm-dev] Pony for Pharo VM: Pony papers

by Shaping

https://www.ponylang.io/community/#papers Shaping From: Pharo-dev [mailto:pharo-dev-bounces@lists.pharo.org] On Behalf Of Shaping Sent: Tuesday, 21 April, 2020 01:05 To: 'Robert' <robert.withers(a)pm.me>; 'Open Smalltalk Virtual Machine Development Discussion' <vm-dev(a)lists.squeakfoundation.org>; 'Pharo Development List' <pharo-dev(a)lists.pharo.org> Subject: Re: [Pharo-dev] [Vm-dev] Pony for Pharo VM The Pony compiler and runtime need to be studied. What better way than to bring the Pony compiler into Squeak? Build a Pony runtime inside Squeak, with the vm simulator. Build a VM. Then people will learn Pony and it would be great! Yes, that is one way. Then we can simulate the new collector with Smalltalk in the usual way, whilst also integrating ref-caps and dynamic types (the main challenge). We already know that Orca works in Pony (in high-performance productionânot an experiment or toy). Still there will be bugs and perhaps room for improvements. Smalltalk simulation would help greatly there. The simulated Pony-Orca (the term used in the Orca paper) or simulated Smalltalk-Orca, if we can tag classes with ref-caps and keep Orca working, will run even more slowly in simulation-mode with all that message-passing added to the mix. The cost of message passing reduces down when using the CogVM JIT. It is indeed somewhat slower when running in the simulator. I think the objective should be to run the Pony bytecodes Pony is a language, compiler and runtime. The compiler converts Pony source to machine code. on the jitting CogVM. This VM allows you to install your own BytecodeEncoderSet. Note that I was definitely promoting a solution of running Pony on the CogVM, not Orca. Pony is not a VM, either--no bytes codes. We would be studying Orca structure in the Pony C/C++, how that fits with the ref-caps, and then determine how to write something similar in the VM or work Smalltalk dynamic types into the existing Pony C/C++ (not nearly as fun, probably). Iâm starting to study the Pharo VM. Can someone suggest what to read. I see what appears to be outdated VM-related material. Iâm not sure what to study (besides the source code) and what to ignore. Iâm especially interested to know what not to read. I would suggest sticking to Squeak, instead of Pharo, as that is where the VM is designed & developed. Okay. How do Pharoâs and Squeakâs VMs differ? I thought OpenSmalltalkVM was the common VM. I also read something recently from Eliot that seemed to indicate a fork. I thought Pharo had the new tools, like GT, but Iâm not sure. I donât follow Squeak anymore. Here's a couple of interesting blogs covering the CogVM [1][2] regarding VM documentation. The problem is easy to understand. It reduces to StW GCing in a large heap and how to make instead may small, well-managed heaps, one per actor. Orca does that already and demonstrates very high performance. Thatâs what the Orca paper is about. The CogVM has a single heap, divided into "segments" I believe they are called, to dynamically grow to gain new heap space. Yeahâno, it wonât work. Sympathies. Empathies. https://ponylang.zulipchat.com/#narrow/search/lzip Read the thread above and watch the video to sharper your imagination and mental model, somewhat, for how real object-oriented programs work at run-time. The video details are fuzzy, but you can get a good feel for message flow. This should have happened first in Smalltalk. The performance of the GC in the CogVM is demonstrated with this profiling result running all Cryptography tests. Load Cryptography with this script, open the Test Runner select Cryptography tests and click 'Run Profiled': Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'. Here are the profiling results. - 12467 tallies, 12696 msec. **Leaves** 13.8% {1752ms} RGSixtyFourBitRegister64>>loadFrom: 8.7% {1099ms} RGSixtyFourBitRegister64>>bitXor: 7.2% {911ms} RGSixtyFourBitRegister64>>+= 6.0% {763ms} SHA256Inlined64>>processBuffer 5.9% {751ms} RGThirtyTwoBitRegister64>>loadFrom: 4.2% {535ms} RGThirtyTwoBitRegister64>>+= 3.9% {496ms} Random>>nextBytes:into:startingAt: 3.5% {450ms} RGThirtyTwoBitRegister64>>bitXor: 3.4% {429ms} LargePositiveInteger(Integer)>>bitShift: 3.3% {413ms} [] SystemProgressMorph(Morph)>>updateDropShadowCache 3.0% {382ms} RGSixtyFourBitRegister64>>leftRotateBy: 2.2% {280ms} RGThirtyTwoBitRegister64>>leftRotateBy: 1.6% {201ms} Random>>generateStates 1.5% {188ms} SHA512p256(SHA512)>>processBuffer 1.5% {184ms} SHA256Test(TestCase)>>timeout:after: 1.4% {179ms} SHA1Inlined64>>processBuffer 1.4% {173ms} RGSixtyFourBitRegister64>>bitAnd: **Memory** old -16,777,216 bytes young +18,039,800 bytes used +1,262,584 bytes free -18,039,800 bytes **GCs** full 1 totalling 86 ms (0.68% uptime), avg 86 ms incr 307 totalling 81 ms (0.6% uptime), avg 0.3 ms tenures 7,249 (avg 0 GCs/tenure) root table 0 overflows As shown, 1 full GC occurred in 86 ms Not acceptable. Too long. and 307 incremental GCs occurred for a total of 81 ms. All of this GC activity occurred within a profile run lasting 12.7 seconds. The total GC time is just 1.31% of the total time. Very fast. Not acceptable. Too long. And, worse, it wonât scale. The problem is not the percentage; itâs the big delays amidst other domain-specific computation. These times must be much smaller and spread out across many pauses during domain-specific computations. No serious real-time apps can be made in this case. I suggest studying the Pony and Orca material, if the video and accompanying explanation donât clarify Pony-Orca speed and scale. The solution for Smalltalk is more complicated, and will involve a concurrent collector. The best one I can find now is Orca. If you know a better one, please share your facts. As different event loops on different cores will use the same externalizing remote interface This idea is not clear. Is there a description of it? So I gather that the Orca/Pony solution does not treat inter-actor messages, within the same process to be remote calls? Why would the idea of âremoteâ enter here? The execution scope is an OS process. Pony actors run on their respective threads in one OS process. Message passing is zero-copy; all âpassingâ is done by reference. No data is actually copied. The scheduler interleaves all threads needing to share a core if there are more actors than cores. Switching time for actor threads, in that case, is 5 to 15 ns. This was mentioned before. Opportunistic work stealing happens. That means that all the cores stay as busy as possible if there is any work at all left to do. All of this happens by design without intervention or thought from the programmer. You can read about this in the links given earlier. I suggest we copy the design for Smalltalk. If each core has a separate thread and thus a separate event loop, it makes sense to have references to actors in other event loops as a remote actor. Thus the parallelism is well defined. to reach other event loops, we do not need a runtime that can run on all of those cores. We just need to start the minimal image on the CogVM with remote capabilities Pony doesnât yet have machine-node remoteness. The networked version is being planned, but is a ways off still. By remote, do you mean: another machine or another OS/CogVM process on the same machine? Yes, I mean both. I also mean between two event loops within the same process, different threads. I think the Pony runtime is still creating by default just one OS process per app and as many threads as needed, with each actor having only one thread of execution by definition of what an actor is (single-threaded, very simple, very small). A scheduler keeps all cores busy, running and interleaving all the current actor threads. Message tracing maintains ref counts. A cycle-detector keep things tidy. Do Squeak and Pharo have those abilities? to share workload. With Pony-Orca, sharing of the workload doesnât need to be managed by the programmer. When I said sharing of workload is a primary challenge, I do not mean explicitly managing concurrency, the event loop ensures that concurrency safety. I meant that the design of a parallelized application into concurrent actors is the challenge, If you can write a state-machine with actors that each do one very simple, preferably reusable thing in response to received async messages, then itâs not a challenge. We do have to learn how to do it. Itâs not what most of us are used to. Pony is a good tool for practicing, even if the syntax is not interesting. Still, as mentioned, we should make tools to help with that state-machine construction. That comes later, but it must happen. Pony has Actors. It also has Classes. The actors have behaviours. Think of these as async methods. Smalltalk would need new syntax for Actors, behaviours, and the ref-caps that type the objects. Doing this last bit well is the task that concerns me most. that exists for Smalltalk capabilities and Pony capabilities. In fact, instead of talking about actors, concurrency & parallel applications, I prefer to speak of a capabilities model, inherently on an event loop which is the foal point for safe concurrency. I suggest a study of the Pony scheduler. There are actors, mailboxes, message queues, and the scheduler, mainly. You donât need to be concerned about safety. Itâs been handled for you by the runtime and ref-caps. Thatâs one of basic reasons for the existence of Pony-Orca. The Pony-Orca dev writes his actors, and they run automatically in load-balance, via the actor-thread scheduler and work-stealing, when possible, on all the cores. Making Smalltalk work with Orca is, at this early stage, about understanding how Orca works (study the C++ and program in Pony) and how to implement it, if possible, in a Smalltalk simulator. Concerning Orca in particular, if you notice at end of the paper, they tested Orca against Erlang VM, C4, and G1, and it performed much better than all. I suppose it should be measured against the CogVM, to know for sure is the single large heap is a performance bottleneck as compared to Pony/Orca performance with tiny per-actor heaps. I donât have time for Pony programming these days--I canât even read about these days. Go ahead if you wish. Your time is better spent in other ways, though. The speed and scale advantages of Orca over the big-heap approach have been demonstrated. That was done some time ago. Read the paper by Clebsch and friends for details. Read Wallaroo Labâs field-experience whilst preparing to use Pony. Or better, learn to write a Pony program. If your resources donât allow that, chat with Rocco Bowling (link above). Everyone on Pony Zulip is very helpful and super-enthusiastic about Ponyâand it doesnât even have its own debugger the last time I checked. The tooling is poor, and people still love this thing. Odd. The biggest challenge, I think you would agree is the system/application design that provides the opportunities to take advantage of parallelism. It kinda fits the microservices arch. So, we would run 64 instances of squeak to take the multicore to town. No, thatâs much slower. Squeak/Pharo still has the basic threading handicap: a single large heap. In my proposal, with 64 separate squeak processes running across 64 cores, there will be 64 heaps, That would be too few actors, in general. We are not thinking on the same scale for speed and actor-count. Expect actor counts to scale into the thousands or tens of thousands. There are about 100 in the app above. 1 per process. There will be a finite number of Capability actors in each event loop. This finite set of actors within one event loop will be GC-able by the global collector, full & incremental. As all inter-event loop interaction occurs through remote message passing, the differences between inter-vat (a vat is the event loop) communication within one process (create two local Vats), inter-vat communication between event-loops in different processes on the same machine and inter-vat communication between event-loops in different processes on different machines are all modeled exactly the same: remote event loops. Hereâs the gist of the problem again: the big heap will not work and must go away, if we are to have extreme speed and a generalized multithreading programming solution. I am not convinced of this. You must read of othersâ measurements, or write your own programs, and do the tests to get those measurements. Read about the measurements made in the academic paper I cited. Thatâs the easy way. You can also read the one from Sebastian Blessing from 2013: https://www.ponylang.io/media/papers/a_string_of_ponies.pdf My current understanding is that Pony-Orca (or Smalltalk-Orca) starts one OS process, and then spawns threads, as new actors begin working. You donât need to do anything special as a programmer to make that happen. You just write the actors, keep them small, use the ref-caps correctly so that the program compiles (the ref-caps must also be applied to Smalltalk classes), and organize your synchronous code into classes, as usual. Functions run synchronous code. Behaviours run asynchronous code. My point was "writing the actors" and "organizing your synchronous code into classes" are challenging in the sense of choosing what is asynchronous and what is synchronous. Yup, but only for a while. Then you get used to it, and canât imagine anything different, like not having a big heap. The parallel design space holds primacy. No, strictly, the state-machine design does. The parallelization is done for you. Youâre not parallelizing anything. Thatâs not your job. (What a relief, yes?) Youâre an application programmer. Youâre writing a state-machine for your app, and distributing its work across specialized actors, which you code and whose async messages to each other change object data slots (wherever they happen to liveâwhich need not concern you), and thus change the state of the state-machine you designed. You canât use the multicore hardware you already own or the goodness in the Orca and ref-cap design if you canât write a state-machine, and use actors, or donât have a tool to help you do that. Most of us will want to use such a tool even if we are fluent at state-machine design. This doesnât even exist in Pony. Itâs very raw over there, but you get used to the patterns, as with any new strategy. Still I want a tool. Donât you? Two tasks: 1) build tools to help us make state-machines in a reliable pleasant way, so that we feel compelled and happy to do it; and 2) implement Pony-style scheduling, ref-caps, and Orca memory management work in Smalltalk. The issue is not whether to use Pony. I donât like Pony, the language; itâs okay, even very good, but itâs not Smalltalk. I like Smalltalk, who concurrency model is painfully lame. Squeak concurrency model. Installer ss project: 'Cryptography'; install: 'CapabilitiesLocal' What abilities does the above install give Squeak? This installs a local only (no remote capabilites) capabilities model that attempts to implement the following in Squeak, the E-Rights capabilities model. [3] This also ensures inter-actor concurrency safety. So your use of Pony is purely to access the Orca vm? Orca is not a VM; itâs a garbage collection protocol for actor-based systems. I suggest using Pony-Orca to learn how Orca works, and then replace the Pony part of Pony-Orca with Smalltalk (dynamic typing), keeping the ref-caps (because they provide the guarantees). I realize that this is a big undertaking. Or: write a new implementation of Orca in Smalltalk for the VM. This is currently second choice, but that could change. I think you will find the CogVM quite interesting and performant. --Not with its current architecture. If the CogVM is not able to: 1) dynamically schedule unlimited actor-threads on all cores Why not separate actor event-loop processes on each core, communicating remotely? [4][5] --Because it will continue the current Smalltalk-concurrency lameness. Itâs a patch. And still it will not allow the system to scale. The concurrency problem has been solved nearly optimally and at high resolution in the current Pony-Orca. Thereâs room for improvement, but itâs already in a completely different performance league compared to any big-heap Smalltalk. If Iâm to work hard on an implementation of this design for Smalltalk, I need a much greater speed-up and scaling ability than what these patches give. 2) automatically load-balance Use of mobility with actors would allow for automated rebalancing. Speed hit. Too slow/wasteful. Moving an actor isnât needed if the each has its own heap. 3) support actor-based programs innately With this code, asynchronous computation of "number eventual * 100" occurs in an event loop and resolves the promise [:number | number eventual * 100] value: 0.03 "returning an unresolved promise until the async computation completes and resolves the promise" Promises and notifications are fine. Both happen in Pony-Orca. But the promises donât fix the big performance problems. Am I wrong to state that this model allows innate support to actors? Or were you somehow stating that the VM would need innate support? Why does the VM have to know? Itâs not enough. We still have the big pauses from GCs in a large heap. 4) guarantee no data-races The issue to observe is whether computations are long running and livelock the event loop from handling other activations. This is a shared issue, as Pony/Orca are also susceptible to this. Yes, and a dedicated cycle-detecting actor watches for this in Pony-Orca. E-right's event loops ensure no data races, as long as actor objects are not accessible from more than one event-loop. Speed hit. No blocking and no write barriers exist in Pony-Orca. You canât wait. If you need to âwait,â you set a timer and respond to the event when the timer fires. Imagine a cloud based compute engine, processing Cassandra events that uses inter-machine actors to process the massively parallel Cassandra database. Inter-thread communication is not sufficient as there are hundreds of separate nodes. Yes; I didnât claim otherwise. The networked version is coming. See above. My point is that the âremoteâ characterization is not needed. Itâs not helping us describe and understand. Design wise, it makes much sense to treat inter-thread, inter-process and inter-machine concurrency as the same remote interface. No new design is needed for concurrency and interfacing. There is much to implement, however. The design is already done, modulo the not-yet-present network extension. Interfacing between actors is always by async messaging. Messaging will work as transparently as possible in the networked version across machine nodes. The issue is how most efficiently to use Orca, which happens to be working in Pony. Pony is in production in two internal, speed-demanding, banking apps and in Wallaroo Labsâ high-rate streaming product. Pony is a convenient way to study and use a working implementation of Orca. Ergo, use Pony, even if we only study it as a good example of how to use Orca. Some tweaks (probably a lot of them) could allow use of dynamic types. We could roll our own implementation of Orca for the current Pharo VM, but that seems like more work than tweaking a working Pony compiler and runtime. Iâm not sure about that. You know the VM better than I. (I was beginning my study of the Pharo/OpenSmalltalkVM when I found Pony.) Sounds like you might regret your choice and took the wrong path. I donât see how you form that conclusion. Iâve not chosen yet. You stated you are not thrilled with using Pony. I donât like the Pony language syntax. I donât like anything that looks like Algo-60. Pony is a language, compiler and runtime implementing Orca. The other stuff is good. And Iâve not had much time to use it; I suspect I could like it more. [â¦] If most of what Squeak/Pharo offers is pleasant/productive VM simulation, much work still remains to achieve even a basic actor system and collector, but the writing of VM code in Smalltalk and compiling it to C may be much more productive than writing C++. The C++ for the Pony compiler and runtime, however, already compiles and works well. Thus, starting the work in C++ is somewhat tempting. Can someone explain the limits of how the VM simulator can be used? How much VM core C is not a part of what can be compiled from Smalltalk? Can all VM C code be compiled from Smalltalk? Can someone answer the above question? [1] Cog Blog - http://www.mirandabanda.org/cogblog/ [2] Smalltalk, Tips 'n Tricks - https://clementbera.wordpress.com/ [3] Capability Computation - http://erights.org/elib/capability/index.html [4] Concurrency (Event Loops) - http://erights.org/elib/concurrency/index.html [5] Distributed Programming - http://erights.org/elib/distrib/index.html Shaping

April 21, 2020

Re: [Pharo-dev] [Vm-dev] Pony for Pharo VM

by Shaping

The Pony compiler and runtime need to be studied. What better way than to bring the Pony compiler into Squeak? Build a Pony runtime inside Squeak, with the vm simulator. Build a VM. Then people will learn Pony and it would be great! Yes, that is one way. Then we can simulate the new collector with Smalltalk in the usual way, whilst also integrating ref-caps and dynamic types (the main challenge). We already know that Orca works in Pony (in high-performance productionânot an experiment or toy). Still there will be bugs and perhaps room for improvements. Smalltalk simulation would help greatly there. The simulated Pony-Orca (the term used in the Orca paper) or simulated Smalltalk-Orca, if we can tag classes with ref-caps and keep Orca working, will run even more slowly in simulation-mode with all that message-passing added to the mix. The cost of message passing reduces down when using the CogVM JIT. It is indeed somewhat slower when running in the simulator. I think the objective should be to run the Pony bytecodes Pony is a language, compiler and runtime. The compiler converts Pony source to machine code. on the jitting CogVM. This VM allows you to install your own BytecodeEncoderSet. Note that I was definitely promoting a solution of running Pony on the CogVM, not Orca. Pony is not a VM, either--no bytes codes. We would be studying Orca structure in the Pony C/C++, how that fits with the ref-caps, and then determine how to write something similar in the VM or work Smalltalk dynamic types into the existing Pony C/C++ (not nearly as fun, probably). Iâm starting to study the Pharo VM. Can someone suggest what to read. I see what appears to be outdated VM-related material. Iâm not sure what to study (besides the source code) and what to ignore. Iâm especially interested to know what not to read. I would suggest sticking to Squeak, instead of Pharo, as that is where the VM is designed & developed. Okay. How do Pharoâs and Squeakâs VMs differ? I thought OpenSmalltalkVM was the common VM. I also read something recently from Eliot that seemed to indicate a fork. I thought Pharo had the new tools, like GT, but Iâm not sure. I donât follow Squeak anymore. Here's a couple of interesting blogs covering the CogVM [1][2] regarding VM documentation. The problem is easy to understand. It reduces to StW GCing in a large heap and how to make instead may small, well-managed heaps, one per actor. Orca does that already and demonstrates very high performance. Thatâs what the Orca paper is about. The CogVM has a single heap, divided into "segments" I believe they are called, to dynamically grow to gain new heap space. Yeahâno, it wonât work. Sympathies. Empathies. https://ponylang.zulipchat.com/#narrow/search/lzip Read the thread above and watch the video to sharper your imagination and mental model, somewhat, for how real object-oriented programs work at run-time. The video details are fuzzy, but you can get a good feel for message flow. This should have happened first in Smalltalk. The performance of the GC in the CogVM is demonstrated with this profiling result running all Cryptography tests. Load Cryptography with this script, open the Test Runner select Cryptography tests and click 'Run Profiled': Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'. Here are the profiling results. - 12467 tallies, 12696 msec. **Leaves** 13.8% {1752ms} RGSixtyFourBitRegister64>>loadFrom: 8.7% {1099ms} RGSixtyFourBitRegister64>>bitXor: 7.2% {911ms} RGSixtyFourBitRegister64>>+= 6.0% {763ms} SHA256Inlined64>>processBuffer 5.9% {751ms} RGThirtyTwoBitRegister64>>loadFrom: 4.2% {535ms} RGThirtyTwoBitRegister64>>+= 3.9% {496ms} Random>>nextBytes:into:startingAt: 3.5% {450ms} RGThirtyTwoBitRegister64>>bitXor: 3.4% {429ms} LargePositiveInteger(Integer)>>bitShift: 3.3% {413ms} [] SystemProgressMorph(Morph)>>updateDropShadowCache 3.0% {382ms} RGSixtyFourBitRegister64>>leftRotateBy: 2.2% {280ms} RGThirtyTwoBitRegister64>>leftRotateBy: 1.6% {201ms} Random>>generateStates 1.5% {188ms} SHA512p256(SHA512)>>processBuffer 1.5% {184ms} SHA256Test(TestCase)>>timeout:after: 1.4% {179ms} SHA1Inlined64>>processBuffer 1.4% {173ms} RGSixtyFourBitRegister64>>bitAnd: **Memory** old -16,777,216 bytes young +18,039,800 bytes used +1,262,584 bytes free -18,039,800 bytes **GCs** full 1 totalling 86 ms (0.68% uptime), avg 86 ms incr 307 totalling 81 ms (0.6% uptime), avg 0.3 ms tenures 7,249 (avg 0 GCs/tenure) root table 0 overflows As shown, 1 full GC occurred in 86 ms Not acceptable. Too long. and 307 incremental GCs occurred for a total of 81 ms. All of this GC activity occurred within a profile run lasting 12.7 seconds. The total GC time is just 1.31% of the total time. Very fast. Not acceptable. Too long. And, worse, it wonât scale. The problem is not the percentage; itâs the big delays amidst other domain-specific computation. These times must be much smaller and spread out across many pauses during domain-specific computations. No serious real-time apps can be made in this case. I suggest studying the Pony and Orca material, if the video and accompanying explanation donât clarify Pony-Orca speed and scale. The solution for Smalltalk is more complicated, and will involve a concurrent collector. The best one I can find now is Orca. If you know a better one, please share your facts. As different event loops on different cores will use the same externalizing remote interface This idea is not clear. Is there a description of it? So I gather that the Orca/Pony solution does not treat inter-actor messages, within the same process to be remote calls? Why would the idea of âremoteâ enter here? The execution scope is an OS process. Pony actors run on their respective threads in one OS process. Message passing is zero-copy; all âpassingâ is done by reference. No data is actually copied. The scheduler interleaves all threads needing to share a core if there are more actors than cores. Switching time for actor threads, in that case, is 5 to 15 ns. This was mentioned before. Opportunistic work stealing happens. That means that all the cores stay as busy as possible if there is any work at all left to do. All of this happens by design without intervention or thought from the programmer. You can read about this in the links given earlier. I suggest we copy the design for Smalltalk. If each core has a separate thread and thus a separate event loop, it makes sense to have references to actors in other event loops as a remote actor. Thus the parallelism is well defined. to reach other event loops, we do not need a runtime that can run on all of those cores. We just need to start the minimal image on the CogVM with remote capabilities Pony doesnât yet have machine-node remoteness. The networked version is being planned, but is a ways off still. By remote, do you mean: another machine or another OS/CogVM process on the same machine? Yes, I mean both. I also mean between two event loops within the same process, different threads. I think the Pony runtime is still creating by default just one OS process per app and as many threads as needed, with each actor having only one thread of execution by definition of what an actor is (single-threaded, very simple, very small). A scheduler keeps all cores busy, running and interleaving all the current actor threads. Message tracing maintains ref counts. A cycle-detector keep things tidy. Do Squeak and Pharo have those abilities? to share workload. With Pony-Orca, sharing of the workload doesnât need to be managed by the programmer. When I said sharing of workload is a primary challenge, I do not mean explicitly managing concurrency, the event loop ensures that concurrency safety. I meant that the design of a parallelized application into concurrent actors is the challenge, If you can write a state-machine with actors that each do one very simple, preferably reusable thing in response to received async messages, then itâs not a challenge. We do have to learn how to do it. Itâs not what most of us are used to. Pony is a good tool for practicing, even if the syntax is not interesting. Still, as mentioned, we should make tools to help with that state-machine construction. That comes later, but it must happen. Pony has Actors. It also has Classes. The actors have behaviours. Think of these as async methods. Smalltalk would need new syntax for Actors, behaviours, and the ref-caps that type the objects. Doing this last bit well is the task that concerns me most. that exists for Smalltalk capabilities and Pony capabilities. In fact, instead of talking about actors, concurrency & parallel applications, I prefer to speak of a capabilities model, inherently on an event loop which is the foal point for safe concurrency. I suggest a study of the Pony scheduler. There are actors, mailboxes, message queues, and the scheduler, mainly. You donât need to be concerned about safety. Itâs been handled for you by the runtime and ref-caps. Thatâs one of basic reasons for the existence of Pony-Orca. The Pony-Orca dev writes his actors, and they run automatically in load-balance, via the actor-thread scheduler and work-stealing, when possible, on all the cores. Making Smalltalk work with Orca is, at this early stage, about understanding how Orca works (study the C++ and program in Pony) and how to implement it, if possible, in a Smalltalk simulator. Concerning Orca in particular, if you notice at end of the paper, they tested Orca against Erlang VM, C4, and G1, and it performed much better than all. I suppose it should be measured against the CogVM, to know for sure is the single large heap is a performance bottleneck as compared to Pony/Orca performance with tiny per-actor heaps. I donât have time for Pony programming these days--I canât even read about these days. Go ahead if you wish. Your time is better spent in other ways, though. The speed and scale advantages of Orca over the big-heap approach have been demonstrated. That was done some time ago. Read the paper by Clebsch and friends for details. Read Wallaroo Labâs field-experience whilst preparing to use Pony. Or better, learn to write a Pony program. If your resources donât allow that, chat with Rocco Bowling (link above). Everyone on Pony Zulip is very helpful and super-enthusiastic about Ponyâand it doesnât even have its own debugger the last time I checked. The tooling is poor, and people still love this thing. Odd. The biggest challenge, I think you would agree is the system/application design that provides the opportunities to take advantage of parallelism. It kinda fits the microservices arch. So, we would run 64 instances of squeak to take the multicore to town. No, thatâs much slower. Squeak/Pharo still has the basic threading handicap: a single large heap. In my proposal, with 64 separate squeak processes running across 64 cores, there will be 64 heaps, That would be too few actors, in general. We are not thinking on the same scale for speed and actor-count. Expect actor counts to scale into the thousands or tens of thousands. There are about 100 in the app above. 1 per process. There will be a finite number of Capability actors in each event loop. This finite set of actors within one event loop will be GC-able by the global collector, full & incremental. As all inter-event loop interaction occurs through remote message passing, the differences between inter-vat (a vat is the event loop) communication within one process (create two local Vats), inter-vat communication between event-loops in different processes on the same machine and inter-vat communication between event-loops in different processes on different machines are all modeled exactly the same: remote event loops. Hereâs the gist of the problem again: the big heap will not work and must go away, if we are to have extreme speed and a generalized multithreading programming solution. I am not convinced of this. You must read of othersâ measurements, or write your own programs, and do the tests to get those measurements. Read about the measurements made in the academic paper I cited. Thatâs the easy way. You can also read the one from Sebastian Blessing from 2013: https://www.ponylang.io/media/papers/a_string_of_ponies.pdf My current understanding is that Pony-Orca (or Smalltalk-Orca) starts one OS process, and then spawns threads, as new actors begin working. You donât need to do anything special as a programmer to make that happen. You just write the actors, keep them small, use the ref-caps correctly so that the program compiles (the ref-caps must also be applied to Smalltalk classes), and organize your synchronous code into classes, as usual. Functions run synchronous code. Behaviours run asynchronous code. My point was "writing the actors" and "organizing your synchronous code into classes" are challenging in the sense of choosing what is asynchronous and what is synchronous. Yup, but only for a while. Then you get used to it, and canât imagine anything different, like not having a big heap. The parallel design space holds primacy. No, strictly, the state-machine design does. The parallelization is done for you. Youâre not parallelizing anything. Thatâs not your job. (What a relief, yes?) Youâre an application programmer. Youâre writing a state-machine for your app, and distributing its work across specialized actors, which you code and whose async messages to each other change object data slots (wherever they happen to liveâwhich need not concern you), and thus change the state of the state-machine you designed. You canât use the multicore hardware you already own or the goodness in the Orca and ref-cap design if you canât write a state-machine, and use actors, or donât have a tool to help you do that. Most of us will want to use such a tool even if we are fluent at state-machine design. This doesnât even exist in Pony. Itâs very raw over there, but you get used to the patterns, as with any new strategy. Still I want a tool. Donât you? Two tasks: 1) build tools to help us make state-machines in a reliable pleasant way, so that we feel compelled and happy to do it; and 2) implement Pony-style scheduling, ref-caps, and Orca memory management work in Smalltalk. The issue is not whether to use Pony. I donât like Pony, the language; itâs okay, even very good, but itâs not Smalltalk. I like Smalltalk, who concurrency model is painfully lame. Squeak concurrency model. Installer ss project: 'Cryptography'; install: 'CapabilitiesLocal' What abilities does the above install give Squeak? This installs a local only (no remote capabilites) capabilities model that attempts to implement the following in Squeak, the E-Rights capabilities model. [3] This also ensures inter-actor concurrency safety. So your use of Pony is purely to access the Orca vm? Orca is not a VM; itâs a garbage collection protocol for actor-based systems. I suggest using Pony-Orca to learn how Orca works, and then replace the Pony part of Pony-Orca with Smalltalk (dynamic typing), keeping the ref-caps (because they provide the guarantees). I realize that this is a big undertaking. Or: write a new implementation of Orca in Smalltalk for the VM. This is currently second choice, but that could change. I think you will find the CogVM quite interesting and performant. --Not with its current architecture. If the CogVM is not able to: 1) dynamically schedule unlimited actor-threads on all cores Why not separate actor event-loop processes on each core, communicating remotely? [4][5] --Because it will continue the current Smalltalk-concurrency lameness. Itâs a patch. And still it will not allow the system to scale. The concurrency problem has been solved nearly optimally and at high resolution in the current Pony-Orca. Thereâs room for improvement, but itâs already in a completely different performance league compared to any big-heap Smalltalk. If Iâm to work hard on an implementation of this design for Smalltalk, I need a much greater speed-up and scaling ability than what these patches give. 2) automatically load-balance Use of mobility with actors would allow for automated rebalancing. Speed hit. Too slow/wasteful. Moving an actor isnât needed if the each has its own heap. 3) support actor-based programs innately With this code, asynchronous computation of "number eventual * 100" occurs in an event loop and resolves the promise [:number | number eventual * 100] value: 0.03 "returning an unresolved promise until the async computation completes and resolves the promise" Promises and notifications are fine. Both happen in Pony-Orca. But the promises donât fix the big performance problems. Am I wrong to state that this model allows innate support to actors? Or were you somehow stating that the VM would need innate support? Why does the VM have to know? Itâs not enough. We still have the big pauses from GCs in a large heap. 4) guarantee no data-races The issue to observe is whether computations are long running and livelock the event loop from handling other activations. This is a shared issue, as Pony/Orca are also susceptible to this. Yes, and a dedicated cycle-detecting actor watches for this in Pony-Orca. E-right's event loops ensure no data races, as long as actor objects are not accessible from more than one event-loop. Speed hit. No blocking and no write barriers exist in Pony-Orca. You canât wait. If you need to âwait,â you set a timer and respond to the event when the timer fires. Imagine a cloud based compute engine, processing Cassandra events that uses inter-machine actors to process the massively parallel Cassandra database. Inter-thread communication is not sufficient as there are hundreds of separate nodes. Yes; I didnât claim otherwise. The networked version is coming. See above. My point is that the âremoteâ characterization is not needed. Itâs not helping us describe and understand. Design wise, it makes much sense to treat inter-thread, inter-process and inter-machine concurrency as the same remote interface. No new design is needed for concurrency and interfacing. There is much to implement, however. The design is already done, modulo the not-yet-present network extension. Interfacing between actors is always by async messaging. Messaging will work as transparently as possible in the networked version across machine nodes. The issue is how most efficiently to use Orca, which happens to be working in Pony. Pony is in production in two internal, speed-demanding, banking apps and in Wallaroo Labsâ high-rate streaming product. Pony is a convenient way to study and use a working implementation of Orca. Ergo, use Pony, even if we only study it as a good example of how to use Orca. Some tweaks (probably a lot of them) could allow use of dynamic types. We could roll our own implementation of Orca for the current Pharo VM, but that seems like more work than tweaking a working Pony compiler and runtime. Iâm not sure about that. You know the VM better than I. (I was beginning my study of the Pharo/OpenSmalltalkVM when I found Pony.) Sounds like you might regret your choice and took the wrong path. I donât see how you form that conclusion. Iâve not chosen yet. You stated you are not thrilled with using Pony. I donât like the Pony language syntax. I donât like anything that looks like Algo-60. Pony is a language, compiler and runtime implementing Orca. The other stuff is good. And Iâve not had much time to use it; I suspect I could like it more. [â¦] If most of what Squeak/Pharo offers is pleasant/productive VM simulation, much work still remains to achieve even a basic actor system and collector, but the writing of VM code in Smalltalk and compiling it to C may be much more productive than writing C++. The C++ for the Pony compiler and runtime, however, already compiles and works well. Thus, starting the work in C++ is somewhat tempting. Can someone explain the limits of how the VM simulator can be used? How much VM core C is not a part of what can be compiled from Smalltalk? Can all VM C code be compiled from Smalltalk? Can someone answer the above question? [1] Cog Blog - http://www.mirandabanda.org/cogblog/ [2] Smalltalk, Tips 'n Tricks - https://clementbera.wordpress.com/ [3] Capability Computation - http://erights.org/elib/capability/index.html [4] Concurrency (Event Loops) - http://erights.org/elib/concurrency/index.html [5] Distributed Programming - http://erights.org/elib/distrib/index.html Shaping

April 21, 2020

Cleanup of some projects on the Pharo contributions Jenkins server

by Gabriel Cotelli

Hi, I need some help to remove the following projects from the Pharo contribution Jenkins Server: - RenoirStRenoirSt-Pharo3 - RenoirSt-SeasideExtensions - RenoirSt-SeasideExtensions-Pharo3 - RenoirSt-UnitsCompatibility Since some years ago we moved the build to use GitHub and Travis CI and disabled the projects but now people are discovering this very old and unmaintained versions using Pharo Launcher so I think it's better to remove them to not generate confusion. I don't have permission to delete the projects even if I created them. So please, anyone with the required permissions can remove them? Or at least rename them to something prefixed with Obsolete- ? Thank you Gabriel

April 20, 2020

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Launcher 2.0 released!

by Christophe Demarey

Hi CÃ©drick, > Le 18 avr. 2020 Ã 10:09, CÃ©drick BÃ©ler <cdrick65(a)gmail.com> a Ã©crit : > > Great Job. It looks indeed nicer. > > I found the docs great too (https://pharo-project.github.io/pharo-launcher <https://pharo-project.github.io/pharo-launcher>). > > One suggestion would be to add the help contextually. I tried to look at. There is description of commands and SpLinkPresenter but donât think itâs easy to add. I wanted to add the ling in the help description. Indeed, good idea. Unfortunately, Spec2 does not yet allow to mix text and links. Probably a new presenter would be needed to support this scenario. Also, it depends on what is doable with the gtk backend. Esteban? Cheers, Christophe > > For instance, the one to create an image for template: > WebBrowser openOn: 'https://pharo-project.github.io/pharo-launcher/create-images/â <https://pharo-project.github.io/pharo-launcher/create-images/%E2%80%98> > > Any way of doing that ? > > <GraphiqueCollÃ©-7.png> > > Cheers, > > CÃ©drick > > > >> Le 17 avr. 2020 Ã 18:08, Christophe Demarey <christophe.demarey(a)inria.fr <mailto:christophe.demarey@inria.fr>> a Ã©crit : >> >> Hi all, >> >> Pharo Launcher 2.0 has just been released! It is available from http://pharo.org/download <http://pharo.org/download>. >> >> This new version introduces major changes: >> The UI has been fully rewritten using the new Spec2 framework <https://github.com/pharo-spec/Spec> and the Commander2 library <https://github.com/pharo-spec/Commander2>. UI has been revamped to increase usability, especially for newcomers. The main window is now composed of a toolbar and the list of images. The template list is now available when clicking on the new image button. >> Documentation web site : All Pharo Launcher features are now explained in the new documentation available at https://pharo-project.github.io/pharo-launcher <https://pharo-project.github.io/pharo-launcher>. You can contribute easily by clicking the *edit on GitHub* button. >> You can now have many launch configurations for an image (VM to use, vm and image arguments). It means you can use headless Pharo VM from Pharo Launcher. >> When creating a new image, you can specify an initialisation script that will be run once at the first image launch. It is useful to load your project code in a stock Pharo image for example. See https://pharo-project.github.io/pharo-launcher/create-images/#image-initial… <https://pharo-project.github.io/pharo-launcher/create-images/#image-initial…> >> You can now define your own template sources in addition to official sources (see https://pharo-project.github.io/pharo-launcher/templates/#create-your-own-l… <https://pharo-project.github.io/pharo-launcher/templates/#create-your-own-l…>), including authenticated sources. >> Improved image metadata. Pharo Launcher now manages all image metadata in a single STON file (including description, Pharo version). >> >> Big thanks to all contributors, including issue reports. It is also the opportunity to thanks Damien Cassou, the original author of Pharo Launcher. >> >> Here is the changelog: >> Pharo Launcher v2.0 <https://github.com/pharo-project/pharo-launcher/releases/tag/2.0> >> >> The list of closed issue is too long (68) to be listed but you can browse it here: https://github.com/pharo-project/pharo-launcher/issues?q=is%3Aissue+is%3Acl… <https://github.com/pharo-project/pharo-launcher/issues?q=is%3Aissue+is%3Acl…> >> Here are some highlights: >> New features: >> >> Documentation web site >> Image initialisation script >> Launch configurations, headless VM support >> User template file in addition to the official template file >> Jenkins server template now support pipeline projects >> Support of private Jenkins server >> Support of authenticated HTTP server >> Improvements: >> >> Monitoring of image launch failures to give back the error message (if any) >> Newly created image is automatically selected in the image list >> Allow to set image description at creation time >> Better error management (you will have the choice to ignore them or debug them) >> Add a poor version column in image list >> Speedup (especially when image repository has a lot of images) >> >> Bug fixes: >> >> Fix use of system unzip on Windows >> >> Regards, >> The Pharo team. >

April 20, 2020

Re: [Pharo-dev] FW: [Pharo-users] Label truncation in new Pharo 8.0 64-bit (stable)

by Sven Van Caekenberghe

Did you try another font ? Another size ? Did you try closing & reopening the Settings Browser ? > On 19 Apr 2020, at 16:31, shaping(a)uurda.org wrote: > > On 2020-04-19 03:01, Sven Van Caekenberghe wrote: > Works for me (this font is larger than the default one). macOS, Pharo > 9 64bit >> The screenshot below is taken on Windows 10 Pro. This time I did not change the theme. I did also update the fonts listed in the dialog before choosing FiraCode. The labels are still truncated. This is another new Pharo 8 64-bit (stable) with only the font changed. Why would formatting not work on Windows 10? >> Can any Windows 10 Pharo users verify this problem? >> <07e94bb1.png> >> >> >>> On 19 Apr 2020, at 06:05, Shaping <shaping(a)uurda.org> wrote: >>> >>> Can anyone suggest a fix for the truncated labels shown below in the >>> screenshot of the Setting window? This happens in the Git client >>> too. >>> >>> Will Pharo tools be rewritten in Spec2, and does its formatting >>> scheme solve generally the text-formatting problem, as font size is >>> changed? >>> >>> Why donât we opt for easy elastic rendering of GUIs by >>> implementing Pharo as a web app? Has this been discussed? Willow >>> is looking very good. >>> >>> Shaping >>> From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On >>> Behalf Of Shaping >>> Sent: Saturday, 18 April, 2020 04:16 >>> To: 'Any question about pharo is welcome' >>> <pharo-users(a)lists.pharo.org>; 'Pharo Development List' >>> <pharo-dev(a)lists.pharo.org> >>> Subject: [Pharo-users] Label truncation in new Pharo 8.0 64-bit >>> (stable) >>> >>> Hi everyone. >>> >>> Below is a Pharo 8.0 64-bit (stable) image (installed with the new >>> Launcher 2.0) with two changes: >>> >>> -Light theme selected. >>> -FiraCode 11 chosen as the default font. >>> >>> <image001.png> >>> >>> Truncated labels in the GUI on font change has always been a >>> problem. Am I the only one experiencing this? >>> >>> Shaping >>> From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On >>> Behalf Of Sanjay Minni >>> Sent: Friday, 17 April, 2020 11:05 >>> To: Any question about pharo is welcome >>> <pharo-users(a)lists.pharo.org> >>> Subject: Re: [Pharo-users] issues in setting up a new project on >>> Github/Iceberg >>> >>> Thanks tried just https and this worked too >>> I suppose once cloned it does not really matter whether it was done >>> using "clone from github.com" or "clone remote repository" >>> --- >>> Sanjay Minni >>> +91-9900-902902 >>> >>> On Fri, 17 Apr 2020 at 19:43, Guillermo Polito >>> <guillermopolito(a)gmail.com> wrote: >>> Hi, >>> >>> El 17 abr 2020, a las 15:30, Sanjay Minni <sm(a)planage.com> >>> escribiÃ³: >>> >>> Hi Guillermo, >>> >>> Ok then given your comments - right now got it working as such >>> - initially created as private in Github >>> - changed to public >>> - created clone from github >>> - (did a repair repository (dont know why) and created "src" and >>> designated it as source folder) >>> - did a commit and push >>> - changed repository back to private >>> seems to work >>> >>> can you give an example on how the "Remote URL" field is to be >>> filled up - note its not taking HTTP as i am not sure about the >>> syntax, assume SSH, Github, MyName, MyRepository, >>> will try that as well >>> >>> you should be able to use the URLs provided by the clone buttons in >>> Github/gitlab/bitbucket, both https and ssh. >>> >>> <image004.png> >>> >>> note Just for Info to other users: >>> I am using Windows 10, it seems to have SSH tools inbuilt and was >>> able to use ssh (with passphrase) without hiccup, by putting my key >>> files path in pharo setting-> ...-> use custom SSH keys, and >>> uploading the .pub key in Github >>> >>> :) >>> >>> Thanks >>> >>> --- >>> Sanjay Minni >>> +91-9900-902902 >>> >>> On Fri, 17 Apr 2020 at 18:36, Guillermo Polito >>> <guillermopolito(a)gmail.com> wrote: >>> Hi Sanjay, >>> >>> This is a bug in Iceberg github integration that tries to access the >>> repository to get metadata using Github REST API using anonymous >>> access apparently. >>> And since your project is private, this fails. >>> >>> As a workaroung, I suggest you to clone using the last option that >>> requires an url only: >>> >>> <PastedGraphic-3.png> >>> >>> This option will not use the Github API, so you should find no >>> problems. >>> >>> Iâm planning doing an Iceberg sprint and fix many issues in the >>> coming weeks. >>> (Also if somebody wants to join, this is an open call :)) >>> Guille >>> >>> El 17 abr 2020, a las 12:49, Sanjay Minni <sm(a)planage.com> >>> escribiÃ³: >>> >>> Using Pharo 8 64 bit on Windows 10 >>> >>> I am trying to setup a new project on Github using Iceberg following >>> the >>> document "Manage your code with Iceberg dtd 25 March 2019) >>> >>> After creating the project in github when I try to clone from Github >>> as (I >>> am at doc item1.4) The system throws an error and the project does >>> not >>> appear in iceberg although the files are downloaded. This happen in >>> both >>> cases whether I use HTTP or SSH >>> >>> What is going wrong ? >>> >>> also: assuming the clone is made, as per next item in doc 1.5, the >>> directory >>> "src" is created after the clone is made. is that correct ?... then >>> how will >>> the packages sources be mapped to ..\src >>> >> <http://forum.world.st/file/t368721/1-GettingFirstEmptyProjFromGit.jpg> >>> >>> >>> <http://forum.world.st/file/t368721/0-GithubProjectCreation.jpg> >>> >>> ----- >>> cheers, >>> Sanjay >>> -- >>> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

April 19, 2020