- Pharo-dev - lists.pharo.org

A weak/leak story

by Guille Polito

Hi list, With Pavel and Christophe we spend some time digging these last weaks chasing the memory leaks we were seeing lately. It is a long story to tell, so this mail is divided in three: 1) A brief intro to weak structures and finalization in Pharo, for those that do not know, 2) A bit of history to explain what happened in pre-spur and post-spur, 3) The actual cause of the memory leak today, 4) How to avoid them in your application, and what are we going to do to prevent this in the future. For those that need/want/prefer just the practical explanation, you can jump over 2) and just read 1) and 3). ======================================================================== 1. A weak explanation ======================================================================== To cleanup objects upon garbage collection, Pharo and Squeak use a finalization mechanism based on a Weak Registry. That is, if you want to execute some cleanup (like closing a file) when an object is about to be collected, you have to put your object inside the weak registry with the corresponding executor/finalizer object. The object you want to 'track' is hold weakly by this weak registry i.e., if the only reference to the object is from the weak registry, it will be chosen for garbage collection. When this object is collected, a special process in the Pharo image will send #finalize to your executor object where you implement your cleanup. To interact with the weak registry, there are two main subscription messages: - #add:executor: Will add an object to the registry with the executor that is send as argument. - #add: Will add an object to the registry, and use as executor a 'shallow copy' of the object. Some conclusions to be made from this: 1) If the executor points strongly to the object that we want to collect, it will never be collected. That is why the #add: message creates a copy of the object. 2) If we do not provide an explicit executor, the registered object should already contain all information required for the finalization (like file handlers or external pointers). If not, the shallow copy will not be able to finalize correctly. Also: - Using weak objects/references do not guarantee that #finalize will be called, you need to put your object inside the registry! - Using weak objects/references do not guarantee that your object will be magically collected. You can still cause memory leaks! ======================================================================== 2. A weak story ======================================================================== Pharo and Squeak use historically the weak registry mentioned above. Because of the limitations that we mentioned, a different kind of weak structure called Ephemerons is required/more useful. To overcome some of these limitations, Igor (Hi Igor! maybe you're reading :)) implemented a couple of years ago a new finalization mechanism that, IIANM, worked as follows: - Some weak objects could have a first instance variable with a special linked list - When the object was about to be collected, instead it was removed from the weak structure and put into its container's linked list - On the image side, a special process iterated all special linked lists and executed #finalize on the weak objects This mechanism was called NewFinalization, in contrast with what was called LegacyFinalization. Of course these names are context dependent, since today's Pharo is back to the so called legacy one ;). NewFinalization was implemented as the default finalization mechanim in Pharo, both in VM and image side. But the VM changes remained in the Pharo branch of development. After some discussions, I remember Igor and Eliot agreed that what they actually needed were Ephemerons, and since Eliot had started working on Spur at that time, he said he would provide Ephemeric classes with the new object format. Basically, for those interested, an ephemeron is an association weak key -> strong value with the special quality that upon garbage collection all references to the weak key that are computed from the strong value (directly or indirectly) are taken as weak. This allows the collection of the weak key even if the strong value points to it, but requires some more machinery in the GC/VM. You can read more in here [1]. Until a couple of months/weeks ago, Pharo was using the NewFinalization mechanism with it's special image and VM support. And Squeak was using the 'Legacy' one. And then Spur arrived. So Spur arrived, and Eliot and Esteban made a lot of effort to simplify the VM's maintenance, and they merged both branches. As a conclusion, Pharo Spur VM did not support any more NewFinalization. This provoked at first some leaks because objects were not being finalized. A couple of weeks ago, we migrated back the image code to use the 'Legacy' mechanism, see issue 17537 [2]. And then finalization was not working either. Nor #finalize was being called on executors, nor objects in the weak registry were collected. As a symptom, opening any tool will cause 30 new everlasting registrations into the weakregistry, and no tools were collected. ======================================================================== 3. The cause ======================================================================== After lots of digging, we finally found what was the particular issue causing objects in the weak registry to not be collected. In some words, it is caused by the normal belief that "weak objects are magical", which caused that weak references and finalizers are really spread over the system with no proper care. And particularly related to the usage of announcements. To explain better, I made some pictures for you :) ***First, imagine you have a morph with its own local announcer. You subscribe to two events, and the graph will look like this. strong-graph - the announcer knows two strong subscriptions - the subscriptions know the announcer to be able to unregister - the subscriptions know the registered object to send the message in case the event happens This forms a closed graph that will be collected. No problem so far. ***Second, let's see what happens if we use weak subsriptions: - the announcer know two weak subscriptions - these weak subscriptions know the announcer strongly to be able to unregister - they also know the subscriber object but weakly - THE difference is made by the weak registry: a global object that manages when and how objects are finalized. In the case of announcers, the weak registry will store weakly the subscriber morph, and strongly the weak announcer subscription. So far so good also: the references to the morph are weak. When the morph is collected, the weak registry will execute finalize on the announcement subscriptions. The subscriptions will unregister from the morph. ***The really problematic case is the third one: mixing weak and strong subscriptions in the same announcer. The object graph is just a mixture of the two other ones. One weak subscription and one strong subscription. BUT: - there is a strong path from a global object (the weak registry) to the subscriber (the morph) - then the morph is never collected - the weak registry never finalizes the weak announcement subscription - the graph remains there forever. And these are the simple cases that show the problem. Imagine that you can have this same configuration but in cycles/chains among different morphs/announcements. Plus this is aggravated by evil globals (e.g., the theme and the HandMorph remembers the last focused morph, the system window class remembers the last top window even if it was closed...). ======================================================================== 4. The solution? ======================================================================== Our solution for the moment is simple. We would like to enforce the following two rules for announcements: - announcers local to a morph should only be used strongly. YES, this may cause small hiccups and leaks, for example if you register a morph A to the announcer to another morph B. But in the long term, these two will form a closed graph and will be collected. - announcers used globally, such as the System announcer, should be used only and uniquely in a weak manner. Like that we ensure that they are loosely coupled for real. So, please, please, do not use weak announcements unless you're really sure of what you're doing. At least, until we have ephemerons and we are sure everything works as expected. Ephemerons would solve this in a more natural way: if we model the weak registry subscription as an ephemeron, any reference to the weak #key that arrives from the #value will be treated as weak also. Other action points we are working on: - fixing tools to follow the rules above - We are also writing tests to check that tools (gt*, Nautilus, Rubric, FT) do not leak. - chasing other small memory leaks created by stepping, focus global variables... ((fogbugz allIssues select: [ :each | each relatedToLeak ]) flatCollect: [ :each | each participants ]) do: #thanks [1] https://en.wikipedia.org/wiki/Ephemeron [2] https://pharo.fogbugz.com/f/cases/17537/SystemAnnouncer-has-far-too-many-su…

April 12, 2016

Re: [Pharo-dev] new pharo cheatsheet

by stepharo

new syntaxsheet

April 12, 2016

Re: [Pharo-dev] new pharo cheatsheet

by J.F. Rick

I agree that enumeration is more useful but it wouldn't be bad to have something in there to indicate the 1-index nature, such as *'abcd' at: 2* will result in $b That's all I was suggesting. Actually, having some cheat sheet on basic enumeration wouldn't be bad as that is a real strength of Smalltalk. Mention and illustrate what do: collect: and select: do. Then mention that there are other ones like count: and sum:. Cheers, Jeff On Sat, Apr 9, 2016 at 4:57 AM Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com> wrote: > Le 08/04/2016 22:08, J.F. Rick a Ã©crit : > > It looks good. I might mention that most data structures are 1 indexed > > as most other languages tend to be 0 indexed and that throws people new > > to the language. > > > > I use Pharo since 1 year now and I had to use indexes only 3-4 times. I > think it is more important to say that Pharo have Collections with an > awesome API. > > -- > Cyril Ferlicot > > http://www.synectique.eu > > 165 Avenue Bretagne > Lille 59000 France > >

April 11, 2016

Phexample based on StateSpecs

by Denis Kudriashov

Hi. I got feedback from Moose guys that Mocketry has problems to be used together with Phexample package. Problem related to own version of "should expressions" in Phexample. So I propose to make Phexample based on StateSpecs. I commit new version 3.0 (with baseline 2.0) with this changes.It allows use Mocketry together with Phexample without problem. Some of Phexample expressions are not exist in StateSpecs. For now they works without changes by own Phexample implementation. One expression was removed because it is not compatible with StateSpecs. It is "should *be* blablabla". Now *#be* expression are based on StateSpecs where you can send any "boolean message" to receiver of should and it will be verified for truth: #(1 2) should be isEmpty 1 should be even. I think it is more simple and practical approach. You can easily browse and debug "predicate" methods of validated objects. And you don't need to extend should expression by "new words" Anyway it is not critical for Moose tests. They are same as before. *There is most noticeable change* in extracting description of some kind of expressions: Set new isEmpty should not beTrue In previous version you will get error "did not expect #isEmpty to be true". There is some magic to implement it. But it is not works generally: Set new isEmpty should not equal: true. It will return "expected true but got false" . So with new version first expression will produce "Got true but it should not be true". To achieve better description StateSpecs provides different approaches without special magic: - boolean expressions (described above) Set new should not *be* isEmpty "fail with: aSet should not be isEmpty" #(1 2) should *be* isEmpty 1 should *be* even. - property expressions by *#where *message (0@0 corner: 2@3) *where* corner x should be: 10 Any messages chain can be sent after #where message to verify deep property. And failure will looked like: "Got 2 from "(0@0 corner: 2@3) corner x" but it should be 10. I hope we can agree on using StateSpecs. It provides more simple and reusable solution. Reusability of StateSpecs achieved by first class specifications of object state and first class validation failures. They are reused to describe message stub arguments like mock someMessageWith: (Kind of: Number) Also they allows to implement different kind of presentation for specific specs and failures to improve debugger. At the end I want suggest to deprecate all kind of beTrue/beFalse/true/false/nil. They provides nothing more than bool should be: true bool should be: false object should be: nil And one expression I completely not understand: 600.123456789 should beCloseTo: 600.1. Why it is not implemented as "(a - b) abs < precision"? (precision = 0.0001 in Phexample) It is implemented as "(a - b) abs / (a abs max: b abs) < precision" In StateSpecs it will looks like 600.123456789 should equal: 600.1 within: 0.0001. Which of course will fail by my definition Best regards Denis

April 11, 2016

Call for Papers: 11th ICOOOLPS Workshop at ECOOP. New Deadline April 22nd, 2016

by Stefan Marr

Please note that the submission deadline is on the 22nd of April, 2016. Call for Papers: ICOOOLPSâ16 ============================ 11th Workshop on Implementation, Compilation, Optimization of OO Languages, Programs and Systems Co-located with ECOOP July 18, 2016, Rome, Italy URL: http://2016.ecoop.org/track/ICOOOLPS-2016 Twitter: @ICOOOLPS The ICOOOLPS workshop series brings together researchers and practitioners working in the field of language implementation and optimization. The goal of the workshop is to discuss emerging problems and research directions as well as new solutions to classic performance challenges. The topics of interest for the workshop include techniques for the implementation and optimization of a wide range of languages including but not limited to object-oriented ones. Furthermore, meta-compilation techniques or language-agnostic approaches are welcome, too. A non-exclusive list of topics follows: - implementation and optimization of fundamental languages features (from automatic memory management to zero-overhead metaprogramming) - runtime systems technology (libraries, virtual machines) - static, adaptive, and speculative optimizations and compiler techniques - meta-compilation techniques and language-agnostic approaches for the efficient implementation of languages - compilers (intermediate representations, offline and online optimizations,...) - empirical studies on language usage, benchmark design, and benchmarking methodology - resource-sensitive systems (real-time, low power, mobile, cloud) - studies on design choices and tradeoffs (dynamic vs. static compilation, heuristics vs. programmer input,...) - tooling support, debuggability and observability of languages as well as their implementations ### Workshop Format and Submissions This workshop welcomes the presentation and discussion of new ideas and emerging problems that give a chance for interaction and exchange. More mature work is welcome as part of a mini-conference format, too. We aim to interleave interactive brainstorming and demonstration sessions between the formal presentations to foster an active exchange of ideas. The workshop papers will be published either in the ACM DL or in the Dagstuhl LIPIcs ECOOP Workshop proceedings. Until further notice, please use the ACM SIGPLAN template with a 10pt font size: http://www.sigplan.org/Resources/Author/. - position and work-in-progress paper: 1-4 pages - technical paper: max. 10 pages - demos and posters: 1-page abstract The page limits include references and appendixes. Note further that the upper page limit is a maximum and not a requirement. For the submission, please use the HotCRP system: http://ssw.jku.at/icooolps/ ### Important Dates - abstract submission: April 18, 2016 - paper submission: April 22, 2016 - notification: May 13, 2016 - all deadlines: Anywhere on Earth (AoE), i.e., GMT/UTCâ12:00 hour - workshop: July 18th, 2016 ### Program Committee Edd Barrett, Kingâs College London, UK Clement Bera, Inria Lille, France Maxime Chevalier-Boisvert, UniversitÃ© de MontrÃ©al, Canada Tim Felgentreff, Hasso Plattner Institute, Germany Roland Ducournau, LIRMM, UniversitÃ© de Montpellier, France Elisa Gonzalez Boix, Vrije Universiteit Brussel, Belgium David Gregg, Trinity College Dublin, Ireland Matthias Grimmer, Johannes Kepler University Linz, Austria Michael Haupt, Oracle, Germany Richard Jones, University of Kent, UK Tomas Kalibera, Northeastern University, USA Hidehiko Masuhara, Tokyo Institute of Technology, Japan Tiark Rompf, Purdue University, USA Jennifer B. Sartor, Ghent University, Belgium Sam Tobin-Hochstadt, Indiana University, USA ### Workshop Organizers Stefan Marr, Johannes Kepler University Linz, Austria Eric Jul, University of Oslo, Norway

April 11, 2016

[pharo-project/pharo-core] dfabd9: 50681

by GitHub

Branch: refs/heads/5.0 Home: https://github.com/pharo-project/pharo-core Commit: dfabd9545ac28cf2636de8bf1ecac3b0a6b62803 https://github.com/pharo-project/pharo-core/commit/dfabd9545ac28cf2636de8bf… Author: Jenkins Build Server <board(a)pharo-project.org> Date: 2016-04-11 (Mon, 11 Apr 2016) Changed paths: M ConfigurationOfRubric.package/ConfigurationOfRubric.class/instance/symbolic versions/stable_.st A ConfigurationOfRubric.package/ConfigurationOfRubric.class/instance/versions/version217_.st A ConfigurationOfRubric.package/ConfigurationOfRubric.class/instance/versions/version218_.st A Morphic-Widgets-Tree.package/MorphTreeNodeMorph.class/instance/announcements/announceDeleted.st M Morphic-Widgets-Tree.package/MorphTreeNodeMorph.class/instance/initialization/initWithContents_prior_forList_indentLevel_.st M Morphic-Widgets-Tree.package/MorphTreeNodeMorph.class/instance/updating/delete.st M Morphic-Widgets-Windows.package/SystemWindow.class/class/top window/noteTopWindowIn_.st M Rubric.package/RubAbstractTextArea.class/instance/initialize/plugFindReplace.st M Rubric.package/RubFindReplaceService.class/instance/accessing/textArea_.st M Rubric.package/RubGhostTextDisplayer.class/instance/submorphs-accessing/noteNewOwner_.st M Rubric.package/RubHighlightSegmentMorph.class/instance/event handling/registerTextArea.st M Rubric.package/RubPluggableTextMorph.class/instance/accessing scrollbars/registerScrollChanges_.st M Rubric.package/RubPrimarySelectionMorph.class/instance/event handling/registerTextArea.st M Rubric.package/RubShoutStylerDecorator.class/instance/editing/style_.st M Rubric.package/RubTextSegmentMorph.class/instance/event handling/registerTextArea.st R ScriptLoader50.package/ScriptLoader.class/instance/pharo - scripts/script50680.st A ScriptLoader50.package/ScriptLoader.class/instance/pharo - scripts/script50681.st R ScriptLoader50.package/ScriptLoader.class/instance/pharo - updates/update50680.st A ScriptLoader50.package/ScriptLoader.class/instance/pharo - updates/update50681.st M ScriptLoader50.package/ScriptLoader.class/instance/public/commentForCurrentUpdate.st Log Message: ----------- 50681 17284 Rubric SubscriptOutOfBounds error https://pharo.fogbugz.com/f/cases/17284 17972 Remove depended in MorphTreeNodeMorph on deletion https://pharo.fogbugz.com/f/cases/17972 17968 a window should not be the top window after its deletion https://pharo.fogbugz.com/f/cases/17968 http://files.pharo.org/image/50/50681.zip

April 11, 2016

[pharo-project/pharo-core]

by GitHub

Branch: refs/tags/50681 Home: https://github.com/pharo-project/pharo-core

April 11, 2016

Re: [Pharo-dev] Missing #removeDuplicates

by Nicolai Hess

2016-04-10 8:55 GMT+02:00 Volkert <volkert(a)komponentenwerkstatt.de>: > Sorry. i meant AthensSkatch ;-) It is a nice demo about using athens. > > http://smalltalkhub.com/#!/~NicolaiHess/AthensSketch Hi Volkert, I just replaced #removeDuplicates with #asSet (this should work now). (and replaced some deprecated on:send:to: calls (for Announcements) with when:send:to: in the latest version. Nicolai > > > > On 10.04.2016 07:42, stepharo wrote: > >> >> >> Le 9/4/16 22:36, Volkert a Ã©crit : >> >>> What happens to OrderedCollection>>removeDuplicates in Pharo 5.0? In >>> Pharo 4.0 it was part of a Package "GroupManager"? >>> >> >> It was probably badly implemented or only used in group manager (whose >> code was terrible) and we cleaned it. >> Now we could reintroduce it but in the right place and with tests. >> >>> It breaks the nice Package "AthensScratch" in Pharo 5.0 >>> >> >> what is athenscracth? >> >>> >>> BW. >>> Volkert >>> >>> >>> >> >> > >

April 11, 2016

MongoCursor>>#execute and MongoTalk changes

by Holger Freyther

Hi guys, I am tracking down a Voyage issue where query/reply is going out of sync when used concurrently. The Mongo wire protocol allows to specify a unique requestId and the response includes it as "responseTo" identifier. Mongo>>#nextRequestID helps in getting a new/unique id and it is already used but the responseTo field is never used. I have modified Mongo>>#cursorQuery: to return the 'id' instead of 'self' and have modified Mongo>>#getMore:collection:chunkSize: to request a new requestId instead of truncating the 64bit cursorId to 32bit and then return the 'id'. Next I modified MongoCursor>>#execute to remember the requestId and then compare it to the response id. This way the code will not treat a response for something else as legitimate response. It is using self error to signal the failure. For the comparison to work in a reliable way even after many requests we need to make sure that Mongo requestID stays within 32bit. Otherwise our internal number will be a LargePositiveInteger using more than four bytes and when we get a reply it will be truncated to a number with four bytes only and the comparison will fail. For my use the failure would occur with < 125 days of operation. I have added a bitAnd: 16r3F...F (which is SmallInteger maxVal on 32bit images) to truncate it to 30 bit. I have put that into the MongoTalk repository. If anything is wrong with the code (unit tests pass) please ping me and I will try my best to fix it. kind regards holger

April 10, 2016

Re: [Pharo-dev] Missing #removeDuplicates

by Volkert

Sorry. i meant AthensSkatch ;-) It is a nice demo about using athens. http://smalltalkhub.com/#!/~NicolaiHess/AthensSketch On 10.04.2016 07:42, stepharo wrote: > > > Le 9/4/16 22:36, Volkert a Ã©crit : >> What happens to OrderedCollection>>removeDuplicates in Pharo 5.0? In >> Pharo 4.0 it was part of a Package "GroupManager"? > > It was probably badly implemented or only used in group manager (whose > code was terrible) and we cleaned it. > Now we could reintroduce it but in the right place and with tests. >> It breaks the nice Package "AthensScratch" in Pharo 5.0 > > what is athenscracth? >> >> BW. >> Volkert >> >> > >

April 10, 2016