April 2016 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] TxText model

by Igor Stasenko

On 6 April 2016 at 17:07, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > 2016-04-06 10:56 GMT+02:00 Igor Stasenko <siguctua(a)gmail.com>: > >> >> >> On 6 April 2016 at 11:36, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >> >>> >>> >>> Thanks Igor. >>> Maybe some more Info? (I don't have much experience with this and don't >>> fully understand what is missing). >>> >>> >>>> 1. No support for vector graphics >>>> >>> >>> I know some parts of Athens (with Cairo font renderer and how it is used >>> in TxText) but at the moment >>> rendering a TextMorph in Athens works: >>> Morph comment asText asMorph openInSceneView >>> >> >> Sorry, missed to reply to that important point. >> It works because one good soul spent effort and helped me to finish >> implementation of a wrapper for Morph, to be able to render any morph into >> Form, and then render Form >> on Athens surface. >> > > Nah, actually it works a bit different. Many (Form)Canvas methods are > reimplemented on Athens API. But yes, I used some hacks as well. > > Didn't knew. Aha.. i see AthensCanvasWrapper. Ohh.. that's so much pain :) > So, that is why it 'works' :) >> >> >> -- >> Best regards, >> Igor Stasenko. >> > > -- Best regards, Igor Stasenko.

April 6, 2016

Re: [Pharo-dev] Two possible Morphic-related issues

by Stephan Eggermont

On 06-04-16 15:27, phil(a)highoctane.be wrote: > Is there anything else than a SmalltalkEditor there? Now there is Rubric > too, but... The superclass of SmalltalkEditor is TextEditor. That is the one I'd expect. Stephan

April 6, 2016

Re: [Pharo-dev] [Pharo-users] STON materialization corrupts a dictionary if the keys are references

by Henrik Johansen

The specified operation of fixing a badly hashed set will, if you go in reverse ;) Removing (or, nilling, then checking for subsequent hits) only ever moves objects at indexes following the one you removed at (and have already verified to be correct) Adding might result in the removed object being put at a lower index, which means you may process it twice, but that's not the end of the world (since the second time, index will be valid, and only one slot is affected). Goes without saying such an operation does some arguably nasty stuff using HashedCollection internals, so would need to be implemented there... Might've been a better compromise to write an optimized removeAllSuchThat: (followed by adding the same over again) fixing slots as it goes, ala the implementation in OrderedCollection. But no, keeping it consistent, it would have to be the values passed as block parameters :( Cheers, Henry > On 06 Apr 2016, at 4:07 , Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Yeah, I just realised that iterating and changing a dictionary or set at the same time won't work ;-) > >> On 06 Apr 2016, at 16:01, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: >> >> If you are iterating over a Set with incorrectly placed objects, remove: calls aren't going to do you much good ;) >> Not to mention, even nilling the slot directly, then add:'ing, still means you have to scan subsequent entries up to the next empty slot for potentially better placement, if the object ended up being added elsewhere. >> (IOW, if you're gonna do it, better iterate in reverse) >> >> Cheers, >> Henry >> >>> On 06 Apr 2016, at 3:46 , Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> >>> >>>> On 06 Apr 2016, at 15:34, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>>> >>>> >>>> >>>> 2016-04-06 15:25 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >>>> Hi Nicolai, >>>> >>>>> On 06 Apr 2016, at 14:56, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> 2016-04-06 14:27 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >>>>> Fix for review: >>>>> >>>>> === >>>>> Name: STON-Core-SvenVanCaekenberghe.71 >>>>> Author: SvenVanCaekenberghe >>>>> Time: 6 April 2016, 2:22:24.782251 pm >>>>> UUID: 64b8b741-365e-41fe-aa98-565e33ca5d24 >>>>> Ancestors: STON-Core-SvenVanCaekenberghe.70 >>>>> >>>>> Fix a bug where STONReferences occurring as keys in Dictionaries or elements in Sets caused those to be unhealthy after materialization. Thx to Peter UhnÃ¡k for reporting this issue. >>>>> >>>>> Add 3 new unit tests to STONReaderTests >>>>> >>>>> #testDictionaryWithReferenceKeys >>>>> #testSetWithReferenceElements >>>>> #testDeepStructure >>>>> >>>>> Fix Details >>>>> >>>>> change the implementation of STONReader>>#processSubObjectsOf: from iterative to recursive (see version 39 of 29 November 2012, this might be a functional regression, see #testDeepStructure; cleanup of stack instance variable for later) so that #stonProcessSubObjects: can be overwritten with code being executed before or after full reference resolution >>>>> >>>>> imho, recursion stack depth will be equal during both writing and reading, and should be acceptable. >>>>> >>>>> overwrite #stonProcessSubObjects: in Dictionary and Set to #rehash at the end, but only when needed (minimal optimalization, see Dictionary>>#containsStonReferenceAsKey and Set>>#containsStonReference) >>>>> === >>>>> Name: STON-Tests-SvenVanCaekenberghe.63 >>>>> Author: SvenVanCaekenberghe >>>>> Time: 6 April 2016, 2:22:45.01986 pm >>>>> UUID: 0beb2322-b81a-46ee-a0e2-6648a808774a >>>>> Ancestors: STON-Tests-SvenVanCaekenberghe.62 >>>>> >>>>> (idem) >>>>> === >>>> >>>> Thanks for looking at the code. >>>> >>>>> Hi Sven, >>>>> instead of rehashing the dictionary for every ston reference, >>>> >>>> (It rehashes only once after resolving all references) >>>> >>>> Ah, of course. I thought this would be called for every ston reference used as key. >>>> >>>> >>>> >>>>> wouldn't it work to remove and readd the value after processing the subobject: >>>>> >>>>> Dictionary>>#stonProcessSubObjects: block >>>>> self keys do:[:key | >>>>> |value| >>>>> value := block value:(self removeKey: key ifAbsent:[ nil]). >>>>> self at: (block value: key) put: value]. >>>> >>>> Interesting idea. I have to think about that approach. >>>> >>>> Now, Object>>#stonProcessSubObjects: is very general and looks at named and indexed instance variables. But this probably could be replaced by something more high level and specific I guess. >>>> >>>> Adding and removing each key/value has a cost too. I try to make the simplest case very efficient and only pay a price when really needed. Anyway, time for some calculations. >>>> >>>> ok, yes running over all keys isn't better than rehashing :-) >>> >>> Still, your idea might be better than you would expect: >>> >>> Now, there is 1 (partial) iteration to do the check before, 2 iterations over the keys and values arrays, then an optional full rehash (which also reallocates and thus generates garbage). >>> >>> Your idea does only 1 iteration over keys, with remove and add (which also happens more efficiently on the arrays above), no check before, no rehash, probably no garbage generation at all in any case. >>> >>> Like I said, I have to study it (especially the cost of remove/add, maybe that can be optimised a bit as well). >>> >>>> Thanks again for the suggestion ! >>>> >>>> Sven >>>> >>>>>> On 06 Apr 2016, at 14:04, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>>> >>>>>> https://pharo.fogbugz.com/f/cases/17946/STON-materializes-unhealthy-Diction… >>>>>> >>>>>> fix coming >>>>>> >>>>>>> On 05 Apr 2016, at 13:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>>>> >>>>>>>> >>>>>>>> On 05 Apr 2016, at 13:02, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2016-04-05 12:32 GMT+02:00 Cyril Ferlicot Delbecque <cyril.ferlicot(a)gmail.com>: >>>>>>>> >>>>>>>> >>>>>>>> On 05/04/2016 12:09, Sven Van Caekenberghe wrote: >>>>>>>> >>>>>>>>> Like I said, it is a hashing issue, sometimes it will be correct by accident. >>>>>>>>> >>>>>>>>> I hope you did not have to much trouble with this bug, I guess it must have been hard to chase. >>>>>>>>> >>>>>>>>> Is it urgent ? >>>>>>>>> >>>>>>>>> I probably can give you a quick fix, but I would like to think a bit more about this, since rehashing each materialised dictionary seems expensive. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Hi Sven, >>>>>>>> >>>>>>>> I got the same kind of problem in a personal application. >>>>>>>> >>>>>>>> I use Sets that I serialize and I had a lot of trouble because sometimes >>>>>>>> some action had strange behaviours. >>>>>>>> >>>>>>>> For example in a set with element `aSet remove: aSet anyOne` raised 'XXX >>>>>>>> not found in aSet'. >>>>>>>> >>>>>>>> I am glad to hear that it is a Ston issue and not me that used sets in a >>>>>>>> bad way :) >>>>>>>> >>>>>>>> For me too it is not urgent since I have a not of university work for >>>>>>>> the moment. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> How are hashed collections created/filled during ston-parsing ? >>>>>>>> If the position in a hashed collection is created by a ston-reference, that is later replaced by the "real" object, >>>>>>>> the index in the dictionary (or other hashed collections) may be wrong. >>>>>>> >>>>>>> Yes, that is indeed it, Nicolai. >>>>>>> >>>>>>> But I would like to try to minimise the rehashing as it seems expensive. But first I need a more reliable failure. >>>>>>> >>>>>>>> -- >>>>>>>> Cyril Ferlicot >>>>>>>> >>>>>>>> http://www.synectique.eu >>>>>>>> >>>>>>>> 165 Avenue Bretagne >>>>>>>> Lille 59000 France >>>>>> >>> >>> >> > >

April 6, 2016

[pharo-project/pharo-core]

by GitHub

Branch: refs/tags/50677 Home: https://github.com/pharo-project/pharo-core

April 6, 2016

[pharo-project/pharo-core] a7a26c: 50677

by GitHub

Branch: refs/heads/5.0 Home: https://github.com/pharo-project/pharo-core Commit: a7a26c014b819d5962a0c92141000a270ea0ca19 https://github.com/pharo-project/pharo-core/commit/a7a26c014b819d5962a0c921… Author: Jenkins Build Server <board(a)pharo-project.org> Date: 2016-04-06 (Wed, 06 Apr 2016) Changed paths: M Kernel-Tests.package/SizeInMemoryTest.class/instance/tests/testSizeInMemoryOfByteObjects.st M Keymapping-KeyCombinations.package/KMAltModifier.class/instance/accessing/eventCode.st M Monticello.package/MCClassDefinition.class/instance/comparing/requirements.st R ScriptLoader50.package/ScriptLoader.class/instance/pharo - scripts/script50676.st A ScriptLoader50.package/ScriptLoader.class/instance/pharo - scripts/script50677.st R ScriptLoader50.package/ScriptLoader.class/instance/pharo - updates/update50676.st A ScriptLoader50.package/ScriptLoader.class/instance/pharo - updates/update50677.st M ScriptLoader50.package/ScriptLoader.class/instance/public/commentForCurrentUpdate.st M Text-Edition.package/TextEditor.class/instance/typing support/setEmphasisHereFromTextForward_.st Log Message: ----------- 50677 17945 TextEditor setEmphasisHereFromTextForward: broken for empty text https://pharo.fogbugz.com/f/cases/17945 17943 It is not possible to write [ in Nautilus on the french keyboard layout https://pharo.fogbugz.com/f/cases/17943 17937 Monticello does not fully unload packages https://pharo.fogbugz.com/f/cases/17937 17621 Failing test SizeInMemoryTest>>#testSizeInMemoryOfByteObjects https://pharo.fogbugz.com/f/cases/17621 http://files.pharo.org/image/50/50677.zip

April 6, 2016

Re: [Pharo-dev] [Pharo-users] STON materialization corrupts a dictionary if the keys are references

by Sven Van Caekenberghe

Yeah, I just realised that iterating and changing a dictionary or set at the same time won't work ;-) > On 06 Apr 2016, at 16:01, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: > > If you are iterating over a Set with incorrectly placed objects, remove: calls aren't going to do you much good ;) > Not to mention, even nilling the slot directly, then add:'ing, still means you have to scan subsequent entries up to the next empty slot for potentially better placement, if the object ended up being added elsewhere. > (IOW, if you're gonna do it, better iterate in reverse) > > Cheers, > Henry > >> On 06 Apr 2016, at 3:46 , Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> >>> On 06 Apr 2016, at 15:34, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>> >>> >>> >>> 2016-04-06 15:25 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >>> Hi Nicolai, >>> >>>> On 06 Apr 2016, at 14:56, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>>> >>>> >>>> >>>> 2016-04-06 14:27 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >>>> Fix for review: >>>> >>>> === >>>> Name: STON-Core-SvenVanCaekenberghe.71 >>>> Author: SvenVanCaekenberghe >>>> Time: 6 April 2016, 2:22:24.782251 pm >>>> UUID: 64b8b741-365e-41fe-aa98-565e33ca5d24 >>>> Ancestors: STON-Core-SvenVanCaekenberghe.70 >>>> >>>> Fix a bug where STONReferences occurring as keys in Dictionaries or elements in Sets caused those to be unhealthy after materialization. Thx to Peter UhnÃ¡k for reporting this issue. >>>> >>>> Add 3 new unit tests to STONReaderTests >>>> >>>> #testDictionaryWithReferenceKeys >>>> #testSetWithReferenceElements >>>> #testDeepStructure >>>> >>>> Fix Details >>>> >>>> change the implementation of STONReader>>#processSubObjectsOf: from iterative to recursive (see version 39 of 29 November 2012, this might be a functional regression, see #testDeepStructure; cleanup of stack instance variable for later) so that #stonProcessSubObjects: can be overwritten with code being executed before or after full reference resolution >>>> >>>> imho, recursion stack depth will be equal during both writing and reading, and should be acceptable. >>>> >>>> overwrite #stonProcessSubObjects: in Dictionary and Set to #rehash at the end, but only when needed (minimal optimalization, see Dictionary>>#containsStonReferenceAsKey and Set>>#containsStonReference) >>>> === >>>> Name: STON-Tests-SvenVanCaekenberghe.63 >>>> Author: SvenVanCaekenberghe >>>> Time: 6 April 2016, 2:22:45.01986 pm >>>> UUID: 0beb2322-b81a-46ee-a0e2-6648a808774a >>>> Ancestors: STON-Tests-SvenVanCaekenberghe.62 >>>> >>>> (idem) >>>> === >>> >>> Thanks for looking at the code. >>> >>>> Hi Sven, >>>> instead of rehashing the dictionary for every ston reference, >>> >>> (It rehashes only once after resolving all references) >>> >>> Ah, of course. I thought this would be called for every ston reference used as key. >>> >>> >>> >>>> wouldn't it work to remove and readd the value after processing the subobject: >>>> >>>> Dictionary>>#stonProcessSubObjects: block >>>> self keys do:[:key | >>>> |value| >>>> value := block value:(self removeKey: key ifAbsent:[ nil]). >>>> self at: (block value: key) put: value]. >>> >>> Interesting idea. I have to think about that approach. >>> >>> Now, Object>>#stonProcessSubObjects: is very general and looks at named and indexed instance variables. But this probably could be replaced by something more high level and specific I guess. >>> >>> Adding and removing each key/value has a cost too. I try to make the simplest case very efficient and only pay a price when really needed. Anyway, time for some calculations. >>> >>> ok, yes running over all keys isn't better than rehashing :-) >> >> Still, your idea might be better than you would expect: >> >> Now, there is 1 (partial) iteration to do the check before, 2 iterations over the keys and values arrays, then an optional full rehash (which also reallocates and thus generates garbage). >> >> Your idea does only 1 iteration over keys, with remove and add (which also happens more efficiently on the arrays above), no check before, no rehash, probably no garbage generation at all in any case. >> >> Like I said, I have to study it (especially the cost of remove/add, maybe that can be optimised a bit as well). >> >>> Thanks again for the suggestion ! >>> >>> Sven >>> >>>>> On 06 Apr 2016, at 14:04, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>> >>>>> https://pharo.fogbugz.com/f/cases/17946/STON-materializes-unhealthy-Diction… >>>>> >>>>> fix coming >>>>> >>>>>> On 05 Apr 2016, at 13:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>>> >>>>>>> >>>>>>> On 05 Apr 2016, at 13:02, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2016-04-05 12:32 GMT+02:00 Cyril Ferlicot Delbecque <cyril.ferlicot(a)gmail.com>: >>>>>>> >>>>>>> >>>>>>> On 05/04/2016 12:09, Sven Van Caekenberghe wrote: >>>>>>> >>>>>>>> Like I said, it is a hashing issue, sometimes it will be correct by accident. >>>>>>>> >>>>>>>> I hope you did not have to much trouble with this bug, I guess it must have been hard to chase. >>>>>>>> >>>>>>>> Is it urgent ? >>>>>>>> >>>>>>>> I probably can give you a quick fix, but I would like to think a bit more about this, since rehashing each materialised dictionary seems expensive. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> Hi Sven, >>>>>>> >>>>>>> I got the same kind of problem in a personal application. >>>>>>> >>>>>>> I use Sets that I serialize and I had a lot of trouble because sometimes >>>>>>> some action had strange behaviours. >>>>>>> >>>>>>> For example in a set with element `aSet remove: aSet anyOne` raised 'XXX >>>>>>> not found in aSet'. >>>>>>> >>>>>>> I am glad to hear that it is a Ston issue and not me that used sets in a >>>>>>> bad way :) >>>>>>> >>>>>>> For me too it is not urgent since I have a not of university work for >>>>>>> the moment. >>>>>>> >>>>>>> >>>>>>> >>>>>>> How are hashed collections created/filled during ston-parsing ? >>>>>>> If the position in a hashed collection is created by a ston-reference, that is later replaced by the "real" object, >>>>>>> the index in the dictionary (or other hashed collections) may be wrong. >>>>>> >>>>>> Yes, that is indeed it, Nicolai. >>>>>> >>>>>> But I would like to try to minimise the rehashing as it seems expensive. But first I need a more reliable failure. >>>>>> >>>>>>> -- >>>>>>> Cyril Ferlicot >>>>>>> >>>>>>> http://www.synectique.eu >>>>>>> >>>>>>> 165 Avenue Bretagne >>>>>>> Lille 59000 France >>>>> >> >> >

April 6, 2016

Re: [Pharo-dev] TxText model

by Nicolai Hess

2016-04-06 10:56 GMT+02:00 Igor Stasenko <siguctua(a)gmail.com>: > > > On 6 April 2016 at 11:36, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > >> >> >> Thanks Igor. >> Maybe some more Info? (I don't have much experience with this and don't >> fully understand what is missing). >> >> >>> 1. No support for vector graphics >>> >> >> I know some parts of Athens (with Cairo font renderer and how it is used >> in TxText) but at the moment >> rendering a TextMorph in Athens works: >> Morph comment asText asMorph openInSceneView >> > > Sorry, missed to reply to that important point. > It works because one good soul spent effort and helped me to finish > implementation of a wrapper for Morph, to be able to render any morph into > Form, and then render Form > on Athens surface. > Nah, actually it works a bit different. Many (Form)Canvas methods are reimplemented on Athens API. But yes, I used some hacks as well. > So, that is why it 'works' :) > > > -- > Best regards, > Igor Stasenko. >

April 6, 2016

Re: [Pharo-dev] [Pharo-users] STON materialization corrupts a dictionary if the keys are references

by Henrik Johansen

If you are iterating over a Set with incorrectly placed objects, remove: calls aren't going to do you much good ;) Not to mention, even nilling the slot directly, then add:'ing, still means you have to scan subsequent entries up to the next empty slot for potentially better placement, if the object ended up being added elsewhere. (IOW, if you're gonna do it, better iterate in reverse) Cheers, Henry > On 06 Apr 2016, at 3:46 , Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> On 06 Apr 2016, at 15:34, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >> >> >> >> 2016-04-06 15:25 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >> Hi Nicolai, >> >>> On 06 Apr 2016, at 14:56, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>> >>> >>> >>> 2016-04-06 14:27 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: >>> Fix for review: >>> >>> === >>> Name: STON-Core-SvenVanCaekenberghe.71 >>> Author: SvenVanCaekenberghe >>> Time: 6 April 2016, 2:22:24.782251 pm >>> UUID: 64b8b741-365e-41fe-aa98-565e33ca5d24 >>> Ancestors: STON-Core-SvenVanCaekenberghe.70 >>> >>> Fix a bug where STONReferences occurring as keys in Dictionaries or elements in Sets caused those to be unhealthy after materialization. Thx to Peter UhnÃ¡k for reporting this issue. >>> >>> Add 3 new unit tests to STONReaderTests >>> >>> #testDictionaryWithReferenceKeys >>> #testSetWithReferenceElements >>> #testDeepStructure >>> >>> Fix Details >>> >>> change the implementation of STONReader>>#processSubObjectsOf: from iterative to recursive (see version 39 of 29 November 2012, this might be a functional regression, see #testDeepStructure; cleanup of stack instance variable for later) so that #stonProcessSubObjects: can be overwritten with code being executed before or after full reference resolution >>> >>> imho, recursion stack depth will be equal during both writing and reading, and should be acceptable. >>> >>> overwrite #stonProcessSubObjects: in Dictionary and Set to #rehash at the end, but only when needed (minimal optimalization, see Dictionary>>#containsStonReferenceAsKey and Set>>#containsStonReference) >>> === >>> Name: STON-Tests-SvenVanCaekenberghe.63 >>> Author: SvenVanCaekenberghe >>> Time: 6 April 2016, 2:22:45.01986 pm >>> UUID: 0beb2322-b81a-46ee-a0e2-6648a808774a >>> Ancestors: STON-Tests-SvenVanCaekenberghe.62 >>> >>> (idem) >>> === >> >> Thanks for looking at the code. >> >>> Hi Sven, >>> instead of rehashing the dictionary for every ston reference, >> >> (It rehashes only once after resolving all references) >> >> Ah, of course. I thought this would be called for every ston reference used as key. >> >> >> >>> wouldn't it work to remove and readd the value after processing the subobject: >>> >>> Dictionary>>#stonProcessSubObjects: block >>> self keys do:[:key | >>> |value| >>> value := block value:(self removeKey: key ifAbsent:[ nil]). >>> self at: (block value: key) put: value]. >> >> Interesting idea. I have to think about that approach. >> >> Now, Object>>#stonProcessSubObjects: is very general and looks at named and indexed instance variables. But this probably could be replaced by something more high level and specific I guess. >> >> Adding and removing each key/value has a cost too. I try to make the simplest case very efficient and only pay a price when really needed. Anyway, time for some calculations. >> >> ok, yes running over all keys isn't better than rehashing :-) > > Still, your idea might be better than you would expect: > > Now, there is 1 (partial) iteration to do the check before, 2 iterations over the keys and values arrays, then an optional full rehash (which also reallocates and thus generates garbage). > > Your idea does only 1 iteration over keys, with remove and add (which also happens more efficiently on the arrays above), no check before, no rehash, probably no garbage generation at all in any case. > > Like I said, I have to study it (especially the cost of remove/add, maybe that can be optimised a bit as well). > >> Thanks again for the suggestion ! >> >> Sven >> >>>> On 06 Apr 2016, at 14:04, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>> >>>> https://pharo.fogbugz.com/f/cases/17946/STON-materializes-unhealthy-Diction… >>>> >>>> fix coming >>>> >>>>> On 05 Apr 2016, at 13:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>>>> >>>>>> >>>>>> On 05 Apr 2016, at 13:02, Nicolai Hess <nicolaihess(a)gmail.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> 2016-04-05 12:32 GMT+02:00 Cyril Ferlicot Delbecque <cyril.ferlicot(a)gmail.com>: >>>>>> >>>>>> >>>>>> On 05/04/2016 12:09, Sven Van Caekenberghe wrote: >>>>>> >>>>>>> Like I said, it is a hashing issue, sometimes it will be correct by accident. >>>>>>> >>>>>>> I hope you did not have to much trouble with this bug, I guess it must have been hard to chase. >>>>>>> >>>>>>> Is it urgent ? >>>>>>> >>>>>>> I probably can give you a quick fix, but I would like to think a bit more about this, since rehashing each materialised dictionary seems expensive. >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> Hi Sven, >>>>>> >>>>>> I got the same kind of problem in a personal application. >>>>>> >>>>>> I use Sets that I serialize and I had a lot of trouble because sometimes >>>>>> some action had strange behaviours. >>>>>> >>>>>> For example in a set with element `aSet remove: aSet anyOne` raised 'XXX >>>>>> not found in aSet'. >>>>>> >>>>>> I am glad to hear that it is a Ston issue and not me that used sets in a >>>>>> bad way :) >>>>>> >>>>>> For me too it is not urgent since I have a not of university work for >>>>>> the moment. >>>>>> >>>>>> >>>>>> >>>>>> How are hashed collections created/filled during ston-parsing ? >>>>>> If the position in a hashed collection is created by a ston-reference, that is later replaced by the "real" object, >>>>>> the index in the dictionary (or other hashed collections) may be wrong. >>>>> >>>>> Yes, that is indeed it, Nicolai. >>>>> >>>>> But I would like to try to minimise the rehashing as it seems expensive. But first I need a more reliable failure. >>>>> >>>>>> -- >>>>>> Cyril Ferlicot >>>>>> >>>>>> http://www.synectique.eu >>>>>> >>>>>> 165 Avenue Bretagne >>>>>> Lille 59000 France >>>> > >

April 6, 2016

Re: [Pharo-dev] [Pharo-users] STON materialization corrupts a dictionary if the keys are references

by Sven Van Caekenberghe

> On 06 Apr 2016, at 15:34, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > 2016-04-06 15:25 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > Hi Nicolai, > > > On 06 Apr 2016, at 14:56, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > > > > > 2016-04-06 14:27 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > Fix for review: > > > > === > > Name: STON-Core-SvenVanCaekenberghe.71 > > Author: SvenVanCaekenberghe > > Time: 6 April 2016, 2:22:24.782251 pm > > UUID: 64b8b741-365e-41fe-aa98-565e33ca5d24 > > Ancestors: STON-Core-SvenVanCaekenberghe.70 > > > > Fix a bug where STONReferences occurring as keys in Dictionaries or elements in Sets caused those to be unhealthy after materialization. Thx to Peter UhnÃ¡k for reporting this issue. > > > > Add 3 new unit tests to STONReaderTests > > > > #testDictionaryWithReferenceKeys > > #testSetWithReferenceElements > > #testDeepStructure > > > > Fix Details > > > > change the implementation of STONReader>>#processSubObjectsOf: from iterative to recursive (see version 39 of 29 November 2012, this might be a functional regression, see #testDeepStructure; cleanup of stack instance variable for later) so that #stonProcessSubObjects: can be overwritten with code being executed before or after full reference resolution > > > > imho, recursion stack depth will be equal during both writing and reading, and should be acceptable. > > > > overwrite #stonProcessSubObjects: in Dictionary and Set to #rehash at the end, but only when needed (minimal optimalization, see Dictionary>>#containsStonReferenceAsKey and Set>>#containsStonReference) > > === > > Name: STON-Tests-SvenVanCaekenberghe.63 > > Author: SvenVanCaekenberghe > > Time: 6 April 2016, 2:22:45.01986 pm > > UUID: 0beb2322-b81a-46ee-a0e2-6648a808774a > > Ancestors: STON-Tests-SvenVanCaekenberghe.62 > > > > (idem) > > === > > Thanks for looking at the code. > > > Hi Sven, > > instead of rehashing the dictionary for every ston reference, > > (It rehashes only once after resolving all references) > > Ah, of course. I thought this would be called for every ston reference used as key. > > > > > wouldn't it work to remove and readd the value after processing the subobject: > > > > Dictionary>>#stonProcessSubObjects: block > > self keys do:[:key | > > |value| > > value := block value:(self removeKey: key ifAbsent:[ nil]). > > self at: (block value: key) put: value]. > > Interesting idea. I have to think about that approach. > > Now, Object>>#stonProcessSubObjects: is very general and looks at named and indexed instance variables. But this probably could be replaced by something more high level and specific I guess. > > Adding and removing each key/value has a cost too. I try to make the simplest case very efficient and only pay a price when really needed. Anyway, time for some calculations. > > ok, yes running over all keys isn't better than rehashing :-) Still, your idea might be better than you would expect: Now, there is 1 (partial) iteration to do the check before, 2 iterations over the keys and values arrays, then an optional full rehash (which also reallocates and thus generates garbage). Your idea does only 1 iteration over keys, with remove and add (which also happens more efficiently on the arrays above), no check before, no rehash, probably no garbage generation at all in any case. Like I said, I have to study it (especially the cost of remove/add, maybe that can be optimised a bit as well). > Thanks again for the suggestion ! > > Sven > > > > On 06 Apr 2016, at 14:04, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > > > https://pharo.fogbugz.com/f/cases/17946/STON-materializes-unhealthy-Diction… > > > > > > fix coming > > > > > >> On 05 Apr 2016, at 13:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> > > >>> > > >>> On 05 Apr 2016, at 13:02, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > >>> > > >>> > > >>> > > >>> 2016-04-05 12:32 GMT+02:00 Cyril Ferlicot Delbecque <cyril.ferlicot(a)gmail.com>: > > >>> > > >>> > > >>> On 05/04/2016 12:09, Sven Van Caekenberghe wrote: > > >>> > > >>>> Like I said, it is a hashing issue, sometimes it will be correct by accident. > > >>>> > > >>>> I hope you did not have to much trouble with this bug, I guess it must have been hard to chase. > > >>>> > > >>>> Is it urgent ? > > >>>> > > >>>> I probably can give you a quick fix, but I would like to think a bit more about this, since rehashing each materialised dictionary seems expensive. > > >>>> > > >>>> > > >>>> > > >>> > > >>> Hi Sven, > > >>> > > >>> I got the same kind of problem in a personal application. > > >>> > > >>> I use Sets that I serialize and I had a lot of trouble because sometimes > > >>> some action had strange behaviours. > > >>> > > >>> For example in a set with element `aSet remove: aSet anyOne` raised 'XXX > > >>> not found in aSet'. > > >>> > > >>> I am glad to hear that it is a Ston issue and not me that used sets in a > > >>> bad way :) > > >>> > > >>> For me too it is not urgent since I have a not of university work for > > >>> the moment. > > >>> > > >>> > > >>> > > >>> How are hashed collections created/filled during ston-parsing ? > > >>> If the position in a hashed collection is created by a ston-reference, that is later replaced by the "real" object, > > >>> the index in the dictionary (or other hashed collections) may be wrong. > > >> > > >> Yes, that is indeed it, Nicolai. > > >> > > >> But I would like to try to minimise the rehashing as it seems expensive. But first I need a more reliable failure. > > >> > > >>> -- > > >>> Cyril Ferlicot > > >>> > > >>> http://www.synectique.eu > > >>> > > >>> 165 Avenue Bretagne > > >>> Lille 59000 France > > >

April 6, 2016

Re: [Pharo-dev] [Pharo-users] STON materialization corrupts a dictionary if the keys are references

by Nicolai Hess

2016-04-06 15:25 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > Hi Nicolai, > > > On 06 Apr 2016, at 14:56, Nicolai Hess <nicolaihess(a)gmail.com> wrote: > > > > > > > > 2016-04-06 14:27 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > > Fix for review: > > > > === > > Name: STON-Core-SvenVanCaekenberghe.71 > > Author: SvenVanCaekenberghe > > Time: 6 April 2016, 2:22:24.782251 pm > > UUID: 64b8b741-365e-41fe-aa98-565e33ca5d24 > > Ancestors: STON-Core-SvenVanCaekenberghe.70 > > > > Fix a bug where STONReferences occurring as keys in Dictionaries or > elements in Sets caused those to be unhealthy after materialization. Thx to > Peter UhnÃ¡k for reporting this issue. > > > > Add 3 new unit tests to STONReaderTests > > > > #testDictionaryWithReferenceKeys > > #testSetWithReferenceElements > > #testDeepStructure > > > > Fix Details > > > > change the implementation of STONReader>>#processSubObjectsOf: from > iterative to recursive (see version 39 of 29 November 2012, this might be a > functional regression, see #testDeepStructure; cleanup of stack instance > variable for later) so that #stonProcessSubObjects: can be overwritten with > code being executed before or after full reference resolution > > > > imho, recursion stack depth will be equal during both writing and > reading, and should be acceptable. > > > > overwrite #stonProcessSubObjects: in Dictionary and Set to #rehash at > the end, but only when needed (minimal optimalization, see > Dictionary>>#containsStonReferenceAsKey and Set>>#containsStonReference) > > === > > Name: STON-Tests-SvenVanCaekenberghe.63 > > Author: SvenVanCaekenberghe > > Time: 6 April 2016, 2:22:45.01986 pm > > UUID: 0beb2322-b81a-46ee-a0e2-6648a808774a > > Ancestors: STON-Tests-SvenVanCaekenberghe.62 > > > > (idem) > > === > > Thanks for looking at the code. > > > Hi Sven, > > instead of rehashing the dictionary for every ston reference, > > (It rehashes only once after resolving all references) > Ah, of course. I thought this would be called for every ston reference used as key. > > > wouldn't it work to remove and readd the value after processing the > subobject: > > > > Dictionary>>#stonProcessSubObjects: block > > self keys do:[:key | > > |value| > > value := block value:(self removeKey: key ifAbsent:[ nil]). > > self at: (block value: key) put: value]. > > Interesting idea. I have to think about that approach. > > Now, Object>>#stonProcessSubObjects: is very general and looks at named > and indexed instance variables. But this probably could be replaced by > something more high level and specific I guess. > > Adding and removing each key/value has a cost too. I try to make the > simplest case very efficient and only pay a price when really needed. > Anyway, time for some calculations. > ok, yes running over all keys isn't better than rehashing :-) > > Thanks again for the suggestion ! > > Sven > > > > On 06 Apr 2016, at 14:04, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > > > > > https://pharo.fogbugz.com/f/cases/17946/STON-materializes-unhealthy-Diction… > > > > > > fix coming > > > > > >> On 05 Apr 2016, at 13:11, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> > > >>> > > >>> On 05 Apr 2016, at 13:02, Nicolai Hess <nicolaihess(a)gmail.com> > wrote: > > >>> > > >>> > > >>> > > >>> 2016-04-05 12:32 GMT+02:00 Cyril Ferlicot Delbecque < > cyril.ferlicot(a)gmail.com>: > > >>> > > >>> > > >>> On 05/04/2016 12:09, Sven Van Caekenberghe wrote: > > >>> > > >>>> Like I said, it is a hashing issue, sometimes it will be correct by > accident. > > >>>> > > >>>> I hope you did not have to much trouble with this bug, I guess it > must have been hard to chase. > > >>>> > > >>>> Is it urgent ? > > >>>> > > >>>> I probably can give you a quick fix, but I would like to think a > bit more about this, since rehashing each materialised dictionary seems > expensive. > > >>>> > > >>>> > > >>>> > > >>> > > >>> Hi Sven, > > >>> > > >>> I got the same kind of problem in a personal application. > > >>> > > >>> I use Sets that I serialize and I had a lot of trouble because > sometimes > > >>> some action had strange behaviours. > > >>> > > >>> For example in a set with element `aSet remove: aSet anyOne` raised > 'XXX > > >>> not found in aSet'. > > >>> > > >>> I am glad to hear that it is a Ston issue and not me that used sets > in a > > >>> bad way :) > > >>> > > >>> For me too it is not urgent since I have a not of university work for > > >>> the moment. > > >>> > > >>> > > >>> > > >>> How are hashed collections created/filled during ston-parsing ? > > >>> If the position in a hashed collection is created by a > ston-reference, that is later replaced by the "real" object, > > >>> the index in the dictionary (or other hashed collections) may be > wrong. > > >> > > >> Yes, that is indeed it, Nicolai. > > >> > > >> But I would like to try to minimise the rehashing as it seems > expensive. But first I need a more reliable failure. > > >> > > >>> -- > > >>> Cyril Ferlicot > > >>> > > >>> http://www.synectique.eu > > >>> > > >>> 165 Avenue Bretagne > > >>> Lille 59000 France > > > > > >

April 6, 2016