- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [Moose-dev] #deepCollect:

by Tudor Girba

Hi Esteban, Indeed, I pondered on the issue of extending Object with the deep* methods. In the end, I chose to extend it because traversals are not a cherry on top, they are rather essential for significant analysis. >From my point of view, Pharo has to become a platform in which understanding and manipulating objects is an essential concern. The reason is that in the long run, it is precisely assessment-related activities that account for the largest development costs. I detailed a bit this idea here: http://www.humane-assessment.com/blog/choose-your-technology-for-the-long-r… For this we need a powerful toolkit that makes crafting custom analyses cheap, and traversals has a prominent place in it. Cheers, Doru On Fri, Dec 13, 2013 at 6:17 PM, Esteban A. Maringolo <emaringolo(a)gmail.com>wrote: > To avoid such kind of situations I would implement all the traversal > methods as > a) Traits or > b) Class side methods of DeepTraversal (you choose the name) to which > you'll have to pass the collection and the block. > > E.g. DeepTraversal deep: aCollection do: aBlock > DeepTraversal flatten: aCollection thenDo: aBlock > > I, personally, don't like adding methods to Object unless it's > unavoidable(*). And in those cases I choose to prefix them with > something that is separate from the semantic, usually the > package/framework prefix. > > Regards, > > (*)I did that for years and in the long term it hits you back. > Esteban A. Maringolo > > > 2013/12/13 Chris Cunningham <cunningham.cb(a)gmail.com>: > > Hi. > > > > I was reading with interest the blog post on Traversal-enabled objects ( > > http://www.humane-assessment.com/blog/traversal-enabled-pharo-objects ) > when > > I noticed the method #deepCollect: referenced. Interestingly, I have a > > method called #deepCollect: that is use (wtih related methods like > #deepDo: > > and #deepSelect:). I suspect these uses may be compatible, with the > > traveral versions being more generic. > > > > My set of #deep methods allow arbitrary flattening of collections. The > > #flatCollect: suite in Pharo today flattens objects 1 level; the > > #deepCollect: flattens the collections as many levels deep as they are > > nested. I found this to be a really useful ability when I work with > > PetitParser parsings, which tend give back massively nested Arrays by > > default. > > > > If you are interested, it is published at: > > http://www.smalltalkhub.com/#!/~cbc/DeepCollection/ . > > > > -cbc > > -- www.tudorgirba.com "Every thing has its own flow"

Dec. 13, 2013

Re: [Pharo-dev] [Moose-dev] #deepCollect:

by Tudor Girba

Hi Chris, This is funny :). I took a look at your code, and if you remove your extensions on Collection, the code using deepCollect: and deep:do should work pretty much the same way. Cheers, Doru On Fri, Dec 13, 2013 at 6:02 PM, Chris Cunningham <cunningham.cb(a)gmail.com>wrote: > Hi. > > I was reading with interest the blog post on Traversal-enabled objects ( > http://www.humane-assessment.com/blog/traversal-enabled-pharo-objects ) > when I noticed the method #deepCollect: referenced. Interestingly, I have > a method called #deepCollect: that is use (wtih related methods like > #deepDo: and #deepSelect:). I suspect these uses may be compatible, with > the traveral versions being more generic. > > My set of #deep methods allow arbitrary flattening of collections. The > #flatCollect: suite in Pharo today flattens objects 1 level; the > #deepCollect: flattens the collections as many levels deep as they are > nested. I found this to be a really useful ability when I work with > PetitParser parsings, which tend give back massively nested Arrays by > default. > > If you are interested, it is published at: > http://www.smalltalkhub.com/#!/~cbc/DeepCollection/ . > > -cbc > -- www.tudorgirba.com "Every thing has its own flow"

Dec. 13, 2013

Re: [Pharo-dev] Tell me about your workflow

by GOUBIER Thierry

Then I will :) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de StÃ©phane Ducasse [stephane.ducasse(a)inria.fr] Date d'envoi : vendredi 13 dÃ©cembre 2013 13:47 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow try with Moose. Stef On Dec 12, 2013, at 4:20 PM, Yuriy Tymchuk <yuriy.tymchuk(a)me.com<mailto:yuriy.tymchuk@me.com>> wrote: Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :) Uko On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits). I'll keep in mind the idea of creating one randomly ;) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Yuriy Tymchuk [yuriy.tymchuk(a)me.com<mailto:yuriy.tymchuk@me.com>] Date d'envoi : jeudi 12 dÃ©cembre 2013 15:37 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow Are you interested in a package or a project? I can provide you information based on size, etcâ¦ Uko On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I gave up running gitfiletree on 1.4 :( It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : jeudi 12 dÃ©cembre 2013 12:12 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : mercredi 11 dÃ©cembre 2013 17:09 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a Ã©crit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a Ã©crit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm(a)gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a Ã©crit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

Dec. 13, 2013

Re: [Pharo-dev] Tell me about your workflow

by GOUBIER Thierry

Hi Sebastian, sorry for the missing instructions... Had to board a train :) In fact, it's not an alternative to filetree, instead it is an extension of filetree (and integrated into filetree, but not completely yet: no real configuration support, no windows support). But yes, I should do a blog post somewhere about it :) Meanwhile, in the train, I tested on Roassal and I profiled the writes... Git is slow: on Roassal, Waiting for the git commit : 67% Writing the package to disk : 20% (writing the version file: 0.4%) -> No real cost associated with the metadata -> git is a lot more than I expected. When I remove git (i.e. pure filetree) Making a snapshot of Roassal: 28.9% basic Store version : 67% writeBasicDefinitions : 30% (of which half of it is reordering some stuff, not writing) writeMethodHolderDefinitions: 27,8% Of which writing to disk effectively is 19.7% Conclusion: gitfiletree will look slow because git is slow on such large commits :( And I don't see much gains to be made on the writing (i.e. a format change won't help much), but a lot to loose (code complexity, bugs). Unless a format change may make it faster for git somehow? I'll keep profiling, or give you the code to use to profile yourself ? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com] Date d'envoi : vendredi 13 dÃ©cembre 2013 16:56 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow so you are working on an alternative to filetree I'm pretty sure if you setup a page (or blog post) somewhere with clear and easy instructions to follow many people will try to use it and, if proven good and reliable, adopt it sebastian<https://about.me/sebastianconcept> o/ On Dec 13, 2013, at 12:04 PM, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: gitfiletree can be had this way (filetree is already in 3.0, gitfiletree is in the filetree configuration for Pharo 3.0 but is a bit hard to load). pharo/pharo pharo/Pharo.image confighttp://smalltalkhub.com/Pharo/MetaRepoForPharo30/main/ ConfigurationOfOSProcess --install=stable pharo/pharo pharo/Pharo.image eval --save Gofer new url:\'http://smalltalkhub.com/mc/ThierryGoubier/Alt30/main\'\<http://smalltalkhub.com/mc/ThierryGoubier/Alt30/main/'/>; package: \'MonticelloFileTree-Git\'\; load Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : vendredi 13 dÃ©cembre 2013 14:57 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow I can try a forced load on a 3.0 ignoring requisites for testing purposes but I'm using this: https://github.com/dalehenrich/filetree I'm not sure what you're are using can you clarify? On Dec 13, 2013, at 11:43 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Ok. I'll try Roassal on a slow netbook to see. I don't see a factor of 10 difference between yours and Roassal, so I'll have a look. Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : vendredi 13 dÃ©cembre 2013 14:32 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow A bit. This is from today's current version (and is not all, it's only the two biggest packages): (MCPackage named: 'flow') workingCopy packageInfo classes size. 363. (MCPackage named: 'flow') workingCopy packageInfo coreMethods size. 4585. (MCPackage named: 'airflowing') workingCopy packageInfo classes size. 377. (MCPackage named: 'airflowing') workingCopy packageInfo coreMethods size. 5818. On Dec 13, 2013, at 11:25 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Roassal: 3493 Number of versions in the package history: 733. Size of the version file: 202796. Is that a lot lower than your count? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : vendredi 13 dÃ©cembre 2013 13:34 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow how many coreMethods? On Dec 13, 2013, at 7:00 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Bad news. Roassal package directory has 355 entries (343 classes + a few extensions) and I don't see much of a slow down (on 3.0). It's not instantaneous, but with a bit of feedback, it doesn't seems long. I'll do some profiling. Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de GOUBIER Thierry Date d'envoi : jeudi 12 dÃ©cembre 2013 17:07 Ã : Pharo Development List Objet : [PROVENANCE INTERNET] Re: [Pharo-dev] Tell me about your workflow Thanks for the pointers. I'll look at Seaside/Moose/Mondrian and Roassal, because I need code I can load and save in an image without destroying the very image I use to test (which would happen if I load Pharo10 stuff in a 3.0 image ;) ). Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Yuriy Tymchuk [yuriy.tymchuk(a)me.com<mailto:yuriy.tymchuk@me.com>] Date d'envoi : jeudi 12 dÃ©cembre 2013 16:24 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow So if you want something big and with a lot of commits you can use Pharo* in general. Pharo10 has the most versions and Pharo30Inbox is the largest one. If you want some other projects then you heve to take a look at Seaside30, Mondrian, Moose, Glamour or Roassal. Uko On 12 Dec 2013, at 16:20, Yuriy Tymchuk <yuriy.tymchuk(a)me.com<mailto:yuriy.tymchuk@me.com>> wrote: Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :) Uko On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits). I'll keep in mind the idea of creating one randomly ;) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Yuriy Tymchuk [yuriy.tymchuk(a)me.com<mailto:yuriy.tymchuk@me.com>] Date d'envoi : jeudi 12 dÃ©cembre 2013 15:37 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow Are you interested in a package or a project? I can provide you information based on size, etcâ¦ Uko On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I gave up running gitfiletree on 1.4 :( It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : jeudi 12 dÃ©cembre 2013 12:12 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : mercredi 11 dÃ©cembre 2013 17:09 Ã : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a Ã©crit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a Ã©crit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm(a)gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a Ã©crit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian(a)flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier(a)cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

Dec. 13, 2013

Re: [Pharo-dev] [Moose-dev] #deepCollect:

by Frank Shearar

On 13 December 2013 17:02, Chris Cunningham <cunningham.cb(a)gmail.com> wrote: > Hi. > > I was reading with interest the blog post on Traversal-enabled objects ( > http://www.humane-assessment.com/blog/traversal-enabled-pharo-objects ) when > I noticed the method #deepCollect: referenced. Interestingly, I have a > method called #deepCollect: that is use (wtih related methods like #deepDo: > and #deepSelect:). I suspect these uses may be compatible, with the > traveral versions being more generic. > > My set of #deep methods allow arbitrary flattening of collections. The > #flatCollect: suite in Pharo today flattens objects 1 level; the > #deepCollect: flattens the collections as many levels deep as they are > nested. I found this to be a really useful ability when I work with > PetitParser parsings, which tend give back massively nested Arrays by > default. What about flattening during the parse, with #flatten? frank > If you are interested, it is published at: > http://www.smalltalkhub.com/#!/~cbc/DeepCollection/ . > > -cbc

Dec. 13, 2013

Re: [Pharo-dev] about the Pharo Catalog

by Guillermo Polito

On Fri, Dec 13, 2013 at 1:51 PM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr > wrote: > > I'm now adding it for OpenDBX and glorp. We should create lint rules for > it ;) > > > please! > I want to add support for catalogMainContact too > I'll add it :) Can I ask for permissions to commit in the Catalog? > > > > > On Thu, Dec 12, 2013 at 10:57 AM, StÃ©phane Ducasse < > stephane.ducasse(a)inria.fr> wrote: > >> Yes :) >> In the output of the catalog itself >> >> >> https://ci.inria.fr/pharo-contribution/job/PharoProjectCatalog/HTML_Report/? >> >> There is probably a typo in the generator catalogkeyClassesAndExample => >> catalogKeyClassesAndExample >> 1. Aconcagua >> >> Please define a class method named catalogDescription returning a >> paragraph of description in the configuration class. >> 1.1. Keywords >> >> Please define a class method named catalogKeywords returning an array of >> symbols in the configuration class. >> 1.2. Key classes and Implementations Notes >> >> Please define a class method named catalogkeyClassesAndExample returning >> a paragraph describing the most important classes of the project and >> relevant information such as an example. >> 1.3. Key change logs >> >> Please define a class method named catalogChangeLog returning a paragraph >> describing the most important changes in the configuration class. We >> suggest to use the following format: >> >> - Version number - Date - topics >> >> >> ConfigurationOfXXX project version: 'xx' ) load >> >> or simply >> >> >> - Version number - Date - topics >> - Version number - Date - topics >> - Version number - Date - topics >> >> >> >> On Dec 12, 2013, at 10:49 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >> wrote: >> >> is there a place where I can see the metadata methods I need to implement? >> >> Esteban >> >> >> On Thu, Dec 12, 2013 at 9:59 AM, StÃ©phane Ducasse < >> stephane.ducasse(a)inria.fr> wrote: >> >>> Hi guys >>> >>> I really want to have an official catalog soon, so please add the >>> missing metadata in your configurations. >>> In addition I was discussing with robert pergl from Prague and he told >>> me that >>> when he has a problem with a package he does not know to whom he should >>> report so >>> I would like to add a catalogContact section. >>> Any idea for improvement. >>> >>> Stef >>> >> >> >> > >

Dec. 13, 2013

Re: [Pharo-dev] [Moose-dev] #deepCollect:

by Yuriy Tymchuk

Wow, this is really cool. I missed this while working with PP. Thanks Chris! Uko On 13 Dec 2013, at 18:02, Chris Cunningham <cunningham.cb(a)gmail.com> wrote: > Hi. > > I was reading with interest the blog post on Traversal-enabled objects ( http://www.humane-assessment.com/blog/traversal-enabled-pharo-objects ) when I noticed the method #deepCollect: referenced. Interestingly, I have a method called #deepCollect: that is use (wtih related methods like #deepDo: and #deepSelect:). I suspect these uses may be compatible, with the traveral versions being more generic. > > My set of #deep methods allow arbitrary flattening of collections. The #flatCollect: suite in Pharo today flattens objects 1 level; the #deepCollect: flattens the collections as many levels deep as they are nested. I found this to be a really useful ability when I work with PetitParser parsings, which tend give back massively nested Arrays by default. > > If you are interested, it is published at: http://www.smalltalkhub.com/#!/~cbc/DeepCollection/ . > > -cbc

Dec. 13, 2013

Re: [Pharo-dev] [Moose-dev] #deepCollect:

by Esteban A. Maringolo

To avoid such kind of situations I would implement all the traversal methods as a) Traits or b) Class side methods of DeepTraversal (you choose the name) to which you'll have to pass the collection and the block. E.g. DeepTraversal deep: aCollection do: aBlock DeepTraversal flatten: aCollection thenDo: aBlock I, personally, don't like adding methods to Object unless it's unavoidable(*). And in those cases I choose to prefix them with something that is separate from the semantic, usually the package/framework prefix. Regards, (*)I did that for years and in the long term it hits you back. Esteban A. Maringolo 2013/12/13 Chris Cunningham <cunningham.cb(a)gmail.com>: > Hi. > > I was reading with interest the blog post on Traversal-enabled objects ( > http://www.humane-assessment.com/blog/traversal-enabled-pharo-objects ) when > I noticed the method #deepCollect: referenced. Interestingly, I have a > method called #deepCollect: that is use (wtih related methods like #deepDo: > and #deepSelect:). I suspect these uses may be compatible, with the > traveral versions being more generic. > > My set of #deep methods allow arbitrary flattening of collections. The > #flatCollect: suite in Pharo today flattens objects 1 level; the > #deepCollect: flattens the collections as many levels deep as they are > nested. I found this to be a really useful ability when I work with > PetitParser parsings, which tend give back massively nested Arrays by > default. > > If you are interested, it is published at: > http://www.smalltalkhub.com/#!/~cbc/DeepCollection/ . > > -cbc

Dec. 13, 2013

[Moose-dev] #deepCollect:

by Chris Cunningham

Hi. I was reading with interest the blog post on Traversal-enabled objects ( http://www.humane-assessment.com/blog/traversal-enabled-pharo-objects ) when I noticed the method #deepCollect: referenced. Interestingly, I have a method called #deepCollect: that is use (wtih related methods like #deepDo: and #deepSelect:). I suspect these uses may be compatible, with the traveral versions being more generic. My set of #deep methods allow arbitrary flattening of collections. The #flatCollect: suite in Pharo today flattens objects 1 level; the #deepCollect: flattens the collections as many levels deep as they are nested. I found this to be a really useful ability when I work with PetitParser parsings, which tend give back massively nested Arrays by default. If you are interested, it is published at: http://www.smalltalkhub.com/#!/~cbc/DeepCollection/ . -cbc

Dec. 13, 2013

Re: [Pharo-dev] Tell me about your workflow

by Sebastian Sastre

so you are working on an alternative to filetree I'm pretty sure if you setup a page (or blog post) somewhere with clear and easy instructions to follow many people will try to use it and, if proven good and reliable, adopt it sebastian o/ On Dec 13, 2013, at 12:04 PM, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: > gitfiletree can be had this way (filetree is already in 3.0, gitfiletree is in the filetree configuration for Pharo 3.0 but is a bit hard to load). > > pharo/pharo pharo/Pharo.image confighttp://smalltalkhub.com/Pharo/MetaRepoForPharo30/main/ ConfigurationOfOSProcess --install=stable > pharo/pharo pharo/Pharo.image eval --save Gofer new url:\'http://smalltalkhub.com/mc/ThierryGoubier/Alt30/main\'\; package: \'MonticelloFileTree-Git\'\; load > > Thierry > > De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com] > Date d'envoi : vendredi 13 dÃ©cembre 2013 14:57 > Ã : Pharo Development List > Objet : Re: [Pharo-dev] Tell me about your workflow > > I can try a forced load on a 3.0 ignoring requisites for testing purposes > > but I'm using this: > https://github.com/dalehenrich/filetree > > I'm not sure what you're are using > > can you clarify? > > > > On Dec 13, 2013, at 11:43 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: > >> Ok. I'll try Roassal on a slow netbook to see. I don't see a factor of 10 difference between yours and Roassal, so I'll have a look. >> >> Thierry >> De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com] >> Date d'envoi : vendredi 13 dÃ©cembre 2013 14:32 >> Ã : Pharo Development List >> Objet : Re: [Pharo-dev] Tell me about your workflow >> >> A bit. >> >> This is from today's current version (and is not all, it's only the two biggest packages): >> >> (MCPackage named: 'flow') workingCopy packageInfo classes size. 363. >> (MCPackage named: 'flow') workingCopy packageInfo coreMethods size. 4585. >> >> (MCPackage named: 'airflowing') workingCopy packageInfo classes size. 377. >> (MCPackage named: 'airflowing') workingCopy packageInfo coreMethods size. 5818. >> >> >> >> >> >> >> >> On Dec 13, 2013, at 11:25 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: >> >>> Roassal: 3493 >>> >>> Number of versions in the package history: 733. Size of the version file: 202796. >>> >>> Is that a lot lower than your count? >>> >>> Thierry >>> >>> De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com] >>> Date d'envoi : vendredi 13 dÃ©cembre 2013 13:34 >>> Ã : Pharo Development List >>> Objet : Re: [Pharo-dev] Tell me about your workflow >>> >>> how many coreMethods? >>> >>> >>> >>> >>> On Dec 13, 2013, at 7:00 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: >>> >>>> Bad news. Roassal package directory has 355 entries (343 classes + a few extensions) and I don't see much of a slow down (on 3.0). It's not instantaneous, but with a bit of feedback, it doesn't seems long. >>>> >>>> I'll do some profiling. >>>> >>>> Thierry >>>> De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de GOUBIER Thierry >>>> Date d'envoi : jeudi 12 dÃ©cembre 2013 17:07 >>>> Ã : Pharo Development List >>>> Objet : [PROVENANCE INTERNET] Re: [Pharo-dev] Tell me about your workflow >>>> >>>> Thanks for the pointers. >>>> >>>> I'll look at Seaside/Moose/Mondrian and Roassal, because I need code I can load and save in an image without destroying the very image I use to test (which would happen if I load Pharo10 stuff in a 3.0 image ;) ). >>>> >>>> Thierry >>>> >>>> De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk(a)me.com] >>>> Date d'envoi : jeudi 12 dÃ©cembre 2013 16:24 >>>> Ã : Pharo Development List >>>> Objet : Re: [Pharo-dev] Tell me about your workflow >>>> >>>> So if you want something big and with a lot of commits you can use Pharo* in general. Pharo10 has the most versions and Pharo30Inbox is the largest one. If you want some other projects then you heve to take a look at Seaside30, Mondrian, Moose, Glamour or Roassal. >>>> >>>> Uko >>>> >>>> On 12 Dec 2013, at 16:20, Yuriy Tymchuk <yuriy.tymchuk(a)me.com> wrote: >>>> >>>>> Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :) >>>>> >>>>> Uko >>>>> >>>>> On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: >>>>> >>>>>> I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits). >>>>>> >>>>>> I'll keep in mind the idea of creating one randomly ;) >>>>>> >>>>>> Thierry >>>>>> >>>>>> De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk(a)me.com] >>>>>> Date d'envoi : jeudi 12 dÃ©cembre 2013 15:37 >>>>>> Ã : Pharo Development List >>>>>> Objet : Re: [Pharo-dev] Tell me about your workflow >>>>>> >>>>>> Are you interested in a package or a project? I can provide you information based on size, etcâ¦ >>>>>> >>>>>> Uko >>>>>> >>>>>> On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: >>>>>> >>>>>>> I gave up running gitfiletree on 1.4 :( >>>>>>> >>>>>>> It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. >>>>>>> >>>>>>> My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? >>>>>>> >>>>>>> Thierry >>>>>>> >>>>>>> De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com] >>>>>>> Date d'envoi : jeudi 12 dÃ©cembre 2013 12:12 >>>>>>> Ã : Pharo Development List >>>>>>> Objet : Re: [Pharo-dev] Tell me about your workflow >>>>>>> >>>>>>> gee the big code package is airflowing which I have, quite conservatively, running on #14438 images >>>>>>> >>>>>>> I load filetree like this: >>>>>>> >>>>>>> Gofer new >>>>>>> url: 'http://ss3.gemstone.com/ss/FileTree'; >>>>>>> package: 'ConfigurationOfFileTree'; >>>>>>> load. >>>>>>> ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. >>>>>>> >>>>>>> and it never complained >>>>>>> >>>>>>> let me know >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: >>>>>>> >>>>>>>> If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Thierry >>>>>>>> >>>>>>>> >>>>>>>> De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Sebastian Sastre [sebastian(a)flowingconcept.com] >>>>>>>> Date d'envoi : mercredi 11 dÃ©cembre 2013 17:09 >>>>>>>> Ã : Pharo Development List >>>>>>>> Objet : Re: [Pharo-dev] Tell me about your workflow >>>>>>>> >>>>>>>> ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. >>>>>>>> >>>>>>>> about 2, I don't know. I'm available to make tests and measure results >>>>>>>> >>>>>>>> have a nice trip, keep us tuned about any progress >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier(a)cea.fr> wrote: >>>>>>>> >>>>>>>>> Yes, you're right in the general case. >>>>>>>>> >>>>>>>>> But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. >>>>>>>>> >>>>>>>>> If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- >>>>>>>>> >>>>>>>>> (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). >>>>>>>>> >>>>>>>>> Thierry >>>>>>>>> >>>>>>>>> Le 11/12/2013 16:44, Sebastian Sastre a Ã©crit : >>>>>>>>>> Without entering in details, a cause for slow package write is dumping >>>>>>>>>> all every time. >>>>>>>>>> >>>>>>>>>> For that strategy, we already have the image save which is magically fast. >>>>>>>>>> >>>>>>>>>> So, if we make something to scan the code and write only when it's >>>>>>>>>> different from what's on disk, then we would be preventing tons of >>>>>>>>>> redundant writes >>>>>>>>>> >>>>>>>>>> sebastian <https://about.me/sebastianconcept> >>>>>>>>>> >>>>>>>>>> o/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier(a)cea.fr >>>>>>>>>> <mailto:thierry.goubier@cea.fr>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Le 11/12/2013 16:27, Esteban Lorenzano a Ã©crit : >>>>>>>>>>>> ah, and IMHO the problem is not about reading... is about writing (if it >>>>>>>>>>>> has to write the metadata each time...). >>>>>>>>>>> >>>>>>>>>>> But, personnaly, I don't know if this is the reason for the lack of >>>>>>>>>>> performance... >>>>>>>>>>> >>>>>>>>>>> I have three hypothesis for Sebastian problem: >>>>>>>>>>> 1 - Slow read time for version metadata >>>>>>>>>>> - Confirmed because of the 16 seconds wait time for reading the >>>>>>>>>>> package metadata in the repository browser. >>>>>>>>>>> 2 - Slow metadata write >>>>>>>>>>> 3 - Slow package write >>>>>>>>>>> >>>>>>>>>>> I have an implemented solution for 1-, a very easy to implement for >>>>>>>>>>> 2-, and none yet for 3- >>>>>>>>>>> >>>>>>>>>>> So I'd really like to check if 3- is confirmed ;) >>>>>>>>>>> >>>>>>>>>>> Thierry >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Esteban >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano >>>>>>>>>>>> <estebanlm(a)gmail.com <mailto:estebanlm@gmail.com> >>>>>>>>>>>> <mailto:estebanlm@gmail.com>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Thierry, I know there is a working version... let me search... >>>>>>>>>>>> >>>>>>>>>>>> (5 mins later) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> here: >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/rjsargent/CypressReferenceImplementation >>>>>>>>>>>> >>>>>>>>>>>> Dale says Richard made a metadata-less version. >>>>>>>>>>>> >>>>>>>>>>>> We should take a look at that. >>>>>>>>>>>> >>>>>>>>>>>> Esteban >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry >>>>>>>>>>>> <thierry.goubier(a)cea.fr >>>>>>>>>>>> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Esteban, Sebastian, >>>>>>>>>>>> >>>>>>>>>>>> In the filetree code, you will find a format without metadata, >>>>>>>>>>>> but it's not in use anymore. >>>>>>>>>>>> >>>>>>>>>>>> If you use gitfiletree, it will write the metadata for >>>>>>>>>>>> compatibility reasons with filetree, but it will never read it >>>>>>>>>>>> back. >>>>>>>>>>>> >>>>>>>>>>>> I'm pushing code to make filetree robust to absence of metadata, >>>>>>>>>>>> but I haven't worked on it for a while. >>>>>>>>>>>> >>>>>>>>>>>> gitfiletree has solved the problem of a slow metadata read. It >>>>>>>>>>>> does not solve any performance problem associated with >>>>>>>>>>>> writing, yet. >>>>>>>>>>>> >>>>>>>>>>>> Thierry >>>>>>>>>>>> >>>>>>>>>>>> Le 11/12/2013 16:12, Esteban Lorenzano a Ã©crit : >>>>>>>>>>>> >>>>>>>>>>>> I know there is a version of filetree without metadata (more >>>>>>>>>>>> compelling >>>>>>>>>>>> for projects that will never use other formats). >>>>>>>>>>>> Dale told me that there was a preview somewhere, but I >>>>>>>>>>>> didn't tested yet >>>>>>>>>>>> (lack of time) and now I cannot find the mail... >>>>>>>>>>>> Dale, can you re-send the link? >>>>>>>>>>>> >>>>>>>>>>>> cheers, >>>>>>>>>>>> Esteban >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre >>>>>>>>>>>> <sebastian(a)flowingconcept.com >>>>>>>>>>>> <mailto:sebastian@flowingconcept.com> >>>>>>>>>>>> <mailto:sebastian@flowingconcept.com> >>>>>>>>>>>> <mailto:sebastian@__flowingconcept.com >>>>>>>>>>>> <mailto:sebastian@flowingconcept.com>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I should breath before I type, but you probably already >>>>>>>>>>>> got that I >>>>>>>>>>>> meant /redundant writes/ (not reads)... >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Anyway.. I was talking with Esteban and he mentions >>>>>>>>>>>> some kind of >>>>>>>>>>>> compatibility metadata. >>>>>>>>>>>> >>>>>>>>>>>> If I'm going to give a leap of faith to filetree repos >>>>>>>>>>>> to save code >>>>>>>>>>>> why should I care about mcz compatibility? Paying a >>>>>>>>>>>> toll for no >>>>>>>>>>>> reason is evil. >>>>>>>>>>>> >>>>>>>>>>>> Maybe we could make that optional so those who don't >>>>>>>>>>>> extract value >>>>>>>>>>>> from that feature can opt-out? >>>>>>>>>>>> >>>>>>>>>>>> sebastian <https://about.me/__sebastianconcept >>>>>>>>>>>> <https://about.me/sebastianconcept>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> o/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Dec 11, 2013, at 12:44 PM, Sebastian Sastre >>>>>>>>>>>> <sebastian(a)flowingconcept.com >>>>>>>>>>>> <mailto:sebastian@flowingconcept.com> >>>>>>>>>>>> <mailto:sebastian@flowingconcept.com> >>>>>>>>>>>> <mailto:sebastian@__flowingconcept.com >>>>>>>>>>>> <mailto:sebastian@flowingconcept.com>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Thierry >>>>>>>>>>>> >>>>>>>>>>>> On Dec 11, 2013, at 12:43 PM, Goubier Thierry >>>>>>>>>>>> <thierry.goubier(a)cea.fr >>>>>>>>>>>> <mailto:thierry.goubier@cea.fr> >>>>>>>>>>>> <mailto:thierry.goubier@cea.fr> >>>>>>>>>>>> <mailto:thierry.goubier@cea.fr >>>>>>>>>>>> <mailto:thierry.goubier@cea.fr>__>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I have packages (in the order of hundreds >>>>>>>>>>>> of classes) and save >>>>>>>>>>>> delays >>>>>>>>>>>> and package click delays are starting to >>>>>>>>>>>> demand patience in a >>>>>>>>>>>> way that >>>>>>>>>>>> doesn't feel like the right path >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Which operations ? I didn't remember noticing >>>>>>>>>>>> much with 179 >>>>>>>>>>>> classes on a laptop without a SSD. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> choose one. Just for clicking the package that will >>>>>>>>>>>> should you >>>>>>>>>>>> UUID, version and author I need to wait ~16 >>>>>>>>>>>> seconds. Sounds like a >>>>>>>>>>>> lot of overhead for reading a small .json file. >>>>>>>>>>>> >>>>>>>>>>>> But the write is the most worrisome >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> All that is with a SSD disk, otherwise save >>>>>>>>>>>> delays would be >>>>>>>>>>>> /way/ beyond >>>>>>>>>>>> unacceptable >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'd like to know more, and understand the >>>>>>>>>>>> reason, for sure. As >>>>>>>>>>>> far as I know, filetree will rewrite the whole >>>>>>>>>>>> package to disk >>>>>>>>>>>> everytime... and maybe optimising that could be >>>>>>>>>>>> the solution. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Well, that explains a lot. Writing all every time >>>>>>>>>>>> is the lazy >>>>>>>>>>>> thing that's okay for a prototype and temporary >>>>>>>>>>>> code in a proof of >>>>>>>>>>>> concept but that massive redundant reads certainly >>>>>>>>>>>> doesn't sounds >>>>>>>>>>>> like pro software. Specially for SSD's which has a >>>>>>>>>>>> limited >>>>>>>>>>>> quantity of writes >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thierry >>>>>>>>>>>> >>>>>>>>>>>> sebastian >>>>>>>>>>>> <https://about.me/__sebastianconcept >>>>>>>>>>>> <https://about.me/sebastianconcept>> >>>>>>>>>>>> >>>>>>>>>>>> o/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thierry Goubier >>>>>>>>>>>> CEA list >>>>>>>>>>>> Laboratoire des Fondations des SystÃ¨mes Temps >>>>>>>>>>>> RÃ©el EmbarquÃ©s >>>>>>>>>>>> 91191 Gif sur Yvette Cedex >>>>>>>>>>>> France >>>>>>>>>>>> Phone/Fax: +33 (0) 1 69 08 32 92 >>>>>>>>>>>> <tel:%2B33%20%280%29%201%2069%2008%2032%2092> >>>>>>>>>>>> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> >>>>>>>>>>>> / 83 95 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thierry Goubier >>>>>>>>>>>> CEA list >>>>>>>>>>>> Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s >>>>>>>>>>>> 91191 Gif sur Yvette Cedex >>>>>>>>>>>> France >>>>>>>>>>>> Phone/Fax: +33 (0) 1 69 08 32 92 >>>>>>>>>>>> <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Thierry Goubier >>>>>>>>>>> CEA list >>>>>>>>>>> Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s >>>>>>>>>>> 91191 Gif sur Yvette Cedex >>>>>>>>>>> France >>>>>>>>>>> Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thierry Goubier >>>>>>>>> CEA list >>>>>>>>> Laboratoire des Fondations des SystÃ¨mes Temps RÃ©el EmbarquÃ©s >>>>>>>>> 91191 Gif sur Yvette Cedex >>>>>>>>> France >>>>>>>>> Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

Dec. 13, 2013