June 2016 - Pharo-users - lists.pharo.org

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/29/16 6:30 AM, Nicolas Passerini wrote: > Thank you Thierry! > > I have more questions inline. > > On Wed, Jun 29, 2016 at 10:44 AM, Thierry Goubier > <thierry.goubier(a)gmail.com <mailto:thierry.goubier@gmail.com>> wrote: > > All the Monticello GUI : track down version numbers to order stuff > in the lists views everywhere. > > > So, if we built a new GUI which does not rely on those version numbers > maybe we could get rid of that problem, what do you think? The package browser for a git repository does not need to list all versions of the package ... listing a single package name (sans version number and author) is sufficient. Of course this new tool needs to provide access to the commit log for the repo and then you need to be allowed to do diffs between git versions without respect to packages as well ... The tool should also provide the git commit log for the package directory (moral equivalent of package history) but then there should also be a way to see the commit history for a class and the methods ... These are things that I have already done for tODE... > > > The Metacello / Gofer stuff (Configurations, Baselines) also use > version numbers. > > > Yes, I understand that they use version numbers, but do they require > that version numbers are correlative? in Metacello, package version numbers are only relevant in the context of a ConfigurationOf. Only the base package name (sans version number and author) is used in a BaselineOf ... technically ... right now ... Metacello does use the version number when determining whether or not to load a package, but I have a Cypress package extension for Metacello that ignores the package version when loading and this extension will become part of the standard Metacello release in support of metadataless repos .... Today a ConfigurationOf should not/cannot be used with a metadataless repository, so version numbers on packages is a moot point. > > > One of the difficulty of switching will be the cohabitation of > number-based systems (smalltalkhub) and SHA-based systems, > especially when you do things like copying a version from a git > repository to a smalltalkhub repository. > > > I think that currently it is not possible to copy a number-based > package version from smalltalkhub to a metadata-less git repository > using git file tree. I mean, you can but the new version will not have > the same version number, right? Yes ... > > > That's why I suggested to change the ordering relation to a > partial order based on the property A is an ancestor of B -> B is > before A. > > > I've lost you there. I agree that it is true that A is an ancestor of > B -> B is before A, but the most common problem goes the other way > arround, i.e. you want to know if A is ancestor of B and the > implication then is not true: B is before A 'does not imply' B is > ancestor of A. So I missed your point here. I think that ordering is a moot point for metadata-less repositories ... There is only one package version present in a checkout of a git repository and it think that it makes sense to turn to explicit git-based tools when dealing with git-based repositories ... I know that GitFileTree has been aimed at preserving the Monticello-ness of packages, but I think that approach is limiting --- because Monticello itself is limited --- When I look at a particular commit that I've done with from tODE I not only see the changes that I've made to package, but I also see the changes that I made along the way to other things I've stashed in the git repository like documentation and scripts so to really support the use of git from within the image, we have to start acknowledging that there are artifacts in the git repository that are not just Monticello packages and those artifacts are important to me as a developer to be able to view in the context of the changes that I've made to my code and non-code artifacts ... For me going metadataless does not imply that "we reconstruct monticello meta data from git" but that "we abandon the Monticello meta data completely" in favor or git-based tools ... Monticello packages with full metadata will not go away anytime soon, and I am not advocating the abandonment Monticello altogether. I am advocating that for developers choosing to use git that we build a set of tools that give direct access to git artifacts ... In tODE I have found that it is possible to provide all of the familiar tools for Monticello packages using a git flavor in parallel with the traditional Monticello metadata-full tools ... at this point in the evolution of Pharo I think that a hybrid approach is called for .. Dale

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/29/16 5:30 AM, Thierry Goubier wrote: > Le 29/06/2016 12:57, Ben Coman a Ã©crit : >> On Wed, Jun 29, 2016 at 4:00 PM, Thierry Goubier >> <thierry.goubier(a)gmail.com> wrote: >>> Le 29/06/2016 00:55, Dale Henrichs a Ã©crit : >>>> >>>> >>> ... >>>>> >>>>> >>>> I'm pretty certain the MCLazyVersionInfo is the real culprit here ... >>>> while reading the code I recognized that many of the basic patterns >>>> were >>>> exactly as I had remembered them from years ago ... however ... >>>> MCLazyVersionInfo this puppy with its "default behavior" to scan the >>>> universe is the real culprit ... I would think that at a minimum the >>>> repository or repository group would/could be know at the time that >>>> the >>>> MCLazyVersionInfo was created and a scan of just those repositories >>>> --- >>>> already associated with the project --- would not be nearly as bad as >>>> when we have now ... >>> >>> >>> The MCLazyVersionInfo thing is mine too; it was a solution to avoid >>> keeping >>> MBs of version info kept inside the image memory, with the cost of >>> having to >>> reload that information when you access the ancestry. >> >> Would it be feasible to drop all those MB except keep the latest >> version of a package in-Image, >> i.e. the one required to determine the next version number of the >> package. ? > > This is not considered enough for Monticello. If you are on a > multi-people team with a smalltalkhub repository, keeping memory of > myPackage.7 is not enough because your team members may push .8, .9, > etc... before your save, and your save should be .10, not .8. So an > access to the remote repository is necessary (and all of them; maybe > you're working with multiple smalltalkhub repositories for that package). when you are doing the next version number calculation, you only need the first ancestor (the ancestor that is loaded in the image) ... to find whether or not team members have committed new versions involves scanning the packages in the repositories in the repository group and picking up the first version in the ancestry ... it seems for a working copy that only the first ancestor is needed in the image ... the rest can be lazily loaded if one wants to see history ... The new versions committed by team members aren't even in the ancestry of the working copy ... I have not looked at the details of the caching that is used by the repository implementation with respect to versionInfo, but if my allInstances yesterday is any indication, it is/was always necessary to scan the repositories in the repository group to look for newer versions of the package but the version info for those scans wasn't necessarilyt cached anyway ... > > Of course, MC does not lock the remote repository while you do that, > so the number it generates at the beginning (just before you get the > window where you fill in the package version comment) is not > guaranteed to be up to date if a team member saves a new version in > the mean-time. > > Of course, MC does not forbid you to change by hand the version number. Right ... so close counts in Monticello version numbers and horse shoes :) Dale

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/29/16 1:00 AM, Thierry Goubier wrote: > Le 29/06/2016 00:55, Dale Henrichs a Ã©crit : >> > ... >>> >> I'm pretty certain the MCLazyVersionInfo is the real culprit here ... >> while reading the code I recognized that many of the basic patterns were >> exactly as I had remembered them from years ago ... however ... >> MCLazyVersionInfo this puppy with its "default behavior" to scan the >> universe is the real culprit ... I would think that at a minimum the >> repository or repository group would/could be know at the time that the >> MCLazyVersionInfo was created and a scan of just those repositories --- >> already associated with the project --- would not be nearly as bad as >> when we have now ... > > The MCLazyVersionInfo thing is mine too; it was a solution to avoid > keeping MBs of version info kept inside the image memory, with the > cost of having to reload that information when you access the ancestry. > > Now, the approach needs to be tuned to avoid spurious "query the > world" searches, but, as you point out, I hasn't been too successfull > yet. And one of the thing MC lack, is that link between a repository > and a working copy. That's not true. Each working copy has repository group where either the developer has either explicitly declared the set of repositories that are associated with the working copy and/or the system has recorded the set of repositories from which the package has been loaded ... you should restrict the search to the repos in the repository group. > > (at the same time, restricting version number determination to a > subset of the repositories is against MC principles when numbering > versions). In principle, yes, but from a practical point of view, the version number search was "always" restricted to the set of repositories in the repository group for the working copy ... keep in mind that putting version numbers in the package name is a _convention_. I don't think that Monticello ever defined a package name syntax ... Technically, Monticello is supposed to use the UUID to disambiguate between to packages with the same name, but as you and I know, this is not really enforced in the code --- Monticello evolved to rely on the version number for ordering package versions, but I don't think that was ever part of the design of Monticello ... if you look at the older implementations of the tools, there was no enformcement of any rationale package naming scheme ... I agree that today when looking at the point to which Monticello usage has evolved that the statement: "the only way to guarantee a unique version number for a package is to scan all available repositories" is absolutely true, but even then there is no guarantee that a duplicate package does not exist in a repository that is not connected to the repo ... However, Monticello is also tolerant of packages with the same name and in practice, the same author rarely if ever produces two different versions of a package with the same name (unless they do it on purpose) and of course two packages with the same name cannot coexist in a single mcz repository, so this condition is even more rare and not necessarily worth scanning all connected repos ....

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/29/16 1:44 AM, Thierry Goubier wrote: > Hi Nicolas, > > Le 29/06/2016 10:24, Nicolas Passerini a Ã©crit : >> Hi Dale, Therry, I am working on a libgit-based git integration for >> Pharo and I am facing the same problem about version numbers. So maybe I >> can help you if you are working on that. > > How is the libgit integration going? > >> I had the same idea as Dale, using short SHAs instead of correlative >> numbers, but I didn't have the time yet to got down that path. > > Ok. > >> Therry, you said that >> >> >> you'll have a bunch of stuff expecting version numbers that will >> stop working. >> >> Do you already know which stuff will stop working? That would be realy >> helpful. >> > All the Monticello GUI : track down version numbers to order stuff in > the lists views everywhere. > > The Metacello / Gofer stuff (Configurations, Baselines) also use > version numbers. > > One of the difficulty of switching will be the cohabitation of > number-based systems (smalltalkhub) and SHA-based systems, especially > when you do things like copying a version from a git repository to a > smalltalkhub repository. My thoughts are that the we'd change the package name parsers to accept hex numbers as version numbers ... granted packages with hex version numbers could wreak havoc if they found their way into an mcz repository, but frankly a developer choosing to go metadata-less isn't likely to be using very many mcz repositories anyway and if a project is migrated to an mcz repository I would think it would be acceptable for the first version in the ancestry to be the Short SHA (allowing one to trace an ancestry manually in a git repo if desired) and subsequent version numbers could use decimal notation ...the version number is not involved in the maintaining monticello ancestry (using a version number is a convention as far as monticello is concerned) > > That's why I suggested to change the ordering relation to a partial > order based on the property A is an ancestor of B -> B is before A. Frankly for git packages, I have toyed with notion of eliminating the version number from the package name altogether. The version number in the name of a package was invented as a convention because otherwise there was no way to distinguish different Monticello packages from each other ... in FileTree we simply use the name of the package to name the directory and technically there is only ever one version of a package in a git repo anyway ... perhaps we should look at completely eliminating the version number from the package altogether? With the cypress package extension, Metacello should be ignoring the version number anyway --- a package without a version number is "obviously metadata-less" ... Hmmm, it might even be easier to parse a package that doesn't even have a version number (or author name for that matter) ... If we are going to be making mods to tools it might be better to go this route ... What do you guys think? Dale

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/29/16 1:24 AM, Nicolas Passerini wrote: > Hi Dale, Therry, I am working on a libgit-based git integration for > Pharo and I am facing the same problem about version numbers. So maybe > I can help you if you are working on that. > I had the same idea as Dale, using short SHAs instead of correlative > numbers, but I didn't have the time yet to got down that path. > > Therry, you said that > > > you'll have a bunch of stuff expecting version numbers that will > stop working. > > Do you already know which stuff will stop working? That would be realy > helpful. > Nicolas, Yes. There are a number of different places in the image where the Monticello package name is parsed. GoferVersionReference>>parseName: is one such place and I know that Metacello has it's own parser for Monticello package names and there are probably a few others that are floating around in the Monticello Browser I don't know whether tools like Kommitter/Versionner are doing their own parsing or not. If I'm not mistaken you will probably find all the Monticello name parsers by looking for senders of #isDigit which should result in some fairly obvious suspects ... then we just need to figure out if we can create a parser that will accept a hex version number and go from there ... If we assume that the short SHA version numbers will only be used with Pharo5.0/6.0/GemStone (platforms for which new tools have been created), then we might be able to pull this off... Dale

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/29/16 1:09 AM, Thierry Goubier wrote: > Dale, > > Le 29/06/2016 00:55, Dale Henrichs a Ã©crit : >> >> > ... > >> Actually, Metacello has a registry of loaded projects (baselines and >> configurations) and using Metacello it is possible to determine which >> pacakges belong to which projects etc. so it is not necessary to >> "register packages with a project" ... the project registry in Metacello >> is enough ... > > How do one adds a new package to a project in that registry? One can > create new packages in a repository, and add them to the baseline (by > editing the baseline method)? Yes, when you add a new package to the baseline the package becomes part of the project ... the exceptions come into play when you add a package to a baseline that is not aimed at the current platform :) For example when I'm working in GemStone I may add a package to the baseline for Pharo and actually create the initial version of the package in the GemStone image ... now as far as Metacello is concerned, the pharo package is not part of the GemStone baseline ... but this is not a common situation and in my case I manually commit the pharo package and don't really expect it to be "automatically managed" ... > > Do we need to rethink the Kommitter/Versionner stuff to provide a GUI > to manipulate such things? I am not (yet) familiar with Kommitter/Versionner, but I imagine that a Metacello Project Browser is needed... In tODE the "Metacello Project Browser" is called the `project list` and does more than just manage the packages and git repo for the project --- there is a one to one relationship between a git repo and a baseline that is very nice --- From the tODE "Metacello Project Browser" one can open a browser on all of the classes in the project, or all of the categories in the project, or all of the packages in the project, or all of the dirty packages in the project, or a changes browser that displays all of the changes to all of the packages in one place, or open a test browser on all of the tests in the project, etc.... So new tools and extensions to existing tools are probably called for ... Dale

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Thierry Goubier

Le 29/06/2016 15:30, Nicolas Passerini a Ã©crit : > Thank you Thierry! > > I have more questions inline. > > On Wed, Jun 29, 2016 at 10:44 AM, Thierry Goubier > <thierry.goubier(a)gmail.com <mailto:thierry.goubier@gmail.com>> wrote: > > All the Monticello GUI : track down version numbers to order stuff > in the lists views everywhere. > > > So, if we built a new GUI which does not rely on those version numbers > maybe we could get rid of that problem, what do you think? Yes, of course. But this still means we have to rebuild the GUI... which probably requires it anyway :) > > The Metacello / Gofer stuff (Configurations, Baselines) also use > version numbers. > > > Yes, I understand that they use version numbers, but do they require > that version numbers are correlative? They need those version numbers to indicate a total ordering of versions, to be able to: - determine what is the HEAD of the repository, - determine if a currently loaded version in the image should be replaced by a higher version numbered from the configuration or baseline. Now we could know what should be the API on there. > One of the difficulty of switching will be the cohabitation of > number-based systems (smalltalkhub) and SHA-based systems, > especially when you do things like copying a version from a git > repository to a smalltalkhub repository. > > > I think that currently it is not possible to copy a number-based package > version from smalltalkhub to a metadata-less git repository using git > file tree. I mean, you can but the new version will not have the same > version number, right? In that way, you're right. The other way works. The other way is more common, as far as I can see. Metacello can only distribute from github and bitbucket; git repositories can happen in many other places, and gitfiletree handles most of them. > That's why I suggested to change the ordering relation to a partial > order based on the property A is an ancestor of B -> B is before A. > > > I've lost you there. I agree that it is true that A is an ancestor of B > -> B is before A, but the most common problem goes the other way > arround, i.e. you want to know if A is ancestor of B and the implication > then is not true: B is before A 'does not imply' B is ancestor of A. So > I missed your point here. Well, this is why I talked about a partial order. If two versions belonging to two different branches are compared, then you can't say anything about the order of the two versions. Thierry

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/29/16 1:05 AM, Thierry Goubier wrote: > Dale, > > you're right, the monticello.meta directory is kept, in part because I > wasn't sure I could get rid of some of the information in some files > in that directory (dependencies, initializers). > > Since those files don't usually create conflicts with git, it seemed Ok. Makes a ton of sense ... the other information on disk can be fabricated on demand, but I am also a fan of "doing the smallest amount needed" .... I got confused because I _expected_ the directory to be gone .... Dale

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Dale Henrichs

On 6/28/16 10:22 PM, Thierry Goubier wrote: > Hi Dale, > > Le 29/06/2016 01:50, Dale Henrichs a Ã©crit : >> Thierry, >> >> Okay ... it is "working" now ... I was also misled by the fact that you >> are continuing to fabricate Monticello version numbers which presumably >> cannot be relied upon in any way. >> >> Tugrik-Help-DaleHenrichs.11 will show up in each branch that includes >> the commit for "Tugrik-Help-DaleHenrichs.10", but the SHA and contents >> would be different for each one of the "Tugrik-Help-DaleHenrichs.11" , >> is that right? > > Yes. > >> Perhaps using the short SHA in place of the "version number" would be >> safer and provide useful information in the version number slot? > > Maybe. But then you'll have a bunch of stuff expecting version numbers > that will stop working. Ah yes, the nasty Monticello version number parsing that is sprinkled around the universe... But at the end of the day, these are tools issues and we shouldn't let tools limitations stand in the way of progress ... With that said I certainly understand the issues ... The design of Metacello was almost completely driven by "tools issues" and of course FileTree was originally written to be completely compatible with the Monticello tools ... But by dropping monticello version data (actually making monticello version data optional) we are breaking the tools ... The fabricated version numbers that you are currently using do the job of not breaking the tools, but they break the semantics of Monticello and fool the developer in believing things that aren't true --- which is a bit dangerous in my opinion ... So at the end of the day, if we are really going to _support_ optional monticello version data, then we need to make a pass through the tools .... I built tODE from the ground up so that I could make the tools directly git aware ... > >> If I support "Metadata" : "false" in GemStone, I do not intend to >> fabricate a "realistic looking Monticello version number" ... but I will >> look into using the short SHA (when in a git repo) and perhaps fall back >> to cypress.1 for non-git repos... >> >> Anyway, I will now be able to move forward with my Metacello Cypress >> experiments and also try to understand how Metacello loads are affected >> by metadtaless, since you _are_ fabricating Monticello version numbers, >> my previous assumptions are not correct ... > > Please tell how it goes, especially the part with the short SHA, > because I haven't tried that; I kept creating version numbers to just > have gitfiletree behaving like filetree (apparently). I understand the motivation ... the basic theory is that when using Baselines and git, the monticello version number is not relevant any more ... as you point out above, the SHA will be problematic with regards to the handful of places in the system that parse package names to extract a version number and solving that problem by itself may not be practical -- it remains to be seen ... The other half of this equation is that new tools should really be built to work with git-based repositories ... a Metacello Project Browser as the "main focus" would relieve the pressure to be completely compatible with the Monticello Browser and open the way for the creation of a Cypress Browser for working with git/filetree repos... Dale

June 29, 2016

Re: [Pharo-users] Issue with NeoCSVReader

by Alexandre Bergel

Ah okay! I understand better now. Thanks Sven! Alexandre > On Jun 29, 2016, at 12:33 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> On 29 Jun 2016, at 16:00, Alexandre Bergel <alexandre.bergel(a)me.com> wrote: >> >> Thanks Sebastian! >> >> Apparently, it is related to UTF encoding. > > To be more correct: the resource (file) is served (by github) with a content-type of 'application/octet-stream', which means it should be interpreted as pure bytes. Maybe that is on purpose (the 'raw' aspect). Still it means a client cannot know what is inside, it could guess. It should be served as 'text/csv' so that clients at least know what it is and can act accordingly (like apply the proper text converter). > >> The following code works well >> >> c := (ZnEasy get: 'https://github.com/sudar/pig-samples/raw/master/data/tweets.csv') contents. >> c := ZnCharacterEncoder utf8 decodeBytes: c. >> content := c readStream. >> lines := (NeoCSVReader on: content) >> skipHeader; >> upToEnd. >> >> Strange >> Alexandre >> >> >> >>> On Jun 28, 2016, at 5:54 PM, Sebastian Heidbrink via Pharo-users <pharo-users(a)lists.pharo.org> wrote: >>> >>> >>> From: Sebastian Heidbrink <sheidev(a)yahoo.de> >>> Subject: Re: [Pharo-users] Issue with NeoCSVReader >>> Date: June 28, 2016 at 5:53:44 PM GMT-4 >>> To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org> >>> >>> >>> Hi Alexandre, >>> >>> I do not have access to a pharo image right now. But I had a look into your csv file. >>> There are rows that do not include any or to few data. I think NeoCSVReader can not handle that and expects a "proper" file >>> >>> I will check tonight, in case nobody else comes up with additional information by then. >>> >>> To be continued >>> Sebastian >>> >>> >>> >>> Am 28.06.2016 um 14:41 schrieb Alexandre Bergel: >>>> Hi! >>>> >>>> I have a simple use of NeoCSVReader, but I get a rollback. >>>> >>>> -=-=-=-=-=-=-=-=-= >>>> content := (ZnEasy get: 'https://github.com/sudar/pig-samples/raw/master/data/tweets.csv') contents readStream. >>>> lines := (NeoCSVReader on: content) >>>> skipHeader; >>>> upToEnd. >>>> -=-=-=-=-=-=-=-=-= >>>> >>>> The url points to a .csv file with a header. No idea why the code . >>>> >>>> I loaded the csv reader using: >>>> Gofer new >>>> smalltalkhubUser: 'SvenVanCaekenberghe' project: 'Neo'; >>>> package: 'ConfigurationOfNeoCSV'; load. >>>> (Smalltalk at: #ConfigurationOfNeoCSV) loadBleedingEdge. >>>> >>>> Alexandre >>> >>> >>> >>> >>> >> >> -- >> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: >> Alexandre Bergel http://www.bergel.eu >> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. > > -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

June 29, 2016