June 2016 - Pharo-users - lists.pharo.org

Re: [Pharo-users] Issue with NeoCSVReader

by Sven Van Caekenberghe

> On 29 Jun 2016, at 16:00, Alexandre Bergel <alexandre.bergel(a)me.com> wrote: > > Thanks Sebastian! > > Apparently, it is related to UTF encoding. To be more correct: the resource (file) is served (by github) with a content-type of 'application/octet-stream', which means it should be interpreted as pure bytes. Maybe that is on purpose (the 'raw' aspect). Still it means a client cannot know what is inside, it could guess. It should be served as 'text/csv' so that clients at least know what it is and can act accordingly (like apply the proper text converter). > The following code works well > > c := (ZnEasy get: 'https://github.com/sudar/pig-samples/raw/master/data/tweets.csv') contents. > c := ZnCharacterEncoder utf8 decodeBytes: c. > content := c readStream. > lines := (NeoCSVReader on: content) > skipHeader; > upToEnd. > > Strange > Alexandre > > > >> On Jun 28, 2016, at 5:54 PM, Sebastian Heidbrink via Pharo-users <pharo-users(a)lists.pharo.org> wrote: >> >> >> From: Sebastian Heidbrink <sheidev(a)yahoo.de> >> Subject: Re: [Pharo-users] Issue with NeoCSVReader >> Date: June 28, 2016 at 5:53:44 PM GMT-4 >> To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org> >> >> >> Hi Alexandre, >> >> I do not have access to a pharo image right now. But I had a look into your csv file. >> There are rows that do not include any or to few data. I think NeoCSVReader can not handle that and expects a "proper" file >> >> I will check tonight, in case nobody else comes up with additional information by then. >> >> To be continued >> Sebastian >> >> >> >> Am 28.06.2016 um 14:41 schrieb Alexandre Bergel: >>> Hi! >>> >>> I have a simple use of NeoCSVReader, but I get a rollback. >>> >>> -=-=-=-=-=-=-=-=-= >>> content := (ZnEasy get: 'https://github.com/sudar/pig-samples/raw/master/data/tweets.csv') contents readStream. >>> lines := (NeoCSVReader on: content) >>> skipHeader; >>> upToEnd. >>> -=-=-=-=-=-=-=-=-= >>> >>> The url points to a .csv file with a header. No idea why the code . >>> >>> I loaded the csv reader using: >>> Gofer new >>> smalltalkhubUser: 'SvenVanCaekenberghe' project: 'Neo'; >>> package: 'ConfigurationOfNeoCSV'; load. >>> (Smalltalk at: #ConfigurationOfNeoCSV) loadBleedingEdge. >>> >>> Alexandre >> >> >> >> >> > > -- > _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: > Alexandre Bergel http://www.bergel.eu > ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

June 29, 2016

Re: [Pharo-users] Issue with NeoCSVReader

by Alexandre Bergel

Thanks Sebastian! Apparently, it is related to UTF encoding. The following code works well c := (ZnEasy get: 'https://github.com/sudar/pig-samples/raw/master/data/tweets.csv') contents. c := ZnCharacterEncoder utf8 decodeBytes: c. content := c readStream. lines := (NeoCSVReader on: content) skipHeader; upToEnd. Strange Alexandre > On Jun 28, 2016, at 5:54 PM, Sebastian Heidbrink via Pharo-users <pharo-users(a)lists.pharo.org> wrote: > > > From: Sebastian Heidbrink <sheidev(a)yahoo.de> > Subject: Re: [Pharo-users] Issue with NeoCSVReader > Date: June 28, 2016 at 5:53:44 PM GMT-4 > To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org> > > > Hi Alexandre, > > I do not have access to a pharo image right now. But I had a look into your csv file. > There are rows that do not include any or to few data. I think NeoCSVReader can not handle that and expects a "proper" file > > I will check tonight, in case nobody else comes up with additional information by then. > > To be continued > Sebastian > > > > Am 28.06.2016 um 14:41 schrieb Alexandre Bergel: >> Hi! >> >> I have a simple use of NeoCSVReader, but I get a rollback. >> >> -=-=-=-=-=-=-=-=-= >> content := (ZnEasy get: 'https://github.com/sudar/pig-samples/raw/master/data/tweets.csv') contents readStream. >> lines := (NeoCSVReader on: content) >> skipHeader; >> upToEnd. >> -=-=-=-=-=-=-=-=-= >> >> The url points to a .csv file with a header. No idea why the code . >> >> I loaded the csv reader using: >> Gofer new >> smalltalkhubUser: 'SvenVanCaekenberghe' project: 'Neo'; >> package: 'ConfigurationOfNeoCSV'; load. >> (Smalltalk at: #ConfigurationOfNeoCSV) loadBleedingEdge. >> >> Alexandre > > > > > -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

June 29, 2016

Re: [Pharo-users] Issue with NeoCSVReader

by Alexandre Bergel

Thanks for all your comments Alexandre > On Jun 29, 2016, at 3:45 AM, Joachim Tuchel <jtuchel(a)objektfabrik.de> wrote: > > Ben, > > > Ben Coman <btc(a)openinworld.com> hat am 29. Juni 2016 um 09:40 geschrieben: > > Your questions are significant. The standard library should not be > > complicated by trying to handle undefined missing fields from a row. > > Such a file is simply invalid. > > cheers -ben > > I fully agree, I just wanted to give an example of questions that are way more difficult to answer than it might seem in the first place. I wanted to make an argument for not blaming NeoCSV for being unable to parse a file like that. I am surely not asking for NeoCSV to do some magic on invalid CSV files. It would be wrong in at least 50% of the cases anyways, no matter what kind of assumptions it would make. > > Joachim > -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Nicolas Passerini

Thank you Thierry! I have more questions inline. On Wed, Jun 29, 2016 at 10:44 AM, Thierry Goubier <thierry.goubier(a)gmail.com > wrote: > All the Monticello GUI : track down version numbers to order stuff in the > lists views everywhere. > So, if we built a new GUI which does not rely on those version numbers maybe we could get rid of that problem, what do you think? > > The Metacello / Gofer stuff (Configurations, Baselines) also use version > numbers. > Yes, I understand that they use version numbers, but do they require that version numbers are correlative? > > One of the difficulty of switching will be the cohabitation of > number-based systems (smalltalkhub) and SHA-based systems, especially when > you do things like copying a version from a git repository to a > smalltalkhub repository. > I think that currently it is not possible to copy a number-based package version from smalltalkhub to a metadata-less git repository using git file tree. I mean, you can but the new version will not have the same version number, right? > That's why I suggested to change the ordering relation to a partial order > based on the property A is an ancestor of B -> B is before A. > I've lost you there. I agree that it is true that A is an ancestor of B -> B is before A, but the most common problem goes the other way arround, i.e. you want to know if A is ancestor of B and the implication then is not true: B is before A 'does not imply' B is ancestor of A. So I missed your point here.

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Thierry Goubier

Le 29/06/2016 12:57, Ben Coman a Ã©crit : > On Wed, Jun 29, 2016 at 4:00 PM, Thierry Goubier > <thierry.goubier(a)gmail.com> wrote: >> Le 29/06/2016 00:55, Dale Henrichs a Ã©crit : >>> >>> >> ... >>>> >>>> >>> I'm pretty certain the MCLazyVersionInfo is the real culprit here ... >>> while reading the code I recognized that many of the basic patterns were >>> exactly as I had remembered them from years ago ... however ... >>> MCLazyVersionInfo this puppy with its "default behavior" to scan the >>> universe is the real culprit ... I would think that at a minimum the >>> repository or repository group would/could be know at the time that the >>> MCLazyVersionInfo was created and a scan of just those repositories --- >>> already associated with the project --- would not be nearly as bad as >>> when we have now ... >> >> >> The MCLazyVersionInfo thing is mine too; it was a solution to avoid keeping >> MBs of version info kept inside the image memory, with the cost of having to >> reload that information when you access the ancestry. > > Would it be feasible to drop all those MB except keep the latest > version of a package in-Image, > i.e. the one required to determine the next version number of the package. ? This is not considered enough for Monticello. If you are on a multi-people team with a smalltalkhub repository, keeping memory of myPackage.7 is not enough because your team members may push .8, .9, etc... before your save, and your save should be .10, not .8. So an access to the remote repository is necessary (and all of them; maybe you're working with multiple smalltalkhub repositories for that package). Of course, MC does not lock the remote repository while you do that, so the number it generates at the beginning (just before you get the window where you fill in the package version comment) is not guaranteed to be up to date if a team member saves a new version in the mean-time. Of course, MC does not forbid you to change by hand the version number. Thierry > cheers -ben > >>> Now, the approach needs to be tuned to avoid spurious "query the world" >> searches, but, as you point out, I hasn't been too successfull yet. And one >> of the thing MC lack, is that link between a repository and a working copy. >> >> (at the same time, restricting version number determination to a subset of >> the repositories is against MC principles when numbering versions). >> >> Thierry >> > >

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Ben Coman

On Wed, Jun 29, 2016 at 4:00 PM, Thierry Goubier <thierry.goubier(a)gmail.com> wrote: > Le 29/06/2016 00:55, Dale Henrichs a Ã©crit : >> >> > ... >>> >>> >> I'm pretty certain the MCLazyVersionInfo is the real culprit here ... >> while reading the code I recognized that many of the basic patterns were >> exactly as I had remembered them from years ago ... however ... >> MCLazyVersionInfo this puppy with its "default behavior" to scan the >> universe is the real culprit ... I would think that at a minimum the >> repository or repository group would/could be know at the time that the >> MCLazyVersionInfo was created and a scan of just those repositories --- >> already associated with the project --- would not be nearly as bad as >> when we have now ... > > > The MCLazyVersionInfo thing is mine too; it was a solution to avoid keeping > MBs of version info kept inside the image memory, with the cost of having to > reload that information when you access the ancestry. Would it be feasible to drop all those MB except keep the latest version of a package in-Image, i.e. the one required to determine the next version number of the package. ? cheers -ben >> Now, the approach needs to be tuned to avoid spurious "query the world" > searches, but, as you point out, I hasn't been too successfull yet. And one > of the thing MC lack, is that link between a repository and a working copy. > > (at the same time, restricting version number determination to a subset of > the repositories is against MC principles when numbering versions). > > Thierry >

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Thierry Goubier

Hi Nicolas, Le 29/06/2016 10:24, Nicolas Passerini a Ã©crit : > Hi Dale, Therry, I am working on a libgit-based git integration for > Pharo and I am facing the same problem about version numbers. So maybe I > can help you if you are working on that. How is the libgit integration going? > I had the same idea as Dale, using short SHAs instead of correlative > numbers, but I didn't have the time yet to got down that path. Ok. > Therry, you said that > > > you'll have a bunch of stuff expecting version numbers that will > stop working. > > Do you already know which stuff will stop working? That would be realy > helpful. > All the Monticello GUI : track down version numbers to order stuff in the lists views everywhere. The Metacello / Gofer stuff (Configurations, Baselines) also use version numbers. One of the difficulty of switching will be the cohabitation of number-based systems (smalltalkhub) and SHA-based systems, especially when you do things like copying a version from a git repository to a smalltalkhub repository. That's why I suggested to change the ordering relation to a partial order based on the property A is an ancestor of B -> B is before A. Thierry

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Nicolas Passerini

Hi Dale, Therry, I am working on a libgit-based git integration for Pharo and I am facing the same problem about version numbers. So maybe I can help you if you are working on that. I had the same idea as Dale, using short SHAs instead of correlative numbers, but I didn't have the time yet to got down that path. Therry, you said that > > you'll have a bunch of stuff expecting version numbers that will stop > working. > > Do you already know which stuff will stop working? That would be realy helpful.

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Thierry Goubier

Dale, Le 29/06/2016 00:55, Dale Henrichs a Ã©crit : > > ... > Actually, Metacello has a registry of loaded projects (baselines and > configurations) and using Metacello it is possible to determine which > pacakges belong to which projects etc. so it is not necessary to > "register packages with a project" ... the project registry in Metacello > is enough ... How do one adds a new package to a project in that registry? One can create new packages in a repository, and add them to the baseline (by editing the baseline method)? Do we need to rethink the Kommitter/Versionner stuff to provide a GUI to manipulate such things? Thierry

June 29, 2016

Re: [Pharo-users] How do I convert a FileTree project to Metadataless GitFileTree project?

by Thierry Goubier

Dale, you're right, the monticello.meta directory is kept, in part because I wasn't sure I could get rid of some of the information in some files in that directory (dependencies, initializers). Since those files don't usually create conflicts with git, it seemed Ok. Thierry Le 29/06/2016 01:11, Dale Henrichs a Ã©crit : > > > On 6/28/16 2:16 PM, Thierry Goubier wrote: >> Dale, >> >> I'm sure it is possible. Wait, wait! If you have in your .filetree >> "Metadata" : "false" then this is fine and it has switched to the >> metadata-less mode. >> >> To see the changes on disk, you need to save a new version of your >> packages, that should be all. >> >> I just tried and that works. >> >> 1- remove the GitFileTree repository from your image >> 2- write the property "Metadata" : "false" in the .filetree on disk >> 3- re-add the GitFileTree repository (local) >> 4- modify then save one of the repository packages >> 5- look on disk: no more monticello.meta/version! >> >> Note that I had no packages in the image linked to that repository at >> 1-, because I'm not sure the simple removal would have really removed >> the repository singleton object. > Well I've found the culprit: MCFileTreeWriter>>addString:at:encodedTo: > ensures that monticello.meta directory exists... > > Of course as I reread your comment "no more monticello.meta/version" I > will have to say that I've only been looking at whether or not the > monticello.meta directory existed or not ... I equate "no monticello > meta data" as no monticello.meta directory and you interpret it as "no > monticallo version file" ... well I can live with that ... I swore that > I saw version files being updated in some of my experiments ... but now > that I know that I should only look at the monticello.meta/version file, > I will try yet again ... Yes > Thanks for you patience and help ... > > Dale > > >

June 29, 2016