- Pharo-users - lists.pharo.org

Re: [Pharo-users] NeoCSV change proposal

by Sven Van Caekenberghe

Peter, > On 22 Jul 2017, at 22:27, Peter Uhnak <i.uhnak(a)gmail.com> wrote: > > Hi Sven, > > my use case was hand-edited CSVs (therefore the quotes are extra clutter), which imples that I would be hand-viewing/editing only small CSVs (no performance issues). > > I agree that we should err on the safe side with CR & LF (e.g. tools may sometimes autoconvert line endings). > > Regarding performance: > > #findFirstInString:inSet:startingAt: didn't work for me (not sure if bug, or I don't know how to use), so I've trried with inCharacterSet: Yes, it is a bitch to use, the the version you used does not use the primitive. Try playing with this: s10 := 'a' repeat: 10. s100 := 'a' repeat: 100. searchSet := ByteArray new: 256 withAll: 0. ',"', String crlf do: [ :each | searchSet at: each asInteger + 1 put: 1 ]. searchSet := (CharacterSet newFrom: ',"', String crlf) byteArrayMap. [ ByteString findFirstInString: s10 inSet: searchSet startingAt: 1 ] bench. "15,160,161 per second" [ ByteString findFirstInString: s100 inSet: searchSet startingAt: 1 ] bench. "8,219,081 per second" ByteString findFirstInString: ',"', String crlf inSet: searchSet startingAt: 1. Sven > Tested on worst-case scenario - strings don't contain tested symbols. > > s10 := 'a' repeat: 10. > s100 := 'a' repeat: 100. > > [ {'"'. ','. String cr. String lf } anySatisfy: [ :each | s10 includesSubstring: each ] ] bench. "'1,200,046 per second'" > [ {'"'. ','. String cr. String lf } anySatisfy: [ :each | s100 includesSubstring: each ] ] bench. "'495,482 per second'" > > [ s10 includesAny: { $,. $". Character cr. Character lf } ] bench. "'2,819,416 per second'" > [ s100 includesAny: { $,. $". Character cr. Character lf } ] bench. "'2,200,668 per second'" > > [ ByteString findFirstInString: s10 inCharacterSet: ',"', String crlf startingAt: 1 ] bench. "'1,187,324 per second'" > [ ByteString findFirstInString: s100 inCharacterSet: ',"', String crlf startingAt: 1 ] bench. "'165,526 per second'" > > > #includesAny: seems to be the best by far. > > Storing the tested characters didn't improve it by much. > > Peter > > On Sat, Jul 22, 2017 at 06:51:31PM +0200, Sven Van Caekenberghe wrote: >> Hi Peter, >> >>> On 22 Jul 2017, at 14:12, Peter Uhnak <i.uhnak(a)gmail.com> wrote: >>> >>> Hi, >>> >>> this is a continuation of an older thread about quoting fields only when necessary. ( http://forum.world.st/NeoCSVWriter-automatic-quotes-td4924781.html ) >>> >>> I've attached fileouts of NeoCSV packages with the addition (I don't know if Metacello can file-out only changesets). >>> >>> The change doesn't affect the default behavior, only when explicitly requested. >>> >>> Peter >> >> I accepted your changes as such, the .mcz's were copied over to the main repositories. This is a pure & clean extension, so that is good. Thank you. >> >> This option is always going to be slower, but the current implementation might be improved, I think. >> >> The key test in #writeOptionalQuotedField: >> >> { >> lineEnd asString. >> separator asString. >> '"' } anySatisfy: [ :each | string includesSubstring: each ] >> >> will be quite slow. >> >> If we accept a little bit of (over safe) error on EOL and use any occurrence of CR or LF as needing a quote, we could switch to characters to test for. There exists a fast (primitive) test, #findFirstInString:inSet:startingAt: that can do all the testing in one go. If your version turns out to be slow, we could try that, if measurements show a difference. >> >> Regards, >> >> Sven >> >> >

July 22, 2017

Re: [Pharo-users] NeoCSV change proposal

by Peter Uhnak

Hi Sven, my use case was hand-edited CSVs (therefore the quotes are extra clutter), which imples that I would be hand-viewing/editing only small CSVs (no performance issues). I agree that we should err on the safe side with CR & LF (e.g. tools may sometimes autoconvert line endings). Regarding performance: #findFirstInString:inSet:startingAt: didn't work for me (not sure if bug, or I don't know how to use), so I've trried with inCharacterSet: Tested on worst-case scenario - strings don't contain tested symbols. s10 := 'a' repeat: 10. s100 := 'a' repeat: 100. [ {'"'. ','. String cr. String lf } anySatisfy: [ :each | s10 includesSubstring: each ] ] bench. "'1,200,046 per second'" [ {'"'. ','. String cr. String lf } anySatisfy: [ :each | s100 includesSubstring: each ] ] bench. "'495,482 per second'" [ s10 includesAny: { $,. $". Character cr. Character lf } ] bench. "'2,819,416 per second'" [ s100 includesAny: { $,. $". Character cr. Character lf } ] bench. "'2,200,668 per second'" [ ByteString findFirstInString: s10 inCharacterSet: ',"', String crlf startingAt: 1 ] bench. "'1,187,324 per second'" [ ByteString findFirstInString: s100 inCharacterSet: ',"', String crlf startingAt: 1 ] bench. "'165,526 per second'" #includesAny: seems to be the best by far. Storing the tested characters didn't improve it by much. Peter On Sat, Jul 22, 2017 at 06:51:31PM +0200, Sven Van Caekenberghe wrote: > Hi Peter, > > > On 22 Jul 2017, at 14:12, Peter Uhnak <i.uhnak(a)gmail.com> wrote: > > > > Hi, > > > > this is a continuation of an older thread about quoting fields only when necessary. ( http://forum.world.st/NeoCSVWriter-automatic-quotes-td4924781.html ) > > > > I've attached fileouts of NeoCSV packages with the addition (I don't know if Metacello can file-out only changesets). > > > > The change doesn't affect the default behavior, only when explicitly requested. > > > > Peter > > I accepted your changes as such, the .mcz's were copied over to the main repositories. This is a pure & clean extension, so that is good. Thank you. > > This option is always going to be slower, but the current implementation might be improved, I think. > > The key test in #writeOptionalQuotedField: > > { > lineEnd asString. > separator asString. > '"' } anySatisfy: [ :each | string includesSubstring: each ] > > will be quite slow. > > If we accept a little bit of (over safe) error on EOL and use any occurrence of CR or LF as needing a quote, we could switch to characters to test for. There exists a fast (primitive) test, #findFirstInString:inSet:startingAt: that can do all the testing in one go. If your version turns out to be slow, we could try that, if measurements show a difference. > > Regards, > > Sven > >

July 22, 2017

Re: [Pharo-users] "Leak"/reference hunting with RefsHunter

by Stephane Ducasse

Hi holger I do not know who is maintinaing the config of ReferenceFinder but yes on the principle. Stef On Sat, Jul 22, 2017 at 9:58 PM, Holger Freyther <holger(a)freyther.de> wrote: > >> On 22. Jul 2017, at 20:40, Pavel Krivanek <pavel.krivanek(a)gmail.com> wrote: >> >> In the most of standard cases you can use ReferenceFinder. It is in the catalog too and I think that we should integrate it into the Pharo because it is extremely useful for the memory leaks detection. It does not require snapshots and I firstly try it before applying of the RefsHunter (and I say that as the author of the RefsHunter ;-)) > > Cool and thank you for RefsHunter! I found RefsHunter by typing "leak". Could we get this attached to the ReferenceFinder in the catalog as well? > > holger

July 22, 2017

Re: [Pharo-users] NeoCSV change proposal

by Sven Van Caekenberghe

> On 22 Jul 2017, at 21:51, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: > > Hi peter > Do you have an example that I understand? In its most simple form, CSV looks like this: a,b,c,d However, if a field is text, it might contain a comma itself, or other problematic characters like line endings, hence there is a provision for quoting fields, as follows: "a","b","c",d Now, "a" could be "a1,a2" and still be one field. Embedded quotes are doubled. Not all fields need quoting, normally only text or string fields. If you know something is an integer, you don't need quotes, like d above. Since CSV is normally a collection of identical records, you set these options for all of them. What Peter wanted, and now implemented himself as an option, is to make the quoting optional per field, per record, by first checking if a field contains dangerous characters and only then adding the quotes. I guess this makes most sense in the context of generating a CSV file for humans to edit. > @Svn Once the change is integrated could you add a little paragraph to > the chapter? > > Stef > > On Sat, Jul 22, 2017 at 6:51 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> Hi Peter, >> >>> On 22 Jul 2017, at 14:12, Peter Uhnak <i.uhnak(a)gmail.com> wrote: >>> >>> Hi, >>> >>> this is a continuation of an older thread about quoting fields only when necessary. ( http://forum.world.st/NeoCSVWriter-automatic-quotes-td4924781.html ) >>> >>> I've attached fileouts of NeoCSV packages with the addition (I don't know if Metacello can file-out only changesets). >>> >>> The change doesn't affect the default behavior, only when explicitly requested. >>> >>> Peter >> >> I accepted your changes as such, the .mcz's were copied over to the main repositories. This is a pure & clean extension, so that is good. Thank you. >> >> This option is always going to be slower, but the current implementation might be improved, I think. >> >> The key test in #writeOptionalQuotedField: >> >> { >> lineEnd asString. >> separator asString. >> '"' } anySatisfy: [ :each | string includesSubstring: each ] >> >> will be quite slow. >> >> If we accept a little bit of (over safe) error on EOL and use any occurrence of CR or LF as needing a quote, we could switch to characters to test for. There exists a fast (primitive) test, #findFirstInString:inSet:startingAt: that can do all the testing in one go. If your version turns out to be slow, we could try that, if measurements show a difference. >> >> Regards, >> >> Sven >> >> >

July 22, 2017

Re: [Pharo-users] Pharo MOOC: Difference between class methods and instance methods

by Stephane Ducasse

Thanks I love so much test first and creating everything automatically from there. We should continue to build on that. I'm working on - a new version of the mooc and may be I will change this. - a new book on Object-Oriented Programming with TDD if you want to read it I will restart to work on it as soon as I finish the lecture slides. Stef On Sat, Jul 22, 2017 at 4:37 PM, Michel Marinier <mobimich(a)gmail.com> wrote: > You're right. I forgot between the lesson and the exercise. My bad. > > -----Original Message----- > From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On Behalf Of Stephane Ducasse > Sent: July 22, 2017 9:56 > To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org> > Subject: Re: [Pharo-users] Pharo MOOC: Difference between class methods and instance methods > > Hello michel > > I checked and we introduced them (without explaining them) in W1S6 > > On Sat, Jul 22, 2017 at 3:50 PM, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: >> Ok I see. We were struggling with order and may be we should remove >> the class methods in the exercise. >> >> On Sat, Jul 22, 2017 at 3:48 PM, Michel Marinier <mobimich(a)gmail.com> wrote: >>> Not yet. I'll start on week 3 today. >>> >>> The lessons you mentioned are in week 3 and week 5 I believe (the W3 and W5 in the PDF file names). However, the exercises I am referring to are in week 1 and week 2, possibly before the concept has been explained? >>> >>> (for example: >>> http://rmod-pharo-mooc.lille.inria.fr/MOOC/Exercises/Exo-W2-TinyBlog- >>> ModelExtensionTests-FR.pdf ) >>> >>> >>> -----Original Message----- >>> From: Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] On >>> Behalf Of Stephane Ducasse >>> Sent: July 22, 2017 7:30 >>> To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org> >>> Subject: Re: [Pharo-users] Pharo MOOC: Difference between class >>> methods and instance methods >>> >>> Hello Michel >>> >>> did you have a look at the mooc videos? >>> >>> >>> C019-W3S06-BasicClassMethods.pdf >>> and >>> W5S02-UnderstandingClassMethods.pdf >>> >>> In Pharo there is only one method lookup. >>> a method is looked up in the class of the receiver and if not found up in superclass hierarchy. >>> >>> This exactly the same for class methods: >>> >>> >>> >>> >>> On Fri, Jul 21, 2017 at 10:38 PM, Michel Marinier <mobimich(a)gmail.com> wrote: >>>> I am an absolute beginner at Pharo and Iâm working my way through >>>> Pharo by Example as well as the MOOC online. While coding the >>>> TinyBlog example in the MOOC, I got stuck until I kind of âguessedâ >>>> the difference between âTBPost >>>>>> methodNameâ and âTBPost class >> methodNameâ. I didnât know I had >>>>>> to >>>> click on the Class button to access the class methods and coded all >>>> the methods as instance methods. As a result, the system complained >>>> when the code tried create a new instance from a method such as âtitle: text:â. >>>> >>>> >>>> >>>> I suppose I might have missed the place where this was explained in >>>> the MOOC, although Iâm not sure it was emphasized early on in the lessons. >>>> However, a suggestion would be that if you ever update the exercises >>>> that a notice be added warning students of the difference, at least >>>> in the first few exercises where there is so much to learn and remember. >>>> Iâm doing it in French, so it might look like this: >>>> >>>> >>>> >>>> â >>>> >>>> Nâoubliez pas la diffÃ©rence entre âTBPost >>â et âTBPost class >>â. >>>> Le premier indique quâil sâagit dâune mÃ©thode dâinstance. Le second >>>> indique quâil sâagit dâune mÃ©thode de classe. Vous devez cliquer le bouton âClassâ >>>> pour accÃ©der Ã la section correspondante. Dans la section pour >>>> lâinstance, la dÃ©finition dÃ©bute par âObject subclass: #TBPostâ >>>> alors que dans la section pour la classe, la dÃ©finition dÃ©bute par âTBPost classâ. >>>> >>>> â >>>> >>>> >>>> >>>> By the way, congrats on the great work providing those resources. I >>>> love the advice to poke around the system with the Browser and the >>>> Finder to find out stuff. Thanks. >>> >>> >>> > > >

July 22, 2017

Re: [Pharo-users] "Leak"/reference hunting with RefsHunter

by Holger Freyther

> On 22. Jul 2017, at 20:40, Pavel Krivanek <pavel.krivanek(a)gmail.com> wrote: > > In the most of standard cases you can use ReferenceFinder. It is in the catalog too and I think that we should integrate it into the Pharo because it is extremely useful for the memory leaks detection. It does not require snapshots and I firstly try it before applying of the RefsHunter (and I say that as the author of the RefsHunter ;-)) Cool and thank you for RefsHunter! I found RefsHunter by typing "leak". Could we get this attached to the ReferenceFinder in the catalog as well? holger

July 22, 2017

Re: [Pharo-users] NeoCSV change proposal

by Stephane Ducasse

Hi peter Do you have an example that I understand? @Svn Once the change is integrated could you add a little paragraph to the chapter? Stef On Sat, Jul 22, 2017 at 6:51 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > Hi Peter, > >> On 22 Jul 2017, at 14:12, Peter Uhnak <i.uhnak(a)gmail.com> wrote: >> >> Hi, >> >> this is a continuation of an older thread about quoting fields only when necessary. ( http://forum.world.st/NeoCSVWriter-automatic-quotes-td4924781.html ) >> >> I've attached fileouts of NeoCSV packages with the addition (I don't know if Metacello can file-out only changesets). >> >> The change doesn't affect the default behavior, only when explicitly requested. >> >> Peter > > I accepted your changes as such, the .mcz's were copied over to the main repositories. This is a pure & clean extension, so that is good. Thank you. > > This option is always going to be slower, but the current implementation might be improved, I think. > > The key test in #writeOptionalQuotedField: > > { > lineEnd asString. > separator asString. > '"' } anySatisfy: [ :each | string includesSubstring: each ] > > will be quite slow. > > If we accept a little bit of (over safe) error on EOL and use any occurrence of CR or LF as needing a quote, we could switch to characters to test for. There exists a fast (primitive) test, #findFirstInString:inSet:startingAt: that can do all the testing in one go. If your version turns out to be slow, we could try that, if measurements show a difference. > > Regards, > > Sven > >

July 22, 2017

Re: [Pharo-users] "Leak"/reference hunting with RefsHunter

by Pavel Krivanek

In the most of standard cases you can use ReferenceFinder. It is in the catalog too and I think that we should integrate it into the Pharo because it is extremely useful for the memory leaks detection. It does not require snapshots and I firstly try it before applying of the RefsHunter (and I say that as the author of the RefsHunter ;-)) -- Pavel 2017-07-22 17:04 GMT+02:00 Holger Freyther <holger(a)freyther.de>: > Hi, > > as I didn't remember the name of the tool and maybe as future reference to > myself. I have installed RefsHunter from the catalogue and it helped me a > lot. > > I was looking at the memory consumption of my Pharo6.0 image and did > something crazy like counting how many objects exist of a specific class > (Object allSubInstances copy do.. and put that into a dictionary). I > noticed that for the ASN1 model the nodes of the parse tree survived. > Yesterday I tried to use the pointer explorer but then most references are > held by the UI code and I gave up after a bit. > > Today I found the RefsHunter again and something as simple (found through > the class comments) > > | rh | > rh := RefsHunter snapshot. > rh wayFrom: ASN1AssignmentNode allInstances first to: myAsn1Model > > > Brings me to a list of references and when searching the list from the end > to the beginning gives a pretty good picture of where things go wrong. > > > Great to have a platform that allows to walk the heap and great that there > are tools that make it manageable! > > > have a nice weekend > > holger >

July 22, 2017

Pharo based system monitoring (Nagios like)?

by Alistair Grant

Hi Everyone, Does anyone know of a system monitoring library written in Pharo (or at least smalltalk)? I'm looking for something that provides functionality similar to nagios and will allow me to monitor the basics, e.g. free disk space, cpu usage, process presence, etc. I found Norbert Hartl's Monitoring library, but that looks like it acts as an agent for nagios or similar. Thanks, Alistair

July 22, 2017

Re: [Pharo-users] NeoCSV change proposal

by Sven Van Caekenberghe

Hi Peter, > On 22 Jul 2017, at 14:12, Peter Uhnak <i.uhnak(a)gmail.com> wrote: > > Hi, > > this is a continuation of an older thread about quoting fields only when necessary. ( http://forum.world.st/NeoCSVWriter-automatic-quotes-td4924781.html ) > > I've attached fileouts of NeoCSV packages with the addition (I don't know if Metacello can file-out only changesets). > > The change doesn't affect the default behavior, only when explicitly requested. > > Peter I accepted your changes as such, the .mcz's were copied over to the main repositories. This is a pure & clean extension, so that is good. Thank you. This option is always going to be slower, but the current implementation might be improved, I think. The key test in #writeOptionalQuotedField: { lineEnd asString. separator asString. '"' } anySatisfy: [ :each | string includesSubstring: each ] will be quite slow. If we accept a little bit of (over safe) error on EOL and use any occurrence of CR or LF as needing a quote, we could switch to characters to test for. There exists a fast (primitive) test, #findFirstInString:inSet:startingAt: that can do all the testing in one go. If your version turns out to be slow, we could try that, if measurements show a difference. Regards, Sven

July 22, 2017