- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [Pharo-users] Download link for latest Pharo6 image on download page

by Alexandre Bergel

Thanks!! This is useful Alexandre > On Jul 14, 2016, at 11:02 AM, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > Hi, > > I have added a link to the download page for Pharo6: > > http://pharo.org/download > > This way it is easier to findâ¦ it is of course in active development, but quite usable > most of the time. > > Marcus -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

July 14, 2016

Re: [Pharo-dev] [Pharo-users] Download link for latest Pharo6 image on download page

by Marcus Denker

I think there is not yetâ¦ There will be more coming in this direction soon. > On 14 Jul 2016, at 11:09, Dimitris Chloupis <kilon.alios(a)gmail.com> wrote: > > Thank you Marcus, where we can find more info about the Pharo Minimal Image ? > > On Thu, Jul 14, 2016 at 12:04 PM Marcus Denker <marcus.denker(a)inria.fr <mailto:marcus.denker@inria.fr>> wrote: > Hi, > > I have added a link to the download page for Pharo6: > > http://pharo.org/download <http://pharo.org/download> > > This way it is easier to findâ¦ it is of course in active development, but quite usable > most of the time. > > Marcus

July 14, 2016

Re: [Pharo-dev] tricks for XML parsing.

by Jan Vrany

On Thu, 2016-07-14 at 11:11 +0200, monty wrote: > > > Sent: Thursday, July 14, 2016 at 2:50 AM > > From: "Jan Vrany" <jan.vrany(a)fit.cvut.cz> > > To: pharo-dev(a)lists.pharo.org > > Subject: Re: [Pharo-dev] tricks for XML parsing. > > > > On Thu, 2016-07-14 at 01:58 +0200, monty wrote: > > > Thanks for the link. > > > > > > In-place parsing is a non-starter because it means storing the > > > entire > > > input as a string in memory, so you could only parse files that > > > fit > > > in Pharo's address space. The multi-gigabyte OpenStreetMap docs > > > the > > > article mentions would be unparsable with SAX in a 32-bit VM. > > > > I do not understand. I only know expat which does - AFAIK - in- > > place > > parsing and surelt does not need the whole input in memory.Â > > From the article, footnote 3: "This creates a lifetime dependencyâthe > entire source buffer must outlive all document nodes for the > technique to work" Ah, I see. I assumed that the consumer would copy data if it needs to retain them...then the parser can do in-place tricks.Â Perhaps the difference is that I do not consider creation of a DOM being part of "parsing" (I'm not saying it's not, just that this is a way I think of the problem which may cause confusion :-)Â > > > > There is always the option of an FFI-based parser, but it > > > shouldn't > > > be a hybrid like Python's minidom (FFI Expat with a Python DOM > > > implementation),Â > > > because something like that already exists in Smalltalk/X (FFI > > > Expat > > > with a Smalltalk DOM)Â > > > > I guess you refer to the implementation I did ages ago.Â > > > > > and it was slower than a St/X port of XMLParser in my tests (I > > > assume > > > due to the FFI overhead), so it's probably not worth it.Â > > > > Very, very interesting. Where can I find the benchmarks?Â > > This was well over a year ago, and it was DOM parsing. I was testing > if St/X (your branch, I think) could be supported by XMLParser in > addition to Pharo, Squeak, and GS, but I ran into too many > incompatibilities, like Monticello not working (had to load in .st > files), #new not sending #initialize, not being able to modify the > value of a dictionary association directly, #lf/#cr weirdness, so I > gave up. But not before hacking it enough to kind-of run and compared > it with the other parsers. Fair enough. If you want to discuss these problems, Â I'd be happy to help. > > > I just run a very simple benchmark on 112MB document (http://www.xm > > l-be > > nchmark.org/downloads.html) and results are quite the opposite:Â > > > > Benchmark resut: > > Generated at :14-07-2016 07:32:25 AM > > > > Â Â Â Â Â Â Â Â Â Â Â BenchmarkÂ Â Â Â Â Â Execution Time [ms]Â Â Â Â Â Â # of M&S GCs > > [1]Â Â Â Â Â Â # of newspace GCs [1]Â Â Â Parameters > > BenchmarkXML > > Â Â Â Â Â Â Â Â Â Â Â Â SAX - > > VWÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 93418Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 0Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > > Â Â Â Â > > Â 2060Â Â Â > > Â Â Â Â Â Â SAX - > > XMLSuiteÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 9921Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 0Â Â Â Â Â Â Â Â Â Â Â Â > > Â Â Â Â > > Â Â Â Â Â Â Â Â 410 Â Â > > > > As you can see, the latter is roughly 10 times faster.Â > > That's the VW parser, which is slower than XMLParser. And again, it > was of DOM parsing. I see. I got confused since VW's parser class is XMLParser... > > > I agree my implementation which uses Expat is clearly suboptimalÂ > > and need to be improved (for example it does not use a ILC-basedÂ > > send to driver so you have a lot of cache misses and does a lotÂ > > of unnecessary memcpy()s, but this can be easily improved) > > Your implementation was fine, and particularly, its XPath/Query was > very impressive. I wasn't attacking you. No worry, I know you were not! Just that I was surprised since my observations - performance wise - were very different. Now I see we were talking about different things.Â > My point was just the hybrid approach built on Expat (which is a non- > validating parser, BTW) should be avoided, in case anyone is > considering it, based on my experience with minidom vs lxml.etree in > Python and with St/X's v2 parser. A parser based on LibXLM2, Xerces, > or something else for SAX, DOM, XPath, etc is probably a better way > of creating an alternative to pure-Smalltalk parsers. I tend to disagree, but I think we have different points of view, that;s why. I have to admit I have not much experience (and need)Â with XML.Â Anyways, I'd like to discuss this more to understand the details. Perhaps privately as it's not strictly Pharo-development related. Best, Jan >Â

July 14, 2016

Re: [Pharo-dev] tricks for XML parsing.

by monty

> Sent: Thursday, July 14, 2016 at 2:50 AM > From: "Jan Vrany" <jan.vrany(a)fit.cvut.cz> > To: pharo-dev(a)lists.pharo.org > Subject: Re: [Pharo-dev] tricks for XML parsing. > > On Thu, 2016-07-14 at 01:58 +0200, monty wrote: > > Thanks for the link. > > > > In-place parsing is a non-starter because it means storing the entire > > input as a string in memory, so you could only parse files that fit > > in Pharo's address space. The multi-gigabyte OpenStreetMap docs the > > article mentions would be unparsable with SAX in a 32-bit VM. > > I do not understand. I only know expat which does - AFAIK - in-place > parsing and surelt does not need the whole input in memory.Â >From the article, footnote 3: "This creates a lifetime dependencyâthe entire source buffer must outlive all document nodes for the technique to work" > > There is always the option of an FFI-based parser, but it shouldn't > > be a hybrid like Python's minidom (FFI Expat with a Python DOM > > implementation), > > because something like that already exists in Smalltalk/X (FFI Expat > > with a Smalltalk DOM) > > I guess you refer to the implementation I did ages ago.Â > > > and it was slower than a St/X port of XMLParser in my tests (I assume > > due to the FFI overhead), so it's probably not worth it. > > Very, very interesting. Where can I find the benchmarks?Â This was well over a year ago, and it was DOM parsing. I was testing if St/X (your branch, I think) could be supported by XMLParser in addition to Pharo, Squeak, and GS, but I ran into too many incompatibilities, like Monticello not working (had to load in .st files), #new not sending #initialize, not being able to modify the value of a dictionary association directly, #lf/#cr weirdness, so I gave up. But not before hacking it enough to kind-of run and compared it with the other parsers. > I just run a very simple benchmark on 112MB document (http://www.xml-be > nchmark.org/downloads.html) and results are quite the opposite:Â > > Benchmark resut: > Generated at :14-07-2016 07:32:25 AM > > Â Â Â Â Â Â Â Â Â Â Â BenchmarkÂ Â Â Â Â Â Execution Time [ms]Â Â Â Â Â Â # of M&S GCs > [1]Â Â Â Â Â Â # of newspace GCs [1]Â Â Â Parameters > BenchmarkXML > Â Â Â Â Â Â Â Â Â Â Â Â SAX - > VWÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 93418Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 0Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > Â 2060Â Â Â > Â Â Â Â Â Â SAX - > XMLSuiteÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 9921Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 0Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â > Â Â Â Â Â Â Â Â 410 Â Â > > As you can see, the latter is roughly 10 times faster.Â That's the VW parser, which is slower than XMLParser. And again, it was of DOM parsing. > I agree my implementation which uses Expat is clearly suboptimalÂ > and need to be improved (for example it does not use a ILC-basedÂ > send to driver so you have a lot of cache misses and does a lotÂ > of unnecessary memcpy()s, but this can be easily improved) Your implementation was fine, and particularly, its XPath/Query was very impressive. I wasn't attacking you. My point was just the hybrid approach built on Expat (which is a non-validating parser, BTW) should be avoided, in case anyone is considering it, based on my experience with minidom vs lxml.etree in Python and with St/X's v2 parser. A parser based on LibXLM2, Xerces, or something else for SAX, DOM, XPath, etc is probably a better way of creating an alternative to pure-Smalltalk parsers. > Jan > Â Â > > > But a non-hybrid parser with everything (including the DOM) done in C > > should definitely be faster. > > > > > Sent: Wednesday, July 13, 2016 at 10:27 AM > > > From: stepharo <stepharo(a)free.fr> > > > To: "Pharo Development List" <pharo-dev(a)lists.pharo.org> > > > Subject: [Pharo-dev] tricks for XML parsing. > > > > > > Hi guys > > > > > > these free books may be interesting for you > > > > > > Â Â Â Â Â http://aosabook.org/ > > > > > > http://aosabook.org/en/posa/parsing-xml-at-the-speed-of-light.html > > > > > > > > > stef > > > > > > > > > > > > >

July 14, 2016

Re: [Pharo-dev] Download link for latest Pharo6 image on download page

by Dimitris Chloupis

Thank you Marcus, where we can find more info about the Pharo Minimal Image ? On Thu, Jul 14, 2016 at 12:04 PM Marcus Denker <marcus.denker(a)inria.fr> wrote: > Hi, > > I have added a link to the download page for Pharo6: > > http://pharo.org/download > > This way it is easier to findâ¦ it is of course in active development, but > quite usable > most of the time. > > Marcus >

July 14, 2016

Download link for latest Pharo6 image on download page

by Marcus Denker

Hi, I have added a link to the download page for Pharo6: http://pharo.org/download This way it is easier to findâ¦ it is of course in active development, but quite usable most of the time. Marcus

July 14, 2016

Re: [Pharo-dev] tricks for XML parsing.

by stepharo

thanks for the analysis :) I just skimmed over the tabl of contents and this caugth my eyes. Le 14/7/16 Ã 01:58, monty a Ã©crit : > Thanks for the link. > > In-place parsing is a non-starter because it means storing the entire input as a string in memory, so you could only parse files that fit in Pharo's address space. The multi-gigabyte OpenStreetMap docs the article mentions would be unparsable with SAX in a 32-bit VM. > > Linked lists for storing child nodes is common (LibXML2 and Xerces do it) and provides constant time insertion and sibling access, but arrays/vectors (Arrays/OrderedCollections) are more cache friendly and faster for sequential access and in Pharo are almost always the correct choice. > > There is always the option of an FFI-based parser, but it shouldn't be a hybrid like Python's minidom (FFI Expat with a Python DOM implementation), because something like that already exists in Smalltalk/X (FFI Expat with a Smalltalk DOM) and it was slower than a St/X port of XMLParser in my tests (I assume due to the FFI overhead), so it's probably not worth it. But a non-hybrid parser with everything (including the DOM) done in C should definitely be faster. > >> Sent: Wednesday, July 13, 2016 at 10:27 AM >> From: stepharo <stepharo(a)free.fr> >> To: "Pharo Development List" <pharo-dev(a)lists.pharo.org> >> Subject: [Pharo-dev] tricks for XML parsing. >> >> Hi guys >> >> these free books may be interesting for you >> >> http://aosabook.org/ >> >> http://aosabook.org/en/posa/parsing-xml-at-the-speed-of-light.html >> >> >> stef >> >> >> >

July 14, 2016

Re: [Pharo-dev] tricks for XML parsing.

by Jan Vrany

On Thu, 2016-07-14 at 01:58 +0200, monty wrote: > Thanks for the link. > > In-place parsing is a non-starter because it means storing the entire > input as a string in memory, so you could only parse files that fit > in Pharo's address space. The multi-gigabyte OpenStreetMap docs the > article mentions would be unparsable with SAX in a 32-bit VM. I do not understand. I only know expat which does - AFAIK - in-place parsing and surelt does not need the whole input in memory.Â > There is always the option of an FFI-based parser, but it shouldn't > be a hybrid like Python's minidom (FFI Expat with a Python DOM > implementation), > because something like that already exists in Smalltalk/X (FFI Expat > with a Smalltalk DOM) I guess you refer to the implementation I did ages ago.Â > and it was slower than a St/X port of XMLParser in my tests (I assume > due to the FFI overhead), so it's probably not worth it. Very, very interesting. Where can I find the benchmarks?Â I just run a very simple benchmark on 112MB document (http://www.xml-be nchmark.org/downloads.html) and results are quite the opposite:Â Benchmark resut: Generated at :14-07-2016 07:32:25 AM Â Â Â Â Â Â Â Â Â Â Â BenchmarkÂ Â Â Â Â Â Execution Time [ms]Â Â Â Â Â Â # of M&S GCs [1]Â Â Â Â Â Â # of newspace GCs [1]Â Â Â Parameters BenchmarkXML Â Â Â Â Â Â Â Â Â Â Â Â SAX - VWÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 93418Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 0Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 2060Â Â Â Â Â Â Â Â Â SAX - XMLSuiteÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 9921Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 0Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 410 Â Â As you can see, the latter is roughly 10 times faster.Â I agree my implementation which uses Expat is clearly suboptimalÂ and need to be improved (for example it does not use a ILC-basedÂ send to driver so you have a lot of cache misses and does a lotÂ of unnecessary memcpy()s, but this can be easily improved) Jan Â Â > But a non-hybrid parser with everything (including the DOM) done in C > should definitely be faster. > > > Sent: Wednesday, July 13, 2016 at 10:27 AM > > From: stepharo <stepharo(a)free.fr> > > To: "Pharo Development List" <pharo-dev(a)lists.pharo.org> > > Subject: [Pharo-dev] tricks for XML parsing. > > > > Hi guys > > > > these free books may be interesting for you > > > > Â Â Â Â Â http://aosabook.org/ > > > > http://aosabook.org/en/posa/parsing-xml-at-the-speed-of-light.html > > > > > > stef > > > > > > >

July 14, 2016

Re: [Pharo-dev] tricks for XML parsing.

by monty

Thanks for the link. In-place parsing is a non-starter because it means storing the entire input as a string in memory, so you could only parse files that fit in Pharo's address space. The multi-gigabyte OpenStreetMap docs the article mentions would be unparsable with SAX in a 32-bit VM. Linked lists for storing child nodes is common (LibXML2 and Xerces do it) and provides constant time insertion and sibling access, but arrays/vectors (Arrays/OrderedCollections) are more cache friendly and faster for sequential access and in Pharo are almost always the correct choice. There is always the option of an FFI-based parser, but it shouldn't be a hybrid like Python's minidom (FFI Expat with a Python DOM implementation), because something like that already exists in Smalltalk/X (FFI Expat with a Smalltalk DOM) and it was slower than a St/X port of XMLParser in my tests (I assume due to the FFI overhead), so it's probably not worth it. But a non-hybrid parser with everything (including the DOM) done in C should definitely be faster. > Sent: Wednesday, July 13, 2016 at 10:27 AM > From: stepharo <stepharo(a)free.fr> > To: "Pharo Development List" <pharo-dev(a)lists.pharo.org> > Subject: [Pharo-dev] tricks for XML parsing. > > Hi guys > > these free books may be interesting for you > > http://aosabook.org/ > > http://aosabook.org/en/posa/parsing-xml-at-the-speed-of-light.html > > > stef > > >

July 13, 2016

Re: [Pharo-dev] tricks for XML parsing.

by Serge Stinckwich

There is also a chapter very similar to the one ObjVLisp here: http://aosabook.org/en/500L/a-simple-object-model.html On Wed, Jul 13, 2016 at 3:27 PM, stepharo <stepharo(a)free.fr> wrote: > Hi guys > > these free books may be interesting for you > > http://aosabook.org/ > > http://aosabook.org/en/posa/parsing-xml-at-the-speed-of-light.html > > > stef > > -- Serge Stinckwich UCBN & UMI UMMISCO 209 (IRD/UPMC) Every DSL ends up being Smalltalk http://www.doesnotunderstand.org/

July 13, 2016