- Pharo-users - lists.pharo.org

Re: [Pharo-users] [vwnc] Parsing in Smalltalk

by Steffen MÃ¤rcker

Dear Sean, thanks for bringing this up and for the interesting link [1]. I'll have a look. Meanwhile, I am still on both lists. ;-) > Steffen MÃ¤rcker wrote >> I wrote a PetitParserGenerator that takes the DSL and builds a >> PetitParser. > > I don't know how I could've missed this gem! I hope Steffen is still > subscribed. I googled "PetitParserGenerator", but only found these ML > posts > :/ It was easy to miss, since I never actually posted that code. Here's what I've done: From Xtreams.PEG syntax: - PetitParserGenerater > PetitParser code - PetitParserParser > PetitParser instance I can send you that code. It should be straight forward to adapt it for PP2. However, I moved away from Xtreams own PEG flavor to Bryan Ford's original PEG syntax (close but more common) and programmed for Xtreams, PP and PP2: From Bryan Ford's original PEG syntax: 1. Xtreams Grammer and Actor that build an Xtreams parser 2. PP(2)PEGParserParser that builds a PP(2) parser instance 3. PP(2)PEGParserCompiler that builds PP(2) parser classes All three are available for VW in the Cincom's public repository: - Xtreasm-Parsing (8.2-3,stm): +PEG parser +some fixes to Xtreams parser - PetitParser-PEG (1.11,stm) - PetitParser2-PEG (1.11,stm) And 2, 3 for Pharo on GitHub, e.g. - https://github.com/kursjan/petitparser2/tree/master/PetitParser2-PEG Speaking of, I just noticed that translator form Xtreams.PEG to PEG is missing to complete the picture here. ;-) > IIRC Xtreams can take a BNF and generate a parser. As far as I know, there is no BNF-like parser generator for Xtreams, available. > wouldn't mind a two step BNF -> Xtreams -> PP process. Is it an option for you to convert your BNF to PEG manually? Best regards, Steffen

Dec. 2, 2019

Porting to Pharo via Ring

by Sean P. DeNigris

Pavel gave an intriguing high-level experience report on porting from VW to Pharo via (among other things) the Ring meta model [1]. Are any more specifics available? It would be valuable to the community to have knowledge of this process freely available. How would someone else actually go about this with another project? Would it help porting from one version of Pharo to another? [1] https://www.youtube.com/watch?v=XrOdlwtVfMM&list=PLJ5nSnWzQXi8DPNpy1jCkjE4y… ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Dec. 1, 2019

Re: [Pharo-users] [vwnc] Parsing in Smalltalk

by Sean P. DeNigris

I rediscovered this thread while pondering an (existential?!) problem: Why do we keep having to write, and rewrite, and rewrite, ad infinitum, parsers for well known domains like rfc specs? The parser world in many ways feels like a modern post-"Tower of Babel". I was really excited about the reverse parsing stuff done in Squeak by Ted Kaehler and Alessandro Warth [1], which never seemed to get picked up. Steffen MÃ¤rcker wrote > I wrote a PetitParserGenerator that takes the DSL and builds a > PetitParser. I don't know how I could've missed this gem! I hope Steffen is still subscribed. I googled "PetitParserGenerator", but only found these ML posts :/ IIRC Xtreams can take a BNF and generate a parser. I was thinking about implementing a BNF parser in PetitParser, but would love to avoid that, wouldn't mind a two step BNF -> Xtreams -> PP process. Although there was this SO reply [2] where Lukas said that one can't necessarily blindly feed a BNF to a PEG. It also got me thinking about sharing parsers between PP and PP2, since in many cases it seems that only the internal class names would differ... [1] http://www.vpri.org/pdf/m2008001_parseback.pdf [2] https://stackoverflow.com/a/9443024/424245 ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Dec. 1, 2019

Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub

by PBKResearch

Sean I used Soup a few times, but found it difficult to interpret the output, because the parse did not seem to reflect the hierarchy of the nodes in the original; in particular, sibling nodes were not necessarily at the same level in the Soup. XMLHTMLParser always gets the structure right, in my experience. I think this is essential if you want to use Xpath to process the parse. The worked examples in the scraping booklet show how the parser and Xpath can work together. HTH Peter Kenny -----Original Message----- From: Pharo-users <pharo-users-bounces(a)lists.pharo.org> On Behalf Of Sean P. DeNigris Sent: 30 November 2019 16:43 To: pharo-users(a)lists.pharo.org Subject: Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub cedreek wrote > To me, far better than using Soup. Ah, interesting! I use Soup almost exclusively. What did you find superior about XMLParserHTML? I may give it a try... cedreek wrote > Google chrome pharo integration helps top to scrap complex full JS web > site like google ;) Also interesting! Any publicly available examples? How does one load "Google chrome pharo integration"? Also, there is often the "poor man's" way (albeit requiring manual intervention) by inspecting the Ajax http requests in a developer console and then recreating directly in Pharo. ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Nov. 30, 2019

Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub

by CÃ©drick BÃ©ler

I couldnât get it from Zn as (I think) there are some js lib that defer the full rendering. I have the same problem with a site in France (leboncoin). They use https://datadome.co <https://datadome.co/> to complicate webscrapping. So an headless browser is the only solution I know. Cheers, CÃ©drick > Le 1 dÃ©c. 2019 Ã 00:23, Esteban Maringolo <emaringolo(a)gmail.com> a Ã©crit : > > Why use Chrome instead of ZnClient? To get a "real" render of the > content? (including JS and whatnot). > > Regards! > > > Esteban A. Maringolo > > On Sat, Nov 30, 2019 at 8:11 PM CÃ©drick BÃ©ler <cdrick65(a)gmail.com> wrote: >> >> >>> >>> Also interesting! Any publicly available examples? How does one load "Google >>> chrome pharo integration Â»? >> >> "https://github.com/astares/Pharo-Chrome" >> "https://github.com/akgrant43/Pharo-Chrome Â» >> >> Cheers, >> >> CÃ©drick >> >> >

Nov. 30, 2019

Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub

by Esteban Maringolo

Why use Chrome instead of ZnClient? To get a "real" render of the content? (including JS and whatnot). Regards! Esteban A. Maringolo On Sat, Nov 30, 2019 at 8:11 PM CÃ©drick BÃ©ler <cdrick65(a)gmail.com> wrote: > > > > > > Also interesting! Any publicly available examples? How does one load "Google > > chrome pharo integration Â»? > > "https://github.com/astares/Pharo-Chrome" > "https://github.com/akgrant43/Pharo-Chrome Â» > > Cheers, > > CÃ©drick > >

Nov. 30, 2019

Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub

by CÃ©drick BÃ©ler

> > Also interesting! Any publicly available examples? How does one load "Google > chrome pharo integration Â»? "https://github.com/astares/Pharo-Chrome" "https://github.com/akgrant43/Pharo-Chrome Â» Cheers, CÃ©drick

Nov. 30, 2019

Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub

by CÃ©drick BÃ©ler

Just a quick try to scrap a query from google search (using Chrome plugin + XMLHTMLParser + Path) : Really coolâ¦ and in few lines you get your own Â« google Â» powered search engine ;-). Nevertheless this is a bit slow (5 seconds for one page, around 20s for the 5 from the script (I think this might be doable to speed things a bit). HTH, CÃ©drick PS: code below in text searches := OrderedCollection new. query := 'black friday' urlEncoded. "chrome plugin" browser := GoogleChrome new. browser headless: true. browser open. page := browser firstTab. 0 to: 50 by: 10 do: [:paging | page get: 'https://www.google.com/search?q=',query,'&start=', paging asString. result := page html. dom := result parseHTML. "uses XMLHTMLParser" searches addAll: (dom xpath: '//div[@class="g"]/div/div/div/a/@href'). ]. searches size "64". searches first. "href=""https://www.fnac.com/black-friday""" "xpath cheat sheet: https://gist.github.com/LeCoupa/8c305ec8c713aad07b14"

Nov. 30, 2019

Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub

by CÃ©drick BÃ©ler

> ï»¿cedreek wrote >> To me, far better than using Soup. > > Ah, interesting! I use Soup almost exclusively. What did you find superior > about XMLParserHTML? I may give it a try... > Itâs mainly xpath which I find easier than navigating the html tree with soup or even The xmlHtmlparser. I usually copy the xpath form a web inspector. I have to tweak it a bit though. > > cedreek wrote >> Google chrome pharo integration helps top to scrap complex full JS web >> site like google ;) > > Also interesting! Any publicly available examples? How does one load "Google > chrome pharo integration"? Also, there is often the "poor man's" way (albeit > requiring manual intervention) by inspecting the Ajax http requests in a > developer console and then recreating directly in Pharo. > I just tried it once. There is a google chrome plugin that allows to use chrome headless to get the fully loaded html page. I need to try it again. A simple example Iâd like to do is to scrap google and remove advertised content ^^ This is btw Torsten package: https://github.com/astares/Pharo-Chrome Happy scrapping ;-) And thx Torsten for all ^^ Cedrick > > > ----- > Cheers, > Sean > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >

Nov. 30, 2019

Re: [Pharo-users] [ANN] XMLParserHTML moved to GitHub

by Sean P. DeNigris

cedreek wrote > To me, far better than using Soup. Ah, interesting! I use Soup almost exclusively. What did you find superior about XMLParserHTML? I may give it a try... cedreek wrote > Google chrome pharo integration helps top to scrap complex full JS web > site like google ;) Also interesting! Any publicly available examples? How does one load "Google chrome pharo integration"? Also, there is often the "poor man's" way (albeit requiring manual intervention) by inspecting the Ajax http requests in a developer console and then recreating directly in Pharo. ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Nov. 30, 2019