Hi, I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any. Cheers, Manuel
Hi, Yes, Pharo regex implementation is very naive. We will be moving to a PCRE binding to match outside world standards but we have not had the time to work on it :( Esteban
On 5 Feb 2019, at 00:27, Manuel Leuenberger <leuenberger@inf.unibe.ch> wrote:
Hi,
I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any.
Cheers, Manuel
The Regex engine inside SmaCC allows for non-greedy REs. But it's integrated as a parser first stage, not as an independent RE engine. Regards, Thierry Le mar. 5 févr. 2019 à 08:34, Esteban Lorenzano <estebanlm@gmail.com> a écrit :
Hi,
Yes, Pharo regex implementation is very naive. We will be moving to a PCRE binding to match outside world standards but we have not had the time to work on it :(
Esteban
On 5 Feb 2019, at 00:27, Manuel Leuenberger <leuenberger@inf.unibe.ch> wrote:
Hi,
I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any.
Cheers, Manuel
Please DON'T move to PCRE. "Outside world standards"? There are so many. There are two important things to know about PCRE: (1) it is a popular open source regexp library for Perl-style regexps, (2) because of that, it is prone to truly horrendous performance problems. There are alternatives, such as re2, https://github.com/google/re2 , which are not subject to PCRE's intrinsic performance pathologies. As it happens, re2 supports *? +? and ??. On Tue, 5 Feb 2019 at 20:34, Esteban Lorenzano <estebanlm@gmail.com> wrote:
Hi,
Yes, Pharo regex implementation is very naive. We will be moving to a PCRE binding to match outside world standards but we have not had the time to work on it :(
Esteban
On 5 Feb 2019, at 00:27, Manuel Leuenberger <leuenberger@inf.unibe.ch> wrote:
Hi,
I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any.
Cheers, Manuel
I am not advocating for PCRE in particular, I just need a regex engine that is just as powerful. I guess re2 serves that purpose, although I haven't used it myself (knowingly). Looking at https://github.com/google/re2/wiki/WhyRE2 <https://github.com/google/re2/wiki/WhyRE2>, re2 actually seems to be a good target. "match time is linear in the length of the input string", sounds like a really nice property.
On 5 Feb 2019, at 12:26, Richard O'Keefe <raoknz@gmail.com> wrote:
Please DON'T move to PCRE. "Outside world standards"? There are so many. There are two important things to know about PCRE: (1) it is a popular open source regexp library for Perl-style regexps, (2) because of that, it is prone to truly horrendous performance problems. There are alternatives, such as re2, https://github.com/google/re2 <https://github.com/google/re2> , which are not subject to PCRE's intrinsic performance pathologies. As it happens, re2 supports *? +? and ??.
On Tue, 5 Feb 2019 at 20:34, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com>> wrote: Hi,
Yes, Pharo regex implementation is very naive. We will be moving to a PCRE binding to match outside world standards but we have not had the time to work on it :(
Esteban
On 5 Feb 2019, at 00:27, Manuel Leuenberger <leuenberger@inf.unibe.ch <mailto:leuenberger@inf.unibe.ch>> wrote:
Hi,
I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any.
Cheers, Manuel
On Wed, Feb 06, 2019 at 12:26:00AM +1300, Richard O'Keefe wrote:
Please DON'T move to PCRE. "Outside world standards"? There are so many. There are two important things to know about PCRE: (1) it is a popular open source regexp library for Perl-style regexps, (2) because of that, it is prone to truly horrendous performance problems. There are alternatives, such as re2, https://github.com/google/re2 , which are not subject to PCRE's intrinsic performance pathologies. As it happens, re2 supports *? +? and ??.
Can you share some examples of PCRE's bad performance? Pierce
PCRE has exponential worst-case time. See for example https://www.regular-expressions.info/catastrophic.html but searching for PCRE exponential time or worst case will find more. That's not the problem. The problem is that it isn't *obvious* which regexps are safe and that people are taught that regular expressions can be matched in linear time, which is sort of the point of them. But PCRE patterns *aren't* regular expressions. On Wed, 6 Feb 2019 at 04:54, Pierce Ng <pierce@samadhiweb.com> wrote:
On Wed, Feb 06, 2019 at 12:26:00AM +1300, Richard O'Keefe wrote:
Please DON'T move to PCRE. "Outside world standards"? There are so many. There are two important things to know about PCRE: (1) it is a popular open source regexp library for Perl-style regexps, (2) because of that, it is prone to truly horrendous performance problems. There are alternatives, such as re2, https://github.com/google/re2 , which are not subject to PCRE's intrinsic performance pathologies. As it happens, re2 supports *? +? and ??.
Can you share some examples of PCRE's bad performance?
Pierce
We can also update pharo version from original VW repositoriy if the current license is appropriate. I think it covers missing parts. 5 ÑевÑ. 2019 г. 7:34 полÑзоваÑÐµÐ»Ñ "Esteban Lorenzano" <estebanlm@gmail.com> напиÑал: Hi, Yes, Pharo regex implementation is very naive. We will be moving to a PCRE binding to match outside world standards but we have not had the time to work on it :( Esteban
On 5 Feb 2019, at 00:27, Manuel Leuenberger <leuenberger@inf.unibe.ch> wrote:
Hi,
I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any.
Cheers, Manuel
Still, there are advantages to an in-image solution, can't says this enough, these external lib dependencies pose their own problems ...
On 5 Feb 2019, at 08:33, Esteban Lorenzano <estebanlm@gmail.com> wrote:
Hi,
Yes, Pharo regex implementation is very naive. We will be moving to a PCRE binding to match outside world standards but we have not had the time to work on it :(
Esteban
On 5 Feb 2019, at 00:27, Manuel Leuenberger <leuenberger@inf.unibe.ch> wrote:
Hi,
I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any.
Cheers, Manuel
Am 05.02.2019 um 16:16 schrieb Sven Van Caekenberghe <sven@stfx.eu>:
Still, there are advantages to an in-image solution, can't says this enough, these external lib dependencies pose their own problems ...
+1
On 5 Feb 2019, at 08:33, Esteban Lorenzano <estebanlm@gmail.com> wrote:
Hi,
Yes, Pharo regex implementation is very naive. We will be moving to a PCRE binding to match outside world standards but we have not had the time to work on it :(
Esteban
On 5 Feb 2019, at 00:27, Manuel Leuenberger <leuenberger@inf.unibe.ch> wrote:
Hi,
I just noticed that the Pharo regexes do not understand non-greedy matches. A regex engine to be PCRE is kind of essential, not having '.*?' to be a parseable and working regex is a bummer. Are there any more powerful regex engines around for Pharo? I could not find any.
Cheers, Manuel
On Tue, Feb 05, 2019 at 07:25:07PM +0100, Norbert Hartl wrote:
Am 05.02.2019 um 16:16 schrieb Sven Van Caekenberghe <sven@stfx.eu>: Still, there are advantages to an in-image solution, can't says this enough, these external lib dependencies pose their own problems ... +1
Libraries like libgit2, libssh2 and quite a few more are already a core part of Pharo, so I'd say philosophically might as well go all in to make FFI to external libraries an intrinsic part of computing with Pharo, not just for developing Pharo itself. The more people use UFFI, the better it will become.
If cre2 meets the standard and community agree, I do not care to be honest, I just mentioned PCRE because it is the âde facto standardâ and we are tired of not being compatible :) About an in-image solutions in general: Yes, it is better than rely in a solution like that. Also about in-image solutions in general: It is harder to maintain them. We are a small community and we cannot allow us to have everything we need implemented in image. Using an FFI solution is a perfect valid way and IMO is preferred many times because of this maintainability issue. Of course, it does not applies to all cases with same intensity :) I donât know the cost of maintain a regex lib in-image. I know at the moment is infinite because there is no-one there doing it (and I know I do now have the time). So we stay for years with a suboptimal solution :( Anyway I want to point to some things I (In my âarchitectâ role, or whatever is what I do here) I always think: 1- This solution is state-of-the-art? 2- is it maintained or the cost to maintain it can be absorbed ? 2.1- Is maintained by whom? 2.2- Will this kick us back if maintainers leave? 3- is it well tested? 4- is it well documented? People often forgets that making libraries is a commitment with your user community, is a lot more than âI do this and then I forgetâ. Of course you can proceed like that (and not few of my own projects are of use-and-throw), but we as âpharo makersâ need to take this into account with priority to other variables. Notice that âperformanceâ is not in this list. I care a lot about performance, but I care a lot more about the other points. Now, that does not means we do not make decisions we later regret (we are humans, after all⦠and this is a learning process). Cheers, Esteban
On 6 Feb 2019, at 02:38, Pierce Ng <pierce@samadhiweb.com> wrote:
On Tue, Feb 05, 2019 at 07:25:07PM +0100, Norbert Hartl wrote:
Am 05.02.2019 um 16:16 schrieb Sven Van Caekenberghe <sven@stfx.eu>: Still, there are advantages to an in-image solution, can't says this enough, these external lib dependencies pose their own problems ... +1
Libraries like libgit2, libssh2 and quite a few more are already a core part of Pharo, so I'd say philosophically might as well go all in to make FFI to external libraries an intrinsic part of computing with Pharo, not just for developing Pharo itself. The more people use UFFI, the better it will become.
An in-image regex engine would always be preferable by me, but creating a fully-flegged engine with all the fancy lookarounds, named/non-capturing groups, non-greedy matches, Unicode support, etc. sounds like a six-month-length full-time project. Any volunteers? ;) I am all for a pragmatic approach: If there is a solution that allows to reuse Smalltalk streams and strings without much memory copying combined with a library dependency that is battle-proven and maintained by capable people - why not? Funny enough, PCRE already seems to be used in the RePlugin (https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/src/plugins/RePlu... <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/src/plugins/RePlu...>). Why is this not used in the image, am I missing something? Regarding third-party dependencies, my only beef is that I do not particularly like that they are distributed and linked together with a copy by the VM. This makes it very portable, but I do not need another copy of a library that I have already installed on my system (git/SSH/SSL/png/FreeType/Cairo). On Linux some libs (Cairo/FreeType/PNG) are already linked to system libs, I would like to have Pharo as an installable package in apt and MacPorts/brew with declared dependencies, which would make it even more lightweight. But that is only marginally related for this thread.
On 6 Feb 2019, at 02:38, Pierce Ng <pierce@samadhiweb.com> wrote:
On Tue, Feb 05, 2019 at 07:25:07PM +0100, Norbert Hartl wrote:
Am 05.02.2019 um 16:16 schrieb Sven Van Caekenberghe <sven@stfx.eu>: Still, there are advantages to an in-image solution, can't says this enough, these external lib dependencies pose their own problems ... +1
Libraries like libgit2, libssh2 and quite a few more are already a core part of Pharo, so I'd say philosophically might as well go all in to make FFI to external libraries an intrinsic part of computing with Pharo, not just for developing Pharo itself. The more people use UFFI, the better it will become.
As an in-image solution, refactoring the SmaCC scanner to a standalone regex engine might be pretty efficient to gain more RE features, but I cannot really judge how big this effort would be.
On 6 Feb 2019, at 11:42, Manuel Leuenberger <leuenberger@inf.unibe.ch> wrote:
An in-image regex engine would always be preferable by me, but creating a fully-flegged engine with all the fancy lookarounds, named/non-capturing groups, non-greedy matches, Unicode support, etc. sounds like a six-month-length full-time project. Any volunteers? ;)
I am all for a pragmatic approach: If there is a solution that allows to reuse Smalltalk streams and strings without much memory copying combined with a library dependency that is battle-proven and maintained by capable people - why not?
Funny enough, PCRE already seems to be used in the RePlugin (https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/src/plugins/RePlu... <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/src/plugins/RePlu...>). Why is this not used in the image, am I missing something?
Regarding third-party dependencies, my only beef is that I do not particularly like that they are distributed and linked together with a copy by the VM. This makes it very portable, but I do not need another copy of a library that I have already installed on my system (git/SSH/SSL/png/FreeType/Cairo). On Linux some libs (Cairo/FreeType/PNG) are already linked to system libs, I would like to have Pharo as an installable package in apt and MacPorts/brew with declared dependencies, which would make it even more lightweight. But that is only marginally related for this thread.
On 6 Feb 2019, at 02:38, Pierce Ng <pierce@samadhiweb.com <mailto:pierce@samadhiweb.com>> wrote:
On Tue, Feb 05, 2019 at 07:25:07PM +0100, Norbert Hartl wrote:
Am 05.02.2019 um 16:16 schrieb Sven Van Caekenberghe <sven@stfx.eu <mailto:sven@stfx.eu>>: Still, there are advantages to an in-image solution, can't says this enough, these external lib dependencies pose their own problems ... +1
Libraries like libgit2, libssh2 and quite a few more are already a core part of Pharo, so I'd say philosophically might as well go all in to make FFI to external libraries an intrinsic part of computing with Pharo, not just for developing Pharo itself. The more people use UFFI, the better it will become.
Manuel Leuenberger wrote
An in-image regex engine would always be preferable by me, but... I am all for a pragmatic approach
I feel similarly: Ideally I would love everything in image, but given limited manpower it seems wise to leverage outside libs for standard things so that we can devote our focus to blue plane invention. Some fantasy future moment when we have already conquered the world and have nothing to do would be a perfect time to circle back and reimplement the delegated tasks in image. ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
participants (9)
-
Denis Kudriashov -
Esteban Lorenzano -
Manuel Leuenberger -
Norbert Hartl -
Pierce Ng -
Richard O'Keefe -
Sean P. DeNigris -
Sven Van Caekenberghe -
Thierry Goubier