- Pharo-dev - lists.pharo.org

Re: [ANN] Libgit 1.0.0 in Pharo 9

by Martin McClure

Hi Pablo, I run Gentoo stable on my main systems (though Ubuntu on others). I ran an experiment this morning. By default, when you build libgit2, it uses the system's PCRE library if there is one. However, if you build libgit2 with cmake -DREGEX_BACKEND=builtin ../libgit2-1.0.0 cmake --build . Then it builds libgit2 with PCRE support built-in from libgit2's own copy of the PCRE sources, and gets rid of the external dependency. Pharo seems to like the resulting library just fine. Shipping libgit2.so with PCRE built-in seems like the cleanest short-term solution. I also noticed that in the stable 9.0 VM from June (which I believe is the latest stable VM?) the libgit2 v1.0.0 library has the wrong name and so P9 always loads the 0.25.1 version. The shipped name: libgit2.1.0.0.so should be libgit2.so.1.0.0 Thanks, -Martin On 10/13/20 1:20 AM, tesonep(a)gmail.com wrote: > Hi Martin, > thanks for testing it. Can you tell me the version / flavour of > Debian are you using? > Maybe we need to ship PCRE with the VM. I am not fun of that, but we > need to do it until we can have a proper packaging for each Linux > distribution (Esteban is working on having a nice OBS configuration > for the VM extending what Holger has previously done). > It takes time and a lot of testing, but the idea is to fix (or at > least control better) the dependency nightmare between different Linux > distributions. > > Cheers, > Pablo > > On Tue, Oct 13, 2020 at 7:49 AM Martin McClure <martin(a)hand2mouse.com> wrote: >> Thanks, this is great news. >> >> Are there instructions on how to make P9 actually use the new version of >> libgit2? I see that the stable Linux VM for 9.0 does include >> "libgit2.1.0.0.so" but Pharo is still loading libgit2.so.0.25.1. >> >> ...this may be because Pharo's libgit2.1.0.0.so has not-found >> dependencies on libpcre.so.3 and libpcreposix.so.3. >> >> I find Debian's versioning of libpcre somewhat confusing. The package >> "libpcre3" is the *old* libraries, and new stuff is supposed to use >> "pcre2." 2 is apparently newer than 3. At any rate PCRE library naming >> is rather distro-specific, and although I have the right packages >> installed, the libraries have different filenames. >> >> But libgit2 is not supposed to have PCRE as a dependency. The source >> code for PCRE is included in the source for libgit2. Seems like it would >> be possible (and nice!) to compile the libgit2.1.0.0.so included with >> the VM to include the PCRE code internally, rather than depend on an >> external library. >> >> Thanks, >> -Martin >> >> On 10/12/20 2:26 AM, tesonep(a)gmail.com wrote: >>> Hi, a later announcement (this have been done some months ago... but I >>> never send the mail). >>> >>> We have upgraded in Pharo 9 to use the latest stable version of Libgit. >>> It required to have some improvements in Libgit to handle the updated >>> version and also to support the previous version. >>> >>> From the point of view of new features, it does not add new things. >>> However, this version fixes a lot of existing problems in Libgit. >>> >>> We can be sure this was a successful deployment as we don't have >>> problems with it. We are really happy that this has been done >>> transparently and keeping the working version with different >>> configurations of images and VMs. As we should support new and old >>> images, on the same VM. And also running new images in old VMs >>> >>> Thanks. >>> > > -- > Pablo Tesone. > tesonep(a)gmail.com

Oct. 13, 2020

Re: P3Converter and DateAndTime>>#readFrom: performance

by Sven Van Caekenberghe

Hi Estaban, I took some time to properly look at your remarks/suggestions. You are right, the chronology conversions could be improved and made faster. I committed the following https://github.com/svenvc/P3/commit/dd06eb33d0147060e0b6f44c704e2e298f55c0d4 "As noted/analysed by Esteban A. Maringolo, incoming conversions for Timestamp/DateAndTime and Date especially were slow. He suggested to take advantage of the known ISO format. Introduction of P3ISO8601Parser that implements a fast, non-checking parser specifically for the PSQL 8601 format. Use P3ISO8601Parser for the chronology conversions in P3Converter, for a 5x to 10x speedup. Note that for this to work, the connection output format must be ISO, the default." I did the following small test to confirm the improvements: p3client := P3Client url: 'psql://sven@localhost:5432/'. p3client execute: 'DROP TABLE IF EXISTS table1'. p3client execute: 'CREATE TABLE table1 (id SERIAL PRIMARY KEY, name TEXT, t1 TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, t2 TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, t0 TIME DEFAULT CURRENT_TIME, d1 DATE DEFAULT CURRENT_DATE)'. 1 to: 100 do: [ :each | p3client execute: 'INSERT INTO table1 (name) VALUES (''n-' , (each + 999 atRandom) asString , ''')' ]. [ p3client query: 'SELECT * FROM table1' ] bench. p3client close. This query (6 columns, 100 rows) went from 68.332 per second to 446.711 per second on my machine. Thanks again for your feedback, it was most useful. Do these changes work for you ? Regards, Sven > On 12 Oct 2020, at 15:57, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hi Esteban, > > Thanks for this detailed feedback, this is most interesting. > > I don't have enough time to look at this in detail, I will do so later, but here is some initial feedback. > > Note that PostreSQL supports many different date/time formats and that the user can set preferences per connection, hence some flexibility in parsing is probably needed, and that is one reason why the parsing is implemented the way it is. > > Check section 8.5.2. from https://www.postgresql.org/docs/13/datatype-datetime.html > > Note that all of the following give the same result: > > Date readFrom: 'March 2 2020' readStream. > Date readFrom: '3/2/2020' readStream. > Date readFrom: '2020-03-02' readStream. > > Date parsing is probably slow because it has to deal with all these formats. > > Of course, parsing a fixed well-known format is always faster. > > The timezones are tricky too, as preferences can be set per connection, so we have to be careful. > > I will come back to this. > > Sven > >> On 12 Oct 2020, at 15:04, Esteban Maringolo <emaringolo(a)gmail.com> wrote: >> >> Hi, >> >> I recently stumbled into some performance issue I had, so I profiled >> it and got it fixed, but by doing that I also noticed a huge time >> spent when reading DateAndTimes and Dates (which also uses the former) >> and the culprit seems to be `Date class>>#readFrom:`. >> >> Digging into this, I noticed that this was called from >> `DateAndTime>>#readFrom:`, but in particular 90% of the time spent was >> when instantiating the Date part of it, because it does unnecessary >> lookup of the index of Month when input stream only have a month >> index. >> >> What I did then was to change the default behavior of DateAndTime >> class>>#readFrom: >> >> readFrom: aStream defaultOffset: defaultOffset >> "Parse and return a new DateAndTime instance from stream, >> as a Date, an optional Time and an optional TimeZone offset. >> The time defaults to midnight, the timezone to defaultOffset" >> "self readFrom: '2013-03-04T23:47:52.876+01:00' readStream" >> >> "From this" >> date := Date readFrom: aStream. >> >> "To this" >> date := Date readFrom: (aStream next: 10) pattern: 'yyyy-mm-dd'. >> >> And also in P3Converter to specify the pattern directly. >> >> P3Converter>>#convertDateFrom: bytes length: length description: description >> ^Date readFrom: (self asciiStreamFor: bytes length: length) pattern: >> 'yyyy-mm-dd' >> >> With these two changes I got a 8x-10x speedup when reading >> Date/DateAndTime from the database. >> >> Then I dig a little further and in P3Converter added a little check to >> avoid translating the offset of a DateAndTime when it is not needed. >> >> P3Converter>>#convertDateAndTimeWithoutTimezoneFrom: bytes length: >> length description: description >> "TIMESTAMP WITHOUT TIME ZONE (TIMESTAMP) is stored internally in >> Postgres the way it was inserted, its representation remains constant, >> with no offset added. We use the timezone of the connection to do the >> necessary shifting. This assumes that the timezones used during >> insert/update and query are the same." >> >> | timestamp offset | >> timestamp := DateAndTime readFrom: (self asciiStreamFor: bytes >> length: length) defaultOffset: Duration zero. >> offset := self timezone offsetForTimestamp: timestamp. >> ^offset isZero "<-- added this check to avoid translation" >> ifTrue: [ timestamp ] >> ifFalse: [ timestamp translateTo: offset ] >> >> And the speedup went even higher. But I don't know whether this last >> part is correct. >> >> Of course this seems as a quick and dirty fix, but I don't know >> whether DateAndTime>>readFrom: should read any format or just ISO 8601 >> formats nor whether the Date/DateAndTime coming from PostgreSQL are >> always formatted that way. I ran my app's small test suite and haven't >> had any issues, neither during development with these changes applied. >> >> I think there is a significant slowdown involved in the instantiation >> of Dates that should be fixed, whether this fix is definitive or not >> is why I'm asking here. >> >> Regards! >> >> Esteban A. Maringolo >

Oct. 13, 2020

Re: [ANN] Libgit 1.0.0 in Pharo 9

by Sean P. DeNigris

tesonep(a)gmail.com wrote > Esteban is working on having a nice OBS configuration > for the VM extending what Holger has previously done Awesome! I code on Mac, but this is super important to the community and often for deployment... ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Oct. 13, 2020

Re: This week (41/2020) on the Issue Tracker

by Sean P. DeNigris

Marcus Denker-4 wrote > This week (41/2020) on the Issue Tracker > This one puts a focus on *merged* PRs to Pharo itself only and sorts them > by category with a > little bit of context. Thanks, Marcus! Indeed this one seems much nicer :) ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Oct. 13, 2020

Re: [ANN] Libgit 1.0.0 in Pharo 9

by tesonep＠gmail.com

Hi Martin, thanks for testing it. Can you tell me the version / flavour of Debian are you using? Maybe we need to ship PCRE with the VM. I am not fun of that, but we need to do it until we can have a proper packaging for each Linux distribution (Esteban is working on having a nice OBS configuration for the VM extending what Holger has previously done). It takes time and a lot of testing, but the idea is to fix (or at least control better) the dependency nightmare between different Linux distributions. Cheers, Pablo On Tue, Oct 13, 2020 at 7:49 AM Martin McClure <martin(a)hand2mouse.com> wrote: > > Thanks, this is great news. > > Are there instructions on how to make P9 actually use the new version of > libgit2? I see that the stable Linux VM for 9.0 does include > "libgit2.1.0.0.so" but Pharo is still loading libgit2.so.0.25.1. > > ...this may be because Pharo's libgit2.1.0.0.so has not-found > dependencies on libpcre.so.3 and libpcreposix.so.3. > > I find Debian's versioning of libpcre somewhat confusing. The package > "libpcre3" is the *old* libraries, and new stuff is supposed to use > "pcre2." 2 is apparently newer than 3. At any rate PCRE library naming > is rather distro-specific, and although I have the right packages > installed, the libraries have different filenames. > > But libgit2 is not supposed to have PCRE as a dependency. The source > code for PCRE is included in the source for libgit2. Seems like it would > be possible (and nice!) to compile the libgit2.1.0.0.so included with > the VM to include the PCRE code internally, rather than depend on an > external library. > > Thanks, > -Martin > > On 10/12/20 2:26 AM, tesonep(a)gmail.com wrote: > > Hi, a later announcement (this have been done some months ago... but I > > never send the mail). > > > > We have upgraded in Pharo 9 to use the latest stable version of Libgit. > > It required to have some improvements in Libgit to handle the updated > > version and also to support the previous version. > > > > From the point of view of new features, it does not add new things. > > However, this version fixes a lot of existing problems in Libgit. > > > > We can be sure this was a successful deployment as we don't have > > problems with it. We are really happy that this has been done > > transparently and keeping the working version with different > > configurations of images and VMs. As we should support new and old > > images, on the same VM. And also running new images in old VMs > > > > Thanks. > > > -- Pablo Tesone. tesonep(a)gmail.com

Oct. 13, 2020

Re: [ANN] Libgit 1.0.0 in Pharo 9

by Martin McClure

Thanks, this is great news. Are there instructions on how to make P9 actually use the new version of libgit2? I see that the stable Linux VM for 9.0 does include "libgit2.1.0.0.so" but Pharo is still loading libgit2.so.0.25.1. ...this may be because Pharo's libgit2.1.0.0.so has not-found dependencies on libpcre.so.3 and libpcreposix.so.3. I find Debian's versioning of libpcre somewhat confusing. The package "libpcre3" is the *old* libraries, and new stuff is supposed to use "pcre2." 2 is apparently newer than 3. At any rate PCRE library naming is rather distro-specific, and although I have the right packages installed, the libraries have different filenames. But libgit2 is not supposed to have PCRE as a dependency. The source code for PCRE is included in the source for libgit2. Seems like it would be possible (and nice!) to compile the libgit2.1.0.0.so included with the VM to include the PCRE code internally, rather than depend on an external library. Thanks, -Martin On 10/12/20 2:26 AM, tesonep(a)gmail.com wrote: > Hi, a later announcement (this have been done some months ago... but I > never send the mail). > > We have upgraded in Pharo 9 to use the latest stable version of Libgit. > It required to have some improvements in Libgit to handle the updated > version and also to support the previous version. > > From the point of view of new features, it does not add new things. > However, this version fixes a lot of existing problems in Libgit. > > We can be sure this was a successful deployment as we don't have > problems with it. We are really happy that this has been done > transparently and keeping the working version with different > configurations of images and VMs. As we should support new and old > images, on the same VM. And also running new images in old VMs > > Thanks. >

Oct. 13, 2020

Re: P3Converter and DateAndTime>>#readFrom: performance

by Sven Van Caekenberghe

Hi Esteban, Thanks for this detailed feedback, this is most interesting. I don't have enough time to look at this in detail, I will do so later, but here is some initial feedback. Note that PostreSQL supports many different date/time formats and that the user can set preferences per connection, hence some flexibility in parsing is probably needed, and that is one reason why the parsing is implemented the way it is. Check section 8.5.2. from https://www.postgresql.org/docs/13/datatype-datetime.html Note that all of the following give the same result: Date readFrom: 'March 2 2020' readStream. Date readFrom: '3/2/2020' readStream. Date readFrom: '2020-03-02' readStream. Date parsing is probably slow because it has to deal with all these formats. Of course, parsing a fixed well-known format is always faster. The timezones are tricky too, as preferences can be set per connection, so we have to be careful. I will come back to this. Sven > On 12 Oct 2020, at 15:04, Esteban Maringolo <emaringolo(a)gmail.com> wrote: > > Hi, > > I recently stumbled into some performance issue I had, so I profiled > it and got it fixed, but by doing that I also noticed a huge time > spent when reading DateAndTimes and Dates (which also uses the former) > and the culprit seems to be `Date class>>#readFrom:`. > > Digging into this, I noticed that this was called from > `DateAndTime>>#readFrom:`, but in particular 90% of the time spent was > when instantiating the Date part of it, because it does unnecessary > lookup of the index of Month when input stream only have a month > index. > > What I did then was to change the default behavior of DateAndTime > class>>#readFrom: > > readFrom: aStream defaultOffset: defaultOffset > "Parse and return a new DateAndTime instance from stream, > as a Date, an optional Time and an optional TimeZone offset. > The time defaults to midnight, the timezone to defaultOffset" > "self readFrom: '2013-03-04T23:47:52.876+01:00' readStream" > > "From this" > date := Date readFrom: aStream. > > "To this" > date := Date readFrom: (aStream next: 10) pattern: 'yyyy-mm-dd'. > > And also in P3Converter to specify the pattern directly. > > P3Converter>>#convertDateFrom: bytes length: length description: description > ^Date readFrom: (self asciiStreamFor: bytes length: length) pattern: > 'yyyy-mm-dd' > > With these two changes I got a 8x-10x speedup when reading > Date/DateAndTime from the database. > > Then I dig a little further and in P3Converter added a little check to > avoid translating the offset of a DateAndTime when it is not needed. > > P3Converter>>#convertDateAndTimeWithoutTimezoneFrom: bytes length: > length description: description > "TIMESTAMP WITHOUT TIME ZONE (TIMESTAMP) is stored internally in > Postgres the way it was inserted, its representation remains constant, > with no offset added. We use the timezone of the connection to do the > necessary shifting. This assumes that the timezones used during > insert/update and query are the same." > > | timestamp offset | > timestamp := DateAndTime readFrom: (self asciiStreamFor: bytes > length: length) defaultOffset: Duration zero. > offset := self timezone offsetForTimestamp: timestamp. > ^offset isZero "<-- added this check to avoid translation" > ifTrue: [ timestamp ] > ifFalse: [ timestamp translateTo: offset ] > > And the speedup went even higher. But I don't know whether this last > part is correct. > > Of course this seems as a quick and dirty fix, but I don't know > whether DateAndTime>>readFrom: should read any format or just ISO 8601 > formats nor whether the Date/DateAndTime coming from PostgreSQL are > always formatted that way. I ran my app's small test suite and haven't > had any issues, neither during development with these changes applied. > > I think there is a significant slowdown involved in the instantiation > of Dates that should be fixed, whether this fix is definitive or not > is why I'm asking here. > > Regards! > > Esteban A. Maringolo

Oct. 12, 2020

P3Converter and DateAndTime>>#readFrom: performance

by Esteban Maringolo

Hi, I recently stumbled into some performance issue I had, so I profiled it and got it fixed, but by doing that I also noticed a huge time spent when reading DateAndTimes and Dates (which also uses the former) and the culprit seems to be `Date class>>#readFrom:`. Digging into this, I noticed that this was called from `DateAndTime>>#readFrom:`, but in particular 90% of the time spent was when instantiating the Date part of it, because it does unnecessary lookup of the index of Month when input stream only have a month index. What I did then was to change the default behavior of DateAndTime class>>#readFrom: readFrom: aStream defaultOffset: defaultOffset "Parse and return a new DateAndTime instance from stream, as a Date, an optional Time and an optional TimeZone offset. The time defaults to midnight, the timezone to defaultOffset" "self readFrom: '2013-03-04T23:47:52.876+01:00' readStream" "From this" date := Date readFrom: aStream. "To this" date := Date readFrom: (aStream next: 10) pattern: 'yyyy-mm-dd'. And also in P3Converter to specify the pattern directly. P3Converter>>#convertDateFrom: bytes length: length description: description ^Date readFrom: (self asciiStreamFor: bytes length: length) pattern: 'yyyy-mm-dd' With these two changes I got a 8x-10x speedup when reading Date/DateAndTime from the database. Then I dig a little further and in P3Converter added a little check to avoid translating the offset of a DateAndTime when it is not needed. P3Converter>>#convertDateAndTimeWithoutTimezoneFrom: bytes length: length description: description "TIMESTAMP WITHOUT TIME ZONE (TIMESTAMP) is stored internally in Postgres the way it was inserted, its representation remains constant, with no offset added. We use the timezone of the connection to do the necessary shifting. This assumes that the timezones used during insert/update and query are the same." | timestamp offset | timestamp := DateAndTime readFrom: (self asciiStreamFor: bytes length: length) defaultOffset: Duration zero. offset := self timezone offsetForTimestamp: timestamp. ^offset isZero "<-- added this check to avoid translation" ifTrue: [ timestamp ] ifFalse: [ timestamp translateTo: offset ] And the speedup went even higher. But I don't know whether this last part is correct. Of course this seems as a quick and dirty fix, but I don't know whether DateAndTime>>readFrom: should read any format or just ISO 8601 formats nor whether the Date/DateAndTime coming from PostgreSQL are always formatted that way. I ran my app's small test suite and haven't had any issues, neither during development with these changes applied. I think there is a significant slowdown involved in the instantiation of Dates that should be fixed, whether this fix is definitive or not is why I'm asking here. Regards! Esteban A. Maringolo

Oct. 12, 2020

[ANN] Libgit 1.0.0 in Pharo 9

by tesonep＠gmail.com

Hi, a later announcement (this have been done some months ago... but I never send the mail). We have upgraded in Pharo 9 to use the latest stable version of Libgit. It required to have some improvements in Libgit to handle the updated version and also to support the previous version. >From the point of view of new features, it does not add new things. However, this version fixes a lot of existing problems in Libgit. We can be sure this was a successful deployment as we don't have problems with it. We are really happy that this has been done transparently and keeping the working version with different configurations of images and VMs. As we should support new and old images, on the same VM. And also running new images in old VMs Thanks. -- Pablo Tesone. tesonep(a)gmail.com

Oct. 12, 2020

Re: [Pharo-users] [ANN] Pharo Consortium New Gold Member: InfOil

by StÃ©phane Ducasse

us too :) S. > On 30 Sep 2020, at 16:39, Esteban Maringolo <emaringolo(a)gmail.com> wrote: > > As a former employee of the company I'm really happy to see this > finally happening! > > Best regards! > > Esteban A. Maringolo > > On Wed, Sep 30, 2020 at 11:13 AM Marcus Denker <marcus.denker(a)inria.fr> wrote: >> >> >> The Pharo Consortium is very happy to announce that InfOil has joined the Consortium as a Gold Member. >> >> InfOil is an Argentinian company focused on software development for the Oil and Gas industry with clients >> in Latin America, Cuba, and Canada. >> >> About >> - InfOil: http://www.infoil.com.ar >> - Pharo Consortium: http://consortium.pharo.org >> >> The goal of the Pharo Consortium is to allow companies and institutions to support the ongoing development >> and future of Pharo. >> >> Individuals can support Pharo via the Pharo Association: >> >> - http://association.pharo.org -------------------------------------------- StÃ©phane Ducasse http://stephane.ducasse.free.fr / http://www.pharo.org 03 59 35 87 52 Assistant: Aurore Dalle FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, BÃ¢t.A, Park Plaza Villeneuve d'Ascq 59650 France

Oct. 11, 2020