- Pharo-users - lists.pharo.org

Re: Introducing new feature dataTypes for the PolyMath/DataFrame project

by Konrad Hinsen

Hi Balaji, > Thanks for pointing this out. You are right with how the data types are > calculated, similar to collection >> commonSuperClass. But this time, it is > calculated only during the creation of DataFrame once and for all. That sounds good. > A nil value, or a Series of nil values yields UndefinedObject as its super > class. There was an error with the dataset used on that example blog post. > I have corrected it now. OK, that explains my confusion. Cheers, Konrad.

Aug. 8, 2021

Logging Frameworks

by Esteban Maringolo

What are the current options for logging? Beacon seems to be the most modern one, but the Github repository doesn't seem very active (maybe it forked somewhere else). Given that the logs might come from different worker images, is there something that can work via HTTP or similar? (it is, forwarding the logged events to a server). Regards! ps: I think I've asked this before, but couldn't find my previous question about it. Esteban A. Maringolo

Aug. 8, 2021

Re: Introducing new feature dataTypes for the PolyMath/DataFrame project

by Richard O'Keefe

I am not quite sure what the point of the datatypes feature is. x := nil. aSequence do: [:each | each ifNotNil: [ x := x ifNil: [each class] ifNotNil: [x commonSuperclassWith: each class]]]. doesn't seem terribly complicated. My difficulty is that from a statistics/data science perspective, it doesn't seem terribly *useful*. I'm currently reading a book about geostatistics with R (based on a survey of the Kola peninsula). For that task, it is ESSENTIAL to know the units in which the items are recorded. If Calcium is measured in mg/kg and Caesium is measured in Âµg/kg, you really really need to know that. This is not information you can derive by looking at the representation of the data in Pharo. Consider for example 1. mass of animals in kg 2. maximum speed of cars in km/h 3. volume of rain in successive dates, in mL (for fixed area) 4. directions taken by sand-hoppers released at different times of day, in degrees 5. region of space illuminated by light bulbs in steradians. These might all have the *same* representation in Pharo, but they are *semantically* very different. 1 and 2 are linear, but cannot be negative. 3 also cannot be negative, but the variable is a *time series*, which 1 and 2 are not. 4 is a circular measure, and taking the usual arithmetic mean or median would be an elementary blunder producing meaningless answers. 5 is perhaps best viewed as a proportion. (These are all actual examples, by the way.) THIS kind of information IS valuable for analysis. The difference between SmallInteger and Float64 is nowhere near as interesting. There's a bunch of weather data that i'm very interested in which has things like air temperature, soil temperature, relative humidity, wind speed and direction (last 5 minutes), gust speed and direction (maximum in last 5 minutes), illumination in W/m^2 (visible, UVB, UVA), rainfall, and of course date+time. Temperatures are measured on an interval scale, so dividing them makes no sense. Nor does adding them. If it's 10C today and 10C tomorrow, nothing is 20C. But oddly enough arithmetic means DO make sense. Humidity is bounded between 0 and 100; adding two relative humidities makes no sense at all. Medians make sense but means do not. Wind speed and direction are reported as separate variables, but they are arguably one 2D vector quantity. Illumination is on a ratio scale. Dividing one illumination by another makes sense, or would if there were no moonless nights... The total illumination over a day makes sense. Rainfall is also on a ratio scale. Dividing the rainfall on one day by that on another would make sense if only the usual measurement were not 0. Total rainfall over a day makes sense. The whole problem a statistician/data scientist faces is that there is important information you need to know even which *basic* operations make sense that has already disappeared by the time Pharo stores it, and cannot be inferred from the DataFrame. I remember one time I was given a CSV file with about 50 variables and it took me about 2 weeks to recover this missing meta-information. On Sat, 7 Aug 2021 at 04:23, Balaji G <gbalaji20002000(a)gmail.com> wrote: > > Hello Everyone, > > I have been working on the addition of a new feature, DataFrame >> dataTypes, which briefs us about the data type of columns in dataframes we work on. > Summarising a dataset is really important during the initial stage of any Data Science and Machine Learning tasks. Knowing the data type of the attribute is one major thing to begin with. > I have tried to work with some sample datasets for a clear understanding of this new feature. > Please go through the following blog post. Any kind of suggestion or feedback is welcome. > > Link to the Post : https://balaji612141526.wordpress.com/2021/08/06/introducing-new-feature-in… > > Previous discussions can be found here : https://lists.pharo.org/empathy/thread/BFOHPRUU72MDYVTJP3YV2DQ5LAZHXELE and here : https://lists.pharo.org/empathy/thread/JZXKXGHSURC3DCDA2NXA7KDWZ2EINAZ5 > > > > Cheers > Balaji G

Aug. 8, 2021

Re: Pharo-SQLite3: closing connections immediately and deleting the database file

by Esteban Maringolo

I've been doing exactly that. It is, creating a new DB file for each test, and I was having the same problem as you. I suggested a change to the driver to cover that situation: https://github.com/pharo-rdbms/Pharo-SQLite3/pull/22/files Try applying that change and see if that works. I still prefer to recreate the database, it will give you a full fresh start, and if you want you can halt before the deletion of the db, to analyze its internal state. Best regards! Esteban A. Maringolo On Sat, Aug 7, 2021 at 11:13 PM <vinref(a)gmail.com> wrote: > > Hi > > Actually I will just do a bunch of > > DROP TABLE IF EXISTS > > instead of dropping the db altogether. > > Thanks, Vince

Aug. 8, 2021

Re: Pharo-SQLite3: closing connections immediately and deleting the database file

by vinref＠gmail.com

Hi Actually I will just do a bunch of DROP TABLE IF EXISTS instead of dropping the db altogether. Thanks, Vince

Aug. 8, 2021

Re: Pharo-SQLite3: closing connections immediately and deleting the database file

by vinref＠gmail.com

Hi Looking at SQLite3Connection>>#close: "Let FFIExternalResourceManager take care." "dbHandle ifNotNil: \[ library close: dbHandle \]." dbHandle := nil. isOpen := false. The commented out lines means that SQLite3Library>>#apiClose is not called when calling SQLite3Connection>>#close. But looking at https://www.sqlite.org/c3ref/close.html and FFIExternalResourceManager it seems like it should take care of releasing resources (i.e., finalizing prepared queries etc) before calling SQLite3Library>>#apiClose. So this is by design? I will have a go at finalizing everything and call SQLite3Library>>#apiClose, at least while developing. Vince

Aug. 8, 2021

Pharo-SQLite3: closing connections immediately and deleting the database file

by vinref＠gmail.com

Hi I like to develop my db schema as I develop the code - TDD-style. So I need to keep dropping the database each and every time I need to update the schema and rerun the code. The problem is that even though I close the connection using #ensure: the reference remains and I cannot delete the database file immediately. ``` buildSchema ``` ``` | fp | ``` ``` ``` ``` (fp := self dbFilePath asFileReference) exists ``` ``` ifTrue: [ fp delete ]. ``` ``` ``` ``` [ ``` ``` self db beginTransaction. ``` ``` self writeSchema; writeData. ``` ``` self db commitTransaction ] ensure: [ self cleanUp ] ``` I even tried running this before everything else to no avail: ``` SQLite3Connection allInstancesDo: [ :each | each close ]. ``` ``` SmalltalkImage cleanUp. ``` ``` SmalltalkImage current garbageCollect. ``` I keep getting a #CannotDeleteFileException because there is still a reference to the database file. If I do wait a seemingly random amount of time, the reference is cleared. How can I get the VM to release the reference to the database file immediately? Pharo 8/64-bit, Windows 10 Regards, Vince

Aug. 8, 2021

Re: Introducing new feature dataTypes for the PolyMath/DataFrame project

by Balaji G

Hi Konrad Thanks for pointing this out. You are right with how the data types are calculated, similar to collection >> commonSuperClass. But this time, it is calculated only during the creation of DataFrame once and for all. Now coming to the next part, Kindly have a look at the below code. ------------------------------ series := #( nil nil nil ) asDataSeries. series calculateDataType. "UndefinedObject" . A nil value, or a Series of nil values yields UndefinedObject as its super class. There was an error with the dataset used on that example blog post. I have corrected it now. The problem was that the dataset was not clean, and had some garbage values of different types in that column. This resulted in a column, with type Object . This was the dataset and specifically the last column resulted in this mistake. On Sat, Aug 7, 2021 at 9:58 PM Konrad Hinsen <konrad.hinsen(a)fastmail.net> wrote: > Hi Balaji, > > > Please go through the following blog post. Any kind of suggestion or > > feedback is welcome. > > That's a nice description, easy to follow. But there is a missing piece: > how do you actually find the dataType for a series of values? My first > guess was that you are using the same method as Collection >> > commonSuperclass, but then a nil value should yield UndefinedObject, not > Object as in your example. > > Cheers > Konrad. >

Aug. 7, 2021

Re: Introducing new feature dataTypes for the PolyMath/DataFrame project

by Konrad Hinsen

Hi Balaji, > Please go through the following blog post. Any kind of suggestion or > feedback is welcome. That's a nice description, easy to follow. But there is a missing piece: how do you actually find the dataType for a series of values? My first guess was that you are using the same method as Collection >> commonSuperclass, but then a nil value should yield UndefinedObject, not Object as in your example. Cheers Konrad.

Aug. 7, 2021

Introducing new feature dataTypes for the PolyMath/DataFrame project

by Balaji G

Hello Everyone, I have been working on the addition of a new feature, DataFrame >> dataTypes, which briefs us about the data type of columns in dataframes we work on. Summarising a dataset is really important during the initial stage of any Data Science and Machine Learning tasks. Knowing the data type of the attribute is one major thing to begin with. I have tried to work with some sample datasets for a clear understanding of this new feature. Please go through the following blog post. Any kind of suggestion or feedback is welcome. Link to the Post : https://balaji612141526.wordpress.com/2021/08/06/introducing-new-feature-in… Previous discussions can be found here : https://lists.pharo.org/empathy/thread/BFOHPRUU72MDYVTJP3YV2DQ5LAZHXELE and here : https://lists.pharo.org/empathy/thread/JZXKXGHSURC3DCDA2NXA7KDWZ2EINAZ5 Cheers Balaji G

Aug. 6, 2021