March 2018 - Pharo-users - lists.pharo.org

Re: [Pharo-users] Ever growing image

by Stephane Ducasse

Hilaire It is up to you to give a real try. 1) Did you look for REAL at our infrastructure? Because you can take the bootstrapped image and change our build scripts. It took us may years to produce the loading scripts and this is the future. In Pharo 7 we have the possibility to have a core and several images. 2) Now I see you complaining but you can also help we never rejected a fix that make the system more modular. And we worked a lot to make it modular and IT IS MUCH MORE MODULAR. Now we do not have the energy to invest in building different image based on the core. So if you that need to not do it then why should we? 3) We will start a new reduction of the bootstrap core with Carolina PhD and you see we will not keep it for us (while we could). We want to have Pharo running on IoT devices. 4) We are starting to remove duplicated tools: Nautilus will go out, EyeInspector too. Old text model and widgets. Now again I personnally do not really care if Pharo has one or two mb more. 5) even on my old iPhone 53 mb is not that large. But I want a much smaller core and we will get there. 6) You see we introduce Athens in particular for Moose and DrGeo. Personnally I do not use at all Athens-Cairo. So this is funny how people react. Stef On Sat, Mar 3, 2018 at 3:21 PM, Hilaire <hilaire(a)drgeo.eu> wrote: > Le 02/03/2018 Ã 22:52, Marcus Denker a Ã©crit : >> >> >> we should again do some analysis where space is goingâ¦ the Pharo7 download >> is now 38MB (the image, decompressed). > > > Situation improved :) Remember P3 was just above 20 MB > >> >> There are things to look for: >> >> 1) wasted space due to not thinking about memory and just âhaving itâ. >> >> Of course some of it are things like duplications (we now have 2 browsers, >> 2 inspectors, >> 2 debuggersâ¦ time to clean up a bitâ¦). >> >> Then every test we add takes space, every âhelp topicâ, every feature... > > > It looks like this topic comes and goes since 20 years: new features added, > image growth then/or it becomes hard to unload ; on Squeak then now with > Pharo at a much higher rate. > > We all know about the right approach (GNU Smalltalk, CUIS Smalltalk), but it > is not the road chosen with Pharo and it is becoming extremely uninteresting > for me to code on DrGeo now. So far I fell ashame to ship end user > application with a 50MB DrGeo image, knowing only 10% has purpose related to > DrGeo domain. When situation improves, motivation may come back. For this > same reason and other, I turned to Python for a planed programming course to > mid high school students. > >> >> But of course sometimes being able to invest memory into a better system >> can be important, too. >> Not all growth in memory is negativ if you can afford it. >> >> I always like to play the game to think âwhat would they have done in 1978 >> if they had this amount >> of memory in even a $5 machine?" Of course one can go very quickly in the >> wrong direction, but > > > Very likely not that much. May be the wonders came because of the hight > constraints. > > >> nevertheless: sometimes it is really worth to think about âspending >> memoryâ to buy abstraction. >> (especially a there are counter measuresâ¦ we under-utilise both >> compression and disk storage) > > Helps could (should?) be file based, and maybe other stuff I don't know. > > How the image is considered in the Pharo team? A kind of sacred place, > where stuff are added following a few guidelines (you will not duplicate > features but replace/imprive/whatever -- aka funny shortcut mess, you will > make your code unloadable, etc...) or just like a forum where teams add > feature according to projects or vision. > > Sorry to discuss these topics again, really don't want but still need to > share a few thoughts. > >> >> 2) wasted space that is just not needed. e.g. there is issue >> https://pharo.fogbugz.com/f/cases/21172/ >> That we have some huge strings to init unicode. Do we need them? >> >> 3) Memory leaks, bit they are more for the case when the system is running >> for a while. >> > > -- > Dr. Geo > http://drgeo.eu > > >

March 4, 2018

Re: [Pharo-users] Ever growing image

by Hilaire

Le 04/03/2018 Ã 11:13, Esteban Lorenzano a Ã©critÂ : > In our community, Pavel is the person that has the bigger experience > on shrinking process images. I followed his work for years and I know > the fragility of the process. Basically each new addition/removal was > breaking it and needing dependency tracking, etc. Now is the opposite: > you just add what you want, and you are sure the resulting image is a) > healthy and b) identical when running. > Indeed it is fragile. And given the latest image size, not sure it could lead to satisfactory results. -- Dr. Geo http://drgeo.eu

March 4, 2018

Re: [Pharo-users] Ever growing image

by Stephane Ducasse

I think that vendors did not go the bootstrap way because it is a lot more difficult to implement than a bunch of scripts to unload packages. On Sun, Mar 4, 2018 at 11:53 AM, Benoit St-Jean via Pharo-users <pharo-users(a)lists.pharo.org> wrote: > > > ---------- Forwarded message ---------- > From: Benoit St-Jean <bstjean(a)yahoo.com> > To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org>, Esteban Lorenzano <estebanlm(a)gmail.com> > Cc: Hilaire <hilaire(a)drgeo.eu> > Bcc: > Date: Sun, 4 Mar 2018 10:53:04 +0000 (UTC) > Subject: Re: [Pharo-users] Ever growing image > Estaban, > > I understand the difference between the 2 approaches my point was just to mention that I guess there are some advantages to shrink the image (as compared to bootstrapping from a minimal image) since all major vendors used that approach (and are still using it). It's just that I was curious to know if anyone knows any paper/document/study comparing the 2 approaches or explaining in detail both of them. I googled for a few hours but couldn't find anything... > > > ----------------- > BenoÃ®t St-Jean > Yahoo! Messenger: bstjean > Twitter: @BenLeChialeux > Pinterest: benoitstjean > Instagram: Chef_Benito > IRC: lamneth > Blogue: endormitoire.wordpress.com > "A standpoint is an intellectual horizon of radius zero". (A. Einstein) > > > On Sunday, March 4, 2018, 5:13:30 AM EST, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > > > > On 3 Mar 2018, at 16:35, Benoit St-Jean via Pharo-users <pharo-users(a)lists.pharo.org> wrote: > > > From: Benoit St-Jean <bstjean(a)yahoo.com> > Subject: Re: [Pharo-users] Ever growing image > Date: 3 March 2018 at 16:35:19 CET > To: pharo-users(a)lists.pharo.org, Hilaire <hilaire(a)drgeo.eu> > Reply-To: Benoit St-Jean <bstjean(a)yahoo.com> > > > On a side note, being a long-time VW & VAST user, I've always felt the Image Stripper & Runtime Packager did a fairly good job at removing crap from the resulting image. > > I don't recall ready anything substantial comparing the Pharo way (bootstrapping from a minimal image and adding stuff) to the VW & VAST way (removing unreferenced stuff) but this would definitely be an interesting topic to read about. > > > > No idea if any âsubstantial" thing has been written (there are some phds around that may talk about), but main difference is very simple to explain: first approach (which was the same for Pharo before) is non-deterministic while the second is. Then, for purpose of repeatability of process (and testing, etc.) bootstrapping is better. > You would not know the amount of effort made to make this possible, it required at least 2 phd works and a considerable amount of engineering effort. > In our community, Pavel is the person that has the bigger experience on shrinking process images. I followed his work for years and I know the fragility of the process. Basically each new addition/removal was breaking it and needing dependency tracking, etc. Now is the opposite: you just add what you want, and you are sure the resulting image is a) healthy and b) identical when running. > > Esteban > > > ----------------- > BenoÃ®t St-Jean > Yahoo! Messenger: bstjean > Twitter: @BenLeChialeux > Pinterest: benoitstjean > Instagram: Chef_Benito > IRC: lamneth > Blogue: endormitoire.wordpress.com > "A standpoint is an intellectual horizon of radius zero". (A. Einstein) > > > On Saturday, March 3, 2018, 9:22:06 AM EST, Hilaire <hilaire(a)drgeo.eu> wrote: > > > Le 02/03/2018 Ã 22:52, Marcus Denker a Ã©crit : >> >> we should again do some analysis where space is goingâ¦ the Pharo7 download >> is now 38MB (the image, decompressed). > > Situation improved :) Remember P3 was just above 20 MB > > > >

March 4, 2018

Re: [Pharo-users] Ever growing image

by Benoit St-Jean

Estaban, I understand the difference between the 2 approaches my point was just to mention that I guess there are some advantages to shrink the image (as compared to bootstrapping from a minimal image) since all major vendors used that approach (and are still using it).Â It's just that I was curious to know if anyone knows any paper/document/study comparing the 2 approaches or explaining in detail both of them.Â I googled for a few hours but couldn't find anything... ----------------- BenoÃ®t St-Jean Yahoo! Messenger: bstjean Twitter: @BenLeChialeux Pinterest: benoitstjean Instagram: Chef_Benito IRC: lamneth Blogue: endormitoire.wordpress.com "A standpoint is an intellectual horizon of radius zero".Â (A. Einstein) On Sunday, March 4, 2018, 5:13:30 AM EST, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: On 3 Mar 2018, at 16:35, Benoit St-Jean via Pharo-users <pharo-users(a)lists.pharo.org> wrote: From: Benoit St-Jean <bstjean(a)yahoo.com> Subject: Re: [Pharo-users] Ever growing image Date: 3 March 2018 at 16:35:19 CET To: pharo-users(a)lists.pharo.org, Hilaire <hilaire(a)drgeo.eu> Reply-To: Benoit St-Jean <bstjean(a)yahoo.com> On a side note, being a long-time VW & VAST user, I've always felt the Image Stripper & Runtime Packager did a fairly good job at removing crap from the resulting image.Â I don't recall ready anything substantial comparing the Pharo way (bootstrapping from a minimal image and adding stuff) to the VW & VAST way (removing unreferenced stuff) but this would definitely be an interesting topic to read about. No idea if any âsubstantial" thing has been written (there are some phds around that may talk about), but main difference is very simple to explain: first approach (which was the same for Pharo before) is non-deterministic while the second is. Then, for purpose of repeatability of process (and testing, etc.) bootstrapping is better.Â You would not know the amount of effort made to make this possible, it required at least 2 phd works and a considerable amount of engineering effort.In our community, Pavel is the person that has the bigger experience on shrinking process images. I followed his work for years and I know the fragility of the process. Basically each new addition/removal was breaking it and needing dependency tracking, etc. Now is the opposite: you just add what you want, and you are sure the resulting image is a) healthy and b) identical when running. Esteban ----------------- BenoÃ®t St-Jean Yahoo! Messenger: bstjean Twitter: @BenLeChialeux Pinterest: benoitstjean Instagram: Chef_Benito IRC: lamneth Blogue: endormitoire.wordpress.com "A standpoint is an intellectual horizon of radius zero".Â (A. Einstein) On Saturday, March 3, 2018, 9:22:06 AM EST, Hilaire <hilaire(a)drgeo.eu> wrote: Le 02/03/2018 Ã 22:52, Marcus Denker a Ã©critÂ : > > we should again do some analysis where space is goingâ¦ the Pharo7 download > is now 38MB (the image, decompressed). Situation improved :) Remember P3 was just above 20 MB

March 4, 2018

Re: [Pharo-users] Ever growing image

by Esteban Lorenzano

> On 3 Mar 2018, at 16:35, Benoit St-Jean via Pharo-users <pharo-users(a)lists.pharo.org> wrote: > > > From: Benoit St-Jean <bstjean(a)yahoo.com> > Subject: Re: [Pharo-users] Ever growing image > Date: 3 March 2018 at 16:35:19 CET > To: pharo-users(a)lists.pharo.org, Hilaire <hilaire(a)drgeo.eu> > Reply-To: Benoit St-Jean <bstjean(a)yahoo.com> > > > On a side note, being a long-time VW & VAST user, I've always felt the Image Stripper & Runtime Packager did a fairly good job at removing crap from the resulting image. > > I don't recall ready anything substantial comparing the Pharo way (bootstrapping from a minimal image and adding stuff) to the VW & VAST way (removing unreferenced stuff) but this would definitely be an interesting topic to read about. No idea if any âsubstantial" thing has been written (there are some phds around that may talk about), but main difference is very simple to explain: first approach (which was the same for Pharo before) is non-deterministic while the second is. Then, for purpose of repeatability of process (and testing, etc.) bootstrapping is better. You would not know the amount of effort made to make this possible, it required at least 2 phd works and a considerable amount of engineering effort. In our community, Pavel is the person that has the bigger experience on shrinking process images. I followed his work for years and I know the fragility of the process. Basically each new addition/removal was breaking it and needing dependency tracking, etc. Now is the opposite: you just add what you want, and you are sure the resulting image is a) healthy and b) identical when running. Esteban > > ----------------- > BenoÃ®t St-Jean > Yahoo! Messenger: bstjean > Twitter: @BenLeChialeux > Pinterest: benoitstjean > Instagram: Chef_Benito > IRC: lamneth > Blogue: endormitoire.wordpress.com > "A standpoint is an intellectual horizon of radius zero". (A. Einstein) > > > On Saturday, March 3, 2018, 9:22:06 AM EST, Hilaire <hilaire(a)drgeo.eu> wrote: > > > Le 02/03/2018 Ã 22:52, Marcus Denker a Ã©crit : > > > > we should again do some analysis where space is goingâ¦ the Pharo7 download > > is now 38MB (the image, decompressed). > > Situation improved :) Remember P3 was just above 20 MB > >

March 4, 2018

Re: [Pharo-users] Ever growing image

by Esteban Lorenzano

Hi Hilaire, > On 3 Mar 2018, at 15:21, Hilaire <hilaire(a)drgeo.eu> wrote: > > Le 02/03/2018 Ã 22:52, Marcus Denker a Ã©crit : >> >> we should again do some analysis where space is goingâ¦ the Pharo7 download >> is now 38MB (the image, decompressed). > > Situation improved :) Remember P3 was just above 20 MB > >> >> There are things to look for: >> >> 1) wasted space due to not thinking about memory and just âhaving itâ. >> >> Of course some of it are things like duplications (we now have 2 browsers, 2 inspectors, >> 2 debuggersâ¦ time to clean up a bitâ¦). >> >> Then every test we add takes space, every âhelp topicâ, every feature... > > It looks like this topic comes and goes since 20 years: new features added, image growth then/or it becomes hard to unload ; on Squeak then now with Pharo at a much higher rate. > > We all know about the right approach (GNU Smalltalk, CUIS Smalltalk), but it is not the road chosen with Pharo and it is becoming extremely uninteresting for me to code on DrGeo now. So far I fell ashame to ship end user application with a 50MB DrGeo image, knowing only 10% has purpose related to DrGeo domain. When situation improves, motivation may come back. For this same reason and other, I turned to Python for a planed programming course to mid high school students. I think thatâs unfair. First, 38m is a really small application nowadays. I think for the amount of features included, 38m is a very small size (just look at any other application aroundâ¦ and of course scripting does not count by itself, it has to be added with all support files it come). Second, âthe right approachâ is always a matter of opinion. Of course, we believe *we* took the right path :) Third, yes, we added new functionality that could or could not be there (for example, I believe Athens should be a loadable project, not part of the image, but since everybody want to use it and even we were planning to move our tools to use, we included it). And yes, we have duplicated tools that we need to clean. But creating an environment that allow the development of any kind of application will always mean we increase the size of the release artefact. Finally fourth, we invested a LOT of effort on bootstrapping so you can do your image of the size you want. So yes, we deliver an image of 38m, and also a minimal image of 18m which you can use as base to create your production images. And if thatâs not enough, you can always do *your own*, since the scripts for doing it are available by just cloning the sources. > >> >> But of course sometimes being able to invest memory into a better system can be important, too. >> Not all growth in memory is negativ if you can afford it. >> >> I always like to play the game to think âwhat would they have done in 1978 if they had this amount >> of memory in even a $5 machine?" Of course one can go very quickly in the wrong direction, but > > Very likely not that much. May be the wonders came because of the hight constraints. > > >> nevertheless: sometimes it is really worth to think about âspending memoryâ to buy abstraction. >> (especially a there are counter measuresâ¦ we under-utilise both compression and disk storage) > Helps could (should?) be file based, and maybe other stuff I don't know. many things could be outside the image, yes. But then we need to distribute them. And that adds complexity. > How the image is considered in the Pharo team? A kind of sacred place, where stuff are added following a few guidelines (you will not duplicate features but replace/imprive/whatever -- aka funny shortcut mess, you will make your code unloadable, etc...) or just like a forum where teams add feature according to projects or vision. We have a close control of what goes inside the image. And we take care things are well documented and with coherence of their components. Of course, we have made mistakes (maybe a lot), but in general things are improving. Esteban > > Sorry to discuss these topics again, really don't want but still need to share a few thoughts. > >> >> 2) wasted space that is just not needed. e.g. there is issue >> https://pharo.fogbugz.com/f/cases/21172/ >> That we have some huge strings to init unicode. Do we need them? >> >> 3) Memory leaks, bit they are more for the case when the system is running for a while. >> > > -- > Dr. Geo > http://drgeo.eu > > >

March 4, 2018

Re: [Pharo-users] Yet another Pharo in docker

by Stephane Ducasse

Could you build a roadmap of the actions to be done to improve the situation (like porting NB...) Stef On Sun, Mar 4, 2018 at 5:11 AM, Pierce Ng <pierce(a)samadhiweb.com> wrote: > On Sat, Mar 03, 2018 at 09:04:59AM -0700, Sean P. DeNigris wrote: >> Cool! I noticed the following [1]: >> ln -s \ >> `/sbin/ldconfig -p | sed -e 's|[^/]*||' | grep sqlite3` \ >> `find . -type f -name SqueakSSL.so -print0 | xargs -0 >> dirname`/sqlite3.so >> What does it do? >> >> Answer from Herby in private mail: fixes linux lib lookup problem in ffi, > > I was just about to do this Docker thing the other way, which is to bundle my > own SQLite library, for these reasons: > > - Customize SQLite to include JSON, FTS, etc. > > - Yesterday I committed functionality to load SQLite extensions (so/dylib/dll > files implementing SQL functions and virtual tables). I'm using a PCRE-based > extension. > > - SQLcipher. This is in NBSQLite3, to be ported to UFFI. > > Pierce > >

March 4, 2018

Re: [Pharo-users] Yet another Pharo in docker

by Pierce Ng

On Sat, Mar 03, 2018 at 09:04:59AM -0700, Sean P. DeNigris wrote: > Cool! I noticed the following [1]: > ln -s \ > `/sbin/ldconfig -p | sed -e 's|[^/]*||' | grep sqlite3` \ > `find . -type f -name SqueakSSL.so -print0 | xargs -0 > dirname`/sqlite3.so > What does it do? > > Answer from Herby in private mail: fixes linux lib lookup problem in ffi, I was just about to do this Docker thing the other way, which is to bundle my own SQLite library, for these reasons: - Customize SQLite to include JSON, FTS, etc. - Yesterday I committed functionality to load SQLite extensions (so/dylib/dll files implementing SQL functions and virtual tables). I'm using a PCRE-based extension. - SQLcipher. This is in NBSQLite3, to be ported to UFFI. Pierce

March 4, 2018

Re: [Pharo-users] HMAC-SHA512

by Pierce Ng

On Fri, Mar 02, 2018 at 11:24:36AM -0300, Esteban A. Maringolo wrote: > There is a PBKDF2 package made by Udo Schneider, and seems to be > easily pluggable with a different hashing algorithm, so I'd need a > SHA512 class. OpenSSL-Pharo wraps the SHA512 in libcrypto. See LcEvpSHA512Test. https://github.com/PierceNg/OpenSSL-Pharo Pierce

March 4, 2018

Re: [Pharo-users] [Pharo-dev] [Issue Tracker] lots of "auto close" happening

by Tudor Girba

Thanks a lot for this! Doru > On Mar 2, 2018, at 7:01 PM, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > Hello, > > As you might now, we automatically close issue tracker entries after 1 year of in-activity. > Due to two factors (the job doing it not working since 1 month *and* now being 1 year > after we started to sort issues for âhas to be fixed for pharo6 yes or noâ, we enter > a period where *a lot* of issue get auto-closed. > > Everyone subscribed to the issue gets a mail and (of course!!) you should re-opne the > issue tracker entry if the case still is relevant. > > Marcus -- www.tudorgirba.com www.feenk.com "Yesterday is a fact. Tomorrow is a possibility. Today is a challenge."

March 3, 2018