Re: [Pharo-users] Ever growing image
Pharo Smalltalk Users mailing list wrote
Estaban, I guess there are some advantages to shrink the image (as compared to bootstrapping from a minimal image)
Also, they are not mutually exclusive. While bootstrapping seems a clear choice for building images. A tool could then scan an image (e.g. for deployment) and pare it down to just what's needed. IIRC Self might have done some work in this direction⦠----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
On 4 Mar 2018, at 17:46, Sean P. DeNigris <sean@clipperadams.com> wrote:
Pharo Smalltalk Users mailing list wrote
Estaban, I guess there are some advantages to shrink the image (as compared to bootstrapping from a minimal image)
Also, they are not mutually exclusive. While bootstrapping seems a clear choice for building images. A tool could then scan an image (e.g. for deployment) and pare it down to just what's needed. IIRC Self might have done some work in this directionâ¦
Pharo7 still has ImageCleaner. It needs some work (even for Pharo6 it would be nice to do a pass). Vincent (I think) did already one improvement to it in Pharo7 (simplified test unloading). I did find some time to do an experiment with Pharo7, and if you remove all tests and documentation with ImageCleaner (after a small fix, will commit that), the image shrinks fro 38 to 35 (I had expected a bit more). *But*: just opening it and running the âDo Image Cleanupâ, then save and quit, the image is again close to 38MB, which is *very* strange! So for size there is something strange going on. We need to have a closer look. I have put on my TODO: - do a pass on ImageCleaner Pharo6 and Pharo7 - start to find duplicated code to not load anymore in the bootstrap (e.g. old compiler, inspectorâ¦). Marcus
Thanks Markus. I already saw an improvement when building drgeo against latest 32bits P7: drgeo image down to 39.9MB... I use image cleaner plus some packages removal of my own[1], left over of the cleaning I was doing on P3. Many of my package cleaning of P4 break on P7. There is still place for improvement. I remembered doing it one by on P3, keeping the safe one, then excluding the one breaking the image. Now if there is a guide to try bootstrap without the need to hit the head on the door, I am interested to read it. Hilaire [1] https://bazaar.launchpad.net/~drgeo-developers/drgeo/trunk/files/506/src/DrG... Le 05/03/2018 à 08:39, Marcus Denker a écrit :
I have put on my TODO: - do a pass on ImageCleaner Pharo6 and Pharo7 - start to find duplicated code to not load anymore in the bootstrap (e.g. old compiler, inspectorâ¦).
-- Dr. Geo http://drgeo.eu
Hi hilaire I think that Pavel sent around a link with the bootstrapped image. For you, you do not want to play changing the bootstrap. What you want is to change and load your own customized baselines on top of the bootstrapped image (probably the one with compiler and Monti/metacello loaded). I do not know where they are (running now out of my office). But the pharo70 build is based on that. Then with carolina we will start building tools to build smaller bootstrap and smaller VMs for IoT. Stef On Mon, Mar 5, 2018 at 6:33 PM, Hilaire <hilaire@drgeo.eu> wrote:
Thanks Markus.
I already saw an improvement when building drgeo against latest 32bits P7: drgeo image down to 39.9MB...
I use image cleaner plus some packages removal of my own[1], left over of the cleaning I was doing on P3. Many of my package cleaning of P4 break on P7. There is still place for improvement. I remembered doing it one by on P3, keeping the safe one, then excluding the one breaking the image.
Now if there is a guide to try bootstrap without the need to hit the head on the door, I am interested to read it.
Hilaire
[1] https://bazaar.launchpad.net/~drgeo-developers/drgeo/trunk/files/506/src/DrG...
Le 05/03/2018 à 08:39, Marcus Denker a écrit :
I have put on my TODO: - do a pass on ImageCleaner Pharo6 and Pharo7 - start to find duplicated code to not load anymore in the bootstrap (e.g. old compiler, inspectorâ¦).
-- Dr. Geo http://drgeo.eu
Marcus Denker-4 wrote
(I had expected a bit more).
I think I definitely read somewhere that there was a system that could analyze the messages that were actually sent and remove everything else... fun to think about the possibilities! ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
yes, Mariano Peck made a PhD on that (when we were exploring âhow to get smaller imagesâ possibilities). We finally went for bootstrap, but we made an analysis on all possibilities around. Maybe he can point to it⦠Esteban
On 6 Mar 2018, at 04:35, Sean P. DeNigris <sean@clipperadams.com> wrote:
Marcus Denker-4 wrote
(I had expected a bit more).
I think I definitely read somewhere that there was a system that could analyze the messages that were actually sent and remove everything else... fun to think about the possibilities!
----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
Maybe this: https://rmod.inria.fr/web/publications/bib?query=Poli17c&display=abstract Guillermo Polito, Luc Fabresse, Noury Bouraqadi, and Stéphane Ducasse. Run-Fail-Grow: Creating Tailored Object-Oriented Runtimes Producing a small deployment version of an application is a challenge because static abstractions such as packages cannot anticipate the use of their parts at runtime. Thus, an application often occupies more memory than actually needed. Tailoring is one of the main solutions to this problem i.e., extracting used code units such as classes and methods of an application. However, existing tailoring techniques are mostly based on static type annotations. These techniques cannot efficiently tailor applications in all their extent (e.g., runtime object graphs and metadata) nor be used in the context of dynamically-typed languages. We propose a run-fail-grow technique to tailor applications using their runtime execution. Run-fail-grow launches (a) a reference application containing the original application to tailor and (b) a nurtured application containing only a seed with a minimal set of code units the user wants to ensure in the final application. The nurtured application is executed, failing when it founds missing objects, classes or methods. On failure, the necessary elements are installed into the nurtured application from the reference one, and the execution resumes. The nurtured application is executed until it finishes, or until the developer explicitly finishes it, for example in the case of a web application. resulting in an object memory (i.e., a heap) with only objects, classes and methods required to execute the application. To validate our approach we implemented a tool based on Virtual Machine modifications, namely Tornado. Tornado succeeds to create very small memory footprint versions of applications e.g., a simple object-oriented heap of 11kb. We show how tailoring works on application code, base and third-party libraries even supporting human interaction with user G. interfaces. These experiments show memory savings ranging from 95\% to 99\%.
yes, Mariano Peck made a PhD on that (when we were exploring âhow to get smaller imagesâ possibilities). We finally went for bootstrap, but we made an analysis on all possibilities around.
Mariano was doing swapping out⦠Mariano Martinez Peck, Noury Bouraqadi, Marcus Denker, Stéphane Ducasse, and Luc Fabresse. Marea: An Efficient Application-Level Object Graph Swapper. Abstract Abstract During the execution of object-oriented applications, several millions of objects are created, used and then collected if they are not referenced. Problems appear when objects are unused but cannot be garbage-collected because they are still referenced from other objects. This is an issue because those objects waste primary memory and applications use more primary memory than they actually need. We claim that relying on the operating system's (OS) virtual memory is not always enough since it cannot take into account the domain and structure of applications. At the same time, applications have no easy way to parametrize nor cooperate with memory management. In this paper, we present Marea, an efficient application-level object graph swapper for object-oriented programming languages. Its main goal is to offer the programmer a novel solution to handle application-level memory. Developers can instruct our system to release primary memory by swapping out unused yet referenced objects to secondary memory. Our approach has been qualitatively and quantitatively validated. Our experiments and benchmarks on real-world applications show that Marea can reduce the memory footprint between 23\% and 36\%.
From the 90's there is the work of Ole Agesen, using type inference to detect (not) dead code and build SELF images.
http://dblp.uni-trier.de/pers/hd/a/Agesen:Ole In any case, I only wanted to say that it is unfair to say that Pharo is big when "you only load your packages". And I feel that's not true, if you're using the graphical environment, you need display, bitblt, morphic, widgets. As soon as you need to dynamically run scripts you need the compiler and parsing machinery. As soon as you write any program you mostly need many core libraries (just think about collections). How much space would take both in disk and in memory to build a similar environment in Java/Python? What I feel the most unfair is that we don't even take the time to measure to make a comparison. It's just a complaint... Now, I understand we could build images without development tools, but for that there is some work needed (not only from the pharo side but also from the developer's point of view). In the file server you can download latest minimal images both 32 and 64 bits (http://files.pharo.org/image/70/): - Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.32bit.zip <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.32bit.zip>2018-03-06 10:314.4 MB <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.49...> - Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.64bit.zip <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.64bit.zip>2018-03-06 10:314.5 MB <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.49...> Those are ~4.5M zip files containing 10MB images and 9MB changes files. Probably that is small enough for you. We can do better but each line of code we remove is indeed a lot of work. Those images have the following packages installed only: - Language Kernel + Traits + Class builder + Slots - RPackage - FFI Kernel - Opal Compiler + AST + Chunk files reader writers - Announcements - Collections - Colors - Files - Others (Hermes, Jobs, Transcript, System packages, UIManager...) - Zinc Character encoders and Buffered streams - Network - Monticello - Metacello + Filetree For a more detailed list, you can browse BaselineOfPharoBootstrap. If you want an image that requires packages that are not there (like morphic), you should install them on top and specify those as dependencies of your project. That is what we do with the Pharo IDE. Take a look at BaselineOfIDE. Again, this could be enhanced and we will greatfully accept any contribution, or discuss on skype about the details or help people to get into this. Guille On Tue, Mar 6, 2018 at 11:14 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
Maybe this:
https://rmod.inria.fr/web/publications/bib?query=Poli17c&display=abstract
Guillermo Polito, Luc Fabresse, Noury Bouraqadi, and Stéphane Ducasse. Run-Fail-Grow: Creating Tailored Object-Oriented Runtimes
Producing a small deployment version of an application is a challenge because static abstractions such as packages cannot anticipate the use of their parts at runtime. Thus, an application often occupies more memory than actually needed. Tailoring is one of the main solutions to this problem i.e., extracting used code units such as classes and methods of an application. However, existing tailoring techniques are mostly based on static type annotations. These techniques cannot efficiently tailor applications in all their extent (e.g., runtime object graphs and metadata) nor be used in the context of dynamically-typed languages. We propose a run-fail-grow technique to tailor applications using their runtime execution. Run-fail-grow launches (a) a reference application containing the original application to tailor and (b) a nurtured application containing only a seed with a minimal set of code units the user wants to ensure in the final application. The nurtured application is executed, failing when it founds missing objects, classes or methods. On failure, the necessary elements are installed into the nurtured application from the reference one, and the execution resumes. The nurtured application is executed until it finishes, or until the developer explicitly finishes it, for example in the case of a web application. resulting in an object memory (i.e., a heap) with only objects, classes and methods required to execute the application. To validate our approach we implemented a tool based on Virtual Machine modifications, namely Tornado. Tornado succeeds to create very small memory footprint versions of applications e.g., a simple object-oriented heap of 11kb. We show how tailoring works on application code, base and third-party libraries even supporting human interaction with user G. interfaces. These experiments show memory savings ranging from 95\% to 99\%.
yes, Mariano Peck made a PhD on that (when we were exploring âhow to get
smaller imagesâ possibilities).
We finally went for bootstrap, but we made an analysis on all possibilities around.
Mariano was doing swapping outâ¦
Mariano Martinez Peck, Noury Bouraqadi, Marcus Denker, Stéphane Ducasse, and Luc Fabresse. Marea: An Efficient Application-Level Object Graph Swapper.
Abstract Abstract During the execution of object-oriented applications, several millions of objects are created, used and then collected if they are not referenced. Problems appear when objects are unused but cannot be garbage-collected because they are still referenced from other objects. This is an issue because those objects waste primary memory and applications use more primary memory than they actually need. We claim that relying on the operating system's (OS) virtual memory is not always enough since it cannot take into account the domain and structure of applications. At the same time, applications have no easy way to parametrize nor cooperate with memory management. In this paper, we present Marea, an efficient application-level object graph swapper for object-oriented programming languages. Its main goal is to offer the programmer a novel solution to handle application-level memory. Developers can instruct our system to release primary memory by swapping out unused yet referenced objects to secondary memory. Our approach has been qualitatively and quantitatively validated. Our experiments and benchmarks on real-world applications show that Marea can reduce the memory footprint between 23\% and 36\%.
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Now, I understand we could build images without development tools, but for that there is some work needed (not only from the pharo side but also from the developer's point of view). In the file server you can download latest minimal images both 32 and 64 bits (http://files.pharo.org/image/70/):
- Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.32bit.zip <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.32bit.zip>2018-03-06 10:314.4 MB <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.49...> - Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.64bit.zip <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.494bbfc.arch.64bit.zip>2018-03-06 10:314.5 MB <http://files.pharo.org/image/70/Pharo-metacello-7.0.0-alpha.build.654.sha.49...>
Those are ~4.5M zip files containing 10MB images and 9MB changes files. Probably that is small enough for you. We can do better but each line of code we remove is indeed a lot of work.
Those images have the following packages installed only:
- Language Kernel + Traits + Class builder + Slots - RPackage - FFI Kernel - Opal Compiler + AST + Chunk files reader writers - Announcements - Collections - Colors - Files - Others (Hermes, Jobs, Transcript, System packages, UIManager...) - Zinc Character encoders and Buffered streams - Network - Monticello - Metacello + Filetree
For a more detailed list, you can browse BaselineOfPharoBootstrap.
If you want an image that requires packages that are not there (like morphic), you should install them on top and specify those as dependencies of your project. That is what we do with the Pharo IDE. Take a look at BaselineOfIDE. Again, this could be enhanced and we will greatfully accept any contribution, or discuss on skype about the details or help people to get into this.
I love it. It opens so many doors. 4.4 mb for such a system is cool. I'm eager to see what we will do with carolina. Stef
A "How to" as we used to have at the linux early time will be very helpful to get started. When I tried on this last summer I lost myself and gave up. Hilaire Le 06/03/2018 à 11:55, Guillermo Polito a écrit :
For a more detailed list, you can browse BaselineOfPharoBootstrap.
If you want an image that requires packages that are not there (like morphic), you should install them on top and specify those as dependencies of your project. That is what we do with the Pharo IDE. Take a look at BaselineOfIDE. Again, this could be enhanced and we will greatfully accept any contribution, or discuss on skype about the details or help people to get into this.
-- Dr. Geo http://drgeo.eu
Le 06/03/2018 à 11:55, Guillermo Polito a écrit :
In any case, I only wanted to say that it is unfair to say that Pharo is big when "you only load your packages". And I feel that's not true, if you're using the graphical environment, you need display, bitblt, morphic, widgets. As soon as you need to dynamically run scripts you need the compiler and parsing machinery. As soon as you write any program you mostly need many core libraries (just think about collections).
It is not unfair, it is just a reality. First version of DrGeo image on Pharo, including display, bitblt, morphic, widgets, compiler, parser was bellow 10MB (including DrGeo code)[1]. Now the question that matter for me, can I get back toward this fair size or close to? I don't buy on the argument we have plenty of place, who cares. It is wrong. It is with this kind of argument we will lead to the exhaustion of the planet Earth resources[2]. Hilaire [1] https://gforge.inria.fr/frs/download.php/file/28307/DrGeo.app-11.03.zip [2] https://en.wikipedia.org/wiki/Rebound_effect_(conservation) -- Dr. Geo http://drgeo.eu
On Sat, Mar 10, 2018 at 11:44 AM, Hilaire <hilaire@drgeo.eu> wrote:
Le 06/03/2018 à 11:55, Guillermo Polito a écrit :
In any case, I only wanted to say that it is unfair to say that Pharo is big when "you only load your packages". And I feel that's not true, if you're using the graphical environment, you need display, bitblt, morphic, widgets. As soon as you need to dynamically run scripts you need the compiler and parsing machinery. As soon as you write any program you mostly need many core libraries (just think about collections).
It is not unfair, it is just a reality. First version of DrGeo image on Pharo, including display, bitblt, morphic, widgets, compiler, parser was bellow 10MB (including DrGeo code)[1]. Now the question that matter for me, can I get back toward this fair size or close to?
What is unfair is that building the same environment on other technology will get closer to the hundreds of MBs, or in the Java/Eclipse case more closer to the GB(s).
I don't buy on the argument we have plenty of place, who cares. It is wrong. It is with this kind of argument we will lead to the exhaustion of the planet Earth resources[2].
We are still far from being a space eater environment... Please let's not exaggerate. That said, I would also like to have a smaller and more malleable system, in that I agree with you, but that takes time and work, and I see few people that is diving in the insides of Pharo to help get us something in that direction. Not because they don't care but because it is difficult and time consuming.
Hilaire
[1] https://gforge.inria.fr/frs/download.php/file/28307/DrGeo.ap p-11.03.zip [2] https://en.wikipedia.org/wiki/Rebound_effect_(conservation)
-- Dr. Geo http://drgeo.eu
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Le 12/03/2018 à 10:16, Guillermo Polito a écrit :
That said, I would also like to have a smaller and more malleable system, in that I agree with you, but that takes time and work, and I see few people that is diving in the insides of Pharo to help get us something in that direction. Not because they don't care but because it is difficult and time consuming.
Which direction should I follow after building a boostrap image following these instructions https://github.com/pharo-project/pharo#bootstrapping-pharo-from-sources ? -- Dr. Geo http://drgeo.eu
Hilaire I do not have the answer and I do not know but you should have a look at the bash script. Because this is where the logic is. I will discuss tomorrow with guille (right now I'm sick). Stef On Mon, Mar 12, 2018 at 9:34 PM, Hilaire <hilaire@drgeo.eu> wrote:
Le 12/03/2018 à 10:16, Guillermo Polito a écrit :
That said, I would also like to have a smaller and more malleable system, in that I agree with you, but that takes time and work, and I see few people that is diving in the insides of Pharo to help get us something in that direction. Not because they don't care but because it is difficult and time consuming.
Which direction should I follow after building a boostrap image following these instructions https://github.com/pharo-project/pharo#bootstrapping-pharo-from-sources ?
-- Dr. Geo http://drgeo.eu
I'll take a look at this from tomorrow morning till the end of the week. On Tue, Mar 13, 2018 at 7:22 PM, Stephane Ducasse <stepharo.self@gmail.com> wrote:
Hilaire
I do not have the answer and I do not know but you should have a look at the bash script. Because this is where the logic is. I will discuss tomorrow with guille (right now I'm sick). Stef
On Mon, Mar 12, 2018 at 9:34 PM, Hilaire <hilaire@drgeo.eu> wrote:
Le 12/03/2018 à 10:16, Guillermo Polito a écrit :
That said, I would also like to have a smaller and more malleable
system,
in that I agree with you, but that takes time and work, and I see few people that is diving in the insides of Pharo to help get us something in that direction. Not because they don't care but because it is difficult and time consuming.
Which direction should I follow after building a boostrap image following these instructions https://github.com/pharo-project/pharo#bootstrapping-pharo-from-sources ?
-- Dr. Geo http://drgeo.eu
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Thanks! Doru
On Mar 13, 2018, at 8:24 PM, Guillermo Polito <guillermopolito@gmail.com> wrote:
I'll take a look at this from tomorrow morning till the end of the week.
On Tue, Mar 13, 2018 at 7:22 PM, Stephane Ducasse <stepharo.self@gmail.com> wrote: Hilaire
I do not have the answer and I do not know but you should have a look at the bash script. Because this is where the logic is. I will discuss tomorrow with guille (right now I'm sick). Stef
On Mon, Mar 12, 2018 at 9:34 PM, Hilaire <hilaire@drgeo.eu> wrote:
Le 12/03/2018 à 10:16, Guillermo Polito a écrit :
That said, I would also like to have a smaller and more malleable system, in that I agree with you, but that takes time and work, and I see few people that is diving in the insides of Pharo to help get us something in that direction. Not because they don't care but because it is difficult and time consuming.
Which direction should I follow after building a boostrap image following these instructions https://github.com/pharo-project/pharo#bootstrapping-pharo-from-sources ?
-- Dr. Geo http://drgeo.eu
--
Guille Polito Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr
Web: http://guillep.github.io Phone: +33 06 52 70 66 13
-- www.tudorgirba.com www.feenk.com "Sometimes the best solution is not the best solution."
Simple explanation about the process will be a good start. For example what is a baseline in the image building? Like in the ConfigurationOf...? Le 13/03/2018 à 20:24, Guillermo Polito a écrit :
I'll take a look at this from tomorrow morning till the end of the week.
-- Dr. Geo http://drgeo.eu
Okay, may be I just understood wrongly what it is about, or my expectation are not correctly aligned. Le 13/03/2018 à 19:22, Stephane Ducasse a écrit :
Hilaire
I do not have the answer and I do not know but you should have a look at the bash script. Because this is where the logic is. I will discuss tomorrow with guille (right now I'm sick). Stef
-- Dr. Geo http://drgeo.eu
participants (7)
-
Esteban Lorenzano -
Guillermo Polito -
Hilaire -
Marcus Denker -
Sean P. DeNigris -
Stephane Ducasse -
Tudor Girba