[Pharo-project] GSoC - Package management with Fuel
Hi This week has ended Google Summer of Code program 2012. I have to thank Marcus and Carlos Ferro, my mentors, as well as Mariano, Guille Polito, Camillo Bruni, and others, who collaborated in different ways during these three months. Also thank, of course, to Carla and Janko... and Google ;) The main purpose of this mail is to summarize the work done for my project, named "Package management with Fuel" [1][2]. Tools for managing code in Pharo export classes, traits, and methods as series of source definitions, which then import by evaluating them with the compiler. This is basically how FileOut/FileIn or Monticello works. This is slow on large pieces of code, so if you want to install a big package, you need a lot of time compiling it. Why should you compile it again, given that it was all compiled when you exported the package? The bytecodes will be the same again, you know... If you just could load the Classes and Compiled Methods instead of compiling it all again, you can save time. With this goal in mind, I started working in FuelPackageLoader[4] (now Tanker[3]), a prototype we wrote last year, which already exports and imports packages without using any compiler. But it has two important issues: it wasn't capable of export source code, and it wasn't able to tolerate class-shape changes in hierarchies (ie, a change on inst vars in a external superclass). So, we decided to start from scratch with a new prototype, code name Carbono. The idea was to reify how FileOut and FileIn are done, but storing binary information so that compilation can be avoided. In a nutshell, Carbono stores a package into two files: - A sources file, imitating the traditional FileOut chunk format. - A binary file, where uses Fuel[5] to serialize a series of code definitions. Each of them knows the position in the sources file where its source code counterpart is. For example, a class definition knows the position of its comment (in the case it exists), and a method definition knows the position of its source code. A basic example, to export a couple of classes: TAExport new binariesOn: aBinaryStream sourcesOn: aTextStream; behaviors: {ClassA. ClassB}; methods: {}; initializeClasses: true; run. which then you can import with: TAImport new binariesFrom: aBinaryStream sourcesFrom: aTextStream; run. Anyway, you can also choose to: - export only binaries or only sources - import only binaries (source code can be decompiled) - import only sources (via #fileIn) Mariano Martinez Peck worked a lot on it, fixing bugs, and especially optimizing it A LOT, so I hope you will enjoy his demonstration this Monday on ESUG Innovation Awards. Seems that he could export Seaside, Pier and Magritte, and import them in another image, with source code, and just in a few seconds ;) Also, some ultra confidential WikiLeaks revealed that he could export almost all Pharo packages, and import them in a kernel image, all with source codes but without using the compiler :) Note1: Tolerating class-shape changes in hierarchies is not yet done, but it should in some days. So far, the only strategy when a change is detected will be to recompile methods, but it may be more sophisticated in a not-so-far future. Note2: Although finally it was not strictly necessary, Guillermo Polito and me wrote tests for a new ClassBuilder. It is part of Slot, a kind of reification of objects layout made by Toon Verwaest and Camillo Bruni. It is too long to explain it here, but the thing is that was not an easy job, and we hope it will be integrated in Pharo. This is a work in progress, but you can install and try it on latest Pharo 2.0, evaluating: Gofer it squeaksource3: 'PharoTaskForces'; package: 'Slot'; package: 'SlotTests'; load. (Smalltalk at: #SlotVirus) perform: #spread. Gofer it squeaksource3: 'PharoTaskForces'; package: 'SlotsClassBuilderIntegration'; load. Gofer new smalltalkhubUser: 'marianopeck' project: 'Tanker'; package: 'ConfigurationOfTanker'; load. (Smalltalk at: #ConfigurationOfTanker) loadBleedingEdge. Gofer new squeaksource: 'FuelExperiments'; package: 'Carbono'; package: 'CarbonoTests'; load. Cheers, MartÃn [1]: http://gsoc2012.esug.org/projects/fuel-packages [2]: https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2012/tinc... [3]: http://marianopeck.wordpress.com/2012/08/11/tanker-transporting-packages-wit... [4]: http://marianopeck.wordpress.com/2011/09/24/importing-and-exporting-packages... [5]: http://rmod.lille.inria.fr/web/pier/software/Fuel
Great work!!! Congrats. Saludos! NicoPaez <http://nicopaez.wordpress.com/> On Fri, Aug 24, 2012 at 5:01 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi
This week has ended Google Summer of Code program 2012. I have to thank Marcus and Carlos Ferro, my mentors, as well as Mariano, Guille Polito, Camillo Bruni, and others, who collaborated in different ways during these three months. Also thank, of course, to Carla and Janko... and Google ;)
The main purpose of this mail is to summarize the work done for my project, named "Package management with Fuel" [1][2].
Tools for managing code in Pharo export classes, traits, and methods as series of source definitions, which then import by evaluating them with the compiler. This is basically how FileOut/FileIn or Monticello works. This is slow on large pieces of code, so if you want to install a big package, you need a lot of time compiling it. Why should you compile it again, given that it was all compiled when you exported the package? The bytecodes will be the same again, you know... If you just could load the Classes and Compiled Methods instead of compiling it all again, you can save time.
With this goal in mind, I started working in FuelPackageLoader[4] (now Tanker[3]), a prototype we wrote last year, which already exports and imports packages without using any compiler. But it has two important issues: it wasn't capable of export source code, and it wasn't able to tolerate class-shape changes in hierarchies (ie, a change on inst vars in a external superclass). So, we decided to start from scratch with a new prototype, code name Carbono. The idea was to reify how FileOut and FileIn are done, but storing binary information so that compilation can be avoided.
In a nutshell, Carbono stores a package into two files: - A sources file, imitating the traditional FileOut chunk format. - A binary file, where uses Fuel[5] to serialize a series of code definitions. Each of them knows the position in the sources file where its source code counterpart is. For example, a class definition knows the position of its comment (in the case it exists), and a method definition knows the position of its source code.
A basic example, to export a couple of classes:
TAExport new binariesOn: aBinaryStream sourcesOn: aTextStream; behaviors: {ClassA. ClassB}; methods: {}; initializeClasses: true; run.
which then you can import with:
TAImport new binariesFrom: aBinaryStream sourcesFrom: aTextStream; run.
Anyway, you can also choose to: - export only binaries or only sources - import only binaries (source code can be decompiled) - import only sources (via #fileIn)
Mariano Martinez Peck worked a lot on it, fixing bugs, and especially optimizing it A LOT, so I hope you will enjoy his demonstration this Monday on ESUG Innovation Awards. Seems that he could export Seaside, Pier and Magritte, and import them in another image, with source code, and just in a few seconds ;) Also, some ultra confidential WikiLeaks revealed that he could export almost all Pharo packages, and import them in a kernel image, all with source codes but without using the compiler :)
Note1: Tolerating class-shape changes in hierarchies is not yet done, but it should in some days. So far, the only strategy when a change is detected will be to recompile methods, but it may be more sophisticated in a not-so-far future.
Note2: Although finally it was not strictly necessary, Guillermo Polito and me wrote tests for a new ClassBuilder. It is part of Slot, a kind of reification of objects layout made by Toon Verwaest and Camillo Bruni. It is too long to explain it here, but the thing is that was not an easy job, and we hope it will be integrated in Pharo.
This is a work in progress, but you can install and try it on latest Pharo 2.0, evaluating:
Gofer it squeaksource3: 'PharoTaskForces'; package: 'Slot'; package: 'SlotTests'; load. (Smalltalk at: #SlotVirus) perform: #spread. Gofer it squeaksource3: 'PharoTaskForces'; package: 'SlotsClassBuilderIntegration'; load. Gofer new smalltalkhubUser: 'marianopeck' project: 'Tanker'; package: 'ConfigurationOfTanker'; load. (Smalltalk at: #ConfigurationOfTanker) loadBleedingEdge. Gofer new squeaksource: 'FuelExperiments'; package: 'Carbono'; package: 'CarbonoTests'; load.
Cheers, MartÃn
[1]: http://gsoc2012.esug.org/projects/fuel-packages [2]: https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2012/tinc... [3]: http://marianopeck.wordpress.com/2012/08/11/tanker-transporting-packages-wit... [4]: http://marianopeck.wordpress.com/2011/09/24/importing-and-exporting-packages... [5]: http://rmod.lille.inria.fr/web/pier/software/Fuel
_______________________________________________ Esug-list mailing list Esug-list@lists.esug.org http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
participants (2)
-
Martin Dias -
Nicolás Paez