[Pharo-dev] [IMPORTANT] Following changes in the bootstrapping process

Mariano Martinez Peck marianopeck at gmail.com
Tue Aug 1 16:52:46 EDT 2017


On Tue, Aug 1, 2017 at 5:31 PM, tesonep at gmail.com <tesonep at gmail.com> wrote:

> Hi,
>     Yes for sure there are objects, and that is very powerful, but
> comparing the algorithm of serialization of Hermes with Fuel (I don't
> really know Parcels) is not possible. It is like comparing Filetree with
> Fuel, The problems they have to take care are completely different.
>
> Maybe I am not seeing the point.
>
>
Let me add one sentence that may explain Eliot questions... if you
serialize the class definition, then it means you would need the compiler.
I guess Eliot may be interested in NOT having the compiler at all and
instead bring classes directly from binary format (like Fuel / Tanker )




> Cheers.
>
> On Tue, Aug 1, 2017 at 10:25 PM, Eliot Miranda <eliot.miranda at gmail.com>
> wrote:
>
>> Hi Pablo,
>>
>> On Tue, Aug 1, 2017 at 12:36 PM, tesonep at gmail.com <tesonep at gmail.com>
>> wrote:
>>
>>> Hi again,
>>>    thanks for the clarification, I haven't understood the question
>>> initially, but now I think I can answer it.
>>>
>>> Hermes is only exporting the classes, not objects. so it does not have
>>> to handle complex graphs of objects. Basically it serializes the definition
>>> of the classes, traits and methods. Then they are loaded in sequence. The
>>> only caring during the generation is to serialize first the superclasses,
>>> and then the subclasses. There is no way of serializing objects outside the
>>> classes, methods, traits and literals in a method.
>>>
>>
>> Classes, traits, methods, literals *are* a graph of objects :-). The
>> Parcel architecture, from which Fuel derived its architecture, was designed
>> for loading code in VisualWorks.  In fact, last time I looked Parcels were
>> used only to store code and not as a general purpose (de)serializer.
>>
>> So the answer is that is not using Parcels or Fuel architecture because
>>> they are intended for different uses.
>>>
>>
>> Well, I don't think that's the reason ;-), but fine.  I understand that
>> it doesn't use this architecture.  Thanks.
>>
>>
>>> Also the format itself has no special design or considerations to be
>>> fast to generate or fast to read (as It happens with Fuel)
>>>
>>> Cheers,
>>> Pablo
>>>
>>> On Tue, Aug 1, 2017 at 9:16 PM, Eliot Miranda <eliot.miranda at gmail.com>
>>> wrote:
>>>
>>>> Hi Pablo,
>>>>
>>>>     I understand that Hermes has its own format.  My question was about
>>>> architecture. The Parcels/Fuel architecture is, AFAIA, the best
>>>> architecture for binary storage because it is virtual, is fast to load and
>>>> cleanly separates loading from initialization.  What do I mean?
>>>>
>>>> - concrete formats like ImageSegment are tied to internal
>>>> representations in the VM and do are difficult to read in forever FB
>>>> systems and very hard to write in foreign systems.  So a virtual format
>>>> (like BOSS, Fuel, Parcels, etc) is better but unless done well can be slow
>>>>
>>>> - a first generation system like BOSS is a flattening of a graph
>>>> traversal using a simple grammar.  Each phrase is either an object
>>>> reference (object id) or an object definition (object is followed by type
>>>> info and contents, contents being a sequence of phrases.  This is slow to
>>>> parse (because each phrase must be decoded), and has invalid intermediate
>>>> states (e.g. a Set whose contents are not all present, hence whose external
>>>> hash may change during loading, etc).  A second generation system like
>>>> Parcels and Fuel separates objects from references (nodes from arcs) and
>>>> batches object creation, grouping all instances of a given class for bulk,
>>>> and hence fast, instantiation.  Following my the objects are the
>>>> references.  So loading has three phases:
>>>> - instantiate the objects
>>>> - connect the graph by filling in object references
>>>> - initialize the objects that require it (e.g. rehash sets)
>>>>
>>>> Consequently the writer must be two pass, the first pass to collect the
>>>> objects in the graph and sort them by class, the second to write the nodes
>>>> followed by the references
>>>>
>>>> So let me ask again, does Hermes use the Parcels/Fuel architecture?
>>>>
>>>> _,,,^..^,,,_ (phone)
>>>>
>>>> On Aug 1, 2017, at 11:52 AM, "tesonep at gmail.com" <tesonep at gmail.com>
>>>> wrote:
>>>>
>>>> Hi Eliot,
>>>>
>>>> The last version of Hermes, that I have generated today and have to be
>>>> tested for real yet, is able to load and save from and to 32 and 64 bits,
>>>> handling the conversion. Because the Hermes representation (if we can call
>>>> it like that, because is really simple, does not care about the
>>>> architecture).
>>>>
>>>> Hermes is using a custom format completely separated from Fuel or any
>>>> other tool.
>>>>
>>>> This is a consequence of the design of Hermes. Hermes has been designed
>>>> to only work as light way of loading code during the bootstrap. So that, it
>>>> is heavily coupled with Pharo (it can be changed, for example implementing
>>>> another loader / installer) and uses only really core classes.
>>>>
>>>> Of course it can be extended and used in another Smalltalk dialects,
>>>> but I am not sure if the tool can be useful for them, or even for the Pharo
>>>> community outside the Pharo bootstrap process.
>>>>
>>>> Today to be useful outside the bootstrap process, some work has to be
>>>> done (perhaps a lot), but mainly new use cases have to be thinked. Once the
>>>> Compiler, Monticello or Metacello is available, there is no need for
>>>> Hermes.  It is only used to generate some binary loadable packages for the
>>>> compiler from the source code.
>>>>
>>>> Again, it was only thinked as a tool to improve the speed during the
>>>> bootstrap. However, if somebody has ideas to use it or want to collaborate
>>>> are  of course welcomed.
>>>>
>>>> Hermes is actually in Github, because to me is more comfortable to have
>>>> it there,  but if somebody is really wanting to use it.... we can manage to
>>>> have an export process to Smalltalkhub.
>>>>
>>>> I hope I have clarified a little the idea behind Hermes.
>>>>
>>>> Maybe you have better ideas of what can we do with it.
>>>>
>>>>
>>>> On Tue, Aug 1, 2017 at 7:59 PM, Eliot Miranda <eliot.miranda at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Guille,
>>>>>
>>>>> On Aug 1, 2017, at 4:29 AM, Guillermo Polito <
>>>>> guillermopolito at gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 1, 2017 at 12:57 PM, philippe.back at highoctane.be <
>>>>> philippe.back at gmail.com> wrote:
>>>>>
>>>>>> Massive.
>>>>>>
>>>>>> What is in the super small image?
>>>>>> Is Hermes going to be a generic binary content loader?
>>>>>>
>>>>>
>>>>> There is still work to do in this front.
>>>>>  - Hermes only works for 32bits format. We should adapt it to 64 bits
>>>>> (immediate floats and so on...)
>>>>>  - There is no format validation. We should add one for safety.
>>>>>
>>>>>
>>>>> Will Hermes be able to save on 32-bits and load on 64-bits and vice
>>>>> verse?
>>>>>
>>>>> Does Hermes use the Parcels/Fuel architecture of saving nodes,
>>>>> grouped by class, followed by the arcs?
>>>>>
>>>>> If yes to both of these, are you willing to keep it in Monticello and
>>>>> collaborate with the Squeak & Cuis communities in developing Hermes?
>>>>>
>>>>>
>>>>>
>>>>>> Phil
>>>>>>
>>>>>> On Aug 1, 2017 12:36, "Stephane Ducasse" <stepharo.self at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Pavel
>>>>>>>
>>>>>>> This is super excellent! IMPRESSIVE. An image without the compiler
>>>>>>> and
>>>>>>> a reloadable compiler.
>>>>>>> Super cool.
>>>>>>>
>>>>>>> Stef
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 1, 2017 at 11:57 AM, Pavel Krivanek
>>>>>>> <pavel.krivanek at gmail.com> wrote:
>>>>>>> > Hello,
>>>>>>> >
>>>>>>> > we are checking a huge pull request #177
>>>>>>> > (https://github.com/pharo-project/pharo/pull/177) that will
>>>>>>> change some
>>>>>>> > basics of the bootstrap process:
>>>>>>> >
>>>>>>> > Now we will bootstrap a smaller image that will not include
>>>>>>> compiler/parser.
>>>>>>> > Compiler and related packages are exported and loaded using a
>>>>>>> binary
>>>>>>> > exporter named Hermes.
>>>>>>> > The compiler is then used to load FileSystem and Monticello. The
>>>>>>> rest of the
>>>>>>> > bootstrap process will be the same as before.
>>>>>>> > As the result we will have faster bootstrap and better system
>>>>>>> modularization
>>>>>>> > and possibilities.
>>>>>>> >
>>>>>>> > It required some modularization efforts:
>>>>>>> >
>>>>>>> > - simplified initialization scripts
>>>>>>> > - Use Zinc converters and encoders instead of FilePathEncoder and
>>>>>>> old
>>>>>>> > TextConverter
>>>>>>> > - Use Stdio instead of FileStream
>>>>>>> > - Using File instead of FileSystem
>>>>>>> > - Deprecated FileStream & childs (Moved to Deprecated70)
>>>>>>> > - Extracted Path classes to their on package: FileSystem-Path
>>>>>>> > - Moved OpalEncoders to their own package. They are required by
>>>>>>> the runtime
>>>>>>> > (not only for compilation)
>>>>>>> > - Introduced AsciiCharset in the kernel to answer to #isLetter
>>>>>>> #isUppercase
>>>>>>> > and so on without requiring full Unicode tables from the beginning
>>>>>>> > - Cleaning up a bit the full exception logging infrastructure
>>>>>>> (streams,
>>>>>>> > transcript, files, stack printing...)
>>>>>>> > - Split Ring methods required for system navigation to the
>>>>>>> Ring-Navigation
>>>>>>> > package
>>>>>>> > - Remove usages of #translated in the kernel
>>>>>>> > - Refactored the bootstrapping classes to remove duplications
>>>>>>> > - Cleaning up dependencies in CompiledMethod>>printOn:
>>>>>>> > - fix path printing
>>>>>>> >
>>>>>>> > We need to merge these changes at once and of course it can cause
>>>>>>> some
>>>>>>> > conflicts with the existing pull requests or external code.
>>>>>>> Anyway, we need
>>>>>>> > to merge it as soon as possible.
>>>>>>> >
>>>>>>> > So please, try to look at the PR and test the resultant image [1]
>>>>>>> to avoid
>>>>>>> > some major problems.
>>>>>>> >
>>>>>>> > [1]
>>>>>>> > https://ci.inria.fr/pharo/view/7.0/job/70-PR-Check-Load/last
>>>>>>> SuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-9c0691d.zip
>>>>>>> >
>>>>>>> > Cheers,
>>>>>>> > -- Pavel
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>> Guille Polito
>>>>>
>>>>>
>>>>> Research Engineer
>>>>>
>>>>> French National Center for Scientific Research - *http://www.cnrs.fr*
>>>>> <http://www.cnrs.fr>
>>>>>
>>>>>
>>>>>
>>>>> *Web:* *http://guillep.github.io* <http://guillep.github.io>
>>>>>
>>>>> *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Pablo Tesone.
>>>> tesonep at gmail.com
>>>>
>>>>
>>>
>>>
>>> --
>>> Pablo Tesone.
>>> tesonep at gmail.com
>>>
>>
>>
>>
>> --
>> _,,,^..^,,,_
>> best, Eliot
>>
>
>
>
> --
> Pablo Tesone.
> tesonep at gmail.com
>



-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20170801/4a39763e/attachment.html>


More information about the Pharo-dev mailing list