Re: [Pharo-dev] Support for git in Pharo
Max wrote:
I donât quite follow. What do you mean by methods?
When mapping methods to files, there is the problem that on disk files take a minimum amount of space, often 4KB. This slows writing a package to disk down a lot. Stephan
On 06.03.2014, at 15:53, Stephan Eggermont <stephan@stack.nl> wrote:
Max wrote:
I donât quite follow. What do you mean by methods?
When mapping methods to files, there is the problem that on disk files take a minimum amount of space, often 4KB. This slows writing a package to disk down a lot.
I see. There are two things to consider: 1. the format we use to store source 2. the fact that Git will create one blob per file Obviously, by choosing a source storage format that avoids this problem, we donât have to deal with the second issue. If you go with something like FileTree, which uses a single file per method, this changes. By combining multiple files (e.g. by zipping) we could avoid the problem of tiny blobs but we would loose the ability to look at the repository from the outside. This is something I donât want to do. It creates a dependency to Pharo (and to the particular implementation of how those files are stored) and unneeded complexity (especially with regard to maintenance). In my opinion there are other factors, like readability, maintenance etc, that are maybe more important than speed. So yes, potentially we could do something about fragmentation but I wonât promise you anything. Does that answer your question? Cheers, Max
Stephan
Le 06/03/2014 16:14, Max Leske a écrit :
On 06.03.2014, at 15:53, Stephan Eggermont <stephan@stack.nl> wrote:
Max wrote:
I donât quite follow. What do you mean by methods?
When mapping methods to files, there is the problem that on disk files take a minimum amount of space, often 4KB. This slows writing a package to disk down a lot.
I see. There are two things to consider: 1. the format we use to store source 2. the fact that Git will create one blob per file
Git will pack blobs in a kind of archive file, in .pack files.
Obviously, by choosing a source storage format that avoids this problem, we donât have to deal with the second issue. If you go with something like FileTree, which uses a single file per method, this changes. By combining multiple files (e.g. by zipping) we could avoid the problem of tiny blobs but we would loose the ability to look at the repository from the outside. This is something I donât want to do. It creates a dependency to Pharo (and to the particular implementation of how those files are stored) and unneeded complexity (especially with regard to maintenance).
+1.
In my opinion there are other factors, like readability, maintenance etc, that are maybe more important than speed.
I agree with that :) Thierry
So yes, potentially we could do something about fragmentation but I wonât promise you anything. Does that answer your question?
Cheers, Max
Stephan
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
On 06.03.2014, at 16:25, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Le 06/03/2014 16:14, Max Leske a écrit :
On 06.03.2014, at 15:53, Stephan Eggermont <stephan@stack.nl> wrote:
Max wrote:
I donât quite follow. What do you mean by methods?
When mapping methods to files, there is the problem that on disk files take a minimum amount of space, often 4KB. This slows writing a package to disk down a lot.
I see. There are two things to consider: 1. the format we use to store source 2. the fact that Git will create one blob per file
Git will pack blobs in a kind of archive file, in .pack files.
That is only partly true: - packs will be generated when objects are transferred over the network (this may be partial packs) - packs can be force generated by using âgit gcâ - packs will be periodically created / updated to reduce size on disk But: the main storage form (and the fastest for access) are loose objects (i.e. single files) Just for completeness :) Max
Obviously, by choosing a source storage format that avoids this problem, we donât have to deal with the second issue. If you go with something like FileTree, which uses a single file per method, this changes. By combining multiple files (e.g. by zipping) we could avoid the problem of tiny blobs but we would loose the ability to look at the repository from the outside. This is something I donât want to do. It creates a dependency to Pharo (and to the particular implementation of how those files are stored) and unneeded complexity (especially with regard to maintenance).
+1.
In my opinion there are other factors, like readability, maintenance etc, that are maybe more important than speed.
I agree with that :)
Thierry
So yes, potentially we could do something about fragmentation but I wonât promise you anything. Does that answer your question?
Cheers, Max
Stephan
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Le 06/03/2014 17:30, Max Leske a écrit :
That is only partly true: - packs will be generated when objects are transferred over the network (this may be partial packs) - packs can be force generated by using âgit gcâ - packs will be periodically created / updated to reduce size on disk
Are you free, when fetching a git repository, to decide whether to pack or not or is the source repository in charge of that? Past a certain point, you can expect all/most of your small files (and small deltas) to be packed, no?
But: the main storage form (and the fastest for access) are loose objects (i.e. single files)
But you have to support both storage form to be able to read a git repository.
Just for completeness :)
You known certainly more about that than I do :) Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
On 07.03.2014, at 09:01, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Le 06/03/2014 17:30, Max Leske a écrit :
That is only partly true: - packs will be generated when objects are transferred over the network (this may be partial packs) - packs can be force generated by using âgit gcâ - packs will be periodically created / updated to reduce size on disk
Are you free, when fetching a git repository, to decide whether to pack or not or is the source repository in charge of that?
Network transfer is always packed.
Past a certain point, you can expect all/most of your small files (and small deltas) to be packed, no?
No, unless you impelement something like post hook to force âgit gcâ. New / changed objects will be stored as loose objects. If you consider a large repository, than maybe âthe larger partâ of all objects may be packed (objects that are only read) but that very much depends on the usage of the repository (e.g. if you have to compile stuff there will be tons of new objects every time).
But: the main storage form (and the fastest for access) are loose objects (i.e. single files)
But you have to support both storage form to be able to read a git repository.
Yes. Thatâs a bit of a downside. But it makes things interesting :)
Just for completeness :)
You known certainly more about that than I do :)
Well, I should, considering the amount of time Iâve spent with Git (and itâs totally useless âthis is an excerpt from the IRC where Linus explains how Git really worksâ documentationâ¦) :p Max
Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
participants (3)
-
Goubier Thierry -
Max Leske -
Stephan Eggermont