Without entering in details, a cause for slow package write is dumping all every time.

For that strategy, we already have the image save which is magically fast.

So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes




On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:



Le 11/12/2013 16:27, Esteban Lorenzano a �crit :
ah, and IMHO the problem is not about reading... is about writing (if it
has to write the metadata each time...).

But, personnaly, I don't know if this is the reason for the lack of performance...

I have three hypothesis for Sebastian problem:
1 - Slow read time for version metadata
 - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser.
2 - Slow metadata write
3 - Slow package write

I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-

So I'd really like to check if 3- is confirmed ;)

Thierry


Esteban


On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com
<mailto:estebanlm@gmail.com>> wrote:

   Thierry, I know there is a working version... let me search...

   (5 mins later)


   here:

   https://github.com/rjsargent/CypressReferenceImplementation

   Dale says Richard made a metadata-less version.

   We should take a look at that.

   Esteban


   On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry
   <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:

       Esteban, Sebastian,

       In the filetree code, you will find a format without metadata,
       but it's not in use anymore.

       If you use gitfiletree, it will write the metadata for
       compatibility reasons with filetree, but it will never read it back.

       I'm pushing code to make filetree robust to absence of metadata,
       but I haven't worked on it for a while.

       gitfiletree has solved the problem of a slow metadata read. It
       does not solve any performance problem associated with writing, yet.

       Thierry

       Le 11/12/2013 16:12, Esteban Lorenzano a �crit :

           I know there is a version of filetree without metadata (more
           compelling
           for projects that will never use other formats).
           Dale told me that there was a preview somewhere, but I
           didn't tested yet
           (lack of time) and now I cannot find the mail...
           Dale, can you re-send the link?

           cheers,
           Esteban


           On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre
           <sebastian@flowingconcept.com
           <mailto:sebastian@flowingconcept.com>
           <mailto:sebastian@__flowingconcept.com
           <mailto:sebastian@flowingconcept.com>>> wrote:

                I should breath before I type, but you probably already
           got that I
                meant /redundant writes/ (not reads)...


                Anyway.. I was talking with Esteban and he mentions
           some kind of
                compatibility metadata.

                If I'm going to give a leap of faith to filetree repos
           to save code
                why should I care about mcz compatibility? Paying a
           toll for no
                reason is evil.

                Maybe we could make that optional so those who don't
           extract value
                from that feature can opt-out?

                sebastian <https://about.me/__sebastianconcept
           <https://about.me/sebastianconcept>>


                o/





                On Dec 11, 2013, at 12:44 PM, Sebastian Sastre
                <sebastian@flowingconcept.com
           <mailto:sebastian@flowingconcept.com>
           <mailto:sebastian@__flowingconcept.com
           <mailto:sebastian@flowingconcept.com>>>
                wrote:

                    Hi Thierry

                    On Dec 11, 2013, at 12:43 PM, Goubier Thierry
                    <thierry.goubier@cea.fr
               <mailto:thierry.goubier@cea.fr>
               <mailto:thierry.goubier@cea.fr
               <mailto:thierry.goubier@cea.fr>__>> wrote:


                            I have packages (in the order of hundreds
                       of classes) and save
                            delays
                            and package click delays are starting to
                       demand patience in a
                            way that
                            doesn't feel like the right path


                        Which operations ? I didn't remember noticing
                   much with 179
                        classes on a laptop without a SSD.


                    choose one. Just for clicking the package that will
               should you
                    UUID, version and author I need to wait ~16
               seconds. Sounds like a
                    lot of overhead for reading a small .json file.

                    But the write is the most worrisome


                            All that is with a SSD disk, otherwise save
                       delays would be
                            /way/ beyond
                            unacceptable


                        I'd like to know more, and understand the
                   reason, for sure. As
                        far as I know, filetree will rewrite the whole
                   package to disk
                        everytime... and maybe optimising that could be
                   the solution.


                    Well, that explains a lot. Writing all every time
               is the lazy
                    thing that's okay for a prototype and temporary
               code in a proof of
                    concept but that massive redundant reads certainly
               doesn't sounds
                    like pro software. Specially for SSD's which has a
               limited
                    quantity of writes


                        Thierry

                            sebastian
                       <https://about.me/__sebastianconcept
                       <https://about.me/sebastianconcept>>

                            o/






                        --
                        Thierry Goubier
                        CEA list
                        Laboratoire des Fondations des Syst�mes Temps
                   R�el Embarqu�s
                        91191 Gif sur Yvette Cedex
                        France
                        Phone/Fax: +33 (0) 1 69 08 32 92
                   <tel:%2B33%20%280%29%201%2069%2008%2032%2092>
                        <tel:%2B33%20%280%29%201%2069%__2008%2032%2092>
                   / 83 95






       --
       Thierry Goubier
       CEA list
       Laboratoire des Fondations des Syst�mes Temps R�el Embarqu�s
       91191 Gif sur Yvette Cedex
       France
       Phone/Fax: +33 (0) 1 69 08 32 92
       <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95




-- 
Thierry Goubier
CEA list
Laboratoire des Fondations des Syst�mes Temps R�el Embarqu�s
91191 Gif sur Yvette Cedex
France
Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95