Tell me about your workflow
Are you using git with your Pharo based software? If you do, are you using filetree? I ask because I'm worried about scale. I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path All that is with a SSD disk, otherwise save delays would be way beyond unacceptable sebastian o/
Le 11/12/2013 15:18, Sebastian Sastre a écrit :
Are you using git with your Pharo based software?
Yes
If you do, are you using filetree?
gitfiletree (it uses the filetree code path for writing, and a git archive path for reading)
I ask because I'm worried about scale.
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Thierry
sebastian <https://about.me/sebastianconcept>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/sebastianconcept>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
I should breath before I type, but you probably already got that I meant redundant writes (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/sebastianconcept>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre < sebastian@flowingconcept.com> wrote:
I should breath before I type, but you probably already got that I meant *redundant writes* (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre < sebastian@flowingconcept.com> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/sebastianconcept>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/sebastianconcept>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr>wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be
/way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/sebastianconcept>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com>wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr>wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be
/way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/sebastianconcept>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a écrit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a écrit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a écrit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0?
Regards,
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so.
about 2, I don't know. I'm available to make tests and measure results
have a nice trip, keep us tuned about any progress
On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
I gave up running gitfiletree on 1.4 :( It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a écrit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a écrit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a écrit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Are you interested in a package or a project? I can provide you information based on size, etc⦠Uko On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I gave up running gitfiletree on 1.4 :(
It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue.
My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough?
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
gee the big code package is airflowing which I have, quite conservatively, running on #14438 images
I load filetree like this:
Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load.
and it never complained
let me know
On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0?
Regards,
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so.
about 2, I don't know. I'm available to make tests and measure results
have a nice trip, keep us tuned about any progress
On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits). I'll keep in mind the idea of creating one randomly ;) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow Are you interested in a package or a project? I can provide you information based on size, etc⦠Uko On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I gave up running gitfiletree on 1.4 :( It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a écrit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a écrit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a écrit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :) Uko On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits).
I'll keep in mind the idea of creating one randomly ;)
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
Are you interested in a package or a project? I can provide you information based on size, etcâ¦
Uko
On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I gave up running gitfiletree on 1.4 :(
It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue.
My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough?
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
gee the big code package is airflowing which I have, quite conservatively, running on #14438 images
I load filetree like this:
Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load.
and it never complained
let me know
On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0?
Regards,
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so.
about 2, I don't know. I'm available to make tests and measure results
have a nice trip, keep us tuned about any progress
On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
So if you want something big and with a lot of commits you can use Pharo* in general. Pharo10 has the most versions and Pharo30Inbox is the largest one. If you want some other projects then you heve to take a look at Seaside30, Mondrian, Moose, Glamour or Roassal. Uko On 12 Dec 2013, at 16:20, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :)
Uko
On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits).
I'll keep in mind the idea of creating one randomly ;)
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
Are you interested in a package or a project? I can provide you information based on size, etcâ¦
Uko
On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I gave up running gitfiletree on 1.4 :(
It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue.
My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough?
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
gee the big code package is airflowing which I have, quite conservatively, running on #14438 images
I load filetree like this:
Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load.
and it never complained
let me know
On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0?
Regards,
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so.
about 2, I don't know. I'm available to make tests and measure results
have a nice trip, keep us tuned about any progress
On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Thanks for the pointers. I'll look at Seaside/Moose/Mondrian and Roassal, because I need code I can load and save in an image without destroying the very image I use to test (which would happen if I load Pharo10 stuff in a 3.0 image ;) ). Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com] Date d'envoi : jeudi 12 décembre 2013 16:24 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow So if you want something big and with a lot of commits you can use Pharo* in general. Pharo10 has the most versions and Pharo30Inbox is the largest one. If you want some other projects then you heve to take a look at Seaside30, Mondrian, Moose, Glamour or Roassal. Uko On 12 Dec 2013, at 16:20, Yuriy Tymchuk <yuriy.tymchuk@me.com<mailto:yuriy.tymchuk@me.com>> wrote: Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :) Uko On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits). I'll keep in mind the idea of creating one randomly ;) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com<mailto:yuriy.tymchuk@me.com>] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow Are you interested in a package or a project? I can provide you information based on size, etc⦠Uko On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I gave up running gitfiletree on 1.4 :( It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a écrit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a écrit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a écrit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
And yes, we miss this data on Smalltalkhub. But I have some suggestions as always :). Will reveal them next week. Uko On 12 Dec 2013, at 16:20, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :)
Uko
On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits).
I'll keep in mind the idea of creating one randomly ;)
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
Are you interested in a package or a project? I can provide you information based on size, etcâ¦
Uko
On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I gave up running gitfiletree on 1.4 :(
It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue.
My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough?
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
gee the big code package is airflowing which I have, quite conservatively, running on #14438 images
I load filetree like this:
Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load.
and it never complained
let me know
On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0?
Regards,
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so.
about 2, I don't know. I'm available to make tests and measure results
have a nice trip, keep us tuned about any progress
On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
try with Moose. Stef On Dec 12, 2013, at 4:20 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :)
Uko
On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits).
I'll keep in mind the idea of creating one randomly ;)
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
Are you interested in a package or a project? I can provide you information based on size, etcâ¦
Uko
On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I gave up running gitfiletree on 1.4 :(
It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue.
My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough?
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
gee the big code package is airflowing which I have, quite conservatively, running on #14438 images
I load filetree like this:
Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load.
and it never complained
let me know
On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0?
Regards,
Thierry
De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so.
about 2, I don't know. I'm available to make tests and measure results
have a nice trip, keep us tuned about any progress
On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Then I will :) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Stéphane Ducasse [stephane.ducasse@inria.fr] Date d'envoi : vendredi 13 décembre 2013 13:47 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow try with Moose. Stef On Dec 12, 2013, at 4:20 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com<mailto:yuriy.tymchuk@me.com>> wrote: Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :) Uko On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits). I'll keep in mind the idea of creating one randomly ;) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com<mailto:yuriy.tymchuk@me.com>] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow Are you interested in a package or a project? I can provide you information based on size, etc⦠Uko On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I gave up running gitfiletree on 1.4 :( It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a écrit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a écrit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a écrit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Stef, how do I load Moose in a 3.0 image? There is no Moose config in the 3.0 meta repository. Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Stéphane Ducasse [stephane.ducasse@inria.fr] Date d'envoi : vendredi 13 décembre 2013 13:47 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow try with Moose. Stef On Dec 12, 2013, at 4:20 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com<mailto:yuriy.tymchuk@me.com>> wrote: Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :) Uko On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits). I'll keep in mind the idea of creating one randomly ;) Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com<mailto:yuriy.tymchuk@me.com>] Date d'envoi : jeudi 12 décembre 2013 15:37 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow Are you interested in a package or a project? I can provide you information based on size, etc⦠Uko On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: I gave up running gitfiletree on 1.4 :( It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue. My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough? Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : jeudi 12 décembre 2013 12:12 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow gee the big code package is airflowing which I have, quite conservatively, running on #14438 images I load filetree like this: Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load. and it never complained let me know On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0? Regards, Thierry ________________________________ De : Pharo-dev [pharo-dev-bounces@lists.pharo.org<mailto:pharo-dev-bounces@lists.pharo.org>] de la part de Sebastian Sastre [sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com>] Date d'envoi : mercredi 11 décembre 2013 17:09 à : Pharo Development List Objet : Re: [Pharo-dev] Tell me about your workflow ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so. about 2, I don't know. I'm available to make tests and measure results have a nice trip, keep us tuned about any progress On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr>> wrote: Yes, you're right in the general case. But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing. If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3- (But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ). Thierry Le 11/12/2013 16:44, Sebastian Sastre a écrit : Without entering in details, a cause for slow package write is dumping all every time. For that strategy, we already have the image save which is magically fast. So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes sebastian <https://about.me/sebastianconcept> o/ On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr>> wrote: Le 11/12/2013 16:27, Esteban Lorenzano a écrit : ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...). But, personnaly, I don't know if this is the reason for the lack of performance... I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3- So I'd really like to check if 3- is confirmed ;) Thierry Esteban On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com<mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com> <mailto:estebanlm@gmail.com>> wrote: Thierry, I know there is a working version... let me search... (5 mins later) here: https://github.com/rjsargent/CypressReferenceImplementation Dale says Richard made a metadata-less version. We should take a look at that. Esteban On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr><mailto:thierry.goubier@cea.fr>> wrote: Esteban, Sebastian, In the filetree code, you will find a format without metadata, but it's not in use anymore. If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back. I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while. gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet. Thierry Le 11/12/2013 16:12, Esteban Lorenzano a écrit : I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link? cheers, Esteban On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)... Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata. If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil. Maybe we could make that optional so those who don't extract value from that feature can opt-out? sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com<mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote: Hi Thierry On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr<mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote: I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD. choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file. But the write is the most worrisome All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution. Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes Thierry sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>> o/ -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95 -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Hi, By following the instructions from any of the Moose webpages: http://www.moosetechnology.org/download/pharo http://www.smalltalkhub.com/#!/~Moose/Moose Gofer new smalltalkhubUser: 'Moose' project: 'Moose'; package: 'ConfigurationOfMoose'; load. (Smalltalk at: #ConfigurationOfMoose) perform: #loadDevelopment Cheers, Doru On Sat, Dec 14, 2013 at 9:01 AM, GOUBIER Thierry <thierry.goubier@cea.fr>wrote:
Stef,
how do I load Moose in a 3.0 image? There is no Moose config in the 3.0 meta repository.
Thierry ------------------------------ *De :* Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Stéphane Ducasse [stephane.ducasse@inria.fr] *Date d'envoi :* vendredi 13 décembre 2013 13:47
*Ã :* Pharo Development List *Objet :* Re: [Pharo-dev] Tell me about your workflow
try with Moose.
Stef
On Dec 12, 2013, at 4:20 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Pharo10 on SmalltalkHub is humongous. You can definitely do a stress test with it :)
Uko
On 12 Dec 2013, at 15:43, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I would need a large project, composed of one or more packages, with more than 150~200 classes, which triggers the slow read and writing times Sebastian experience. And, probably, to be complete, a long and complex commit history in git (> 100 commits).
I'll keep in mind the idea of creating one randomly ;)
Thierry
------------------------------ *De :* Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Yuriy Tymchuk [yuriy.tymchuk@me.com] *Date d'envoi :* jeudi 12 décembre 2013 15:37 *à :* Pharo Development List *Objet :* Re: [Pharo-dev] Tell me about your workflow
Are you interested in a package or a project? I can provide you information based on size, etcâ¦
Uko
On 12 Dec 2013, at 15:30, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
I gave up running gitfiletree on 1.4 :(
It's possible to use gitfiletree from a 2.0 or a 3.0 image to browse your git repository, but testing the writing will be an issue.
My best chance would be to find a large enough package I can use on 2.0 or 3.0 to test and profile. Does anybody has a large enough package which could fit? Anything that doesn't require a NDA to read it, of course. Is Roassal large enough?
Thierry
------------------------------ *De :* Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] *Date d'envoi :* jeudi 12 décembre 2013 12:12 *à :* Pharo Development List *Objet :* Re: [Pharo-dev] Tell me about your workflow
gee the big code package is airflowing which I have, quite conservatively, running on #14438 images
I load filetree like this:
Gofer new url: 'http://ss3.gemstone.com/ss/FileTree'; package: 'ConfigurationOfFileTree'; load. ((Smalltalk at: #ConfigurationOfFileTree) project version: #'stable') load.
and it never complained
let me know
On Dec 12, 2013, at 3:53 AM, GOUBIER Thierry <thierry.goubier@cea.fr> wrote:
If you would be ready to profile a package save on your repository, it would be great. In the mean time, I'll make available a special gitfiletree package to test. Which version of Pharo you are using? 2.0 or 3.0?
Regards,
Thierry
------------------------------ *De :* Pharo-dev [pharo-dev-bounces@lists.pharo.org] de la part de Sebastian Sastre [sebastian@flowingconcept.com] *Date d'envoi :* mercredi 11 décembre 2013 17:09 *à :* Pharo Development List *Objet :* Re: [Pharo-dev] Tell me about your workflow
ok, if saving is dumping all, then 3 is confirmed? After the first commit, I'd say so.
about 2, I don't know. I'm available to make tests and measure results
have a nice trip, keep us tuned about any progress
On Dec 11, 2013, at 2:09 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
Yes, you're right in the general case.
But a solution to that general problem will take time to be implemented (time I lack at the moment, sadly) and if the main gain is a few % because it's writing the version file and the metadata for methods which are the "slow" factors, then we'll have worked hard for nothing.
If you want to help, I'd really like to see either 2- or 3- confirmed. I can produce a special gitfiletree to remove writing the metadata, that you can try on a large project temporary copy; if the slow writing (and reading) is confirmed, then this is 3-
(But I'm leaving on a trip tomorrow early, so I have no idea of when I'll have the time to do that :( ).
Thierry
Le 11/12/2013 16:44, Sebastian Sastre a écrit :
Without entering in details, a cause for slow package write is dumping all every time.
For that strategy, we already have the image save which is magically fast.
So, if we make something to scan the code and write only when it's different from what's on disk, then we would be preventing tons of redundant writes
sebastian <https://about.me/sebastianconcept>
o/
On Dec 11, 2013, at 1:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr <thierry.goubier@cea.fr>>> wrote:
Le 11/12/2013 16:27, Esteban Lorenzano a écrit :
ah, and IMHO the problem is not about reading... is about writing (if it has to write the metadata each time...).
But, personnaly, I don't know if this is the reason for the lack of performance...
I have three hypothesis for Sebastian problem: 1 - Slow read time for version metadata - Confirmed because of the 16 seconds wait time for reading the package metadata in the repository browser. 2 - Slow metadata write 3 - Slow package write
I have an implemented solution for 1-, a very easy to implement for 2-, and none yet for 3-
So I'd really like to check if 3- is confirmed ;)
Thierry
Esteban
On Wed, Dec 11, 2013 at 4:24 PM, Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com <estebanlm@gmail.com>> <mailto:estebanlm@gmail.com <estebanlm@gmail.com>>> wrote:
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr <thierry.goubier@cea.fr>>< mailto:thierry.goubier@cea.fr <thierry.goubier@cea.fr>>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com <sebastian@flowingconcept.com>> <mailto:sebastian@flowingconcept.com<sebastian@flowingconcept.com>
<mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com<sebastian@flowingconcept.com>>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com <sebastian@flowingconcept.com>> <mailto:sebastian@flowingconcept.com<sebastian@flowingconcept.com>
<mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com<sebastian@flowingconcept.com>
wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr <thierry.goubier@cea.fr>> <mailto:thierry.goubier@cea.fr <thierry.goubier@cea.fr>> <mailto:thierry.goubier@cea.fr <thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <thierry.goubier@cea.fr>>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
-- www.tudorgirba.com "Every thing has its own flow"
Thanks. I'll have a look. If this format solves the git merge conflicts, I'll be porting it. But I'll make sure first it doesn't have the performance problems Sebastian is telling about. Because if it is just removing writing the metadata in gitfiletree, it's a 5 minutes job :). Thierry Le 11/12/2013 16:24, Esteban Lorenzano a écrit :
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com <mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept <https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
that would be sooo cool. Thanks Thierry! Esteban On Wed, Dec 11, 2013 at 4:39 PM, Goubier Thierry <thierry.goubier@cea.fr>wrote:
Thanks. I'll have a look.
If this format solves the git merge conflicts, I'll be porting it. But I'll make sure first it doesn't have the performance problems Sebastian is telling about.
Because if it is just removing writing the metadata in gitfiletree, it's a 5 minutes job :).
Thierry
Le 11/12/2013 16:24, Esteban Lorenzano a écrit :
Thierry, I know there is a working version... let me search...
(5 mins later)
here:
https://github.com/rjsargent/CypressReferenceImplementation
Dale says Richard made a metadata-less version.
We should take a look at that.
Esteban
On Wed, Dec 11, 2013 at 4:28 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr>> wrote:
Esteban, Sebastian,
In the filetree code, you will find a format without metadata, but it's not in use anymore.
If you use gitfiletree, it will write the metadata for compatibility reasons with filetree, but it will never read it back.
I'm pushing code to make filetree robust to absence of metadata, but I haven't worked on it for a while.
gitfiletree has solved the problem of a slow metadata read. It does not solve any performance problem associated with writing, yet.
Thierry
Le 11/12/2013 16:12, Esteban Lorenzano a écrit :
I know there is a version of filetree without metadata (more compelling for projects that will never use other formats). Dale told me that there was a preview somewhere, but I didn't tested yet (lack of time) and now I cannot find the mail... Dale, can you re-send the link?
cheers, Esteban
On Wed, Dec 11, 2013 at 4:08 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com
<mailto:sebastian@flowingconcept.com>>> wrote:
I should breath before I type, but you probably already got that I meant /redundant writes/ (not reads)...
Anyway.. I was talking with Esteban and he mentions some kind of compatibility metadata.
If I'm going to give a leap of faith to filetree repos to save code why should I care about mcz compatibility? Paying a toll for no reason is evil.
Maybe we could make that optional so those who don't extract value from that feature can opt-out?
sebastian <https://about.me/__sebastianconcept
<https://about.me/sebastianconcept>>
o/
On Dec 11, 2013, at 12:44 PM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com> <mailto:sebastian@__flowingconcept.com
<mailto:sebastian@flowingconcept.com>>> wrote:
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr <mailto:thierry.goubier@cea.fr> <mailto:thierry.goubier@cea.fr
<mailto:thierry.goubier@cea.fr>__>> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
But the write is the most worrisome
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Thierry
sebastian <https://about.me/__sebastianconcept
<https://about.me/sebastianconcept>>
o/
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> <tel:%2B33%20%280%29%201%2069%__2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 <tel:%2B33%20%280%29%201%2069%2008%2032%2092> / 83 95
-- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
Le 11/12/2013 15:44, Sebastian Sastre a écrit :
Hi Thierry
On Dec 11, 2013, at 12:43 PM, Goubier Thierry <thierry.goubier@cea.fr> wrote:
I have packages (in the order of hundreds of classes) and save delays and package click delays are starting to demand patience in a way that doesn't feel like the right path
Which operations ? I didn't remember noticing much with 179 classes on a laptop without a SSD.
choose one. Just for clicking the package that will should you UUID, version and author I need to wait ~16 seconds. Sounds like a lot of overhead for reading a small .json file.
This operation is fast with gitfiletree because I cache the commit history and avoid reading the version file everytime: the version reader is very slow, especially with large version files. You're using FileTree and not gitfiletree ? gitfiletree may solve the reading problem, if it fits with your workflow (and it reads cleanly pre-existing git/filetree repository without changing them).
But the write is the most worrisome
This is something I haven't found a need to optimise yet, so maybe you're on something important.
All that is with a SSD disk, otherwise save delays would be /way/ beyond unacceptable
I'd like to know more, and understand the reason, for sure. As far as I know, filetree will rewrite the whole package to disk everytime... and maybe optimising that could be the solution.
Well, that explains a lot. Writing all every time is the lazy thing that's okay for a prototype and temporary code in a proof of concept but that massive redundant reads certainly doesn't sounds like pro software. Specially for SSD's which has a limited quantity of writes
Well, it ensures correctness and simplify the code ;) When it is confirmed to be a bottleneck, then we can act and change it. Thierry -- Thierry Goubier CEA list Laboratoire des Fondations des Systèmes Temps Réel Embarqués 91191 Gif sur Yvette Cedex France Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95
participants (7)
-
Esteban Lorenzano -
GOUBIER Thierry -
Goubier Thierry
-
Sebastian Sastre -
Stéphane Ducasse -
Tudor Girba -
Yuriy Tymchuk