Hi, I'll write down some of the reasons of the project's design, like that I can afterwards copy paste it in the wiki :). First, this design did not came up from an egg. We worked on it for about two months. And it is thought to be backwards compatible and manage lots of metacello particularities. It may have things that are perfectible, sure, so let's discuss it. One of the main problems we saw in metacello, and that Iceberg inherited, was the "source subdirectory" thing. This source directory had to be specified in the CLIENT, meaning that every time we clone a repository we should know by heart the directory chose by its developer. Moreover, we lack a standard way to do it, so everybody does as he feels (root directory, src, source, repository, mc....). This has some bad consequences: - once a repository is referenced by some other project, it is more complicated to change its source directory. Imagine that tomorrow we set as standard that all git repos should have the code in src. Then voyage should change. And all its clients too. - Making a typo in the code subdirectory means sometimes super ugly errors from metacello that are difficult to debug and understand (e.g., "Cannot resolve BaselineOfMetacello WTF") Moreover, there was another problem that people started stumbling on: the fact that iceberg got confused sometimes thinking that an empty project was in filetree (to keep backwards compatibility with projects without a .properties). So we decided that for this release we wanted to revert a bit that situation. Think object: let's put the meta-data used to interpret a project's structure inside the project itself. The idea is that: - each project should contain both a .project and a .properties file. The first can contain arbitrary project meta-data (such as the source directory). The second contains the cypress properties, which are needed to correctly interpret the code inside the source directory. - a project without a .project file is an old project and cannot be interpreted, because we don't know the source directory - a project without a .properties file is an old project and is by default transformed in a project with a #filetree properties file - an old project cloned from iceberg detects the missing .project file and gives the user the opportunity to declare it (and then commit it explicitly) - an old project cloned and loaded from a Metacello expression defining a source directory will honnor the source directory defined in the Metacello expression (for backwards compatibility, and we have ~500 tests about this). # About defaults values / forcing the user to define a project First, notice that even when the repositories you load are just marked as "dirty". This is because in memory we add a project to your repository. But you're not forced to commit it. Actually, you can still load packages and baselines from that repository without committing. This is in line with Iceberg's "explicitness". We try to not do any destructive operation without asking the user first (that's why we have several preview windows for pushing, pulling, checkout, merge..., and why contrastingly with monticello we show the committed changes on the commit window...). So, instead of transparently "adding the file" we have decided to modify the project in memory and let the user the responsibility to commit that file. If there's a drawback, is that the repository is marked as dirty. Which is a bit noisy, yes, but still I think it's not so bad compared with the previous drawbacks. To solve this, we could have some default values, yes, and only mark it as dirty if the project does not follow the default value. This could work, but right now all projects use different names for their source directories. So the question is, what would be a good default? I'd like to use 'src' since this is short, well known and less alien (all these in the sense that we do not lose anything and we have a lot to gain by using it). However, not much repositories use 'src' so it will still produce a lot of "noise"... But still! Committing that file is a one-time operation. Once people fix their repositories adding the project meta-data, you will not see them dirty anymore. So we can see this as a transition noise too... Of course, new ideas are welcome. I'll let Pablo and Esteban add their points of view on this too. Guille

Am 07.08.2018 um 16:00 schrieb Guillermo Polito <guillermopolito@gmail.com>:

Hi,

I'll write down some of the reasons of the project's design, like that I can afterwards copy paste it in the wiki :).

First, this design did not came up from an egg. We worked on it for about two months. And it is thought to be backwards compatible and manage lots of metacello particularities. It may have things that are perfectible, sure, so let's discuss it.

One of the main problems we saw in metacello, and that Iceberg inherited, was the "source subdirectory" thing. This source directory had to be specified in the CLIENT, meaning that every time we clone a repository we should know by heart the directory chose by its developer. Moreover, we lack a standard way to do it, so everybody does as he feels (root directory, src, source, repository, mc....).

This has some bad consequences: - once a repository is referenced by some other project, it is more complicated to change its source directory. Imagine that tomorrow we set as standard that all git repos should have the code in src. Then voyage should change. And all its clients too. - Making a typo in the code subdirectory means sometimes super ugly errors from metacello that are difficult to debug and understand (e.g., "Cannot resolve BaselineOfMetacello WTF")

Moreover, there was another problem that people started stumbling on: the fact that iceberg got confused sometimes thinking that an empty project was in filetree (to keep backwards compatibility with projects without a .properties).

So we decided that for this release we wanted to revert a bit that situation. Think object: let's put the meta-data used to interpret a project's structure inside the project itself. The idea is that:

- each project should contain both a .project and a .properties file. The first can contain arbitrary project meta-data (such as the source directory). The second contains the cypress properties, which are needed to correctly interpret the code inside the source directory. - a project without a .project file is an old project and cannot be interpreted, because we don't know the source directory - a project without a .properties file is an old project and is by default transformed in a project with a #filetree properties file - an old project cloned from iceberg detects the missing .project file and gives the user the opportunity to declare it (and then commit it explicitly) - an old project cloned and loaded from a Metacello expression defining a source directory will honnor the source directory defined in the Metacello expression (for backwards compatibility, and we have ~500 tests about this).

# About defaults values / forcing the user to define a project

First, notice that even when the repositories you load are just marked as "dirty". This is because in memory we add a project to your repository. But you're not forced to commit it. Actually, you can still load packages and baselines from that repository without committing.

This is in line with Iceberg's "explicitness". We try to not do any destructive operation without asking the user first (that's why we have several preview windows for pushing, pulling, checkout, merge..., and why contrastingly with monticello we show the committed changes on the commit window...). So, instead of transparently "adding the file" we have decided to modify the project in memory and let the user the responsibility to commit that file.

If there's a drawback, is that the repository is marked as dirty. Which is a bit noisy, yes, but still I think it's not so bad compared with the previous drawbacks. To solve this, we could have some default values, yes, and only mark it as dirty if the project does not follow the default value. This could work, but right now all projects use different names for their source directories. So the question is, what would be a good default? I'd like to use 'src' since this is short, well known and less alien (all these in the sense that we do not lose anything and we have a lot to gain by using it). However, not much repositories use 'src' so it will still produce a lot of "noise"...

But still! Committing that file is a one-time operation. Once people fix their repositories adding the project meta-data, you will not see them dirty anymore. So we can see this as a transition noise too...

Of course, new ideas are welcome. I'll let Pablo and Esteban add their points of view on this too.

I think I can see what is the rationale behind it but I’m not sure this can be the way to go: - I don’t think there can be a „standard way“ of defining source directory. And I don’t think that a tool should enforce this however. I keep frontend and backend code in some repositories together so the source is in my case in backend/source. What does it mean for users not using the „standard“ name? - I don’t see why there needs to be a 1:1 relationship between a repository and working copy in pharo. It is like this at the moment already but the .project file manifests this. So it should not be supported to have more source dirs in one git repo? It might be not a good idea that the client has to write the source dir but it opens the possibility that there can be more than one. - My mode of working is to have an eye on dirty repositories because that shows what the impact of your work is. If I have a lot of dirty repositories in my repository list it does not feel good and I don’t want that. Especially for projects I don’t have write access to. How can I change this? I’m not sure that assuming everyone will add these files is a likely one. Norbert

Damien Pollet wrote on 8. 8. 2018 13:53:

First of all, quick stupid question: I'm currently loading my code with gitlocal://./src as the repository URL (my workflow starts in a terminal rather than in a Pharo image) Should I just remove the /src part, now that my repo has the project metadata?

Also, are more features planned for the .project file? E.g. what about storing a default selection for Calypso and the Test Runner in there?

On Wed, 8 Aug 2018 at 10:28, Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>> wrote:

- I don’t think there can be a „standard way“ of defining source directory. And I don’t think that a tool should enforce this however. I keep frontend and backend code in some repositories together so the source is in my case in backend/source. What does it mean for users not using the „standard“ name?

Sure there can. Look at any ruby or maven project, they all have strong conventions for organizing projects and standard config files for deviating from those conventions. I would have preferred if Iceberg picked one convention (arbitrarily) in the absence of a .project file instead of forcing its explicit presence. IMHO the choice of default directory per se (be it ./, ./src, ./source or whatever) matters less than the fact that there is a convention in place.

- I don’t see why there needs to be a 1:1 relationship between a repository and working copy in pharo. It is like this at the moment already but the .project file manifests this. So it should not be supported to have more source dirs in one git repo? It might be not a good idea that the client has to write the source dir but it opens the possibility that there can be more than one.

I see your point here, but by using separate source directories you're sort of creating a hydra project… What I mean here is that the source directories are separate, but their histories are tangled. If you want separate source dirs it kinda means that you want separate change histories, doesn't it? What if the same class has diverging definitions in separate directories (I wonder what maven does in that case…)?

There are monorepos. Some people / companies love them.

On the other hand, separate source directories would be helpful to work with git-subrepo and similar tools…

-- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet

On Thu, Aug 9, 2018 at 11:18 AM Norbert Hartl <norbert@hartl.name> wrote:

Am 08.08.2018 um 14:12 schrieb Herbert Vojčík <herby@mailbox.sk>:

Damien Pollet wrote on 8. 8. 2018 13:53:

First of all, quick stupid question: I'm currently loading my code with gitlocal://./src as the repository URL (my workflow starts in a terminal rather than in a Pharo image) Should I just remove the /src part, now that my repo has the project metadata? Also, are more features planned for the .project file? E.g. what about storing a default selection for Calypso and the Test Runner in there? On Wed, 8 Aug 2018 at 10:28, Norbert Hartl <norbert@hartl.name < mailto:norbert@hartl.name <norbert@hartl.name>>> wrote: - I don’t think there can be a „standard way“ of defining source directory. And I don’t think that a tool should enforce this however. I keep frontend and backend code in some repositories together so the source is in my case in backend/source. What does it mean for users not using the „standard“ name? Sure there can. Look at any ruby or maven project, they all have strong conventions for organizing projects and standard config files for deviating from those conventions. I would have preferred if Iceberg picked one convention (arbitrarily) in the absence of a .project file instead of forcing its explicit presence. IMHO the choice of default directory per se (be it ./, ./src, ./source or whatever) matters less than the fact that there is a convention in place. - I don’t see why there needs to be a 1:1 relationship between a repository and working copy in pharo. It is like this at the moment already but the .project file manifests this. So it should not be supported to have more source dirs in one git repo? It might be not a good idea that the client has to write the source dir but it opens the possibility that there can be more than one. I see your point here, but by using separate source directories you're sort of creating a hydra project… What I mean here is that the source directories are separate, but their histories are tangled. If you want separate source dirs it kinda means that you want separate change histories, doesn't it? What if the same class has diverging definitions in separate directories (I wonder what maven does in that case…)?

There are monorepos. Some people / companies love them.

One of these companies we know for good. It is google. Google has one (!) repositoryf for all their software.

And that is the point I want to stretch here. One repo with one project and a standard source directory is too theoretical and too narrow in thoughts.

I think that you underestimate people when you say this.

Today there is no possibility to reliably make one product out of multiple repos. Git submodule and git subtree are not even close to something reliable and usable.

We are well aware of this. And that's why Pharo is not using any of them. But this goes against what you said before, or I'm not getting something.

Descriptions like metacello are language centric.

Yes. But I don't understand why this comes up. You would like language... "agnostic"? For example?

So what is the most secure way of defining a single product? It is the mono repo. That is the main reason I have my sources in backend/source because there is a javascript frontend in the same repo because I want to have one reliable source that defines my product and I can be sure that the backend and frontend are always the ones that match.

Just to repeat myself (and just to avoid missing it in the previous emails ^^). Can you describe better your needs for "multiple iceberg subdirectories"?

I don’t think it is a good idea if we once leave the island to embrace something like git just to make git islandish a little later.

I did not understand this...

Just to be very clear. We are talking about two files: .project and .properties. I have nothing against .properties because we have multiple formats possible so making it explicit is a good idea. But .project makes every repo a single source repo. If we enforce .properties files why do we not scan the repo for directories containing this file? If none is found we cannot know (or we add looking at some default locations). If multiple directory are found I can add that to iceberg as a real project out of the combination for repository-source directory.

This way I could have multiple projects out of one repo. And with the

possibility of having multiple source directories it is obvious that the path still needs to be added to a url in metacello.

Unless there is one of this directories marked to be the default which

would be loaded if no directory is given. This way the default directory would be the one selected if only one is there. So you can have multiple projects in one repo and if this is only one per project the directory can be automatically resolved. Use case kept.

Yes it could be done, but it's not so easy. I think you're underestimating the problem. For example, - tons of old projects have no .properties file. - so far iceberg only explicitly supported one code directory, and assumed that "no subdirectory" means root directory, not "default" one - If we discover two subdirectories in your repository, and you add a package into your repository, in which subdirectory should it be? - Should in this case be able to add the same package to both subdirectories? Do we commit them at the same time? Or they have different versions? Let's add that all these changes may involve changes in: - UI (and doing it properly takes time to design it, not just "coding" it) - merge Now, again, I can understand that the change make you uncomfortable, but I'd prefer that we talk about what are your needs instead of "how easy it could be done if we weren't stupid".

Norbert

On the other hand, separate source directories would be helpful to work with git-subrepo and similar tools… -- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet

-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

[SNIP] I think I can see what is the rationale behind it but I’m not sure this can be the way to go:

- I don’t think there can be a „standard way“ of defining source directory. And I don’t think that a tool should enforce this however. I keep frontend and backend code in some repositories together so the source is in my case in backend/source. What does it mean for users not using the „standard“ name?

Well, I think the opposite. The tool should enforce some standard way, and then, if you want to diverge from it you should specify it explicitly.

- I don’t see why there needs to be a 1:1 relationship between a repository and working copy in pharo. It is like this at the moment already but the .project file manifests this. So it should not be supported to have more source dirs in one git repo? It might be not a good idea that the client has to write the source dir but it opens the possibility that there can be more than one.

Our solution doesn't close the door either. Actually, we were thinking on having composite projects (a project with subprojects), which would support your use case. We are aware we wanted this for Pharo so for example we can split pharo itself in several smaller subprojects instead of having the mess of ~500 packages we have today. However, we were not aware of any project out there using such setup, and previous iceberg versions was maybe supporting this just "by chance" and not by design. (Just imagine that having in the same image two repositories with the same git repository behind but different source directories could be a way, but it's more of a workaround). This said, could you explain better how you were using this so far? - you have different source directories for frontend/backend? - you interact with them as different repositories in the same image? or from different images pointing to the same repo? - every time you commit in one of the iceberg repositories you get detached in the other(s)? This said, I'd like to support this scenarios, but I want to do it by design :)

- My mode of working is to have an eye on dirty repositories because that shows what the impact of your work is. If I have a lot of dirty repositories in my repository list it does not feel good and I don’t want that. Especially for projects I don’t have write access to. How can I change this? I’m not sure that assuming everyone will add these files is a likely one.

Maybe we need to be less invasive with colors? My point of view is that Monticello shows tons of packages and repositories, automatic rewrite and auto-deprecation rules rewrite code dynamically and mark monticello packages as dirty. But nobody ever complained about monticello or tried to change it. What are the packages you depend on and you have no write access to? If we make a pull request per day, I think we can solve this situation really fast...

Norbert

-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13