On 7 Nov 2018, at 14:19, Ben Coman <btc@openinworld.com> wrote:

On Wed, 7 Nov 2018 at 17:16, Esteban Lorenzano <estebanlm@gmail.com> wrote:

It is not so easy.

A huge project as Pharo would profit to be split in order to manage its parts correctly. For example, we recently released iceberg 1.3.0. And there were some problems around and we evaluate the possibility of revert that release until we fix the problems. At the end, problems were not so important and the fix was made with a 1.3.1 release and almost immediately an 1.4.0.

Now, imagine If iceberg would be integrated in pharo.

1. The nature of ���iceberg��� would be lost in other parts. We cannot talk anymore of releasing 1.3.1 or whatever, we just release Pharo. No clear way to know what changed in Iceberg without digging in the full list of changes.
2. If we need to revert. How do we do it? There is no release so what do we revert? A commit? How many commits? How do we revert if development tree continued growing and iceberg commit are now mixed with others?
Of course, we could maintain a branch. But then I'd arg��e is more or less the same as having a separated project.

Situation is not ideal (even if it is a lot better as before).
But I���m envisioning this solution: add multiple source directories capability.

Then we can have:

src/kernel
src/system
src/spec
src/iceberg
src/calypso
etc.

This will not just help to improve modularisation but we���ll gives a ���free��� feature : using git subtree to manage those directories.
For those who do not know: subtree allow us to merge different project trees into one. But the important part is how it will behave in Pharo (or other projects adopting it): For Pharo it will be JUST ONE REPOSITORY. So one single commit will spread to all repositories.
Now, the ���price to pay��� later is that time to time we need to push subtree from Pharo to their external projects (but this can be automated easily), and of course, doing ���an iceberg release��� means to do a pull subtree from external repository to Pharo (also very easy).

Why not a submodule approach?

 I do vaguely remember you having some
pain with it integrating the opensmalltalk-vm repo under the pharo-vm
repo before Pharo started more directly using the opensmallk-vm CI
products.

I thought I might have something useful to contribute to discussion by
finding the good article [1] I remembered
that compared subtree and submodule, the summary of which is...

  * Is the external repository something you own yourself and are
likely to push code back to?
    Then use a submodule. This gives you the quickest and easiest way
for you to push your changes back.

  * Is the external repository third party code that you are unlikely
to push anything back to?
    Then use a subtree. This gives the advantage of not having to
give people permissions to an extra repo when you are giving them
access to the code base, and also reduces the chance that someone will
forget to run a git submodule update. [But this could be automated in
Pharo]


But then I came across [2] which says...
   * Submodules [...] work well when you want to include a
third-party library in your project that only occasionally needs to be
updated.

so I don't know.
[1] https://codewinsarguments.co/2016/05/01/git-submodules-vs-git-subtrees/
[2] http://www.mos6581.org/git_subtree_alternative


I think main issues to consider between the two options are:
   A. the effect on history between Pharo repo and dependent repos,
of commits to either repo
   B. how branching is handled between Pharo repo and dependent repos
and I don't know really have my head around that, so for now I'll shut it. :)


anyway, my understanding was that subtree was implemented in shell
scripts and not libgit so does that make it more difficult?
  https://stackoverflow.com/a/25474895

cheers -ben

Esteban

Ps: remember: programming is an iterative improving process. ���Process" is part of programming so iterations applies to it too.

On 7 Nov 2018, at 07:02, Tim Mackinnon <tim@testit.works> wrote:

It���s a tricky trade off as Norbert alludes too - in my recent example I needed some underlying base pieces in place (for the compiler���s ast) and those needed reviewing to then be committed - then there were two parts to Calypso that relied on those changes) that also needed reviewing and I wasn���t sure if Denis waited for the core approval or loaded up my PR - but anyway it took him a bit of time to process that and he came back with some better suggestions that I also needed to implement (which I had to find time to do as well).

When I noticed that the core Pharo pieces were merged, I then had to chase Denis to see if he was now happy and could merge my changes.

I guess it would be helpful if there was a way to easily load up multiple project pr���s in one go (like the suggested slice concept) so maintainers can easily review.  Probably more importantly is an easy way to track the status of multiple submissions so you can follow up with relevant people and push things along and also ensure things get committed in the right order (eg doc the dependency chain a bit better).

For me, after a few days I forgot about my Calypso changes and realised a few weeks later (by accident) and so could chase Denis.

I think it���s this latter case that Steph alludes to - you lose interest after a few days without some useful prompts and easy status tracking. If we can make that easier I think it would help.

Tim

Sent from my iPhone

On 7 Nov 2018, at 05:48, Ben Coman <btc@openinworld.com> wrote:

I get the feeling what is needed is mirroring all dependent repos from
the canonical location under http://github.com/pharo-project
and a Slice-like tool (probably keeping the name "Slice") which...
1. Pulls all dependent repos to the local machine
2. Simultaneously commits to the local repos with the same commit message
3. Updates a bootstrap-configuration file holding commit-hashes of all
the dependencies and commits with same commit message
4. Pushes that bootstrap-configuration file and all changed dependent
repos to user's github account
5. Issues a pull request for the bootstrap-configuration file
6. Our CI then builds a test-image by commit-hash direct from each
user's repo and if it passes, pulls dependent repo commits under
pharo-project
7. CI can then issue PRs to the dependency canonical repos

cheers -ben

On Wed, 7 Nov 2018 at 02:55, Stephane Ducasse <stepharo.self@gmail.com> wrote:

Calypso is integral part of Pharo as Iceberg.
We started to discuss the problem in the team. Right now this project
spread kills us.

Stef

On Mon, Nov 5, 2018 at 11:56 AM Stephan Eggermont <stephan@stack.nl> wrote:

Tim Mackinnon <tim@testit.works> wrote:

In retrospect,  I���m wondering if successful projects that have proved
integration usefulness should be moved into the core repo?
(Iceberg/Calypso?) or are we missing something to help easily track the
journey of a multi faceted change (although this sounds overkill?). Or
are there sprint days to try and knock these things through easily with
everyone on board to do it together?

We are sort of damned if you do and damned if you don���t. But certainly we
want to endure that progress can be made without losing the will to contribute.

Indeed. Putting things in one repo cannot scale and cannot be a solution
for something that is neither core pharo nor an application. I encourage
everyone who wants to get a good description of this problem to read

"Managing Design Data: The Five Dimensions of CAD Frameworks, Configuration
Management, and Product Data Management" by Peter van den Hamer & Kees
Lepoeter.

With git and github a solution to decouple fast-moving from slow-moving
projects seems to be indeed to fork and make PRs.
That only works if the quality of the PRs is high enough and we manage to
use the feedback from slower-moving projects well.

Earlier, we���ve seen projects like Magma being overwhelmed by the number of
needed changes, and Pier being broken by Pillar not respecting its
constraints.

With tools like Travis, it is quickly clear if a PR would result in a green
build in the original repo.

With projects where Pharo uses only the core, and applications use more
than that, the applications still have a dependency problem: if the core
changes in Pharo influence the other parts, someone needs to do the work to
make those parts work again. With forked repos, that can be a pharo
maintainer, the project maintainer or the application maintainer. All three
need to be able to make those changes. And they need to be decoupled from
having to make them immediately. And being able to have the discussion
about the exact implementation independently from implementing a stop-gap
solution now is also valuable.

So if Calypso is supposed to be extendable and only the core part is part
of Pharo, having it as an external project makes sense. With a fork for
Pharo to move at its own speed. If Iceberg is Pharo-only, just having
different branches for different Pharo versions, it sounds to me like it
might be better of in the Pharo project. Tonel is supposed to be
cross-platform so should be separate.

Is this helpful?

Stephan