On 1/23/16 8:59 AM, Stephan Eggermont wrote:
We want to use Metacello to manage smalltalk dependencies in a way that results in working software using a minimum amount of manual work. At the moment there are several ways in which Metacello is used, some of which result in more work and instability than others.

An inherent conflict in managing these dependencies is the level of repeatability that is wanted and achievable, vs. the amount of manual work needed. On the one hand we would like perfect repeatability. That needs an exact description of the order in which specific versions of packages were loaded in a specific image. In Metacello, we can achieve this by making sure all package versions are fixed and the dependencies are fixed to numeric versions.
This is one of the reasons that I have pushed for the use of git ... with git ALL of the packages are committed together so a single commit SHA is sufficient to ensure "that all package versions are fixed" without regard to numeric versions - the packages that work together are committed together.

The Metacello registry records the SHA of the commit that was loaded into an image so there is absolutely no ambiguity.

Dependencies are recorded in a BaselineOf that is also committed with the packages, so the dependencies are "guaranteed" to be in sync with the package versions ... a single SHA is all that is needed to exactly duplicate the formula used to load a project into an image ...


The problem with this is that that not necessarily results in working software. There are several reasons for this:
- Metacello only manages the smalltalk level dependencies. It does not take into account changes in the vm necessitating image-level changes or operating system and library changes.
There is no reason that project attirbutes should be restricted to only platform and platform versions .... the attribute list is generated by the running image, so a list of attributes that included additional information like vm type and version could be included and then it would only be necessary to use these new attributes in the baseline ...
- The image itself is not fully managed with Metacello (yet). Bug fixes made after the creation of the Metacello version might be needed for the software to work.
Yes, this is a critical point ...
- Bugs might have been found and fixed in dependencies.
In the situation of a top-level configuration, i.e. one that no other configuration depends on, this might be acceptable. Metacello dependencies are defined one-directional. A configuration knows on which configuration it depends, not which configurations depend on it. Configurations that are published should therefore always be regarded as possibly being used by others, and not only top-level.

If we consider the situation where others (B) are depending on configurations (A) that are written like this, then the situation gets worse:
- B might need an update because it is affected by an issue that does not affect A.
- B might need a different load ordering, e.g. need to only partially load dependencies of A, then something itself and then the other dependencies of A.
Nevertheless, sometimes we need near-perfect repeatability. For that, a snapshot of all loaded packages provides a reasonable alternative to a Metacello configuration version.
... and a list of project/sha/branch like the following:

project list

provides near-perfect repeatability on a project by project basis --- there are ways using existing Metacello features to ensure that the exact SHA associated with each of the projects will be used to the exclusion of all others ...
On the other hand the least amount of manual updates results from always depending on #development or #stable versions of dependencies. This results in the following problems:
- If the API of a dependency changes, the configuration might break silently.
- Projects might be slow to promote versions from #bleedingEdge to #development to #stable, resulting in a slow uptake of improvements.
For configurations of simple packages that have an API that is very stable or only expands, depending on #stable can be appropriate.
The historical problem with generic symbolic versions like #development and #stable has�� been that the semantics of the #stable version changes over time in the beginning it meant the version that was "stable on Pharo2.0", but when the same symbolic version to imply "stable on Pharo3.0", all was lost:)...

From a modularity point of view, a configuration should be specific on the packages in its own span of control, and as generous as possible in what it accepts from its dependencies. In that way a high cohesion and low coupling is achieved. We can achieve that by defining symbolic versions that reflect observable change in behaviour. If we make changes that should not affect users of the configuration, we create a new numeric version and update the symbolic version in place to use this new version. By default we only depend on these symbolic versions.


You are basically describing the principles behind semantic versioning[1] and�� I've always been a fan of the symbolic version scheme that the Seaside project has been using (#release3, #release3.1, etc.), because it makes it possible to express a project dependency at the appropriate semantic version level...

For git repositories, one can create a tag for a particular SHA ("specific on the packages in its own span of control") and then use "tag pattern matching"[2], to provide a reasonable approximation of symbolic versions with a "git" repo.

-----

4 years ago, I was very aware of the fragility of the�� ConfigurationOf ecosystem and at the time, was very glad to see that responsibility for the implementation for versions, branches and tags could be off-loaded onto a disk-based SCM - leaving Metacello to continue to specify package load order and external project dependencies...

In the subsequent years, I have been adding features to Metacello that I found were needed to support a "git-based" workflow (primarily the lock command[3])....

I have also been working on building a Smalltalk development environment that embraces "git" from the ground up and I consider that I've learned quite a few important lessons in this area ... I am not going to say that tODE[4] should be ported to Pharo, but I will say that tODE _is_ a proof of concept that a git-based development environment for Smalltalk can be built that preserves image-based Smalltalk development while leveraging "git" as the SCM... I should note that tODE bridges the ConfigurationOf and BaselineOf worlds with support for both ....

I have been "patiently waiting" for folks to come to accept/tolerate "git" and it seems that that time is getting close enough to begin trying to share the lessons that I've learned�� --- but I can't "push on a rope" ---

Tool support is critical at this juncture and I am willing to share my experience (and code if desired) with the folks who will be building these next generation tools ---

As I mentioned in another post, the focus for the tools should shift from being package-centric to being project-centric ... a "git" repository contains all of the artifacts associated with a project, and in the "tODE model" there is a one to one mapping between a project and a git repository.

Of course there is more, but I will save that for sharing with folks who are curious and interested in learning from my mistakes and my successes:)

Dale


[1] http://semver.org
[2] https://github.com/dalehenrich/metacello-work/issues/277#issuecomment-58970696
[3] https://github.com/dalehenrich/metacello-work/blob/master/docs/LockCommandReference.md#lock-command-reference
[4] https://github.com/dalehenrich/tode#tode-the-object-centric-development-environment-