Hi all,

We had reports of issues observed with this version, we will rollback during the morning and after proceed to study the indicent.

G

Thanks.

And please tell us what data we can generate in order to help. It happened on our CI and this is a place where I can introduce stuff which generates data if it does not become too slow.

Norbert

Hi all,

I've rolled the stable version back to 10.3.1, please test.

@Norbert et al, we identified at least two issues:

• a glibC incompatibility with Ubuntu 20.04 that arises because recently the build servers changed to a Debian, so all zeroconf users are impacted (except those using stable debian).
• a vm crash that we are still investigating

If you have the second issue, and you have a small reproducible case (like some small test set in your open source project) that we can use to reproduce the issue, that will accelerate our debugging process.

Thanks all for understanding,
G

thanks, that means the 10.3.1 is still build with ubuntu and 10.3.2 is debian based? Just to be sure. In that case it would be nice to have a way to download precisely 10.3.1 in order to stay safe and we have time to plan countermeasure. I think on one of the issues there was a mentioning that Kris add some patch for their CI. Maybe this is a good addition to zeroconf? Or can this be achieved differently

Overall this is quite unfortunate. I hope there will be a better solution then everyone has to build their own vms. I think I need to plan some time to do it.

I can confirm that. The segfault was always in ZdcSocketStream when accessing a native glibc function. Sadly the console I cannot see anymore because too many builds in between.

Norbert

The binary for version 10.3.1 was built on ubuntu 16 or 18 I think, maybe Christophe or Pablo can give more details, I don’t recall it right now.

10.3.2 was built on the new linux server infrastructure.

$ uname -a
Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux

We are evaluating different alternatives (e.g., downgrading the build server for a while).

If you want to control what you download, right now the safest way is to go to files.pharo.org http://files.pharo.org/, more precisely to https://files.pharo.org/vm/pharo-spur64/, and look for your architecture.

There you will find the archives with the binaries we built.
Details about naming conventions are found here: https://github.com/pharo-project/pharo-vm/wiki/About-Build-and-Artifacts

That file server stores both released artifacts and intermediate build results.
The released artifact is the first artifact with the corresponding version tag (e.g., 10.3.1, 10.2.0).
This is generally the oldest in timestime, but you can confirm with the tags on github.

Moreover, the general advice is to avoid zeroconf as much as possible.
We don’t have the manpower to maintain build farms for each possible linux flavor.

Internally, we recently started discussing the deprecation of zeroconf and the migration path to OBS for *nixes.
The early sketch is that each released version will have its own package in OBS.
Then a virtual package will gather them all together (and help people update or downgrade on need)
More of this will come soon.

I’m not aware of this, can you post a link here?

Zeroconf worked very well until now, and we have put lot of effort in its modernization lately.
However, as it is it has problems to scale:

• executing an untrusted script that downloads unverified binaries
• we manage only one nix build, which works only as soon as all nixes keep compatibility with the version we built on
• it is designed to expose two versions: latest and stable.

Much of this is solved by OBS, and we hope this improves our stability and visibility.

We understand, and believe me, it’s unfortunate for us too.
We are working on it and hope to have some news in the following weeks.

Thanks, could you share your OS/setup ? (uname -a probably is enough).

Also, anybody having a couple of minutes to test and help, if you see this issue locally, could you

• build a VM (https://github.com/pharo-project/pharo-vm/), checking out tag v10.3.2
• run locally the failing tests

This information could help us reduce the search space, and identify if it’s just the build environment or there is something else we have to deal with.

Cheers,
G

Also, we improved the artifact naming to include also:

• suffixes (alpha, RC, SNAPSHOT)
• the distance to the released tag in the format [+x] where x is the commits.

https://github.com/pharo-project/pharo-vm/pull/870

We have tested this in the dev branch, I’ll backport this in the stable branch.
This will allow us to clearly identify the release commit (the one without the +x tag in the name).

G

ok, thanks.

Agreed. It is really hard to do. The only way would be to extract them from the OBS build. OBS's purpose is to have that build farm building all flavors. So if there would be a way to extract files from there and adjusting the zeroconf bash script it would be feasible. But probably not really easy to do.

That would be really sad because it is super handy. But it would also be understandable to shut down it down when it does the wrong thing.

This is inevitable.

This is nice but less important then the above.

cool

https://discord.com/channels/223421264751099906/492279831216783360/1314169201883353090

that is a user choice, putting a warning label is sufficient.

see above

to my memory these versions are fully arbitrary so you could remove them to gain at least clarity.

Ubuntu 22.04
Linux fe1102acfb5f 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
that is based on a docker ubuntu:22.04

I will take that for creating a vm build on my CI.

Yes, you should have some procedure how to get data so we can help. And everyone complaining about instability is suspect to exactly follow that :)

thanks again,

Norbert

perfect. We use

git describe | sed -e 's/-([0-9][0-9])-./.\1/'

for that.

super! Thank you for the good work!!

Norbert

Haha, this is indeed a nice idea, it’s true we have a lot of this in our heads.
I’ll come up with something.