Hi all, I've rolled the stable version back to 10.3.1, please test. @Norbert et al, we identified at least two issues: - a glibC incompatibility with Ubuntu 20.04 that arises because recently the build servers changed to a Debian, so all zeroconf users are impacted (except those using stable debian). - a vm crash that we are still investigating If you have the second issue, and you have a small reproducible case (like some small test set in your open source project) that we can use to reproduce the issue, that will accelerate our debugging process. Thanks all for understanding, G

Le 5 déc. 2024 à 10:45 AM, Norbert Hartl <norbert@hartl.name> a écrit :

Thanks.

And please tell us what data we can generate in order to help. It happened on our CI and this is a place where I can introduce stuff which generates data if it does not become too slow.

Norbert

...

Am 05.12.2024 um 08:42 schrieb Guille Polito <guillermo.polito@inria.fr>:

Hi all,

We had reports of issues observed with this version, we will rollback during the morning and after proceed to study the indicent.

G

...

Le 4 déc. 2024 à 10:56 AM, Guille Polito <guillermo.polito@inria.fr> a écrit :

Hi all,

We’ve published a new VM release (v10.3.2) to be used with Pharo versions 10 through 13. Please check the change log here:

https://github.com/pharo-project/pharo-vm/releases/tag/v10.3.2

The build is on the oven, new VMs should be available as soon as the build farm finishes.

Cheers, G _______________________________________________ Pharo-vm mailing list -- pharo-vm@lists.pharo.org To unsubscribe send an email to pharo-vm-leave@lists.pharo.org

_______________________________________________ Pharo-vm mailing list -- pharo-vm@lists.pharo.org To unsubscribe send an email to pharo-vm-leave@lists.pharo.org

...

I've rolled the stable version back to 10.3.1, please test.

thanks, that means the 10.3.1 is still build with ubuntu and 10.3.2 is debian based?

The binary for version 10.3.1 was built on ubuntu 16 or 18 I think, maybe Christophe or Pablo can give more details, I don’t recall it right now. 10.3.2 was built on the new linux server infrastructure. $ uname -a Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux We are evaluating different alternatives (e.g., downgrading the build server for a while).

Just to be sure. In that case it would be nice to have a way to download precisely 10.3.1 in order to stay safe and we have time to plan countermeasure.

If you want to control what you download, right now the safest way is to go to files.pharo.org <http://files.pharo.org/>, more precisely to https://files.pharo.org/vm/pharo-spur64/, and look for your architecture. There you will find the archives with the binaries we built. Details about naming conventions are found here: https://github.com/pharo-project/pharo-vm/wiki/About-Build-and-Artifacts That file server stores both released artifacts and intermediate build results. The released artifact is the first artifact with the corresponding version tag (e.g., 10.3.1, 10.2.0). This is generally the oldest in timestime, but you can confirm with the tags on github. Moreover, the general advice is to avoid zeroconf as much as possible. We don’t have the manpower to maintain build farms for each possible linux flavor. Internally, we recently started discussing the deprecation of zeroconf and the migration path to OBS for *nixes. The early sketch is that each released version will have its own package in OBS. Then a virtual package will gather them all together (and help people update or downgrade on need) More of this will come soon.

I think on one of the issues there was a mentioning that Kris add some patch for their CI.

I’m not aware of this, can you post a link here?

Maybe this is a good addition to zeroconf? Or can this be achieved differently

Zeroconf worked very well until now, and we have put lot of effort in its modernization lately. However, as it is it has problems to scale: - executing an untrusted script that downloads unverified binaries - we manage only one nix build, which works only as soon as all nixes keep compatibility with the version we built on - it is designed to expose two versions: latest and stable. Much of this is solved by OBS, and we hope this improves our stability and visibility.

Overall this is quite unfortunate. I hope there will be a better solution then everyone has to build their own vms. I think I need to plan some time to do it.

We understand, and believe me, it’s unfortunate for us too. We are working on it and hope to have some news in the following weeks.

...

@Norbert et al, we identified at least two issues: - a glibC incompatibility with Ubuntu 20.04 that arises because recently the build servers changed to a Debian, so all zeroconf users are impacted (except those using stable debian).

I can confirm that. The segfault was always in ZdcSocketStream when accessing a native glibc function. Sadly the console I cannot see anymore because too many builds in between.

Thanks, could you share your OS/setup ? (uname -a probably is enough). Also, anybody having a couple of minutes to test and help, if you see this issue locally, could you - build a VM (https://github.com/pharo-project/pharo-vm/), checking out tag v10.3.2 - run locally the failing tests This information could help us reduce the search space, and identify if it’s just the build environment or there is something else we have to deal with. Cheers, G

Am 05.12.2024 um 14:04 schrieb Guillermo Polito <guillermopolito@gmail.com>:

...

Le 5 déc. 2024 à 1:49 PM, Guillermo Polito <guillermopolito@gmail.com> a écrit :

...

Just to be sure. In that case it would be nice to have a way to download precisely 10.3.1 in order to stay safe and we have time to plan countermeasure.

If you want to control what you download, right now the safest way is to go to files.pharo.org <http://files.pharo.org/>, more precisely to https://files.pharo.org/vm/pharo-spur64/, and look for your architecture.

There you will find the archives with the binaries we built. Details about naming conventions are found here: https://github.com/pharo-project/pharo-vm/wiki/About-Build-and-Artifacts

That file server stores both released artifacts and intermediate build results. The released artifact is the first artifact with the corresponding version tag (e.g., 10.3.1, 10.2.0). This is generally the oldest in timestime, but you can confirm with the tags on github.

Also, we improved the artifact naming to include also: - suffixes (alpha, RC, SNAPSHOT) - the distance to the released tag in the format [+x] where x is the commits.

perfect. We use git describe | sed -e 's/\-\([0-9][0-9]*\)\-.*/.\1/' for that.

https://github.com/pharo-project/pharo-vm/pull/870

We have tested this in the dev branch, I’ll backport this in the stable branch. This will allow us to clearly identify the release commit (the one without the +x tag in the name).

super! Thank you for the good work!! Norbert

G

Am 05.12.2024 um 13:49 schrieb Guillermo Polito <guillermopolito@gmail.com>:

...

...

I've rolled the stable version back to 10.3.1, please test.

thanks, that means the 10.3.1 is still build with ubuntu and 10.3.2 is debian based?

The binary for version 10.3.1 was built on ubuntu 16 or 18 I think, maybe Christophe or Pablo can give more details, I don’t recall it right now.

10.3.2 was built on the new linux server infrastructure.

$ uname -a Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux

We are evaluating different alternatives (e.g., downgrading the build server for a while).

...

Just to be sure. In that case it would be nice to have a way to download precisely 10.3.1 in order to stay safe and we have time to plan countermeasure.

If you want to control what you download, right now the safest way is to go to files.pharo.org <http://files.pharo.org/>, more precisely to https://files.pharo.org/vm/pharo-spur64/, and look for your architecture.

ok, thanks.

There you will find the archives with the binaries we built. Details about naming conventions are found here: https://github.com/pharo-project/pharo-vm/wiki/About-Build-and-Artifacts

That file server stores both released artifacts and intermediate build results. The released artifact is the first artifact with the corresponding version tag (e.g., 10.3.1, 10.2.0). This is generally the oldest in timestime, but you can confirm with the tags on github.

Moreover, the general advice is to avoid zeroconf as much as possible. We don’t have the manpower to maintain build farms for each possible linux flavor.

Agreed. It is really hard to do. The only way would be to extract them from the OBS build. OBS's purpose is to have that build farm building all flavors. So if there would be a way to extract files from there and adjusting the zeroconf bash script it would be feasible. But probably not really easy to do.

Internally, we recently started discussing the deprecation of zeroconf and the migration path to OBS for *nixes.

That would be really sad because it is super handy. But it would also be understandable to shut down it down when it does the wrong thing.

The early sketch is that each released version will have its own package in OBS.

This is inevitable.

Then a virtual package will gather them all together (and help people update or downgrade on need)

This is nice but less important then the above.

More of this will come soon.

cool

...

I think on one of the issues there was a mentioning that Kris add some patch for their CI.

I’m not aware of this, can you post a link here?

https://discord.com/channels/223421264751099906/492279831216783360/131416920...

...

Maybe this is a good addition to zeroconf? Or can this be achieved differently

Zeroconf worked very well until now, and we have put lot of effort in its modernization lately. However, as it is it has problems to scale: - executing an untrusted script that downloads unverified binaries

that is a user choice, putting a warning label is sufficient.

- we manage only one nix build, which works only as soon as all nixes keep compatibility with the version we built on

see above

- it is designed to expose two versions: latest and stable.

to my memory these versions are fully arbitrary so you could remove them to gain at least clarity.

Much of this is solved by OBS, and we hope this improves our stability and visibility.

...

Overall this is quite unfortunate. I hope there will be a better solution then everyone has to build their own vms. I think I need to plan some time to do it.

We understand, and believe me, it’s unfortunate for us too. We are working on it and hope to have some news in the following weeks.

...

...

@Norbert et al, we identified at least two issues: - a glibC incompatibility with Ubuntu 20.04 that arises because recently the build servers changed to a Debian, so all zeroconf users are impacted (except those using stable debian).

I can confirm that. The segfault was always in ZdcSocketStream when accessing a native glibc function. Sadly the console I cannot see anymore because too many builds in between.

Thanks, could you share your OS/setup ? (uname -a probably is enough).

Ubuntu 22.04 Linux fe1102acfb5f 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux that is based on a docker ubuntu:22.04

Also, anybody having a couple of minutes to test and help, if you see this issue locally, could you - build a VM (https://github.com/pharo-project/pharo-vm/), checking out tag v10.3.2 - run locally the failing tests

I will take that for creating a vm build on my CI.

This information could help us reduce the search space, and identify if it’s just the build environment or there is something else we have to deal with.

Yes, you should have some procedure how to get data so we can help. And everyone complaining about instability is suspect to exactly follow that :) thanks again, Norbert

Cheers, G

Am 05.12.2024 um 14:26 schrieb Norbert Hartl <norbert@hartl.name>:

...

Also, anybody having a couple of minutes to test and help, if you see this issue locally, could you - build a VM (https://github.com/pharo-project/pharo-vm/), checking out tag v10.3.2 - run locally the failing tests

I will take that for creating a vm build on my CI.

a small update. I managed in 20 minutes to create a Dockerfile that seems to work: --- # # Dockerfile for building pharo-10 branch under ubuntu 22.04 # FROM ubuntu:22.04 RUN \ apt-get update && \ apt-get install -y wget curl unzip inetutils-ping gnupg libcurl3-gnutls && \ apt-get install -y \ build-essential \ cmake \ git \ python3-minimal \ pkg-config \ libgit2-dev \ libssh-dev \ libsdl2-dev \ libfreetype-dev \ libffi-dev && \ rm -rf /var/lib/apt/lists/* WORKDIR /opt RUN mkdir -p /opt/src WORKDIR /opt/src RUN git clone https://github.com/pharo-project/pharo-vm.git && cd pharo-vm && git checkout pharo-10 RUN cmake -S pharo-vm -B pharo-vm-build RUN cmake --build pharo-vm-build --target install RUN rm -rf pharo-vm-build/build/dist/pharo-vm RUN mv pharo-vm-build/build/dist /opt/pharo-vm RUN rm -rf pharo-vm-build pharo-vm --- Doing a docker build . in that directory builds the vm and installs it under /opt/pharo-vm/pharo In the end it copies the files to a special place. But this could also be a way to zip them and put them somewhere else. The most simple use would be to have a Dockerfile per combination OS-pharoVmVersion. Maybe this would work on github. If I find time I might try somethig along that line. That would make zeroconf survive and still we can support a bit more. Of course it duplicates some work on the OBS but that deals with much more to integrate in the system. For me two uses cases. What I do now is that the Dockerfile above builds and installs the vm in the docker image. I upload these and make them the basis of my docker builds so this is the closest I could get and apart from obstacles that might occur it works. Just in case some has interests in the Dockerfile or volunteering building a repo with those Dockerfiles. Norbert

Am 05.12.2024 um 13:49 schrieb Guillermo Polito <guillermopolito@gmail.com>:

Also, anybody having a couple of minutes to test and help, if you see this issue locally, could you - build a VM (https://github.com/pharo-project/pharo-vm/), checking out tag v10.3.2 - run locally the failing tests

This information could help us reduce the search space, and identify if it’s just the build environment or there is something else we have to deal with.

It seems you have figured it out. Since I started to compile my own vm I cannot build anything on my jenkins. In order to at least confirm a few things I've built a little infrastructure for me to generate vms in different combinations. I went down the rabbit hole and tried v10.21, v10.3.1 and v10.3.2 under debian 12 and all failed. Then I downgraded gcc to gcc-11 and retried. All are still failing. I read that the gcc 12 and gcc 11 more or less the same in that regard. Sadly it is not easy to go much back from where I am. I need to check on the jenkins and this is based on debian 12. So I could retry with older gccs but I'm not sure what will be the outcome. Right now I'm retrying with pharo-10 HEAD because your commit message sounded it is worth it (well, nothing else works) If you want me to test somethig please just say so. Norbert

Hi Norbert, the patch has not been merged yet. Are we talking about the same? the pharo-10 branch still has the issue

Le 11 déc. 2024 à 3:50 PM, Norbert Hartl <norbert@hartl.name> a écrit :

...

Am 11.12.2024 um 15:19 schrieb Norbert Hartl <norbert@hartl.name>:

...

Am 05.12.2024 um 13:49 schrieb Guillermo Polito <guillermopolito@gmail.com>:

Also, anybody having a couple of minutes to test and help, if you see this issue locally, could you - build a VM (https://github.com/pharo-project/pharo-vm/), checking out tag v10.3.2 - run locally the failing tests

This information could help us reduce the search space, and identify if it’s just the build environment or there is something else we have to deal with.

It seems you have figured it out. Since I started to compile my own vm I cannot build anything on my jenkins.

In order to at least confirm a few things I've built a little infrastructure for me to generate vms in different combinations. I went down the rabbit hole and tried v10.21, v10.3.1 and v10.3.2 under debian 12 and all failed. Then I downgraded gcc to gcc-11 and retried. All are still failing. I read that the gcc 12 and gcc 11 more or less the same in that regard. Sadly it is not easy to go much back from where I am. I need to check on the jenkins and this is based on debian 12. So I could retry with older gccs but I'm not sure what will be the outcome. Right now I'm retrying with pharo-10 HEAD because your commit message sounded it is worth it (well, nothing else works)

Running a vm from HEAD build let's it live longer but in the end it collapses as well. What is really wild now is that I cannot use the old binaries anymore. I have segfaults there, too. So I'm a bit out of ideas

Norbert

Am 11.12.2024 um 15:51 schrieb Guillermo Polito <guillermopolito@gmail.com>:

Hi Norbert, the patch has not been merged yet.

Are we talking about the same?

the pharo-10 branch still has the issue

The PR looks much better. But now I notice that with the compile and install options documented on github the vm in build/dist does not include some libraries, e.g, libcrypto.* libssl.* etc. How is the packaging done for the archives on files.pharo.org? thanks, Norbert

...

Le 11 déc. 2024 à 3:50 PM, Norbert Hartl <norbert@hartl.name> a écrit :

...

Am 11.12.2024 um 15:19 schrieb Norbert Hartl <norbert@hartl.name>:

...

Am 05.12.2024 um 13:49 schrieb Guillermo Polito <guillermopolito@gmail.com>:

Also, anybody having a couple of minutes to test and help, if you see this issue locally, could you - build a VM (https://github.com/pharo-project/pharo-vm/), checking out tag v10.3.2 - run locally the failing tests

This information could help us reduce the search space, and identify if it’s just the build environment or there is something else we have to deal with.

It seems you have figured it out. Since I started to compile my own vm I cannot build anything on my jenkins.

In order to at least confirm a few things I've built a little infrastructure for me to generate vms in different combinations. I went down the rabbit hole and tried v10.21, v10.3.1 and v10.3.2 under debian 12 and all failed. Then I downgraded gcc to gcc-11 and retried. All are still failing. I read that the gcc 12 and gcc 11 more or less the same in that regard. Sadly it is not easy to go much back from where I am. I need to check on the jenkins and this is based on debian 12. So I could retry with older gccs but I'm not sure what will be the outcome. Right now I'm retrying with pharo-10 HEAD because your commit message sounded it is worth it (well, nothing else works)

Running a vm from HEAD build let's it live longer but in the end it collapses as well. What is really wild now is that I cannot use the old binaries anymore. I have segfaults there, too. So I'm a bit out of ideas

Norbert