Status update of Libgit2 and CI issues
A couple of weeks ago we got reported a big recurrent failure on CI jobs while cloning using iceberg/libgit https://github.com/pharo-project/pharo/issues/11481. TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight. =-=-=-=-=-=-=-= More on the situation: Errors happened randomly with ``` error reading from the zlib stream ``` or ``` bad packet length ``` Several people helped in getting a reliable script to reproduce the issue, plus some instructions to produce a reproduction environment using a docker container. With that in our hands, we managed to corner the issue to a new incompatibility between old libgit2 versions, ssh2, OpenSSL, and (at least) Github. After testing some configurations, we decided to upgrade the libgit and related binaries to newer versions: - libgit 1.4.4 - libssh 1.9.0 - OpenSSL 1.1.1k This was of course not as straightforward as we would have liked. Libgit2 1.4.4 was not compatible with its older versions and broke our ffi bindings. We needed to provide new VM binaries for all our major platforms, make a new VM release, upgrade the libgit2 bindings, and make releases of the libgit2 bindings, iceberg, and backport to Pharo 10 and Pharo 9. TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight. Libgit+related compilation instructions are updated in the VM wiki https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-ARM64-Third-Part... https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-X86_64-Third-Par... https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM64-Third-Pa... https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM32-Third-Pa... Libgit bindings Upgrade changes: https://github.com/pharo-vcs/libgit2-pharo-bindings/commit/0e0100f7d0f3a11c3... release: https://github.com/pharo-vcs/libgit2-pharo-bindings/releases/tag/v2.2.6 Iceberg Upgrade pr: https://github.com/pharo-vcs/iceberg/pull/1605 release: https://github.com/pharo-vcs/iceberg/releases/tag/v2.0.7 Upgrading library dependencies in the VM https://github.com/pharo-project/pharo-vm/pull/447 Release VM 9.0.16 https://github.com/pharo-project/pharo-vm/releases/tag/v9.0.16 Backports to Pharo Pharo10 https://github.com/pharo-project/pharo/pull/11545 Pharo9 https://github.com/pharo-project/pharo/pull/11546 We have left for later to upgrade library dependencies for the Windows VM (x86-64, ARM64). Of course, any help in this direction is welcome. Cheers, G in behalf of the team
Thank you for the update. Once the new VM and image versions are built by the CI I will update also the ba-st docker Pharo images, so they start working flawlessly again. On Fri, Aug 19, 2022 at 11:45 AM Guillermo Polito <guillermopolito@gmail.com> wrote:
A couple of weeks ago we got reported a big recurrent failure on CI jobs while cloning using iceberg/libgit https://github.com/pharo-project/pharo/issues/11481.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
=-=-=-=-=-=-=-=
More on the situation: Errors happened randomly with
``` error reading from the zlib stream ```
or ``` bad packet length ```
Several people helped in getting a reliable script to reproduce the issue, plus some instructions to produce a reproduction environment using a docker container. With that in our hands, we managed to corner the issue to a new incompatibility between old libgit2 versions, ssh2, OpenSSL, and (at least) Github.
After testing some configurations, we decided to upgrade the libgit and related binaries to newer versions: - libgit 1.4.4 - libssh 1.9.0 - OpenSSL 1.1.1k
This was of course not as straightforward as we would have liked. Libgit2 1.4.4 was not compatible with its older versions and broke our ffi bindings. We needed to provide new VM binaries for all our major platforms, make a new VM release, upgrade the libgit2 bindings, and make releases of the libgit2 bindings, iceberg, and backport to Pharo 10 and Pharo 9.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
Libgit+related compilation instructions are updated in the VM wiki
https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-ARM64-Third-Part...
https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-X86_64-Third-Par...
https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM64-Third-Pa...
https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM32-Third-Pa...
Libgit bindings Upgrade changes: https://github.com/pharo-vcs/libgit2-pharo-bindings/commit/0e0100f7d0f3a11c3... release: https://github.com/pharo-vcs/libgit2-pharo-bindings/releases/tag/v2.2.6
Iceberg Upgrade pr: https://github.com/pharo-vcs/iceberg/pull/1605 release: https://github.com/pharo-vcs/iceberg/releases/tag/v2.0.7
Upgrading library dependencies in the VM https://github.com/pharo-project/pharo-vm/pull/447 Release VM 9.0.16 https://github.com/pharo-project/pharo-vm/releases/tag/v9.0.16
Backports to Pharo Pharo10 https://github.com/pharo-project/pharo/pull/11545 Pharo9 https://github.com/pharo-project/pharo/pull/11546
We have left for later to upgrade library dependencies for the Windows VM (x86-64, ARM64). Of course, any help in this direction is welcome.
Cheers, G in behalf of the team
Thank you for the update. I followed the thread and I imagine it was not easy to work on this problem. Best regards, Hernán El vie, 19 ago 2022 a las 16:45, Guillermo Polito (< guillermopolito@gmail.com>) escribió:
A couple of weeks ago we got reported a big recurrent failure on CI jobs while cloning using iceberg/libgit https://github.com/pharo-project/pharo/issues/11481.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
=-=-=-=-=-=-=-=
More on the situation: Errors happened randomly with
``` error reading from the zlib stream ```
or ``` bad packet length ```
Several people helped in getting a reliable script to reproduce the issue, plus some instructions to produce a reproduction environment using a docker container. With that in our hands, we managed to corner the issue to a new incompatibility between old libgit2 versions, ssh2, OpenSSL, and (at least) Github.
After testing some configurations, we decided to upgrade the libgit and related binaries to newer versions: - libgit 1.4.4 - libssh 1.9.0 - OpenSSL 1.1.1k
This was of course not as straightforward as we would have liked. Libgit2 1.4.4 was not compatible with its older versions and broke our ffi bindings. We needed to provide new VM binaries for all our major platforms, make a new VM release, upgrade the libgit2 bindings, and make releases of the libgit2 bindings, iceberg, and backport to Pharo 10 and Pharo 9.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
Libgit+related compilation instructions are updated in the VM wiki
https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-ARM64-Third-Part...
https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-X86_64-Third-Par...
https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM64-Third-Pa...
https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM32-Third-Pa...
Libgit bindings Upgrade changes: https://github.com/pharo-vcs/libgit2-pharo-bindings/commit/0e0100f7d0f3a11c3... release: https://github.com/pharo-vcs/libgit2-pharo-bindings/releases/tag/v2.2.6
Iceberg Upgrade pr: https://github.com/pharo-vcs/iceberg/pull/1605 release: https://github.com/pharo-vcs/iceberg/releases/tag/v2.0.7
Upgrading library dependencies in the VM https://github.com/pharo-project/pharo-vm/pull/447 Release VM 9.0.16 https://github.com/pharo-project/pharo-vm/releases/tag/v9.0.16
Backports to Pharo Pharo10 https://github.com/pharo-project/pharo/pull/11545 Pharo9 https://github.com/pharo-project/pharo/pull/11546
We have left for later to upgrade library dependencies for the Windows VM (x86-64, ARM64). Of course, any help in this direction is welcome.
Cheers, G in behalf of the team
Hi all, * after some hiccups, CI servers are up un running. See for example: https://github.com/moosetechnology/Moose/runs/7976358526?check_suite_focus=t... <https://github.com/moosetechnology/Moose/runs/7976358526?check_suite_focus=t...> * Another extra fix to an issue affecting only OSX machines is on the pipe with backports for Pharo10 and Pharo9. https://github.com/pharo-project/pharo/issues/11561 <https://github.com/pharo-project/pharo/issues/11561> Please, tell us if you find some other problem. G
El 19 ago 2022, a las 21:31, Hernán Morales Durand <hernan.morales@gmail.com> escribió:
Thank you for the update. I followed the thread and I imagine it was not easy to work on this problem.
Best regards,
Hernán
El vie, 19 ago 2022 a las 16:45, Guillermo Polito (<guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>>) escribió: A couple of weeks ago we got reported a big recurrent failure on CI jobs while cloning using iceberg/libgit https://github.com/pharo-project/pharo/issues/11481 <https://github.com/pharo-project/pharo/issues/11481>.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
=-=-=-=-=-=-=-=
More on the situation: Errors happened randomly with
``` error reading from the zlib stream ```
or ``` bad packet length ```
Several people helped in getting a reliable script to reproduce the issue, plus some instructions to produce a reproduction environment using a docker container. With that in our hands, we managed to corner the issue to a new incompatibility between old libgit2 versions, ssh2, OpenSSL, and (at least) Github.
After testing some configurations, we decided to upgrade the libgit and related binaries to newer versions: - libgit 1.4.4 - libssh 1.9.0 - OpenSSL 1.1.1k
This was of course not as straightforward as we would have liked. Libgit2 1.4.4 was not compatible with its older versions and broke our ffi bindings. We needed to provide new VM binaries for all our major platforms, make a new VM release, upgrade the libgit2 bindings, and make releases of the libgit2 bindings, iceberg, and backport to Pharo 10 and Pharo 9.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
Libgit+related compilation instructions are updated in the VM wiki https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-ARM64-Third-Part... <https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-ARM64-Third-Part...> https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-X86_64-Third-Par... <https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-X86_64-Third-Par...> https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM64-Third-Pa... <https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM64-Third-Pa...> https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM32-Third-Pa... <https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM32-Third-Pa...>
Libgit bindings Upgrade changes: https://github.com/pharo-vcs/libgit2-pharo-bindings/commit/0e0100f7d0f3a11c3... <https://github.com/pharo-vcs/libgit2-pharo-bindings/commit/0e0100f7d0f3a11c3...> release: https://github.com/pharo-vcs/libgit2-pharo-bindings/releases/tag/v2.2.6 <https://github.com/pharo-vcs/libgit2-pharo-bindings/releases/tag/v2.2.6>
Iceberg Upgrade pr: https://github.com/pharo-vcs/iceberg/pull/1605 <https://github.com/pharo-vcs/iceberg/pull/1605> release: https://github.com/pharo-vcs/iceberg/releases/tag/v2.0.7 <https://github.com/pharo-vcs/iceberg/releases/tag/v2.0.7>
Upgrading library dependencies in the VM https://github.com/pharo-project/pharo-vm/pull/447 <https://github.com/pharo-project/pharo-vm/pull/447> Release VM 9.0.16 https://github.com/pharo-project/pharo-vm/releases/tag/v9.0.16 <https://github.com/pharo-project/pharo-vm/releases/tag/v9.0.16>
Backports to Pharo Pharo10 https://github.com/pharo-project/pharo/pull/11545 <https://github.com/pharo-project/pharo/pull/11545> Pharo9 https://github.com/pharo-project/pharo/pull/11546 <https://github.com/pharo-project/pharo/pull/11546>
We have left for later to upgrade library dependencies for the Windows VM (x86-64, ARM64). Of course, any help in this direction is welcome.
Cheers, G in behalf of the team
yes indeed thanks a lot for handling this during ESUG ! nicolas On 23/08/2022 17:07, Guillermo Polito wrote:
Hi all,
* after some hiccups, CI servers are up un running.
See for example:
https://github.com/moosetechnology/Moose/runs/7976358526?check_suite_focus=t...
* Another extra fix to an issue affecting only OSX machines is on the pipe with backports for Pharo10 and Pharo9.
https://github.com/pharo-project/pharo/issues/11561
Please, tell us if you find some other problem. G
El 19 ago 2022, a las 21:31, Hernán Morales Durand <hernan.morales@gmail.com> escribió:
Thank you for the update. I followed the thread and I imagine it was not easy to work on this problem.
Best regards,
Hernán
El vie, 19 ago 2022 a las 16:45, Guillermo Polito (<guillermopolito@gmail.com>) escribió:
A couple of weeks ago we got reported a big recurrent failure on CI jobs while cloning using iceberg/libgit https://github.com/pharo-project/pharo/issues/11481.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
=-=-=-=-=-=-=-=
More on the situation: Errors happened randomly with
``` error reading from the zlib stream ```
or ``` bad packet length ```
Several people helped in getting a reliable script to reproduce the issue, plus some instructions to produce a reproduction environment using a docker container. With that in our hands, we managed to corner the issue to a new incompatibility between old libgit2 versions, ssh2, OpenSSL, and (at least) Github.
After testing some configurations, we decided to upgrade the libgit and related binaries to newer versions: Â - libgit 1.4.4 Â - libssh 1.9.0 Â - OpenSSL 1.1.1k
This was of course not as straightforward as we would have liked. Libgit2 1.4.4 was not compatible with its older versions and broke our ffi bindings. We needed to provide new VM binaries for all our major platforms, make a new VM release, upgrade the libgit2 bindings, and make releases of the libgit2 bindings, iceberg, and backport to Pharo 10 and Pharo 9.
TL;DR; So far (Friday ~17h Paris time) we managed to get everything up and running. A new VM is being released by our CI, all pull requests are issued and we should have everything integrated by tonight.
Libgit+related compilation instructions are updated in the VM wiki https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-ARM64-Third-Part... https://github.com/pharo-project/pharo-vm/wiki/Building-OSX-X86_64-Third-Par... https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM64-Third-Pa... https://github.com/pharo-project/pharo-vm/wiki/Building-Linux-ARM32-Third-Pa...
Libgit bindings Upgrade   changes: https://github.com/pharo-vcs/libgit2-pharo-bindings/commit/0e0100f7d0f3a11c3...   release: https://github.com/pharo-vcs/libgit2-pharo-bindings/releases/tag/v2.2.6
Iceberg Upgrade   pr: https://github.com/pharo-vcs/iceberg/pull/1605   release: https://github.com/pharo-vcs/iceberg/releases/tag/v2.0.7
Upgrading library dependencies in the VM https://github.com/pharo-project/pharo-vm/pull/447 Â Release VM 9.0.16 https://github.com/pharo-project/pharo-vm/releases/tag/v9.0.16
Backports to Pharo  Pharo10 https://github.com/pharo-project/pharo/pull/11545  Pharo9 https://github.com/pharo-project/pharo/pull/11546
We have left for later to upgrade library dependencies for the Windows VM (x86-64, ARM64). Of course, any help in this direction is welcome.
Cheers, G in behalf of the team
-- Nicolas Anquetil RMod team -- Inria Lille
participants (4)
-
Gabriel Cotelli -
Guillermo Polito -
Hernán Morales Durand -
Nicolas Anquetil