Am 04.07.2018 um 17:52 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

Where can I get my hands on that image that you say has no sender of #newBinaryFileNamed:do: in it?

I cannot give away the image, sorry. I can try to reproduce it somehow. Do you expect the method to be there? Norbert

Max

...

On 4 Jul 2018, at 17:30, Norbert Hartl wrote:

I tried to see how hard it would be to load my current project in pharo7. Cyril helped me to see that I load an old version of zinc and it does not work that way. Upgrading to zinc 2.9.2 made it work in pharo 7 but my jenkins build do not work anymore. I don’t fully understand what is happening. I see in the log

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolMessageNotUnderstood: ZnFileSystemUtils class>>newBinaryFileNamed:do: ZnFileSystemUtils class(Object)>>doesNotUnderstand: #newBinaryFileNamed:do: ZnClient>>downloadEntityTo: ZnClient>>downloadTo: IceMetacelloPharoPlatform(MetacelloPharoPlatform)>>downloadZipArchive:to:

The strange thing about it is that when I download that image where the error occurred I cannot find a sender of #newBinaryFileNamed:do: and the #downloadEntityTo: does not seem to use it. Downgrading to zinc 2.8.10 makes it work again.

I feel a little bit stupid because the image does not load any new package on startup and still there is this error message which does not make sense to me at all.

Any ideas?

Norbert

Ok, I think I’ve found something. If you look at the screenshot (that is my image from jenkins, 6.1 with my code loaded and that throw the error) you can see that Zinc-FileSystem.15 is already loaded. But the Author gives the hint that this was loaded by the image integrator. And it differs from the .15 from Sven’s repo. Exactly the difference that could cause my error. I’m not sure if metacello upgrades only packages that have a higher version number. But that would be the explanation why it doesn’t work for me. I checked it by download a fresh 6.1 image and the situation is the same here. So reproducing the problem should be as easy as creating a baseline that loads zinc 2.9.2 and later something from github. My case was that in the first step the project code was loaded in the image including zinc 2.9.2. That image was saved and on the next invocation it loads the test code and parasol was the first dependency to load from github and it failed. Norbert

Am 05.07.2018 um 08:44 schrieb Max Leske <maxleske@gmail.com>:

On 4 Jul 2018, at 23:00, Norbert Hartl wrote:

...

...

Am 04.07.2018 um 17:52 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

Where can I get my hands on that image that you say has no sender of #newBinaryFileNamed:do: in it?

I cannot give away the image, sorry. I can try to reproduce it somehow. Do you expect the method to be there?

I would have, yes. But as you have discovered, the problem may lie with ZnFileSystemUtils (although I would have expected that package to be updated along with Zinc).

Max

...

Norbert

...

Max

...

On 4 Jul 2018, at 17:30, Norbert Hartl wrote:

I tried to see how hard it would be to load my current project in pharo7. Cyril helped me to see that I load an old version of zinc and it does not work that way. Upgrading to zinc 2.9.2 made it work in pharo 7 but my jenkins build do not work anymore. I don’t fully understand what is happening. I see in the log

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolMessageNotUnderstood: ZnFileSystemUtils class>>newBinaryFileNamed:do: ZnFileSystemUtils class(Object)>>doesNotUnderstand: #newBinaryFileNamed:do: ZnClient>>downloadEntityTo: ZnClient>>downloadTo: IceMetacelloPharoPlatform(MetacelloPharoPlatform)>>downloadZipArchive:to:

The strange thing about it is that when I download that image where the error occurred I cannot find a sender of #newBinaryFileNamed:do: and the #downloadEntityTo: does not seem to use it. Downgrading to zinc 2.8.10 makes it work again.

I feel a little bit stupid because the image does not load any new package on startup and still there is this error message which does not make sense to me at all.

Any ideas?

Norbert

Norbert, I did (on macOS): $ mkdir pharo61 $ cd pharo61/ $ curl get.pharo.org/61+vm | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3036 100 3036 0 0 24386 0 --:--:-- --:--:-- --:--:-- 24483 Downloading the latest 61 Image: http://files.pharo.org/get-files/61/pharo.zip Pharo.image Downloading the latest pharoVM: http://files.pharo.org/get-files/61/pharo-mac-stable.zip pharo-vm/Pharo.app/Contents/MacOS/Pharo Downloading PharoV61.sources: http://files.pharo.org/get-files/61/sources.zip Creating starter scripts pharo and pharo-ui $ ./pharo-ui Pharo.image I loaded ConfigurationOfZincHTTPComponents via the Catalog, but since that was not the latest, I loaded it manually and did ConfigurationOfZincHTTPComponents project latestVersion load. Here is the Transcript Loading 2.8.3 of ConfigurationOfZincHTTPComponents... Fetched -> Zodiac-Core-SvenVanCaekenberghe.42 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Tests-SvenVanCaekenberghe.13 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.29 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-FileSystem-SvenVanCaekenberghe.12 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-HTTP-SvenVanCaekenberghe.453 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Zodiac-SvenVanCaekenberghe.36 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Loaded -> Zodiac-Core-SvenVanCaekenberghe.42 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Tests-SvenVanCaekenberghe.13 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.29 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-FileSystem-SvenVanCaekenberghe.12 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-HTTP-SvenVanCaekenberghe.453 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Zodiac-SvenVanCaekenberghe.36 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ...finished 2.8.3 Loading 2.9.2 of ConfigurationOfZincHTTPComponents... Fetched -> Zodiac-Core-TorstenBergmann.48 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Tests-SvenVanCaekenberghe.22 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Extra-SvenVanCaekenberghe.8 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zinc-Character-Encoding-Core-SvenVanCaekenberghe.64 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.43 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-Core-SvenVanCaekenberghe.68 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.38 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-FileSystem-SvenVanCaekenberghe.15 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-HTTP-SvenVanCaekenberghe.474 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Tests-SvenVanCaekenberghe.245 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Zodiac-SvenVanCaekenberghe.44 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-FileSystem-SvenVanCaekenberghe.9 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Loaded -> Zodiac-Core-TorstenBergmann.48 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Tests-SvenVanCaekenberghe.22 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Extra-SvenVanCaekenberghe.8 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache ZnBase64Encoder (DefaultInverse is Undeclared) ZnBase64Encoder (DefaultAlphabet is Undeclared) ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared) ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared) ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared) ZnSimplifiedByteEncoder class>>byteTextConverters (ByteTextConverters is Undeclared) Loaded -> Zinc-Character-Encoding-Core-SvenVanCaekenberghe.64 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.43 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-Core-SvenVanCaekenberghe.68 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.38 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-FileSystem-SvenVanCaekenberghe.15 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ZnSingleThreadedServer class>>adoptAsDefault: (Default is Undeclared) ZnSingleThreadedServer class>>initialize (Default is Undeclared) ZnSingleThreadedServer class>>default (Default is Undeclared) ZnSingleThreadedServer class>>stopDefault (Default is Undeclared) ZnSingleThreadedServer class>>stopDefault (Default is Undeclared) ZnSingleThreadedServer class>>stopDefault (Default is Undeclared) ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared) ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared) ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared) Loaded -> Zinc-HTTP-SvenVanCaekenberghe.474 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Tests-SvenVanCaekenberghe.245 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Zodiac-SvenVanCaekenberghe.44 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-FileSystem-SvenVanCaekenberghe.9 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ...finished 2.9.2 (The Undeclared references are eventually resolved). The code you mentioned is present in the image. Unit tests (apart from one) are green. === Name: Zinc-FileSystem-SvenVanCaekenberghe.15 Author: SvenVanCaekenberghe Time: 14 March 2018, 2:08:47.155993 pm UUID: 9c55f9cf-c023-0d00-9b4a-76f3014de7ca Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.14 === Sven

On 6 Jul 2018, at 12:50, Norbert Hartl <norbert@hartl.name> wrote:

Bump. I wonder nobody sees the problem. Or do I misunderstand there is something wrong in pharo?

Norbert

...

Am 05.07.2018 um 09:09 schrieb Norbert Hartl <norbert@hartl.name>:

Ok,

I think I’ve found something. If you look at the screenshot (that is my image from jenkins, 6.1 with my code loaded and that throw the error) you can see that Zinc-FileSystem.15 is already loaded. But the Author gives the hint that this was loaded by the image integrator. And it differs from the .15 from Sven’s repo. Exactly the difference that could cause my error. I’m not sure if metacello upgrades only packages that have a higher version number. But that would be the explanation why it doesn’t work for me. I checked it by download a fresh 6.1 image and the situation is the same here. So reproducing the problem should be as easy as creating a baseline that loads zinc 2.9.2 and later something from github. My case was that in the first step the project code was loaded in the image including zinc 2.9.2. That image was saved and on the next invocation it loads the test code and parasol was the first dependency to load from github and it failed.

Norbert

<Bildschirmfoto 2018-07-05 um 08.59.10.png>

...

Am 05.07.2018 um 08:44 schrieb Max Leske <maxleske@gmail.com>:

On 4 Jul 2018, at 23:00, Norbert Hartl wrote:

...

...

Am 04.07.2018 um 17:52 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

Where can I get my hands on that image that you say has no sender of #newBinaryFileNamed:do: in it?

I cannot give away the image, sorry. I can try to reproduce it somehow. Do you expect the method to be there?

I would have, yes. But as you have discovered, the problem may lie with ZnFileSystemUtils (although I would have expected that package to be updated along with Zinc).

Max

...

Norbert

...

Max

...

On 4 Jul 2018, at 17:30, Norbert Hartl wrote:

I tried to see how hard it would be to load my current project in pharo7. Cyril helped me to see that I load an old version of zinc and it does not work that way. Upgrading to zinc 2.9.2 made it work in pharo 7 but my jenkins build do not work anymore. I don’t fully understand what is happening. I see in the log

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolMessageNotUnderstood: ZnFileSystemUtils class>>newBinaryFileNamed:do: ZnFileSystemUtils class(Object)>>doesNotUnderstand: #newBinaryFileNamed:do: ZnClient>>downloadEntityTo: ZnClient>>downloadTo: IceMetacelloPharoPlatform(MetacelloPharoPlatform)>>downloadZipArchive:to:

The strange thing about it is that when I download that image where the error occurred I cannot find a sender of #newBinaryFileNamed:do: and the #downloadEntityTo: does not seem to use it. Downgrading to zinc 2.8.10 makes it work again.

I feel a little bit stupid because the image does not load any new package on startup and still there is this error message which does not make sense to me at all.

Any ideas?

Norbert

Can anyone explain how zinc is managed? The package Zinc-FileSystem in pharo 6.1 is Name: Zinc-FileSystem-TheIntegrator.15 Author: TheIntegrator Time: 15 March 2017, 1:16:56.199425 pm UUID: b9684fa8-2507-0d00-a4da-a21207d152a4 Ancestors: Zinc-FileSystem-HenrikNergaard.14 19838 ZnMonticelloServerDelegate>>#repositoryListing sends self self https://pharo.fogbugz.com/f/cases/19838 <https://pharo.fogbugz.com/f/cases/19838> In Sven’s repo there is Name: Zinc-FileSystem-SvenVanCaekenberghe.14 Author: SvenVanCaekenberghe Time: 14 March 2018, 11:29:06.12468 am UUID: 885ce694-be23-0d00-a4f2-9f24027c8ee7 Ancestors: Zinc-FileSystem-GitHub.1520938892, Zinc-FileSystem-SvenVanCaekenberghe.13 tracking pharo 7 changes and Name: Zinc-FileSystem-SvenVanCaekenberghe.15 Author: SvenVanCaekenberghe Time: 14 March 2018, 2:08:47.155993 pm UUID: 9c55f9cf-c023-0d00-9b4a-76f3014de7ca Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.14 patch to dynamically work on older pharo versions How do this packages relate? Norbert

Am 06.07.2018 um 16:34 schrieb Norbert Hartl <norbert@hartl.name>:

Sven,

I use Metacello and that seems to be different in behaviour. I prepared the case for you. What I did

- save the attachment of this mail into a clean directory and open a terminal there

$> unzip baseline-of-zincproblem.zip $> curl get.pharo.org/64/61 <http://get.pharo.org/64/61> | bash $> /Applications/Pharo64.app/Contents/MacOS/Pharo ./Pharo.image eval „Metacello new repository: 'filetree://src <filetree://src>'; baseline: 'ZincProblem'; load"

Look at ZnClient>>#downloadEntityTo:

Norbert

<baseline-of-zincproblem.zip>

...

Am 06.07.2018 um 13:19 schrieb Sven Van Caekenberghe <sven@stfx.eu <mailto:sven@stfx.eu>>:

Norbert,

I did (on macOS):

$ mkdir pharo61

$ cd pharo61/

$ curl get.pharo.org/61+vm <http://get.pharo.org/61+vm> | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3036 100 3036 0 0 24386 0 --:--:-- --:--:-- --:--:-- 24483 Downloading the latest 61 Image: http://files.pharo.org/get-files/61/pharo.zip <http://files.pharo.org/get-files/61/pharo.zip> Pharo.image Downloading the latest pharoVM: http://files.pharo.org/get-files/61/pharo-mac-stable.zip <http://files.pharo.org/get-files/61/pharo-mac-stable.zip> pharo-vm/Pharo.app/Contents/MacOS/Pharo Downloading PharoV61.sources: http://files.pharo.org/get-files/61/sources.zip Creating starter scripts pharo and pharo-ui

$ ./pharo-ui Pharo.image

I loaded ConfigurationOfZincHTTPComponents via the Catalog, but since that was not the latest, I loaded it manually and did

ConfigurationOfZincHTTPComponents project latestVersion load.

Here is the Transcript

Loading 2.8.3 of ConfigurationOfZincHTTPComponents... Fetched -> Zodiac-Core-SvenVanCaekenberghe.42 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Tests-SvenVanCaekenberghe.13 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.29 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-FileSystem-SvenVanCaekenberghe.12 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-HTTP-SvenVanCaekenberghe.453 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Zodiac-SvenVanCaekenberghe.36 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Loaded -> Zodiac-Core-SvenVanCaekenberghe.42 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Tests-SvenVanCaekenberghe.13 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.29 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-FileSystem-SvenVanCaekenberghe.12 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-HTTP-SvenVanCaekenberghe.453 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Zodiac-SvenVanCaekenberghe.36 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ...finished 2.8.3

Loading 2.9.2 of ConfigurationOfZincHTTPComponents... Fetched -> Zodiac-Core-TorstenBergmann.48 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Tests-SvenVanCaekenberghe.22 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Extra-SvenVanCaekenberghe.8 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zinc-Character-Encoding-Core-SvenVanCaekenberghe.64 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.43 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-Core-SvenVanCaekenberghe.68 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.38 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-FileSystem-SvenVanCaekenberghe.15 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-HTTP-SvenVanCaekenberghe.474 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Tests-SvenVanCaekenberghe.245 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Zodiac-SvenVanCaekenberghe.44 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-FileSystem-SvenVanCaekenberghe.9 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Loaded -> Zodiac-Core-TorstenBergmann.48 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Tests-SvenVanCaekenberghe.22 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Extra-SvenVanCaekenberghe.8 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache ZnBase64Encoder (DefaultInverse is Undeclared) ZnBase64Encoder (DefaultAlphabet is Undeclared) ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared)

ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared)

ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared)

ZnSimplifiedByteEncoder class>>byteTextConverters (ByteTextConverters is Undeclared)

Loaded -> Zinc-Character-Encoding-Core-SvenVanCaekenberghe.64 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.43 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-Core-SvenVanCaekenberghe.68 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.38 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-FileSystem-SvenVanCaekenberghe.15 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ZnSingleThreadedServer class>>adoptAsDefault: (Default is Undeclared)

ZnSingleThreadedServer class>>initialize (Default is Undeclared)

ZnSingleThreadedServer class>>default (Default is Undeclared)

ZnSingleThreadedServer class>>stopDefault (Default is Undeclared)

ZnSingleThreadedServer class>>stopDefault (Default is Undeclared)

ZnSingleThreadedServer class>>stopDefault (Default is Undeclared)

ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared)

ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared)

ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared)

Loaded -> Zinc-HTTP-SvenVanCaekenberghe.474 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Tests-SvenVanCaekenberghe.245 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Zodiac-SvenVanCaekenberghe.44 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-FileSystem-SvenVanCaekenberghe.9 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ...finished 2.9.2

(The Undeclared references are eventually resolved).

The code you mentioned is present in the image. Unit tests (apart from one) are green.

=== Name: Zinc-FileSystem-SvenVanCaekenberghe.15 Author: SvenVanCaekenberghe Time: 14 March 2018, 2:08:47.155993 pm UUID: 9c55f9cf-c023-0d00-9b4a-76f3014de7ca Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.14 ===

Sven

...

On 6 Jul 2018, at 12:50, Norbert Hartl <norbert@hartl.name> wrote:

Bump. I wonder nobody sees the problem. Or do I misunderstand there is something wrong in pharo?

Norbert

...

Am 05.07.2018 um 09:09 schrieb Norbert Hartl <norbert@hartl.name>:

Ok,

I think I’ve found something. If you look at the screenshot (that is my image from jenkins, 6.1 with my code loaded and that throw the error) you can see that Zinc-FileSystem.15 is already loaded. But the Author gives the hint that this was loaded by the image integrator. And it differs from the .15 from Sven’s repo. Exactly the difference that could cause my error. I’m not sure if metacello upgrades only packages that have a higher version number. But that would be the explanation why it doesn’t work for me. I checked it by download a fresh 6.1 image and the situation is the same here. So reproducing the problem should be as easy as creating a baseline that loads zinc 2.9.2 and later something from github. My case was that in the first step the project code was loaded in the image including zinc 2.9.2. That image was saved and on the next invocation it loads the test code and parasol was the first dependency to load from github and it failed.

Norbert

<Bildschirmfoto 2018-07-05 um 08.59.10.png>

...

Am 05.07.2018 um 08:44 schrieb Max Leske <maxleske@gmail.com>:

On 4 Jul 2018, at 23:00, Norbert Hartl wrote:

...

...

Am 04.07.2018 um 17:52 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

Where can I get my hands on that image that you say has no sender of #newBinaryFileNamed:do: in it?

I cannot give away the image, sorry. I can try to reproduce it somehow. Do you expect the method to be there?

I would have, yes. But as you have discovered, the problem may lie with ZnFileSystemUtils (although I would have expected that package to be updated along with Zinc).

Max

...

Norbert

...

Max

...

On 4 Jul 2018, at 17:30, Norbert Hartl wrote:

I tried to see how hard it would be to load my current project in pharo7. Cyril helped me to see that I load an old version of zinc and it does not work that way. Upgrading to zinc 2.9.2 made it work in pharo 7 but my jenkins build do not work anymore. I don’t fully understand what is happening. I see in the log

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolMessageNotUnderstood: ZnFileSystemUtils class>>newBinaryFileNamed:do: ZnFileSystemUtils class(Object)>>doesNotUnderstand: #newBinaryFileNamed:do: ZnClient>>downloadEntityTo: ZnClient>>downloadTo: IceMetacelloPharoPlatform(MetacelloPharoPlatform)>>downloadZipArchive:to:

The strange thing about it is that when I download that image where the error occurred I cannot find a sender of #newBinaryFileNamed:do: and the #downloadEntityTo: does not seem to use it. Downgrading to zinc 2.8.10 makes it work again.

I feel a little bit stupid because the image does not load any new package on startup and still there is this error message which does not make sense to me at all.

Any ideas?

Norbert

Am 07.07.2018 um 12:53 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

Obviously something went wrong somewhere, but I cannot see what exactly. It is a while ago, and integration processes have been in flux.

Zinc is being developed in its own repo(s), but sometimes changes are made in the main Pharo repo that get copied back upstream. From time to time, Zinc itself is updated in Pharo itself.

If Zinc-FileSystem-TheIntegrator.15 is indeed different from Zinc-FileSystem-SvenVanCaekenberghe.15 then the best solution would be to merge them and create a newer, more correct version.

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight. I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image. I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step. Norbert

...

On 7 Jul 2018, at 12:29, Norbert Hartl <norbert@hartl.name> wrote:

Can anyone explain how zinc is managed? The package Zinc-FileSystem in pharo 6.1 is

Name: Zinc-FileSystem-TheIntegrator.15 Author: TheIntegrator Time: 15 March 2017, 1:16:56.199425 pm UUID: b9684fa8-2507-0d00-a4da-a21207d152a4 Ancestors: Zinc-FileSystem-HenrikNergaard.14

19838 ZnMonticelloServerDelegate>>#repositoryListing sends self self https://pharo.fogbugz.com/f/cases/19838

In Sven’s repo there is

Name: Zinc-FileSystem-SvenVanCaekenberghe.14 Author: SvenVanCaekenberghe Time: 14 March 2018, 11:29:06.12468 am UUID: 885ce694-be23-0d00-a4f2-9f24027c8ee7 Ancestors: Zinc-FileSystem-GitHub.1520938892, Zinc-FileSystem-SvenVanCaekenberghe.13

tracking pharo 7 changes

and

Name: Zinc-FileSystem-SvenVanCaekenberghe.15 Author: SvenVanCaekenberghe Time: 14 March 2018, 2:08:47.155993 pm UUID: 9c55f9cf-c023-0d00-9b4a-76f3014de7ca Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.14

patch to dynamically work on older pharo versions

How do this packages relate?

Norbert

...

Am 06.07.2018 um 16:34 schrieb Norbert Hartl <norbert@hartl.name>:

Sven,

I use Metacello and that seems to be different in behaviour. I prepared the case for you. What I did

- save the attachment of this mail into a clean directory and open a terminal there

$> unzip baseline-of-zincproblem.zip $> curl get.pharo.org/64/61 | bash $> /Applications/Pharo64.app/Contents/MacOS/Pharo ./Pharo.image eval „Metacello new repository: 'filetree://src'; baseline: 'ZincProblem'; load"

Look at ZnClient>>#downloadEntityTo:

Norbert

<baseline-of-zincproblem.zip>

...

Am 06.07.2018 um 13:19 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

Norbert,

I did (on macOS):

$ mkdir pharo61

$ cd pharo61/

$ curl get.pharo.org/61+vm | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3036 100 3036 0 0 24386 0 --:--:-- --:--:-- --:--:-- 24483 Downloading the latest 61 Image: http://files.pharo.org/get-files/61/pharo.zip Pharo.image Downloading the latest pharoVM: http://files.pharo.org/get-files/61/pharo-mac-stable.zip pharo-vm/Pharo.app/Contents/MacOS/Pharo Downloading PharoV61.sources: http://files.pharo.org/get-files/61/sources.zip Creating starter scripts pharo and pharo-ui

$ ./pharo-ui Pharo.image

I loaded ConfigurationOfZincHTTPComponents via the Catalog, but since that was not the latest, I loaded it manually and did

ConfigurationOfZincHTTPComponents project latestVersion load.

Here is the Transcript

Loading 2.8.3 of ConfigurationOfZincHTTPComponents... Fetched -> Zodiac-Core-SvenVanCaekenberghe.42 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Tests-SvenVanCaekenberghe.13 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.29 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-FileSystem-SvenVanCaekenberghe.12 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-HTTP-SvenVanCaekenberghe.453 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Zodiac-SvenVanCaekenberghe.36 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Loaded -> Zodiac-Core-SvenVanCaekenberghe.42 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Tests-SvenVanCaekenberghe.13 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.29 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-FileSystem-SvenVanCaekenberghe.12 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-HTTP-SvenVanCaekenberghe.453 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Zodiac-SvenVanCaekenberghe.36 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ...finished 2.8.3

Loading 2.9.2 of ConfigurationOfZincHTTPComponents... Fetched -> Zodiac-Core-TorstenBergmann.48 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Tests-SvenVanCaekenberghe.22 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zodiac-Extra-SvenVanCaekenberghe.8 --- http://mc.stfx.eu/Zodiac --- http://mc.stfx.eu/Zodiac Fetched -> Zinc-Character-Encoding-Core-SvenVanCaekenberghe.64 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.43 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-Core-SvenVanCaekenberghe.68 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.38 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-FileSystem-SvenVanCaekenberghe.15 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-HTTP-SvenVanCaekenberghe.474 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Tests-SvenVanCaekenberghe.245 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Zodiac-SvenVanCaekenberghe.44 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Fetched -> Zinc-Resource-Meta-FileSystem-SvenVanCaekenberghe.9 --- http://mc.stfx.eu/ZincHTTPComponents --- http://mc.stfx.eu/ZincHTTPComponents Loaded -> Zodiac-Core-TorstenBergmann.48 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Tests-SvenVanCaekenberghe.22 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache Loaded -> Zodiac-Extra-SvenVanCaekenberghe.8 --- http://mc.stfx.eu/Zodiac --- /private/tmp/pharo61/pharo-local/package-cache ZnBase64Encoder (DefaultInverse is Undeclared) ZnBase64Encoder (DefaultAlphabet is Undeclared) ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared)

ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared)

ZnSimplifiedByteEncoder class>>initializeByteTextConverters (ByteTextConverters is Undeclared)

ZnSimplifiedByteEncoder class>>byteTextConverters (ByteTextConverters is Undeclared)

Loaded -> Zinc-Character-Encoding-Core-SvenVanCaekenberghe.64 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Character-Encoding-Tests-SvenVanCaekenberghe.43 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-Core-SvenVanCaekenberghe.68 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-Tests-SvenVanCaekenberghe.38 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-FileSystem-SvenVanCaekenberghe.15 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ZnSingleThreadedServer class>>adoptAsDefault: (Default is Undeclared)

ZnSingleThreadedServer class>>initialize (Default is Undeclared)

ZnSingleThreadedServer class>>default (Default is Undeclared)

ZnSingleThreadedServer class>>stopDefault (Default is Undeclared)

ZnSingleThreadedServer class>>stopDefault (Default is Undeclared)

ZnSingleThreadedServer class>>stopDefault (Default is Undeclared)

ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared)

ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared)

ZnSingleThreadedServer class>>defaultOn: (Default is Undeclared)

Loaded -> Zinc-HTTP-SvenVanCaekenberghe.474 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Tests-SvenVanCaekenberghe.245 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Zodiac-SvenVanCaekenberghe.44 --- http://mc.stfx.eu/ZincHTTPComponents --- cache Loaded -> Zinc-Resource-Meta-FileSystem-SvenVanCaekenberghe.9 --- http://mc.stfx.eu/ZincHTTPComponents --- cache ...finished 2.9.2

(The Undeclared references are eventually resolved).

The code you mentioned is present in the image. Unit tests (apart from one) are green.

=== Name: Zinc-FileSystem-SvenVanCaekenberghe.15 Author: SvenVanCaekenberghe Time: 14 March 2018, 2:08:47.155993 pm UUID: 9c55f9cf-c023-0d00-9b4a-76f3014de7ca Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.14 ===

Sven

...

On 6 Jul 2018, at 12:50, Norbert Hartl <norbert@hartl.name> wrote:

Bump. I wonder nobody sees the problem. Or do I misunderstand there is something wrong in pharo?

Norbert

...

Am 05.07.2018 um 09:09 schrieb Norbert Hartl <norbert@hartl.name>:

Ok,

I think I’ve found something. If you look at the screenshot (that is my image from jenkins, 6.1 with my code loaded and that throw the error) you can see that Zinc-FileSystem.15 is already loaded. But the Author gives the hint that this was loaded by the image integrator. And it differs from the .15 from Sven’s repo. Exactly the difference that could cause my error. I’m not sure if metacello upgrades only packages that have a higher version number. But that would be the explanation why it doesn’t work for me. I checked it by download a fresh 6.1 image and the situation is the same here. So reproducing the problem should be as easy as creating a baseline that loads zinc 2.9.2 and later something from github. My case was that in the first step the project code was loaded in the image including zinc 2.9.2. That image was saved and on the next invocation it loads the test code and parasol was the first dependency to load from github and it failed.

Norbert

<Bildschirmfoto 2018-07-05 um 08.59.10.png>

...

Am 05.07.2018 um 08:44 schrieb Max Leske <maxleske@gmail.com>:

On 4 Jul 2018, at 23:00, Norbert Hartl wrote:

...

...

Am 04.07.2018 um 17:52 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

Where can I get my hands on that image that you say has no sender of #newBinaryFileNamed:do: in it?

I cannot give away the image, sorry. I can try to reproduce it somehow. Do you expect the method to be there?

I would have, yes. But as you have discovered, the problem may lie with ZnFileSystemUtils (although I would have expected that package to be updated along with Zinc).

Max

...

Norbert

...

Max

...

On 4 Jul 2018, at 17:30, Norbert Hartl wrote:

I tried to see how hard it would be to load my current project in pharo7. Cyril helped me to see that I load an old version of zinc and it does not work that way. Upgrading to zinc 2.9.2 made it work in pharo 7 but my jenkins build do not work anymore. I don’t fully understand what is happening. I see in the log

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolMessageNotUnderstood: ZnFileSystemUtils class>>newBinaryFileNamed:do: ZnFileSystemUtils class(Object)>>doesNotUnderstand: #newBinaryFileNamed:do: ZnClient>>downloadEntityTo: ZnClient>>downloadTo: IceMetacelloPharoPlatform(MetacelloPharoPlatform)>>downloadZipArchive:to:

The strange thing about it is that when I download that image where the error occurred I cannot find a sender of #newBinaryFileNamed:do: and the #downloadEntityTo: does not seem to use it. Downgrading to zinc 2.8.10 makes it work again.

I feel a little bit stupid because the image does not load any new package on startup and still there is this error message which does not make sense to me at all.

Any ideas?

Norbert

Really? I thought the same but then I didn’t believe it works like that. Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high. Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com>:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr <https://ferlicot.fr/>

Hi Norbert, This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1. Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries). Cheers, Max On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com <mailto:cyril.ferlicot@gmail.com>>:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr <https://ferlicot.fr/>

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file. I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start... Max On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

...

Am 09.07.2018 um 18:18 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

...

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

...

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com <mailto:cyril.ferlicot@gmail.com>>:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr <https://ferlicot.fr/>

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta... BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely). Cheers, Max On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

...

Am 09.07.2018 um 19:10 schrieb Max Leske <maxleske@gmail.com>:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

...

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

...

Am 09.07.2018 um 18:18 schrieb Max Leske <maxleske@gmail.com <mailto:maxleske@gmail.com>>:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

...

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

...

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name <mailto:norbert@hartl.name>>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com <mailto:cyril.ferlicot@gmail.com>>:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr <https://ferlicot.fr/>

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

...

Am 10.07.2018 um 08:37 schrieb Max Leske <maxleske@gmail.com>:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

...

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

...

Am 09.07.2018 um 19:10 schrieb Max Leske <maxleske@gmail.com>:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

...

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

...

Am 09.07.2018 um 18:18 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

...

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

...

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com>:

...

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

...

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr

Thank you Sven. That fixed my first problem. Sadly the other problem is more persitent than this. Norbert

Am 11.07.2018 um 16:46 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

FWIW, I created a new #stable version 2.9.3 that includes a newer Zinc-FileSystem-SvenVanCaekenberghe.16 - maybe this helps.

=== Name: Zinc-FileSystem-SvenVanCaekenberghe.16 Author: SvenVanCaekenberghe Time: 11 July 2018, 4:30:36.923152 pm UUID: 830fbcd3-1b2d-0d00-bd45-164704867404 Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.15

Force an update (class comment changed) ===

...

On 10 Jul 2018, at 09:17, Max Leske <maxleske@gmail.com> wrote:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

...

Max,

thanks for taking the effort.

No worries.

...

...

Am 10.07.2018 um 08:37 schrieb Max Leske <maxleske@gmail.com>:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

...

Norbert

...

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

...

Am 09.07.2018 um 19:10 schrieb Max Leske <maxleske@gmail.com>:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

...

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

...

Am 09.07.2018 um 18:18 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

...

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

...

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com>:

...

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

...

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr

Am 11.07.2018 um 20:19 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

...

On 11 Jul 2018, at 19:50, Norbert Hartl <norbert@hartl.name> wrote:

Thank you Sven. That fixed my first problem. Sadly the other problem is more persitent than this.

I would guess that the other problem (the ZipArchive usage) might be related to the recent file/stream changes in Pharo 7 and not related to the loading of Zinc. The bug might be a binary/text stream mixup or the usage of deprecated code.

That’d be my guess, too. Unfortunately it occurs where debugging is difficult. Norbert

...

Norbert

...

Am 11.07.2018 um 16:46 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

FWIW, I created a new #stable version 2.9.3 that includes a newer Zinc-FileSystem-SvenVanCaekenberghe.16 - maybe this helps.

=== Name: Zinc-FileSystem-SvenVanCaekenberghe.16 Author: SvenVanCaekenberghe Time: 11 July 2018, 4:30:36.923152 pm UUID: 830fbcd3-1b2d-0d00-bd45-164704867404 Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.15

Force an update (class comment changed) ===

...

On 10 Jul 2018, at 09:17, Max Leske <maxleske@gmail.com> wrote:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

...

Max,

thanks for taking the effort.

No worries.

...

...

Am 10.07.2018 um 08:37 schrieb Max Leske <maxleske@gmail.com>:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

...

Norbert

...

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

...

Am 09.07.2018 um 19:10 schrieb Max Leske <maxleske@gmail.com>:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

...

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

...

Am 09.07.2018 um 18:18 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

...

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

...

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com>:

...

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

...

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though. Max

Norbert

...

Am 10.07.2018 um 09:17 schrieb Max Leske <maxleske@gmail.com>:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

...

Max,

thanks for taking the effort.

No worries.

...

...

Am 10.07.2018 um 08:37 schrieb Max Leske <maxleske@gmail.com>:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

...

Norbert

...

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

...

Am 09.07.2018 um 19:10 schrieb Max Leske <maxleske@gmail.com>:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

...

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

...

Am 09.07.2018 um 18:18 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

...

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

...

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com>:

...

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

...

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr

Hi Norbert, I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc. Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding `outputStream flush` as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple `close()` on the file which, according to the man page on `close()`, does *not* guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again. Here's a smaller test case for you to play with Sven: ``` ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size. ``` The output in the Transcript should be: ``` 315392 318420 ``` Cheers, Max On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

...

Am 12.07.2018 um 08:05 schrieb Max Leske <maxleske@gmail.com>:

...

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

...

Max

...

Norbert

...

Am 10.07.2018 um 09:17 schrieb Max Leske <maxleske@gmail.com>:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

...

Max,

thanks for taking the effort.

No worries.

...

...

Am 10.07.2018 um 08:37 schrieb Max Leske <maxleske@gmail.com>:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

...

Norbert

...

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

...

Am 09.07.2018 um 19:10 schrieb Max Leske <maxleske@gmail.com>:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

...

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

...

Am 09.07.2018 um 18:18 schrieb Max Leske <maxleske@gmail.com>:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

...

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

...

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasolError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

...

Am 07.07.2018 um 13:28 schrieb Norbert Hartl <norbert@hartl.name>:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

...

...

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot <cyril.ferlicot@gmail.com>:

...

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

...

Norbert

...

-- Cyril Ferlicot https://ferlicot.fr

Gosh , I’m always impressed how you guys figure this stuff out. Many thanks to all of you for coming up with a test case and solution(s). It’s a great community. Tim Sent from my iPhone

On 13 Jul 2018, at 00:19, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com> wrote:

Isn't there a recent PR on opensmalltalk that address just this?

...

Le jeu. 12 juil. 2018 à 23:16, Max Leske <maxleske@gmail.com> a écrit : Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does not guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size. The output in the Transcript should be:

315392 318420 Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr

Max, what do you think will be a proper fix for this? I cannot see if this is a vm problem or not. Norbert

Am 14.07.2018 um 08:25 schrieb Max Leske <maxleske@gmail.com>:

Hi Nicolas,

The PR you are referring [1] to concerns a missing fflush() when truncating a file. I believe that is a different case to this one, as a) the file is not truncated and b) in the case of truncation it is an explicit requirement that the file be empty before the first write happens. In our case, and in general, it depends on the situation whether flushing must occur immediately, e.g. because the file will be read again immediately, or not, e.g. when writing a data to a backup file.

Thanks for the pointer though.

Cheers, Max

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/271

...

On 12 Jul 2018, at 23:19, Nicolas Cellier wrote:

Isn't there a recent PR on opensmalltalk that address just this?

...

Le jeu. 12 juil. 2018 à 23:16, Max Leske <maxleske@gmail.com> a écrit :

Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does *not* guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

The output in the Transcript should be:

315392 318420

Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr

I tried running Max's snippet (Pharo 6.1 on Ubuntu 16.04 LTS), ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size. And it gives me, 318420 318420 For me there is nothing nothing to see here ... and I have the impression we are all talking about different things.

On 21 Jul 2018, at 14:18, Max Leske <maxleske@gmail.com> wrote:

On 21 Jul 2018, at 12:07, Norbert Hartl wrote:

...

Max,

what do you think will be a proper fix for this? I cannot see if this is a vm problem or not.

In my opinion it should be fixed in Zinc as there's no way for the user of Zinc to flush the stream. The alternative would be for Zinc to expose a method or setting that allows the user to specify the expected behaviour. I do not think that this should be handled by the VM plugin or the stream class, as there are indeed different behaviours that make sense.

Cheers, Max

...

Norbert

...

Am 14.07.2018 um 08:25 schrieb Max Leske <maxleske@gmail.com>:

Hi Nicolas,

The PR you are referring [1] to concerns a missing fflush() when truncating a file. I believe that is a different case to this one, as a) the file is not truncated and b) in the case of truncation it is an explicit requirement that the file be empty before the first write happens. In our case, and in general, it depends on the situation whether flushing must occur immediately, e.g. because the file will be read again immediately, or not, e.g. when writing a data to a backup file.

Thanks for the pointer though.

Cheers, Max

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/271

...

On 12 Jul 2018, at 23:19, Nicolas Cellier wrote:

Isn't there a recent PR on opensmalltalk that address just this?

...

Le jeu. 12 juil. 2018 à 23:16, Max Leske <maxleske@gmail.com> a écrit :

Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does *not* guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

The output in the Transcript should be:

315392 318420

Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr

Sven, you're right. What I found last time works as a side-effect. The real problem is in ZnFileSystemUtils class>>newBinaryFileNamed:do:. In Pharo 6.1 the logic there leads to the following evaluation: `block value: (self binaryFileStreamFor: fileName)` This is the only place where the stream is created without a surrounding safety block (e.g. #writeStream:do:), which means, no one closes the stream. The line should be: `| s | s := self binaryFileStreamFor: fileName. [ block value: s ] ensure: [ s close ]` ZnFileSystemUtils class>>newBinaryFileNamed:do: is only used by ZnClient after loading Zinc-HTTP-SvenVanCaekenberghe.472, which is being loaded by version 2.9.1 of ConfigurationOfZincHTTPComponents. Cheers, Max On 21 Jul 2018, at 17:21, Max Leske wrote:

Just to be safe, I'll redo my experiment and get back to you.

On 21 Jul 2018, at 16:59, Sven Van Caekenberghe wrote:

...

I tried running Max's snippet (Pharo 6.1 on Ubuntu 16.04 LTS),

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

And it gives me,

318420 318420

For me there is nothing nothing to see here ... and I have the impression we are all talking about different things.

...

On 21 Jul 2018, at 14:18, Max Leske <maxleske@gmail.com> wrote:

On 21 Jul 2018, at 12:07, Norbert Hartl wrote:

...

Max,

what do you think will be a proper fix for this? I cannot see if this is a vm problem or not.

In my opinion it should be fixed in Zinc as there's no way for the user of Zinc to flush the stream. The alternative would be for Zinc to expose a method or setting that allows the user to specify the expected behaviour. I do not think that this should be handled by the VM plugin or the stream class, as there are indeed different behaviours that make sense.

Cheers, Max

...

Norbert

...

Am 14.07.2018 um 08:25 schrieb Max Leske <maxleske@gmail.com>:

Hi Nicolas,

The PR you are referring [1] to concerns a missing fflush() when truncating a file. I believe that is a different case to this one, as a) the file is not truncated and b) in the case of truncation it is an explicit requirement that the file be empty before the first write happens. In our case, and in general, it depends on the situation whether flushing must occur immediately, e.g. because the file will be read again immediately, or not, e.g. when writing a data to a backup file.

Thanks for the pointer though.

Cheers, Max

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/271

...

On 12 Jul 2018, at 23:19, Nicolas Cellier wrote:

Isn't there a recent PR on opensmalltalk that address just this?

...

Le jeu. 12 juil. 2018 à 23:16, Max Leske <maxleske@gmail.com> a écrit :

Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does *not* guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

The output in the Transcript should be:

315392 318420

Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr

Thank you guys! It is really great to see that there are people like Max jumping onto problems like this and solving it. @sven I’m eager to test when the new configuration is available. Please drop us a note when you are done! thanks again, Norbert

Am 22.07.2018 um 11:06 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

Ah, that makes total sense. Great catch. Thank you.

=== Name: Zinc-FileSystem-SvenVanCaekenberghe.17 Author: SvenVanCaekenberghe Time: 22 July 2018, 11:04:40.148339 am UUID: e97a508e-f42d-0d00-a127-f016078b1897 Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.16

Bugfix to ZnFileSystemUtils class>>#newBinaryFileNamed:do: we must ensure that the binaryStream opened here is closed as well (thx Max Leske for the bug hunting and fix) ===

The method in question now look as follows:

newBinaryFileNamed: fileName do: block | fileReference | fileReference := fileName asFileReference. ^ (fileReference respondsTo: #binaryWriteStreamDo:ifPresent:) ifTrue: [ fileName asFileReference binaryWriteStreamDo: block ifPresent: [ FileExists signalWith: fileReference ] ] ifFalse: [ fileReference isFile ifTrue: [ FileExists signalWith: fileReference ] ifFalse: [ | binaryStream | binaryStream := self binaryFileStreamFor: fileName. [ block value: binaryStream ] ensure: [ binaryStream close ] ] ]

I create a new 2.9.4 of the config later on.

...

On 21 Jul 2018, at 22:00, Max Leske <maxleske@gmail.com> wrote:

Sven, you're right. What I found last time works as a side-effect. The real problem is in ZnFileSystemUtils class>>newBinaryFileNamed:do:. In Pharo 6.1 the logic there leads to the following evaluation:

block value: (self binaryFileStreamFor: fileName)

This is the only place where the stream is created without a surrounding safety block (e.g. #writeStream:do:), which means, no one closes the stream. The line should be:

| s | s := self binaryFileStreamFor: fileName. [ block value: s ] ensure: [ s close ]

ZnFileSystemUtils class>>newBinaryFileNamed:do: is only used by ZnClient after loading Zinc-HTTP-SvenVanCaekenberghe.472, which is being loaded by version 2.9.1 of ConfigurationOfZincHTTPComponents.

Cheers, Max

On 21 Jul 2018, at 17:21, Max Leske wrote:

Just to be safe, I'll redo my experiment and get back to you.

On 21 Jul 2018, at 16:59, Sven Van Caekenberghe wrote: I tried running Max's snippet (Pharo 6.1 on Ubuntu 16.04 LTS),

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

And it gives me,

318420 318420

For me there is nothing nothing to see here ... and I have the impression we are all talking about different things. On 21 Jul 2018, at 14:18, Max Leske <maxleske@gmail.com> wrote:

On 21 Jul 2018, at 12:07, Norbert Hartl wrote: Max,

what do you think will be a proper fix for this? I cannot see if this is a vm problem or not.

In my opinion it should be fixed in Zinc as there's no way for the user of Zinc to flush the stream. The alternative would be for Zinc to expose a method or setting that allows the user to specify the expected behaviour. I do not think that this should be handled by the VM plugin or the stream class, as there are indeed different behaviours that make sense.

Cheers, Max Norbert Am 14.07.2018 um 08:25 schrieb Max Leske <maxleske@gmail.com>:

Hi Nicolas,

The PR you are referring [1] to concerns a missing fflush() when truncating a file. I believe that is a different case to this one, as a) the file is not truncated and b) in the case of truncation it is an explicit requirement that the file be empty before the first write happens. In our case, and in general, it depends on the situation whether flushing must occur immediately, e.g. because the file will be read again immediately, or not, e.g. when writing a data to a backup file.

Thanks for the pointer though.

Cheers, Max

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/271 On 12 Jul 2018, at 23:19, Nicolas Cellier wrote:

Isn't there a recent PR on opensmalltalk that address just this? Le jeu. 12 juil. 2018 à 23:16, Max Leske <maxleske@gmail.com> a écrit :

Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does *not* guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

The output in the Transcript should be:

315392 318420

Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr

Sven, thank you very much for quickly providing a new version. I changed the dependencies of my project for 2.9.4 and everything is fine now. So I can start using pharo7 on my laptop and keep 6.1 images for building the deployment artefact. Great! thanks, Norberr

Am 22.07.2018 um 12:59 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

Done:

=== Name: ConfigurationOfZincHTTPComponents-SvenVanCaekenberghe.117 Author: SvenVanCaekenberghe Time: 22 July 2018, 12:58:12.362 pm UUID: 949d5a24-f62d-0d00-a12c-ead9078b1897 Ancestors: ConfigurationOfZincHTTPComponents-SvenVanCaekenberghe.116

stable v 2.9.4 ===

Please tell us if this works for you

...

On 22 Jul 2018, at 12:10, Norbert Hartl <norbert@hartl.name> wrote:

Thank you guys! It is really great to see that there are people like Max jumping onto problems like this and solving it.

@sven I’m eager to test when the new configuration is available. Please drop us a note when you are done!

thanks again,

Norbert

...

Am 22.07.2018 um 11:06 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

Ah, that makes total sense. Great catch. Thank you.

=== Name: Zinc-FileSystem-SvenVanCaekenberghe.17 Author: SvenVanCaekenberghe Time: 22 July 2018, 11:04:40.148339 am UUID: e97a508e-f42d-0d00-a127-f016078b1897 Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.16

Bugfix to ZnFileSystemUtils class>>#newBinaryFileNamed:do: we must ensure that the binaryStream opened here is closed as well (thx Max Leske for the bug hunting and fix) ===

The method in question now look as follows:

newBinaryFileNamed: fileName do: block | fileReference | fileReference := fileName asFileReference. ^ (fileReference respondsTo: #binaryWriteStreamDo:ifPresent:) ifTrue: [ fileName asFileReference binaryWriteStreamDo: block ifPresent: [ FileExists signalWith: fileReference ] ] ifFalse: [ fileReference isFile ifTrue: [ FileExists signalWith: fileReference ] ifFalse: [ | binaryStream | binaryStream := self binaryFileStreamFor: fileName. [ block value: binaryStream ] ensure: [ binaryStream close ] ] ]

I create a new 2.9.4 of the config later on.

...

On 21 Jul 2018, at 22:00, Max Leske <maxleske@gmail.com> wrote:

Sven, you're right. What I found last time works as a side-effect. The real problem is in ZnFileSystemUtils class>>newBinaryFileNamed:do:. In Pharo 6.1 the logic there leads to the following evaluation:

block value: (self binaryFileStreamFor: fileName)

This is the only place where the stream is created without a surrounding safety block (e.g. #writeStream:do:), which means, no one closes the stream. The line should be:

| s | s := self binaryFileStreamFor: fileName. [ block value: s ] ensure: [ s close ]

ZnFileSystemUtils class>>newBinaryFileNamed:do: is only used by ZnClient after loading Zinc-HTTP-SvenVanCaekenberghe.472, which is being loaded by version 2.9.1 of ConfigurationOfZincHTTPComponents.

Cheers, Max

On 21 Jul 2018, at 17:21, Max Leske wrote:

Just to be safe, I'll redo my experiment and get back to you.

On 21 Jul 2018, at 16:59, Sven Van Caekenberghe wrote: I tried running Max's snippet (Pharo 6.1 on Ubuntu 16.04 LTS),

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

And it gives me,

318420 318420

For me there is nothing nothing to see here ... and I have the impression we are all talking about different things. On 21 Jul 2018, at 14:18, Max Leske <maxleske@gmail.com> wrote:

On 21 Jul 2018, at 12:07, Norbert Hartl wrote: Max,

what do you think will be a proper fix for this? I cannot see if this is a vm problem or not.

In my opinion it should be fixed in Zinc as there's no way for the user of Zinc to flush the stream. The alternative would be for Zinc to expose a method or setting that allows the user to specify the expected behaviour. I do not think that this should be handled by the VM plugin or the stream class, as there are indeed different behaviours that make sense.

Cheers, Max Norbert Am 14.07.2018 um 08:25 schrieb Max Leske <maxleske@gmail.com>:

Hi Nicolas,

The PR you are referring [1] to concerns a missing fflush() when truncating a file. I believe that is a different case to this one, as a) the file is not truncated and b) in the case of truncation it is an explicit requirement that the file be empty before the first write happens. In our case, and in general, it depends on the situation whether flushing must occur immediately, e.g. because the file will be read again immediately, or not, e.g. when writing a data to a backup file.

Thanks for the pointer though.

Cheers, Max

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/271 On 12 Jul 2018, at 23:19, Nicolas Cellier wrote:

Isn't there a recent PR on opensmalltalk that address just this? Le jeu. 12 juil. 2018 à 23:16, Max Leske <maxleske@gmail.com> a écrit :

Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does *not* guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

The output in the Transcript should be:

315392 318420

Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr

On 22 Jul 2018, at 12:10, Norbert Hartl wrote:

Thank you guys! It is really great to see that there are people like Max jumping onto problems like this and solving it.

You're welcome!

@sven I’m eager to test when the new configuration is available. Please drop us a note when you are done!

thanks again,

Norbert

...

Am 22.07.2018 um 11:06 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

Ah, that makes total sense. Great catch. Thank you.

=== Name: Zinc-FileSystem-SvenVanCaekenberghe.17 Author: SvenVanCaekenberghe Time: 22 July 2018, 11:04:40.148339 am UUID: e97a508e-f42d-0d00-a127-f016078b1897 Ancestors: Zinc-FileSystem-SvenVanCaekenberghe.16

Bugfix to ZnFileSystemUtils class>>#newBinaryFileNamed:do: we must ensure that the binaryStream opened here is closed as well (thx Max Leske for the bug hunting and fix) ===

The method in question now look as follows:

newBinaryFileNamed: fileName do: block | fileReference | fileReference := fileName asFileReference. ^ (fileReference respondsTo: #binaryWriteStreamDo:ifPresent:) ifTrue: [ fileName asFileReference binaryWriteStreamDo: block ifPresent: [ FileExists signalWith: fileReference ] ] ifFalse: [ fileReference isFile ifTrue: [ FileExists signalWith: fileReference ] ifFalse: [ | binaryStream | binaryStream := self binaryFileStreamFor: fileName. [ block value: binaryStream ] ensure: [ binaryStream close ] ] ]

I create a new 2.9.4 of the config later on.

...

On 21 Jul 2018, at 22:00, Max Leske <maxleske@gmail.com> wrote:

Sven, you're right. What I found last time works as a side-effect. The real problem is in ZnFileSystemUtils class>>newBinaryFileNamed:do:. In Pharo 6.1 the logic there leads to the following evaluation:

block value: (self binaryFileStreamFor: fileName)

This is the only place where the stream is created without a surrounding safety block (e.g. #writeStream:do:), which means, no one closes the stream. The line should be:

| s | s := self binaryFileStreamFor: fileName. [ block value: s ] ensure: [ s close ]

ZnFileSystemUtils class>>newBinaryFileNamed:do: is only used by ZnClient after loading Zinc-HTTP-SvenVanCaekenberghe.472, which is being loaded by version 2.9.1 of ConfigurationOfZincHTTPComponents.

Cheers, Max

On 21 Jul 2018, at 17:21, Max Leske wrote:

Just to be safe, I'll redo my experiment and get back to you.

On 21 Jul 2018, at 16:59, Sven Van Caekenberghe wrote: I tried running Max's snippet (Pharo 6.1 on Ubuntu 16.04 LTS),

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

And it gives me,

318420 318420

For me there is nothing nothing to see here ... and I have the impression we are all talking about different things. On 21 Jul 2018, at 14:18, Max Leske <maxleske@gmail.com> wrote:

On 21 Jul 2018, at 12:07, Norbert Hartl wrote: Max,

what do you think will be a proper fix for this? I cannot see if this is a vm problem or not.

In my opinion it should be fixed in Zinc as there's no way for the user of Zinc to flush the stream. The alternative would be for Zinc to expose a method or setting that allows the user to specify the expected behaviour. I do not think that this should be handled by the VM plugin or the stream class, as there are indeed different behaviours that make sense.

Cheers, Max Norbert Am 14.07.2018 um 08:25 schrieb Max Leske <maxleske@gmail.com>:

Hi Nicolas,

The PR you are referring [1] to concerns a missing fflush() when truncating a file. I believe that is a different case to this one, as a) the file is not truncated and b) in the case of truncation it is an explicit requirement that the file be empty before the first write happens. In our case, and in general, it depends on the situation whether flushing must occur immediately, e.g. because the file will be read again immediately, or not, e.g. when writing a data to a backup file.

Thanks for the pointer though.

Cheers, Max

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/271 On 12 Jul 2018, at 23:19, Nicolas Cellier wrote:

Isn't there a recent PR on opensmalltalk that address just this? Le jeu. 12 juil. 2018 à 23:16, Max Leske <maxleske@gmail.com> a écrit :

Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does *not* guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

The output in the Transcript should be:

315392 318420

Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr

Sven, for me the problem is only on linux Norbert

Am 13.07.2018 um 13:26 schrieb Sven Van Caekenberghe <sven@stfx.eu>:

...

On 12 Jul 2018, at 23:15, Max Leske <maxleske@gmail.com> wrote:

Hi Norbert,

I was able to reproduce the problem and then identify the culprit, although what I don't yet understand is how this is related to the changes in Zinc.

Anyway, the problem is that the file stream isn't properly flushed in ZnUtils class>>streamFrom:to:size:. Adding outputStream flush as the last statement fixes your problem. Apparently, StandardFileStream / MultiByteFileStream perform a simple close() on the file which, according to the man page on close(), does not guarantee that the contents have all been written to file. In this case a flush is necessary because the entire file is immediately read again.

Here's a smaller test case for you to play with Sven:

ZnClient new url: 'https://github.com/zweidenker/Parasol/archive/master.zip'; downloadTo: '/tmp/foobar.zip'. bytes := '/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]. Transcript open; show: bytes size; cr. 5 seconds asDelay wait. Transcript show: ('/tmp/foobar.zip' asFileReference binaryReadStreamDo: [ :s | s contents ]) size.

The output in the Transcript should be:

315392 318420

Thanks for the effort, Max, but I am confused about your runnable snippet. For me (macOS, Pharo 7) is gives

318420 318420

as it should, reading the same file twice. Did I miss something or did you make a typo ?

As for #close, it is my understanding that Pharo did a #flush before doing a #close, but maybe that is no longer true with the new primitive streams, we'll have to check carefully.

Good catch. I'll think about your suggestion (there are already #flush messages sent, but indeed the last one would be missing).

...

Cheers, Max

On 12 Jul 2018, at 8:17, Norbert Hartl wrote:

Am 12.07.2018 um 08:05 schrieb Max Leske maxleske@gmail.com:

On 11 Jul 2018, at 22:44, Norbert Hartl wrote:

Hi Max,

I constructed a case that fails exactly like I experience it. Could you try? Just unpack the attachment on a linux, set PHARO_VM env to your executable and execute build.sh

I will. Might take a couple of days though.

No problem. I‘m happy if you find time.

Norbert

Max

Norbert

Am 10.07.2018 um 09:17 schrieb Max Leske maxleske@gmail.com:

On 10 Jul 2018, at 8:48, Norbert Hartl wrote:

Max,

thanks for taking the effort.

No worries.

Am 10.07.2018 um 08:37 schrieb Max Leske maxleske@gmail.com:

I did my best to reproduce the issue but didn't have any luck. I really need access to a Linux VM to debug this. So I'm praying that Apple fixes the access restrictions to kernel extensions in their next beta...

BTW, Metacello uses ZnClient, which uses ZnFileSystemUtils, so there is indeed a chance that something goes wrong during download of the zip archive and that something may be tied to a difference in the versions of Zinc (although I still think that's unlikely).

Yes there is potential but I don‘t see it. I take a fresh 6.1 image and load my project into. I‘m not sure but I think zinc 2.9.2 is loaded rather early in that process. So I wonder why it does not go wrong in the first phase. And also not if I load the test group within the first phase. It must be either the second Metacello invocation or the stopping, copying and starting of the image. I try to isolate this case more and provide a script that goes wrong on my machine. But it will take some time because I needed to stop trying to solve this as I wasted nearly two days on that already.

Let me know once you have something and I'll try to help out.

Norbert

Max

On 9 Jul 2018, at 19:43, Norbert Hartl wrote:

Am 09.07.2018 um 19:10 schrieb Max Leske maxleske@gmail.com:

I checked the Parasol Archive and it does not appear to be in Zip64 format (Metacello uses ZipArchive which can't cope with Zip64 but ZipArchive can read the Parasol zip). So my next guess is that there's either a problem with Metacello or Pharo in the way that ZipArchive is used (e.g. endianness problem or non-binary stream data). It would therefore be helpful to find out what happens in ZipArchive>>readFrom:, i.e. what kind of stream is passed / opened, is it binary, does the file exist and is it still the correct file.

I couldn’t see anything obvious. The file in the debug message exists, it is a readable zip file. The way Metacello uses it it is a StandardFileStream. It switches it to binary in the code. But the only difference between a working and non-working build is the upgrade to zinc 2.9.2.

Norbert

I'd debug it myself but I can't run VirtualBox at the moment because I'm on the macOS beta and it won't start...

Max

On 9 Jul 2018, at 18:31, Norbert Hartl wrote:

Hi Max,

Am 09.07.2018 um 18:18 schrieb Max Leske maxleske@gmail.com:

Hi Norbert,

This is a bit of a guess, but it's possible that the archive that is downloaded from github is in Zip64 format and that the libraries for extracting Zip64 are missing on your Linux. That would of course contradict the experience that the same operation appears to work in 6.1.

to be honest I don’t know what Zip64 format is. I thought the Zip classes are pure smalltalk for unpacking.

Try extracting the archive manually on your Linux machine with the same method that Metacello uses (I assume, Metacello uses the ZipPlugin, which will probably use the system libraries).

I have no ZipPlugin as library in any of my vms.

But there are zips downloaded and unpacked. I start the image the first time loading all the code of my project. Then it is saved, copied to a new name and reopened to load the tests and then it fails. I did try to load the tests in the first run and then it works.

I’m running out of ideas

thanks,

Norbert

Cheers, Max

On 9 Jul 2018, at 17:14, Norbert Hartl wrote:

With the help of Esteban I got one step further. If I do

MCWorkingCopy managersForClass: ZnFileSystemUtils do: [ :each | each ancestry initialize ]

before loading my project it updates Zinc-FileSystem as well. Sadly it still does not work for me because I get

Loading baseline of BaselineOfMobilityMap... ...RETRY->BaselineOfParasol ...RETRY->BaselineOfParasol[31mError: can't find EOCD position

and I don’t know why. zip file is downloaded from github and present but it fails and only on my jenkins on linux. On my Mac this works.

I try a few things but then I’m back on pharo6.1 for the time being :(

Norbert

Am 07.07.2018 um 13:28 schrieb Norbert Hartl norbert@hartl.name:

Really? I thought the same but then I didn’t believe it works like that.

Anyway this would be very easy to solve. We just need to ask Sven if he is fine with doing an empty .16 version for Zinc-FileSystem and does an in-place version reset of 2.9.2 or a new 2.9.3. I’m not fully convinced that will solve it but the cost won’t be very high.

Norbert

Am 07.07.2018 um 13:22 schrieb Cyril Ferlicot cyril.ferlicot@gmail.com:

Need to investigate more. There are two .15 versions but there is 1 year in between (if you didn’t notice TheIntegratior.15 is from 2017). Just want to have more context information because I can only see that this is strange but lacking insight.

I’m trying to figure out why it does not update Zinc-FileSystem. No matter what I do I cannot get Metacello to load the newer package. That would fix the issue because I’m loading 2.9.2 which should have Zinc-FileSystem-SVC.15 and not stay on the one included in the image.

I think this is important for everyone that has a product based on 6.1 and that want to migrate someday to pharo7. This way it is impossible to do it step by step.

If there is a package .15 in the image and a package .15 in the repo, I think Metacello will not update because it rely on the numbers to know when to update. If it find a .15 and currently have a .15 I think Metacello will think they are at the same version.

Norbert

-- Cyril Ferlicot https://ferlicot.fr