Re: [Pharo-dev] Random corrupted data when copying from very large byte array

Jan. 18, 2018

      As a separate data point:

- I've seen corruption twice out of several runs (the process hits
"self halt") in a 32 bit VM.
-- In both cases the first four bytes of the array were 0.
- I haven't yet seen a crash or corruption in the 64 bit VM (it's in
the process of looping 1000 times).

OS: Ubuntu 16.04.
Images:
- Pharo-7.0+alpha.build.425.sha.eb0a6fb140ac4a42b1f158ed37717e0650f778b4
(32 Bit)
- Pharo-7.0+alpha.build.436.sha.7e0f6d30dca546f3859b32764483b42f4fdcd63c
(64 Bit)
VM:

5.0-201801110739  Thursday 11 January  09:22:24 CET 2018 gcc 4.8.5
[Production Spur 64-bit VM]
CoInterpreter VMMaker.oscog-eem.2302 uuid:
55ec8f63-cdbe-4e79-8f22-48fdea585b88 Jan 11 2018
StackToRegisterMappingCogit VMMaker.oscog-eem.2302 uuid:
55ec8f63-cdbe-4e79-8f22-48fdea585b88 Jan 11 2018
VM: 201801110739
alistair@alistair-xps13:snap/pharo-snap/pharo-vm/opensmalltalk-vm $
Date: Wed Jan 10 23:39:30 2018 -0800 $
Plugins: 201801110739
alistair@alistair-xps13:snap/pharo-snap/pharo-vm/opensmalltalk-vm $
Linux b07d7880072c 4.13.0-26-generic #29~16.04.2-Ubuntu SMP Tue Jan 9
22:00:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
plugin path: /snap/pharo7/x1/usr/bin/pharo-vm/5.0-201801110739
[default: /snap/pharo7/x1/usr/bin/pharo-vm/5.0-201801110739/]

The 32 bit VM is the same version as above.

Cheers,
Alistair

On 18 January 2018 at 16:42, Cyrille Delaunay <cy.delaunay@gmail.com> wrote:
...
Hi everyone,
I just added a new bug entry for an issue we are experimenting since some
times:
https://pharo.fogbugz.com/f/cases/20982/Random-corrupted-data-when-copying-f...
Here is the description:
History:
This issue has been spotted after experimenting strange behavior with
seaside upload.
After uploading a big file from a web browser, the modeled file generated
within pharo image begins with 4 unexpected bytes.
This issue occurs randomly: sometimes the first 4 bytes are right. Sometimes
the first 4 bytes are wrong.
This issue only occurs with Pharo 6.
This issue occurs for all platforms (Mac, Ubuntu, Windows)
Steps to reproduce:
I have been able to set up a small scenario that highlight the issue.
Download Pharo 6.1 on my Mac (Sierra 10.12.6):
https://pharo.org/web/download
Then, iterate over this process till spotting the issue:
=> start the pharo image
=> execute this piece of code in a playground
1:
2:
3:
4:
5:
6:
ZnServer startDefaultOn: 1701.
ZnServer default maximumEntitySize: 80* 1024 * 1024.
'/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out |
out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].
18202065 timesRepeat: [ out nextPut: 0 ]
].
=> Open a web browser page on: http://localhost:1701/form-test-3
=> Upload the file zip file previously generated ('myzip.zip').
=> If the web page displays: "contents=000000000a00..." (or anything
unexpected), THIS IS THE ISSUE !
=> If the web page displays: "contents=504b03040a00..", the upload worked
fine. You can close the image (without saving)
Debugging:
Bob Arning has been able to reproduce the issue with my scenario.
He dived into the code involved during this process, till reaching some
"basic" methods where he saw the issue occuring.
Here are the conclusion till there:
=> A corruption occurs while reading an input stream with method ZnUtils
class>>readUpToEnd:limit:
The first 4 bytes may be altered randomely.
=> The first 4 bytes are initially correctly written to an outputStream.
But, the first 4 bytes of this outputStream gets altered (corrupted),
sometimes when the inner byte array grows OR when performing the final
"outputStream contents"
=> Here is a piece of code that reproduce the issue (still randomely.
stopping an restarting the image may change the behavior)
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
test4"self test4"    | species bufferSize buffer totalRead outputStream
answer inputStream ba byte1 |            ba := ByteArray new: 18202085.
ba atAllPut: 99.        1 to: 20 do: [  :i | ba at: i put: (#[80 75 3 4 10 7
7 7 7 7 125 83 67 73 7 7 7 7 7 7] at: i) ].    inputStream := ba readStream.
bufferSize := 16384.    species := ByteArray.
    buffer := species new: bufferSize.
    totalRead := 0.
    outputStream := nil.
    [ inputStream atEnd ] whileFalse: [ | readCount |
        readCount := inputStream readInto: buffer startingAt: 1 count:
bufferSize.
        totalRead = 0 ifTrue: [
            byte1 := buffer first.
        ].
        totalRead := totalRead + readCount.
outputStream ifNil: [
            inputStream atEnd
                ifTrue: [ ^ buffer copyFrom: 1 to: readCount ]
                ifFalse: [ outputStream := (species new: bufferSize)
writeStream ] ].
        outputStream next: readCount putAll: buffer startingAt: 1.
        byte1 = outputStream contents first ifFalse: [ self halt ].
    ].
    answer := outputStream ifNil: [ species new ] ifNotNil: [ outputStream
contents ].
    byte1 = answer first ifFalse: [ self halt ].    ^answer
suspicions
This issue appeared with Pharo 6.
Some people suggested that it could be a vm issue, and to try my little
scenario with the last available vm.
I am not sure where to find the last available vm.
I did the test using these elements:
https://files.pharo.org/image/60/latest.zip
https://files.pharo.org/get-files/70/pharo-mac-latest.zip/
The issue is still present
--
Cyrille Delaunay

Re: [Pharo-dev] Random corrupted data when copying from very large byte array

Alistair Grant