- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Random corrupted data when copying from very large byte array

by Alistair Grant

Hi ClÃ©ment, On 19 January 2018 at 17:21, Alistair Grant <akgrant0710(a)gmail.com> wrote: > Hi ClÃ©ment, > > On 19 January 2018 at 17:04, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: >> Does not seem to be related to prim 105. >> >> I am confused. Has the size of the array an impact at all ? > > Yes, I tried reducing the size of the array by a factor of 10 and > wasn't able to reproduce the problem at all. > > With the full size array it failed over half the time (32 bit). > > I ran the test about 180 times on 64 bit and didn't get a single failure. > >> It seems the >> problem shows since the first copy of 16k elements. >> >> I can't really reproduce the bug - it happened once but I cannot do it >> again. Does the bug happen with the StackVM/PharoS VM you can find here the >> 32 bits versions : http://files.pharo.org/vm/pharoS-spur32/ ? The >> StackVM/PharoS VM is the VM without the JIT, it may be since the bug is >> unreliable that it happens only in jitted code, so trying that out may be >> worth it. > > I'll try and have a look at this over the weekend. This didn't fail once in 55 runs. OS: Ubuntu 16.04 Image: Pharo 6.0 Latest update: #60528 VM: 5.0 #1 Wed Oct 12 15:48:53 CEST 2016 gcc 4.6.3 [Production Spur ITHB VM] StackInterpreter VMMaker.oscog-EstebanLorenzano.1881 uuid: ed616067-a57c-409b-bfb6-dab51f058235 Oct 12 2016 https://github.com/pharo-project/pharo-vm.git Commit: 01a03276a2e2b243cd4a7d3427ba541f835c07d3 Date: 2016-10-12 14:31:09 +0200 By: Esteban Lorenzano <estebanlm(a)gmail.com> Jenkins build #606 Linux pharo-linux 3.2.0-31-generic-pae #50-Ubuntu SMP Fri Sep 7 16:39:45 UTC 2012 i686 i686 i386 GNU/Linux plugin path: /home/alistair/pharo7/Issue20982/bin/ [default: /home/alistair/pharo7/Issue20982/bin/] I then went back and attempted to reproduce the failures in my regular 32 bit image, but only got 1 corruption in 10 runs. I've been working in this image without restarting for most of the day. Quitting out and restarting the image and then running the corruption check resulted in 11 corruptions from 11 runs. Image: Pharo 7.0 Build information: Pharo-7.0+alpha.build.425.sha.eb0a6fb140ac4a42b1f158ed37717e0650f778b4 (32 Bit) VM: 5.0-201801110739 Thursday 11 January 09:30:12 CET 2018 gcc 4.8.5 [Production Spur VM] CoInterpreter VMMaker.oscog-eem.2302 uuid: 55ec8f63-cdbe-4e79-8f22-48fdea585b88 Jan 11 2018 StackToRegisterMappingCogit VMMaker.oscog-eem.2302 uuid: 55ec8f63-cdbe-4e79-8f22-48fdea585b88 Jan 11 2018 VM: 201801110739 alistair@alistair-xps13:snap/pharo-snap/pharo-vm/opensmalltalk-vm $ Date: Wed Jan 10 23:39:30 2018 -0800 $ Plugins: 201801110739 alistair@alistair-xps13:snap/pharo-snap/pharo-vm/opensmalltalk-vm $ Linux b07d7880072c 4.13.0-26-generic #29~16.04.2-Ubuntu SMP Tue Jan 9 22:00:44 UTC 2018 i686 i686 i686 GNU/Linux plugin path: /snap/core/3748/lib/i386-linux-gnu/ [default: /snap/core/3748/lib/i386-linux-gnu/] So, as well as restarting the image before running the test, just wondering if the gcc compiler version could have an impact? HTH, Alistair > Cheers, > Alistair > > > >> On Thu, Jan 18, 2018 at 7:12 PM, ClÃ©ment Bera <bera.clement(a)gmail.com> >> wrote: >>> >>> I would suspect a bug in primitive 105 on byte objects (it was changed >>> recently in the VM), called by copyFrom: 1 to: readCount. The bug would >>> likely by due to specific alignment in readCount or something like that. >>> (Assuming you're in 32 bits since the 4 bytes are corrupted). >>> >>> When I get better I can have a look (I am currently quite sick). >>> >>> On Thu, Jan 18, 2018 at 4:51 PM, Thierry Goubier >>> <thierry.goubier(a)gmail.com> wrote: >>>> >>>> Hi Cyril, >>>> >>>> try with the last vms available at: >>>> >>>> https://bintray.com/opensmalltalk/vm/cog/ >>>> >>>> For example, the last Ubuntu 64bits vm is at: >>>> >>>> https://bintray.com/opensmalltalk/vm/cog/201801170946#files >>>> >>>> Regards, >>>> >>>> Thierry >>>> >>>> 2018-01-18 16:42 GMT+01:00 Cyrille Delaunay <cy.delaunay(a)gmail.com>: >>>>> >>>>> Hi everyone, >>>>> >>>>> I just added a new bug entry for an issue we are experimenting since >>>>> some times: >>>>> >>>>> >>>>> https://pharo.fogbugz.com/f/cases/20982/Random-corrupted-data-when-copying-… >>>>> >>>>> Here is the description: >>>>> >>>>> >>>>> History: >>>>> >>>>> This issue has been spotted after experimenting strange behavior with >>>>> seaside upload. >>>>> After uploading a big file from a web browser, the modeled file >>>>> generated within pharo image begins with 4 unexpected bytes. >>>>> This issue occurs randomly: sometimes the first 4 bytes are right. >>>>> Sometimes the first 4 bytes are wrong. >>>>> This issue only occurs with Pharo 6. >>>>> This issue occurs for all platforms (Mac, Ubuntu, Windows) >>>>> >>>>> Steps to reproduce: >>>>> >>>>> I have been able to set up a small scenario that highlight the issue. >>>>> >>>>> Download Pharo 6.1 on my Mac (Sierra 10.12.6): >>>>> https://pharo.org/web/download >>>>> Then, iterate over this process till spotting the issue: >>>>> >>>>> => start the pharo image >>>>> => execute this piece of code in a playground >>>>> >>>>> 1: >>>>> 2: >>>>> 3: >>>>> 4: >>>>> 5: >>>>> 6: >>>>> >>>>> ZnServer startDefaultOn: 1701. >>>>> ZnServer default maximumEntitySize: 80* 1024 * 1024. >>>>> '/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out | >>>>> out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 >>>>> 0]. >>>>> 18202065 timesRepeat: [ out nextPut: 0 ] >>>>> ]. >>>>> >>>>> => Open a web browser page on: http://localhost:1701/form-test-3 >>>>> => Upload the file zip file previously generated ('myzip.zip'). >>>>> => If the web page displays: "contents=000000000a00..." (or anything >>>>> unexpected), THIS IS THE ISSUE ! >>>>> => If the web page displays: "contents=504b03040a00..", the upload >>>>> worked fine. You can close the image (without saving) >>>>> >>>>> >>>>> >>>>> Debugging: >>>>> >>>>> >>>>> >>>>> Bob Arning has been able to reproduce the issue with my scenario. >>>>> He dived into the code involved during this process, till reaching some >>>>> "basic" methods where he saw the issue occuring. >>>>> >>>>> Here are the conclusion till there: >>>>> => A corruption occurs while reading an input stream with method ZnUtils >>>>> class>>readUpToEnd:limit: >>>>> The first 4 bytes may be altered randomely. >>>>> => The first 4 bytes are initially correctly written to an outputStream. >>>>> But, the first 4 bytes of this outputStream gets altered (corrupted), >>>>> sometimes when the inner byte array grows OR when performing the final >>>>> "outputStream contents" >>>>> => Here is a piece of code that reproduce the issue (still randomely. >>>>> stopping an restarting the image may change the behavior) >>>>> >>>>> 1: >>>>> 2: >>>>> 3: >>>>> 4: >>>>> 5: >>>>> 6: >>>>> 7: >>>>> 8: >>>>> 9: >>>>> 10: >>>>> 11: >>>>> 12: >>>>> 13: >>>>> 14: >>>>> 15: >>>>> 16: >>>>> 17: >>>>> 18: >>>>> 19: >>>>> 20: >>>>> >>>>> test4"self test4" | species bufferSize buffer totalRead outputStream >>>>> answer inputStream ba byte1 | ba := ByteArray new: 18202085. >>>>> ba atAllPut: 99. 1 to: 20 do: [ :i | ba at: i put: (#[80 75 3 4 10 7 >>>>> 7 7 7 7 125 83 67 73 7 7 7 7 7 7] at: i) ]. inputStream := ba readStream. >>>>> bufferSize := 16384. species := ByteArray. >>>>> buffer := species new: bufferSize. >>>>> totalRead := 0. >>>>> outputStream := nil. >>>>> [ inputStream atEnd ] whileFalse: [ | readCount | >>>>> readCount := inputStream readInto: buffer startingAt: 1 count: >>>>> bufferSize. >>>>> totalRead = 0 ifTrue: [ >>>>> byte1 := buffer first. >>>>> ]. >>>>> totalRead := totalRead + readCount. >>>>> >>>>> outputStream ifNil: [ >>>>> inputStream atEnd >>>>> ifTrue: [ ^ buffer copyFrom: 1 to: readCount ] >>>>> ifFalse: [ outputStream := (species new: bufferSize) >>>>> writeStream ] ]. >>>>> outputStream next: readCount putAll: buffer startingAt: 1. >>>>> byte1 = outputStream contents first ifFalse: [ self halt ]. >>>>> ]. >>>>> answer := outputStream ifNil: [ species new ] ifNotNil: [ >>>>> outputStream contents ]. >>>>> byte1 = answer first ifFalse: [ self halt ]. ^answer >>>>> >>>>> >>>>> >>>>> suspicions >>>>> >>>>> This issue appeared with Pharo 6. >>>>> >>>>> Some people suggested that it could be a vm issue, and to try my little >>>>> scenario with the last available vm. >>>>> >>>>> I am not sure where to find the last available vm. >>>>> >>>>> I did the test using these elements: >>>>> >>>>> https://files.pharo.org/image/60/latest.zip >>>>> >>>>> https://files.pharo.org/get-files/70/pharo-mac-latest.zip/ >>>>> >>>>> >>>>> >>>>> The issue is still present >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cyrille Delaunay >>>> >>>> >>> >>> >>> >>> -- >>> ClÃ©ment BÃ©ra >>> Pharo consortium engineer >>> https://clementbera.wordpress.com/ >>> BÃ¢timent B 40, avenue Halley 59650 Villeneuve d'Ascq >> >> >> >> >> -- >> ClÃ©ment BÃ©ra >> Pharo consortium engineer >> https://clementbera.wordpress.com/ >> BÃ¢timent B 40, avenue Halley 59650 Villeneuve d'Ascq

Jan. 19, 2018

Re: [Pharo-dev] [ANN] Fari â a shell tool to fetch & prepare your fresh images

by Damien Pollet

Le ven. 19 janv. 2018 Ã 20:59, HernÃ¡n Morales Durand < hernan.morales(a)gmail.com> a Ã©crit : > $ ./fari.sh > ls: cannot access './*.load.st': No such file or directory > > You may need to add a shell option in the script to include hidden file > names in expansions > > shopt -s dotglob > Ah no, this is about nullglob. I used it somewhere else but missed that spot. Will have a look later this evening. $./fari.sh: line 176: ./pharo-ui: No such file or directory > Missing feature ð youâll have to get the VM with zeroconf for now. I don't know but only sources file was downloaded. > Expected: the script stops as soon as a problem os encountered. 2018-01-19 15:02 GMT-03:00 Damien Pollet <damien.pollet(a)gmail.com>: > >> Here it is, tell me what you think: https://github.com/cdlm/fari.sh >> >> We discussed it with Christophe and of course there is some overlap with >> PharoLauncher, and in the future, it might get replaced by a command-line >> version of PharoLauncher based on Clap. >> >> But for the moment it works and fits my twisted mind better ð >> >> -- >> Damien Pollet >> type less, do more [ | ] http://people.untyped.org/damien.pollet >> > > -- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet

Jan. 19, 2018

Re: [Pharo-dev] [ANN] Fari â a shell tool to fetch & prepare your fresh images

by HernÃ¡n Morales Durand

Hi Damien, Thanks for sharing this. This is what I did using mingw: $ wget https://github.com/cdlm/fari.sh/raw/master/fari.sh $ cat > .load.st Metacello new baseline: 'CCBC'; repository: 'gitlocal://./src'; load. $ ./fari.sh ls: cannot access './*.load.st': No such file or directory You may need to add a shell option in the script to include hidden file names in expansions shopt -s dotglob then $./fari.sh: line 176: ./pharo-ui: No such file or directory $ ll total 31861 drwxr-xr-x 1 mvs 197609 0 Jan 19 16:30 BaselineOfCCBC.package drwxr-xr-x 1 mvs 197609 0 Jan 19 16:44 CCBC.package drwxr-xr-x 1 mvs 197609 0 Jan 19 16:30 CCBC-Core.package -rw-r--r-- 1 mvs 197609 76951 Jan 19 16:30 'Collections - Strings.png' -rwxr-xr-x 1 mvs 197609 7399 Jan 19 16:53 fari.sh -rw-r--r-- 1 mvs 197609 44415 Jan 19 16:30 'Iceberg Class Critiques (Pharo 6.1).png' -rw-r--r-- 1 mvs 197609 1100 Jan 19 16:30 LICENSE -rw-r--r-- 1 mvs 197609 32371497 Jan 19 16:55 Pharo7.0-32bit-e359bc0.sources -rw-r--r-- 1 mvs 197609 549 Jan 19 16:30 README.md -rw-r--r-- 1 mvs 197609 101557 Jan 19 16:30 'Spec Method Critiques (Pharo 6.1).png' I don't know but only sources file was downloaded. Cheers, HernÃ¡n 2018-01-19 15:02 GMT-03:00 Damien Pollet <damien.pollet(a)gmail.com>: > Here it is, tell me what you think: https://github.com/cdlm/fari.sh > > We discussed it with Christophe and of course there is some overlap with > PharoLauncher, and in the future, it might get replaced by a command-line > version of PharoLauncher based on Clap. > > But for the moment it works and fits my twisted mind better ð > > -- > Damien Pollet > type less, do more [ | ] http://people.untyped.org/damien.pollet >

Jan. 19, 2018

[ANN] Fari â a shell tool to fetch & prepare your fresh images

by Damien Pollet

Here it is, tell me what you think: https://github.com/cdlm/fari.sh We discussed it with Christophe and of course there is some overlap with PharoLauncher, and in the future, it might get replaced by a command-line version of PharoLauncher based on Clap. But for the moment it works and fits my twisted mind better ð -- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet

Jan. 19, 2018

Re: [Pharo-dev] Random corrupted data when copying from very large byte array

by Alistair Grant

Hi ClÃ©ment, On 19 January 2018 at 17:04, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: > Does not seem to be related to prim 105. > > I am confused. Has the size of the array an impact at all ? Yes, I tried reducing the size of the array by a factor of 10 and wasn't able to reproduce the problem at all. With the full size array it failed over half the time (32 bit). I ran the test about 180 times on 64 bit and didn't get a single failure. > It seems the > problem shows since the first copy of 16k elements. > > I can't really reproduce the bug - it happened once but I cannot do it > again. Does the bug happen with the StackVM/PharoS VM you can find here the > 32 bits versions : http://files.pharo.org/vm/pharoS-spur32/ ? The > StackVM/PharoS VM is the VM without the JIT, it may be since the bug is > unreliable that it happens only in jitted code, so trying that out may be > worth it. I'll try and have a look at this over the weekend. Cheers, Alistair > On Thu, Jan 18, 2018 at 7:12 PM, ClÃ©ment Bera <bera.clement(a)gmail.com> > wrote: >> >> I would suspect a bug in primitive 105 on byte objects (it was changed >> recently in the VM), called by copyFrom: 1 to: readCount. The bug would >> likely by due to specific alignment in readCount or something like that. >> (Assuming you're in 32 bits since the 4 bytes are corrupted). >> >> When I get better I can have a look (I am currently quite sick). >> >> On Thu, Jan 18, 2018 at 4:51 PM, Thierry Goubier >> <thierry.goubier(a)gmail.com> wrote: >>> >>> Hi Cyril, >>> >>> try with the last vms available at: >>> >>> https://bintray.com/opensmalltalk/vm/cog/ >>> >>> For example, the last Ubuntu 64bits vm is at: >>> >>> https://bintray.com/opensmalltalk/vm/cog/201801170946#files >>> >>> Regards, >>> >>> Thierry >>> >>> 2018-01-18 16:42 GMT+01:00 Cyrille Delaunay <cy.delaunay(a)gmail.com>: >>>> >>>> Hi everyone, >>>> >>>> I just added a new bug entry for an issue we are experimenting since >>>> some times: >>>> >>>> >>>> https://pharo.fogbugz.com/f/cases/20982/Random-corrupted-data-when-copying-… >>>> >>>> Here is the description: >>>> >>>> >>>> History: >>>> >>>> This issue has been spotted after experimenting strange behavior with >>>> seaside upload. >>>> After uploading a big file from a web browser, the modeled file >>>> generated within pharo image begins with 4 unexpected bytes. >>>> This issue occurs randomly: sometimes the first 4 bytes are right. >>>> Sometimes the first 4 bytes are wrong. >>>> This issue only occurs with Pharo 6. >>>> This issue occurs for all platforms (Mac, Ubuntu, Windows) >>>> >>>> Steps to reproduce: >>>> >>>> I have been able to set up a small scenario that highlight the issue. >>>> >>>> Download Pharo 6.1 on my Mac (Sierra 10.12.6): >>>> https://pharo.org/web/download >>>> Then, iterate over this process till spotting the issue: >>>> >>>> => start the pharo image >>>> => execute this piece of code in a playground >>>> >>>> 1: >>>> 2: >>>> 3: >>>> 4: >>>> 5: >>>> 6: >>>> >>>> ZnServer startDefaultOn: 1701. >>>> ZnServer default maximumEntitySize: 80* 1024 * 1024. >>>> '/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out | >>>> out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 >>>> 0]. >>>> 18202065 timesRepeat: [ out nextPut: 0 ] >>>> ]. >>>> >>>> => Open a web browser page on: http://localhost:1701/form-test-3 >>>> => Upload the file zip file previously generated ('myzip.zip'). >>>> => If the web page displays: "contents=000000000a00..." (or anything >>>> unexpected), THIS IS THE ISSUE ! >>>> => If the web page displays: "contents=504b03040a00..", the upload >>>> worked fine. You can close the image (without saving) >>>> >>>> >>>> >>>> Debugging: >>>> >>>> >>>> >>>> Bob Arning has been able to reproduce the issue with my scenario. >>>> He dived into the code involved during this process, till reaching some >>>> "basic" methods where he saw the issue occuring. >>>> >>>> Here are the conclusion till there: >>>> => A corruption occurs while reading an input stream with method ZnUtils >>>> class>>readUpToEnd:limit: >>>> The first 4 bytes may be altered randomely. >>>> => The first 4 bytes are initially correctly written to an outputStream. >>>> But, the first 4 bytes of this outputStream gets altered (corrupted), >>>> sometimes when the inner byte array grows OR when performing the final >>>> "outputStream contents" >>>> => Here is a piece of code that reproduce the issue (still randomely. >>>> stopping an restarting the image may change the behavior) >>>> >>>> 1: >>>> 2: >>>> 3: >>>> 4: >>>> 5: >>>> 6: >>>> 7: >>>> 8: >>>> 9: >>>> 10: >>>> 11: >>>> 12: >>>> 13: >>>> 14: >>>> 15: >>>> 16: >>>> 17: >>>> 18: >>>> 19: >>>> 20: >>>> >>>> test4"self test4" | species bufferSize buffer totalRead outputStream >>>> answer inputStream ba byte1 | ba := ByteArray new: 18202085. >>>> ba atAllPut: 99. 1 to: 20 do: [ :i | ba at: i put: (#[80 75 3 4 10 7 >>>> 7 7 7 7 125 83 67 73 7 7 7 7 7 7] at: i) ]. inputStream := ba readStream. >>>> bufferSize := 16384. species := ByteArray. >>>> buffer := species new: bufferSize. >>>> totalRead := 0. >>>> outputStream := nil. >>>> [ inputStream atEnd ] whileFalse: [ | readCount | >>>> readCount := inputStream readInto: buffer startingAt: 1 count: >>>> bufferSize. >>>> totalRead = 0 ifTrue: [ >>>> byte1 := buffer first. >>>> ]. >>>> totalRead := totalRead + readCount. >>>> >>>> outputStream ifNil: [ >>>> inputStream atEnd >>>> ifTrue: [ ^ buffer copyFrom: 1 to: readCount ] >>>> ifFalse: [ outputStream := (species new: bufferSize) >>>> writeStream ] ]. >>>> outputStream next: readCount putAll: buffer startingAt: 1. >>>> byte1 = outputStream contents first ifFalse: [ self halt ]. >>>> ]. >>>> answer := outputStream ifNil: [ species new ] ifNotNil: [ >>>> outputStream contents ]. >>>> byte1 = answer first ifFalse: [ self halt ]. ^answer >>>> >>>> >>>> >>>> suspicions >>>> >>>> This issue appeared with Pharo 6. >>>> >>>> Some people suggested that it could be a vm issue, and to try my little >>>> scenario with the last available vm. >>>> >>>> I am not sure where to find the last available vm. >>>> >>>> I did the test using these elements: >>>> >>>> https://files.pharo.org/image/60/latest.zip >>>> >>>> https://files.pharo.org/get-files/70/pharo-mac-latest.zip/ >>>> >>>> >>>> >>>> The issue is still present >>>> >>>> >>>> >>>> >>>> -- >>>> Cyrille Delaunay >>> >>> >> >> >> >> -- >> ClÃ©ment BÃ©ra >> Pharo consortium engineer >> https://clementbera.wordpress.com/ >> BÃ¢timent B 40, avenue Halley 59650 Villeneuve d'Ascq > > > > > -- > ClÃ©ment BÃ©ra > Pharo consortium engineer > https://clementbera.wordpress.com/ > BÃ¢timent B 40, avenue Halley 59650 Villeneuve d'Ascq

Jan. 19, 2018

Re: [Pharo-dev] Random corrupted data when copying from very large byte array

by ClÃ©ment Bera

Does not seem to be related to prim 105. I am confused. Has the size of the array an impact at all ? It seems the problem shows since the first copy of 16k elements. I can't really reproduce the bug - it happened once but I cannot do it again. Does the bug happen with the StackVM/PharoS VM you can find here the 32 bits versions : http://files.pharo.org/vm/pharoS-spur32/ ? The StackVM/PharoS VM is the VM without the JIT, it may be since the bug is unreliable that it happens only in jitted code, so trying that out may be worth it. On Thu, Jan 18, 2018 at 7:12 PM, ClÃ©ment Bera <bera.clement(a)gmail.com> wrote: > I would suspect a bug in primitive 105 on byte objects (it was changed > recently in the VM), called by copyFrom: 1 to: readCount. The bug would > likely by due to specific alignment in readCount or something like that. > (Assuming you're in 32 bits since the 4 bytes are corrupted). > > When I get better I can have a look (I am currently quite sick). > > On Thu, Jan 18, 2018 at 4:51 PM, Thierry Goubier < > thierry.goubier(a)gmail.com> wrote: > >> Hi Cyril, >> >> try with the last vms available at: >> >> https://bintray.com/opensmalltalk/vm/cog/ >> >> For example, the last Ubuntu 64bits vm is at: >> >> https://bintray.com/opensmalltalk/vm/cog/201801170946#files >> >> Regards, >> >> Thierry >> >> 2018-01-18 16:42 GMT+01:00 Cyrille Delaunay <cy.delaunay(a)gmail.com>: >> >>> Hi everyone, >>> >>> I just added a new bug entry for an issue we are experimenting since >>> some times: >>> >>> https://pharo.fogbugz.com/f/cases/20982/Random-corrupted-dat >>> a-when-copying-from-very-large-byte-array >>> >>> Here is the description: >>> >>> >>> *History:* >>> >>> This issue has been spotted after experimenting strange behavior with >>> seaside upload. >>> After uploading a big file from a web browser, the modeled file >>> generated within pharo image begins with 4 unexpected bytes. >>> This issue occurs randomly: sometimes the first 4 bytes are right. >>> Sometimes the first 4 bytes are wrong. >>> This issue only occurs with Pharo 6. >>> This issue occurs for all platforms (Mac, Ubuntu, Windows) >>> >>> *Steps to reproduce:* >>> >>> I have been able to set up a small scenario that highlight the issue. >>> >>> Download Pharo 6.1 on my Mac (Sierra 10.12.6): https://pharo.org/we >>> b/download >>> Then, iterate over this process till spotting the issue: >>> >>> => start the pharo image >>> => execute this piece of code in a playground >>> >>> 1: >>> 2: >>> 3: >>> 4: >>> 5: >>> 6: >>> >>> ZnServer startDefaultOn: 1701.ZnServer default maximumEntitySize: 80* 1024 * 1024.'/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out |out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].18202065 timesRepeat: [ out nextPut: 0 ]]. >>> >>> => Open a web browser page on: http://localhost:1701/form-test-3 >>> => Upload the file zip file previously generated ('myzip.zip'). >>> => If the web page displays: "contents=000000000a00 >>> <https://pharo.kilnhg.com/Search?search=000000000a00>..." (or anything >>> unexpected), THIS IS THE ISSUE ! >>> => If the web page displays: "contents=504b03040a00 >>> <https://pharo.kilnhg.com/Search?search=504b03040a00>..", the upload >>> worked fine. You can close the image (without saving) >>> >>> >>> >>> *Debugging:* >>> >>> >>> >>> Bob Arning has been able to reproduce the issue with my scenario. >>> He dived into the code involved during this process, till reaching some >>> "basic" methods where he saw the issue occuring. >>> >>> Here are the conclusion till there: >>> => A corruption occurs while reading an input stream with method ZnUtils >>> class>>readUpToEnd:limit: >>> The first 4 bytes may be altered randomely. >>> => The first 4 bytes are initially correctly written to an >>> outputStream. >>> But, the first 4 bytes of this outputStream gets altered (corrupted), >>> sometimes when the inner byte array grows OR when performing the >>> final "outputStream contents" >>> => Here is a piece of code that reproduce the issue (still randomely. >>> stopping an restarting the image may change the behavior) >>> >>> 1: >>> 2: >>> 3: >>> 4: >>> 5: >>> 6: >>> 7: >>> 8: >>> 9: >>> 10: >>> 11: >>> 12: >>> 13: >>> 14: >>> 15: >>> 16: >>> 17: >>> 18: >>> 19: >>> 20: >>> >>> test4"self test4" | species bufferSize buffer totalRead outputStream answer inputStream ba byte1 | ba := ByteArray new: 18202085. ba atAllPut: 99. 1 to: 20 do: [ :i | ba at: i put: (#[80 75 3 4 10 7 7 7 7 7 125 83 67 73 7 7 7 7 7 7] at: i) ]. inputStream := ba readStream. bufferSize := 16384. species := ByteArray. >>> buffer := species new: bufferSize. >>> totalRead := 0. >>> outputStream := nil. >>> [ inputStream atEnd ] whileFalse: [ | readCount | >>> readCount := inputStream readInto: buffer startingAt: 1 count: bufferSize. >>> totalRead = 0 ifTrue: [ >>> byte1 := buffer first. >>> ]. >>> totalRead := totalRead + readCount. >>> >>> outputStream ifNil: [ >>> inputStream atEnd >>> ifTrue: [ ^ buffer copyFrom: 1 to: readCount ] >>> ifFalse: [ outputStream := (species new: bufferSize) writeStream ] ]. >>> outputStream next: readCount putAll: buffer startingAt: 1. >>> byte1 = outputStream contents first ifFalse: [ self halt ]. >>> ]. >>> answer := outputStream ifNil: [ species new ] ifNotNil: [ outputStream contents ]. >>> byte1 = answer first ifFalse: [ self halt ]. ^answer >>> >>> >>> >>> *suspicions* >>> >>> This issue appeared with Pharo 6. >>> >>> Some people suggested that it could be a vm issue, and to try my little >>> scenario with the last available vm. >>> >>> I am not sure where to find the last available vm. >>> >>> I did the test using these elements: >>> https://files.pharo.org/image/60/latest.zip >>> >>> https://files.pharo.org/get-files/70/pharo-mac-latest.zip/ >>> <https://files.pharo.org/get-files/70/> >>> >>> >>> >>> The issue is still present >>> >>> >>> >>> -- >>> Cyrille Delaunay >>> >> >> > > > -- > ClÃ©ment BÃ©ra > Pharo consortium engineer > https://clementbera.wordpress.com/ > BÃ¢timent B 40, avenue Halley 59650 Villeneuve d'Ascq > -- ClÃ©ment BÃ©ra Pharo consortium engineer https://clementbera.wordpress.com/ BÃ¢timent B 40, avenue Halley 59650 Villeneuve d'Ascq

Jan. 19, 2018

Re: [Pharo-dev] [IMPORTANT] PHARO CONTRIBUTION PROCESS UPDATED! PLEASE READ!

by Stephan Eggermont

Luke Gorrie <luke(a)snabb.co> wrote: > For me it's another story. I'm on an unsupported platform (NixOS Linux), > I'm building the VM from random git commits because the source releases are > all antiquated, Iceberg segfaults the moment I start it, and > epica+monticello+metacello+iceberg+fogbugz+jenkins feels like a series of > obstacles between me and maintaining my application. Isn't nix supported by travis? I suppose that means that you should be able to automatically build a variant of opensmalltalk I have absolutely no idea how much work that would be though. Stephan

Jan. 19, 2018

Re: [Pharo-dev] [IMPORTANT] PHARO CONTRIBUTION PROCESS UPDATED! PLEASE READ!

by Denis Kudriashov

2018-01-19 8:39 GMT+01:00 Esteban Lorenzano <estebanlm(a)gmail.com>: > hi :) > > On 18 Jan 2018, at 18:39, Stephane Ducasse <stepharo.self(a)gmail.com> > wrote: > > Thanks Luke for your feedback > Guillermo and Pablo are working on a new Iceberg model and UI. > > > Model is done, we will integrate it as 0.7 version. > UIâ¦ is harder. We could do a simple UI that allows just ONE path but at > the end, no one will use it, because almost no use-case follows that. > Believe me, I made a research on thatâ¦ and if you do not believe me, go and > try github desktop, which basically does that (and nobody uses it ;) ) > I use github desktop and I am very like it. But also I do not have problems with Iceberg process (if not count bugs) > > true story: actually the UI of iceberg was pretty simple when we started. > It became the mess it is now just adding what people asked to have. > but also true story: it *is* a mess, so we have to get it right before > release of Pharo7 :) > > May be > we can ask you if you want to give them feedback > and try the new version. > It is important for us that people can contribute. > > Stef > > On Wed, Jan 17, 2018 at 12:20 PM, Luke Gorrie <luke(a)snabb.co> wrote: > > Just one data point... > > I am completely overwhelmed by all of the different ways to manage source > code in the Pharo universe. I would love to contribute to upstream but this > seems way out of reach: I'm not even managing to keep track of the code in > my own home directory properly yet. > > > there are just two ways: > > - iceberg > - monticello > > compare with the amount of ways you have in other languages. > you can use either way you want to keep yours sources. > > but to contribute to Pharo there is just *one* way, following this > instructions: https://github.com/pharo-project/pharo/wiki/ > Contribute-a-fix-to-Pharo > > people has reported this version is a lot easier than previous ones (and > is not much different on how to contribute to other languages) but we are > still looking for ways to simplify it (I want to test github issue tracker > soon, for example, to enhance communication that today is broken in two > partsâ¦ and honest reality check: people finds easier to discuss on github > over the PR than over fogbugz). > > Just now I am looking at upstream code that should be fixed: CairoLibrary > unix32ModuleName and unix64ModuleName are giving higher priority to guessed > hard-coded paths than to LD_LIBRARY_PATH and this is backwards i.e. the > carefully selected paths a user supplies with LD_LIBRARY_PATH should come > first (imho.) > > > follow the contribution instructions ;) > > On the one hand it would be fine to dive into the upstream contribution > process to work out how to do this. On the other hand I am still trying to > master tools like Epica to recover useful code that I have written and lost > track of when images crash and get rebuilt etc. I really need to get this > under control before spending time learning Iceberg, FogBugz, etc. > > > wellâ¦ not so much to say here. My only advice is to save often you image, > when you are working with things that you know may crash the VM (like using > FFI). > > cheers! > Esteban > > > > > > On 14 December 2017 at 13:19, Esteban Lorenzano <estebanlm(a)gmail.com> > wrote: > > > Hi! > > Iâm working on simplifying the contribution process, after collecting > opinions/experiences last couple of months. > As you know, Pharo contribution process is still WIP and we aim to have it > as smooth as possible for Pharo 7.0 release. Now, after observe the idea of > the âsystem repositoriesâ was a bad idea because it introduced extra and > non > standard âpathâ to contribution, I managed to remove that to reestablish > âthe regular wayâ: you will now need to add pharo repository just as any > other repository you add, by cloning or adding local repository. > > I took Guilleâs doc and moved it to pharo project (it does not has sense > to have it living in a contributorâs repository when is so important). You > can find it here: > > https://github.com/pharo-project/pharo/wiki/Contribute-a-fix-to-Pharo > > This document is also updated to reveal this new process, please read it. > > How to update your startup scripts? > Some people has added startup scripts to easy the first part of > contribution. Instead enabling system repositories, etc. you now need to > replace that with this: > > (IceRepositoryCreator new > location: '/path/to/pharo-project/pharo' asFileReference; > subdirectory: 'src'; > createRepository) > register > > PLEASE, PLEASE, PLEASEâ¦ take a moment to read and try the document. Is > very important that document reflects new process and works reliable in > different scenarios (I validated it on macOS and Windows, and assumed it > worked fine on linux but you knowâ¦ bad assumptions is the base of failure > ;) > ) > > Iâm eager to hear your feedback and continue enhancing the process. > > (yes, Stef, I know UI is still cumbersomeâ¦ Iâm working on that :) ) > > cheers! > Esteban > > > > > >

Jan. 19, 2018

Re: [Pharo-dev] Esteban's ChangeLog week of 8 January 2018

by Esteban Lorenzano

Sorry, script was broken. I just spent some time fixing itâ¦ now they are back :) Esteban > On 19 Jan 2018, at 13:25, estebanlm(a)gmail.com wrote: > > Hello! > > This is my weekly ChangeLog, from 8 January 2018 to 14 January 2018. > You can see it in a better format by going here: http://log.smallworks.eu/web/search?from=8/1/2018&to=14/1/2018 > > ChangeLog > ========= > > 12 January 2018: > ---------------- > > * So well... it was the full week trying to compile all dependences for win64. > > and failed. > > Not that I didn't make any progress... but I still cann't compile +libssh2+ not +cairo+ :( > > (all the others are being built correctly) > > > 8 January 2018: > --------------- > > * Still trying to compile dependences for win64 (stuck on libssh2 for now) :( > > > cheers! > Esteban

Jan. 19, 2018

Esteban's ChangeLog week of 8 January 2018

by estebanlm＠gmail.com

Hello! This is my weekly ChangeLog, from 8 January 2018 to 14 January 2018. You can see it in a better format by going here: http://log.smallworks.eu/web/search?from=8/1/2018&to=14/1/2018 ChangeLog ========= 12 January 2018: ---------------- * So well... it was the full week trying to compile all dependences for win64. and failed. Not that I didn't make any progress... but I still cann't compile +libssh2+ not +cairo+ :( (all the others are being built correctly) 8 January 2018: --------------- * Still trying to compile dependences for win64 (stuck on libssh2 for now) :( cheers! Esteban

Jan. 19, 2018