- Pharo-dev - lists.pharo.org

Lots of "UTF8InvalidText: Invalid utf8 input detected" from OMBU

by Mariano Martinez Peck

Guys, I am getting "UTF8InvalidText: Invalid utf8 input detected" quite frequently on Pharo 6.0 which come from ombu background job. Has anything seen this too? Quite annoying... This is not the first time I hit it. This time I analyzed a bit why it was failing and it's because I removed a method manually...I mean, I did MyClass removeSelector: #myMethod And the omEntry is a EpMethodRemoval which represents the exact method I deleted... Also I can see the MultiByteFileStream internal 'collection' is full of none ascii things. I can share a fuelout as well as the image with the files if you want. I can also share the ombu file in question. Contact me in private. ---- [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] in [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :error | ... BlockClosure>>cull: Context>>evaluateSignal: Context>>handleSignal: UTF8InvalidText(Exception)>>signal UTF8InvalidText(Exception)>>signal: UTF8TextConverter>>errorMalformedInput UTF8TextConverter>>nextFromStream: MultiByteFileStream>>next MultiByteFileStream(PositionableStream)>>match: MultiByteFileStream(PositionableStream)>>upToAll: OmSTONEntryReader>>nextEntryPositionIfFound:ifNone: OmBlockFileStore>>refreshNewBlocksFrom: [ :readStream | self refreshNewBlocksFrom: readStream ] in OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since: in Block: [ :readStream | self refreshNewBlocksFrom: readStr...etc... [ ^ aBlockClosure value: readStream ascii ] in [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ ^ aBlockClosure value: readStream ascii ] BlockClosure>>on:do: [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :readStream | ... [ aBlock value: stream ] in FileReference(AbstractFileReference)>>readStreamDo: in Block: [ aBlock value: stream ] BlockClosure>>ensure: FileReference(AbstractFileReference)>>readStreamDo: OmBlockFileStore(OmFileStore)>>readEntriesWith: OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since: [ | initialPosition initialLocalName | self entryBuffer isEmpty ifTrue: [ ^ self ]. fileReference writeStreamDo: [ :fileStream | fileStream setToEnd. initialPosition := fileStream position. initialLocalName := self entryBuffer first value. ZnBufferedWriteStream on: fileStream do: [ :aWriteStream | | anEntryWriter | anEntryWriter := self newEntryWriter. [ self entryBuffer isEmpty ] whileFalse: [ | next entry | next := self entryBuffer removeFirst. entry := next key. "Write entry to file" anEntryWriter on: aWriteStream nextEntryPut: entry ] ]. "In Linux it was necessary to explicitely flush the file stream" fileStream flush. lastStreamPosition := fileStream size ]. self refreshEntryPositionsByLocalNameStartingAt: initialPosition since: initialLocalName ] in OmBlockFileStore(OmFileStore)>>flushEntryBuffer in Block: [ | initialPosition initialLocalName |... [ caught := true. self wait. blockValue := mutuallyExcludedBlock value ] in Semaphore>>critical: in Block: [ caught := true.... BlockClosure>>ensure: Semaphore>>critical: OmBlockFileStore(OmFileStore)>>critical: OmBlockFileStore(OmFileStore)>>flushEntryBuffer OmDeferrer>>sendMessage [ self sendMessage ] in OmDeferrer>>flush in Block: [ self sendMessage ] -- Mariano http://marianopeck.wordpress.com

July 25, 2017

Re: [Pharo-dev] [Vm-dev] FileSystem file attributes and #isSymlink patch

by Stephane Ducasse

Hi Alistair Yes it should become part of the core of Pharo :). Stef On Tue, Jul 25, 2017 at 5:21 PM, Alistair Grant <akgrant0710(a)gmail.com> wrote: > Hi David, > > Thanks very much for your follow-up. > > On Mon, Jul 24, 2017 at 07:28:07PM -0400, David T. Lewis wrote: >> >> Hi Alistair, >> >> I am copying this to vm-dev for follow up on the plugin, see below. >> >> >> On Mon, Jul 24, 2017 at 08:09:10AM +0000, Alistair Grant wrote: >> > Hi All, >> > >> > I'm nearly ready to submit a patch that started with the goal of being >> > able to retrieve the device id and fixing FileReference>>isSymlink and >> > also follows Esteban's suggestion of splitting out file existence and >> > other attributes (which provides a performace gain). See the summary >> > below for a description of the changes. >> > >> > The patch involves adding a new VM plugin, FileAttriubutesPlugin. To >> > minimise the chance of any problems along the way I'd like to submit the >> > patch in three steps: >> > >> > 1. Add the VM plugin (FileAttributesPlugin) >> > 2. Add the code the image that allows testing of the code and plugin, >> > but doesn't do any integration with existing functionality. >> > This will allow the plugin to be tested by a few volunteers >> > (hopefully). >> > 3. Add the patches that make the switch over to the new implementation. >> > >> >> I think you are handling this in exactly the right way, kudos. >> >> > >> > Can someone point me to how to submit a new VM plugin? The code is >> > contained in a subclass of InterpreterPlugin. >> >> Following up on the vm-dev list: If your plugin is available in a Monticello >> repository, that would be great because VM builders can easily include it >> and try it out. Any repository would be fine for starters, or if you have >> an account on squeaksource.com I can add you as developer in the somewhat >> loosely-related DirectoryPlugin project if that is of any help. Whatever is >> convenient for you. > > smalltalkhub.com is probably easiest for me, thanks. > > >> Your plugin sounds like something that would be stable and require little >> maintenance over time, so it might make sense to pull it directly into >> the VMMaker package. We can discuss that vm-dev list. > > I'm hoping this will become a core part of Pharo, so ultimately it > should live with the other Pharo core plugins like FilePlugin (assuming > it is accepted, of course). > > >> Once your plugin code is available, it should be straighforward to start >> including it in the various VM build configurations. > > Great, thanks. build.linux32x86/pharo.cog.spur is probably the best > place to start. > > I should be able to post the plugin to smalltalkhub.com quite > quickly. The smalltalk code will take me a while to repackage as I'll > want to test it fairly carefully, and my time is very scattered (this is > my hobby, but lots of family demands :-)). > > > As a side effect, once this has been fully integrated it will be > possible to get rid of at least some of the "#if Pharo" type > conditionals in the FilePlugin, allowing the code to be tidied up. > > I'll post another reply once I've got the plugin code in > smalltalkhub.com. > > > Thanks again, > Alistair > > > >> Dave >> >> >> > >> > I've been using this as my production environment for about 2 months now >> > on a linux 32 bit VM. Running the full test suite results in the same >> > set of test failing before and after applying the patch. >> > >> > I've also ran file related tests on linux 64 bit (run the Test Runner, >> > select all packages with "file" as part of the name and run all the >> > available tests) and the full test suite on Windows 32 bit. >> > >> > The summary is: >> > >> > 1. #isSymlink now works properly on Linux (and it should also work on >> > MacOS and BSD). >> > 2. The list of file attributes available from FileReference now is: >> > #accessTime (new) >> > #changeTime (new) >> > #creationTime >> > #deviceId (new) >> > #exists >> > #gid (new) >> > #inode (new) >> > #isBlock (new) >> > #isCharacter (new) >> > #isDirectory >> > #isExecutable (new) >> > #isFIFO (new) >> > #isFile >> > #isReadable >> > #isRegular (new) >> > #isSocket (new) >> > #isSymlink (works) >> > #isWritable >> > #modificationTime >> > #numberOfHardLinks (new) >> > #permissions >> > #size >> > #targetFile (new) >> > #uid (new) >> > 3. FileReference>>exists is faster than before (well, at least on my >> > linux laptop). This is useful as it is called quite often. >> > 4. It is possible to retrieve symbolic link attributes, e.g. all the >> > attributes listed above plus the target path. >> > >> > >> > Given how similar MacOS and BSD are to linux, I assume that this will >> > all work without problem on those platforms (but it obviously needs to >> > be tested). >> > >> > As implied above, the changes are all backward compatible (except >> > the broken #isSymlink), although a couple deserve mention: >> > >> > 1. Obviously #isSymlink now answers correctly (previously it would only >> > answer correctly for a broken link). >> > 2. Requesting any of the attributes listed above (except #isSymlink) >> > will return the value of the target path. If the FileReference is to a >> > broken symbolic link, it will return the attributes of the symbolic >> > link (keeping existing behaviour). >> > 3. The attributes of a symbolic link can be retrieved using >> > FileReference>>symlinkAttributes. >> > >> > Overall, performance is slightly better than before. Code that >> > needs to access multiple attributes and is written to take advantage of >> > the new behaviour will see significant performance improvements. >> > >> > >> > If you've got this far and forgotten the original question :-) >> > >> > Can someone point me to how to submit a new VM plugin? >> > >> > >> > Thanks, >> > Alistair >

July 25, 2017

Re: [Pharo-dev] Can I still download the old 64bit 6.0 vm?

by Tim Mackinnon

Actually it looks like the minimal 6.1 image is broken in a different way to the 6.0 image (this is the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccess… one, and I recall there were some other variants to try). Tim Sent from my iPhone Sent from my iPhone > On 25 Jul 2017, at 18:10, Tim Mackinnon <tim(a)testit.works> wrote: > > Hi - since the update to pharo 6.1, my PharoLambda project has stopped building. Iâm trying to diagnose what is going wrong, and I wanted to zeroconf download the older Pharo6.0 vm and image - is that still possible? > > I seem to be getting an error with loading a metacello package as I am getting a null repository error. > > I am retrying now with a non minimal image to try and diagnose the issue, but thought it worth asking about version control of these things. > > Tim

July 25, 2017

Can I still download the old 64bit 6.0 vm?

by Tim Mackinnon

Hi - since the update to pharo 6.1, my PharoLambda project has stopped building. Iâm trying to diagnose what is going wrong, and I wanted to zeroconf download the older Pharo6.0 vm and image - is that still possible? I seem to be getting an error with loading a metacello package as I am getting a null repository error. I am retrying now with a non minimal image to try and diagnose the issue, but thought it worth asking about version control of these things. Tim

July 25, 2017

Re: [Pharo-dev] [ANN] Pharo 6.1 (summer) released!

by Hilaire

Hi, It is the other minor bugs fixes details which were missing. What is hot-fixes? Hilaire -- Dr. Geo http://drgeo.eu

July 25, 2017

Re: [Pharo-dev] [ANN] Pharo 6.1 (summer) released!

by Esteban Lorenzano

> On 25 Jul 2017, at 17:29, Hilaire <hilaire(a)drgeo.eu> wrote: > > As the P6.1 change log is not complete enough I have to try out to discover: > > - fix for PNG regression is included > > - the Athen Wrap regression is still present :( > is very complete. Is just that it does not has so many changes :) this was a version we told it was going to be made available to update iceberg (as we use iceberg for develop P7, we need to have recurrent updates even in Pharo 6). And since there was important modifications, we needed to change the backend library. And since we changed the backend library, we needed to change the vm. Finally, since we changed the VM, we were unable to apply this update as a âhot-fixâ (what we do regularly, but version number does not change, just build number). So, Pharo 6.1 is exactly the same as before, but with: new iceberg, new vm, new backend library. cheers! Esteban ps: the other regressions, when fixed, can be applied as we usual apply backported hot-fixes. > > Le 25/07/2017 Ã 08:59, Hilaire a Ã©crit : >> Meaning these tow ones ? >> >> http://forum.world.st/Regression-with-PNGReaderWriter-in-P6-td4951796.html#… <http://forum.world.st/Regression-with-PNGReaderWriter-in-P6-td4951796.html#…> >> http://forum.world.st/WrapAthen-World-coordinate-td4951750.html#a4951966 <http://forum.world.st/WrapAthen-World-coordinate-td4951750.html#a4951966> > -- > Dr. Geo > http://drgeo.eu <http://drgeo.eu/>

July 25, 2017

Re: [Pharo-dev] [ANN] Pharo 6.1 (summer) released!

by Hilaire

As the P6.1 change log is not complete enough I have to try out to discover: - fix for PNG regression is included - the Athen Wrap regression is still present :( Le 25/07/2017 Ã 08:59, Hilaire a Ã©crit : > > Meaning these tow ones ? > > http://forum.world.st/Regression-with-PNGReaderWriter-in-P6-td4951796.html#… > > http://forum.world.st/WrapAthen-World-coordinate-td4951750.html#a4951966 > -- Dr. Geo http://drgeo.eu

July 25, 2017

Re: [Pharo-dev] [Vm-dev] FileSystem file attributes and #isSymlink patch

by Alistair Grant

Hi David, Thanks very much for your follow-up. On Mon, Jul 24, 2017 at 07:28:07PM -0400, David T. Lewis wrote: > > Hi Alistair, > > I am copying this to vm-dev for follow up on the plugin, see below. > > > On Mon, Jul 24, 2017 at 08:09:10AM +0000, Alistair Grant wrote: > > Hi All, > > > > I'm nearly ready to submit a patch that started with the goal of being > > able to retrieve the device id and fixing FileReference>>isSymlink and > > also follows Esteban's suggestion of splitting out file existence and > > other attributes (which provides a performace gain). See the summary > > below for a description of the changes. > > > > The patch involves adding a new VM plugin, FileAttriubutesPlugin. To > > minimise the chance of any problems along the way I'd like to submit the > > patch in three steps: > > > > 1. Add the VM plugin (FileAttributesPlugin) > > 2. Add the code the image that allows testing of the code and plugin, > > but doesn't do any integration with existing functionality. > > This will allow the plugin to be tested by a few volunteers > > (hopefully). > > 3. Add the patches that make the switch over to the new implementation. > > > > I think you are handling this in exactly the right way, kudos. > > > > > Can someone point me to how to submit a new VM plugin? The code is > > contained in a subclass of InterpreterPlugin. > > Following up on the vm-dev list: If your plugin is available in a Monticello > repository, that would be great because VM builders can easily include it > and try it out. Any repository would be fine for starters, or if you have > an account on squeaksource.com I can add you as developer in the somewhat > loosely-related DirectoryPlugin project if that is of any help. Whatever is > convenient for you. smalltalkhub.com is probably easiest for me, thanks. > Your plugin sounds like something that would be stable and require little > maintenance over time, so it might make sense to pull it directly into > the VMMaker package. We can discuss that vm-dev list. I'm hoping this will become a core part of Pharo, so ultimately it should live with the other Pharo core plugins like FilePlugin (assuming it is accepted, of course). > Once your plugin code is available, it should be straighforward to start > including it in the various VM build configurations. Great, thanks. build.linux32x86/pharo.cog.spur is probably the best place to start. I should be able to post the plugin to smalltalkhub.com quite quickly. The smalltalk code will take me a while to repackage as I'll want to test it fairly carefully, and my time is very scattered (this is my hobby, but lots of family demands :-)). As a side effect, once this has been fully integrated it will be possible to get rid of at least some of the "#if Pharo" type conditionals in the FilePlugin, allowing the code to be tidied up. I'll post another reply once I've got the plugin code in smalltalkhub.com. Thanks again, Alistair > Dave > > > > > > I've been using this as my production environment for about 2 months now > > on a linux 32 bit VM. Running the full test suite results in the same > > set of test failing before and after applying the patch. > > > > I've also ran file related tests on linux 64 bit (run the Test Runner, > > select all packages with "file" as part of the name and run all the > > available tests) and the full test suite on Windows 32 bit. > > > > The summary is: > > > > 1. #isSymlink now works properly on Linux (and it should also work on > > MacOS and BSD). > > 2. The list of file attributes available from FileReference now is: > > #accessTime (new) > > #changeTime (new) > > #creationTime > > #deviceId (new) > > #exists > > #gid (new) > > #inode (new) > > #isBlock (new) > > #isCharacter (new) > > #isDirectory > > #isExecutable (new) > > #isFIFO (new) > > #isFile > > #isReadable > > #isRegular (new) > > #isSocket (new) > > #isSymlink (works) > > #isWritable > > #modificationTime > > #numberOfHardLinks (new) > > #permissions > > #size > > #targetFile (new) > > #uid (new) > > 3. FileReference>>exists is faster than before (well, at least on my > > linux laptop). This is useful as it is called quite often. > > 4. It is possible to retrieve symbolic link attributes, e.g. all the > > attributes listed above plus the target path. > > > > > > Given how similar MacOS and BSD are to linux, I assume that this will > > all work without problem on those platforms (but it obviously needs to > > be tested). > > > > As implied above, the changes are all backward compatible (except > > the broken #isSymlink), although a couple deserve mention: > > > > 1. Obviously #isSymlink now answers correctly (previously it would only > > answer correctly for a broken link). > > 2. Requesting any of the attributes listed above (except #isSymlink) > > will return the value of the target path. If the FileReference is to a > > broken symbolic link, it will return the attributes of the symbolic > > link (keeping existing behaviour). > > 3. The attributes of a symbolic link can be retrieved using > > FileReference>>symlinkAttributes. > > > > Overall, performance is slightly better than before. Code that > > needs to access multiple attributes and is written to take advantage of > > the new behaviour will see significant performance improvements. > > > > > > If you've got this far and forgotten the original question :-) > > > > Can someone point me to how to submit a new VM plugin? > > > > > > Thanks, > > Alistair

July 25, 2017

Re: [Pharo-dev] [ANN] Pharo 6.1 (summer) released!

by Hilaire

Meaning these tow ones ? http://forum.world.st/Regression-with-PNGReaderWriter-in-P6-td4951796.html#… http://forum.world.st/WrapAthen-World-coordinate-td4951750.html#a4951966 Hilaire Le 24/07/2017 Ã 15:11, Hilaire a Ã©crit : > I hope it includes the fixes for the bugs impacting Dr. Geo. -- Dr. Geo http://drgeo.eu

July 25, 2017

Re: [Pharo-dev] FileSystem file attributes and #isSymlink patch

by Tudor Girba

Thank you very much, indeed! Doru > On Jul 24, 2017, at 10:04 PM, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: > > HI alistair > > this is supercool that you push this! > I mean it for real. Because this simplnk stuff was always getting on > our way for scripts. > This is really great to see this happening. > Stef > > On Mon, Jul 24, 2017 at 10:43 AM, Max Leske <maxleske(a)gmail.com> wrote: >> Nice work Alistair! >> >> >> >> On 24 July 2017 at 10:10:09, Alistair Grant (akgrant0710(a)gmail.com) wrote: >> >> Hi All, >> >> I'm nearly ready to submit a patch that started with the goal of being >> able to retrieve the device id and fixing FileReference>>isSymlink and >> also follows Esteban's suggestion of splitting out file existence and >> other attributes (which provides a performace gain). See the summary >> below for a description of the changes. >> >> The patch involves adding a new VM plugin, FileAttriubutesPlugin. To >> minimise the chance of any problems along the way I'd like to submit the >> patch in three steps: >> >> 1. Add the VM plugin (FileAttributesPlugin) >> 2. Add the code the image that allows testing of the code and plugin, >> but doesn't do any integration with existing functionality. >> This will allow the plugin to be tested by a few volunteers >> (hopefully). >> 3. Add the patches that make the switch over to the new implementation. >> >> >> Can someone point me to how to submit a new VM plugin? The code is >> contained in a subclass of InterpreterPlugin. >> >> I've been using this as my production environment for about 2 months now >> on a linux 32 bit VM. Running the full test suite results in the same >> set of test failing before and after applying the patch. >> >> I've also ran file related tests on linux 64 bit (run the Test Runner, >> select all packages with "file" as part of the name and run all the >> available tests) and the full test suite on Windows 32 bit. >> >> The summary is: >> >> 1. #isSymlink now works properly on Linux (and it should also work on >> MacOS and BSD). >> 2. The list of file attributes available from FileReference now is: >> #accessTime (new) >> #changeTime (new) >> #creationTime >> #deviceId (new) >> #exists >> #gid (new) >> #inode (new) >> #isBlock (new) >> #isCharacter (new) >> #isDirectory >> #isExecutable (new) >> #isFIFO (new) >> #isFile >> #isReadable >> #isRegular (new) >> #isSocket (new) >> #isSymlink (works) >> #isWritable >> #modificationTime >> #numberOfHardLinks (new) >> #permissions >> #size >> #targetFile (new) >> #uid (new) >> 3. FileReference>>exists is faster than before (well, at least on my >> linux laptop). This is useful as it is called quite often. >> 4. It is possible to retrieve symbolic link attributes, e.g. all the >> attributes listed above plus the target path. >> >> >> Given how similar MacOS and BSD are to linux, I assume that this will >> all work without problem on those platforms (but it obviously needs to >> be tested). >> >> As implied above, the changes are all backward compatible (except >> the broken #isSymlink), although a couple deserve mention: >> >> 1. Obviously #isSymlink now answers correctly (previously it would only >> answer correctly for a broken link). >> 2. Requesting any of the attributes listed above (except #isSymlink) >> will return the value of the target path. If the FileReference is to a >> broken symbolic link, it will return the attributes of the symbolic >> link (keeping existing behaviour). >> 3. The attributes of a symbolic link can be retrieved using >> FileReference>>symlinkAttributes. >> >> Overall, performance is slightly better than before. Code that >> needs to access multiple attributes and is written to take advantage of >> the new behaviour will see significant performance improvements. >> >> >> If you've got this far and forgotten the original question :-) >> >> Can someone point me to how to submit a new VM plugin? >> >> >> Thanks, >> Alistair >> > -- www.tudorgirba.com www.feenk.com "Innovation comes in the least expected form. That is, if it is expected, it already happened."

July 25, 2017