- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [pharo-project/pharo-core] 0635d1: 40371

by Ben Coman

Marcus Denker wrote: > > > I can confirm those failing tests, in both slice.4 and slice.9, > so strange they didn't show up in the Validator report. > > Now if its less work, you can revert to the original code by doing > "Delay newCodeEnabled: false", and both tests will pass. It late > here, so I'll work on fixing those tests tomorrow. > > > The problem is that we now have a wrong version in the build while > getting a correct one does not work due to the failure of the shrink pass. > > Maybe for now reverting is the best, and then dealing with the issue as > one change and not two. > > Marcus > I am only guessing at possibilities without knowing all that is involved. Whatever you think is best practice / least effort. btw, slice.10 resolves those failing tests. cheers -ben

Nov. 17, 2014

Re: [Pharo-dev] [Vm-dev] Reproducible VM crash with excessive garbage collection

by Nicolai Hess

2014-11-16 17:46 GMT+01:00 Max Leske <maxleske(a)gmail.com>: > > Hi. > > I may have discovered a bug in the PharoVM (stable and latest), which may > have been around for at least two years (I had a very similiar situation > about two years ago with Fuel). > > I have a test that fails when a particular method contains â3 timesRepeat: > [ Smalltalk garbageCollect ]â. In order to be sure that itâ not the > #timesRepeat message thatâs causing the crash, I replaced that with > multiple âSmalltalk garbageCollectâ sends. > > The test Iâm talking about ObsoleteTest>>testFixObsoleteSharedPools. To > make the test crash the VM, simply add a couple of garbageCollects at the > end of SmalltalkImage>>fixObsoleteSharedPools. I tried to come up with a > little script to demonstrate the issue but I can only reproduce the problem > with Nautilus (which makes me suspect that the problem is related to > Nautilus). > > Steps to reproduce: > 1. get a fresh image and the latest vm: > curl get.pharo.org/40+vmLastest | bash && ./pharo-ui Pharo.image > 2. open the class browser and navigate to > SmalltalkImage>>fixObsoleteSharedPools > 3. Append ten âself garbageCollectâ statements to the method. > 4. navigate to ObsoleteTest>>testFixObsoleteSharedPools and click on the > test runner icon to the left of the method name. > > I can reproduce this with the Mac VM (OS X 10.10) as well as with the > linux VM (Ubuntu 12.04.1). > > Iâve opened an issue for this: > https://pharo.fogbugz.com/f/cases/14478/Reproducible-VM-crash-with-excessiv… > > > Cheers, > Max > > Can not reproduce this on linux with may own vm build from an older version (on ubuntu 10.04) /home/nicolai/devel/pharo-vm/pharo-vm/results/pharo NBCoInterpreter NativeBoost-CogPlugin-HolgerHansPeterFreyther.21 uuid: e0df6e2d-5694-40e2-8035-dc217200b424 Oct 14 2014 NBCogit NativeBoost-CogPlugin-HolgerHansPeterFreyther.21 uuid: e0df6e2d-5694-40e2-8035-dc217200b424 Oct 14 2014 https://github.com/pharo-project/pharo-vm.git Commit: 81b5d19917dcb78f22482a780deec48c53738396 Date: 2014-09-20 14:36:18 +0200 By: Esteban Lorenzano <estebanlm(a)gmail.com> Unix built on Oct 14 2014 20:55:01 Compiler: 4.4.3 VMMaker versionString https://github.com/pharo-project/pharo-vm.git Commit: 81b5d19917dcb78f22482a780deec48c53738396 Date: 2014-09-20 14:36:18 +0200 By: Esteban Lorenzano <estebanlm(a)gmail.com> NBCoInterpreter NativeBoost-CogPlugin-HolgerHansPeterFreyther.21 uuid: e0df6e2d-5694-40e2-8035-dc217200b424 Oct 14 2014 NBCogit NativeBoost-CogPlugin-HolgerHansPeterFreyther.21 uuid: e0df6e2d-5694-40e2-8035-dc217200b424 Oct 14 2014 reproducable on ubuntu 12.? 64 bit (32 bit latest vm): --------------- /home/nicolai/Downloads/pharo-vm/pharo NBCoInterpreter NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Nov 13 2014 NBCogit NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Nov 13 2014 https://github.com/pharo-project/pharo-vm.git Commit: 27d505ceffde98d854ed887903a2a94908637a49 Date: 2014-11-04 12:04:48 +0100 By: Esteban Lorenzano <estebanlm(a)gmail.com> Jenkins build #14876 Unix built on Nov 13 2014 00:43:51 Compiler: 4.6.3 VMMaker versionString https://github.com/pharo-project/pharo-vm.git Commit: 27d505ceffde98d854ed887903a2a94908637a49 Date: 2014-11-04 12:04:48 +0100 By: Esteban Lorenzano <estebanlm(a)gmail.com> Jenkins build #14876 NBCoInterpreter NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Nov 13 2014 NBCogit NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Nov 13 2014 both with image 40370

Nov. 17, 2014

Re: [Pharo-dev] [pharo-project/pharo-core] 0635d1: 40371

by Marcus Denker

> > I can confirm those failing tests, in both slice.4 and slice.9, > so strange they didn't chow up in the Validator report. > > Now if its less work, you can revert to the original code by doing "Delay > newCodeEnabled: false", and both tests will pass. It late here, so I'll > work on fixing those tests tomorrow. > The problem is that we now have a wrong version in the build while getting a correct one does not work due to the failure of the shrink pass. Maybe for now reverting is the best, and then dealing with the issue as one change and not two. Marcus

Nov. 17, 2014

WhatsUp from: 2014-11-17 until: 2014-11-30

by seaside＠rmod.lille.inria.fr

Hi! We're sending this automatic email twice a month, to give the community an opportunity to easily know what's happening and to coordinate efforts. Just answer informally, and feel free to spawn discussions thereafter! ### Here's what I've been up to since the last WhatsUp: - $HEROIC_ACHIEVEMENTS_OR_DISMAL_FAILURES_OR_SIMPLE_BORING_NECESSARY_TASKS ### What's next, until 2014-11-30 (*): - $NEXT_STEPS_TOWARDS_WORLD_DOMINATION (*) we'll be expecting results by then ;)

Nov. 17, 2014

Re: [Pharo-dev] [NB] NativeBoost and optCoerceNilToNull

by Nicolai Hess

2014-11-16 19:11 GMT+01:00 Max Leske <maxleske(a)gmail.com>: > > On 15.11.2014, at 22:28, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > > On 15 November 2014 15:53, Max Leske <maxleske(a)gmail.com> wrote: > >> Once again Iâm struggling with NativeBoost. Iâm trying to use >> optCoerceNilToNull with this function: >> >> ^self call: #(LGitReturnCodeEnum git_commit_create( >> LGitId * theCommitId, >> LGitRepository repo, >> String update_ref, >> LGitSignature * theAuthor, >> LGitSignature * theCommitter, >> String message_encoding, >> String theMessage, >> LGitTree theTree, >> int parent_count, >> LGitCommit * parentsPointer)) options: #( optCoerceNilToNull ) >> >> The last argument can be NULL, per documentation, (no parents). >> LGitCommit is a subclass of NBExternalObject and apparently >> NBExternalObjctType>>pushAsPointer: does *not* use the optCoerceNilToNull >> option (the super implementation in NBExternalType>>pushAsPointer: however >> does). The upshot of course is that the call fails with âan instance of >> LGitCommit expectedâ. >> >> My questions: >> - The documentation says: >> >> >> "#optCoerceNilToNull" >> "passing nil as a pointer-type argument, converts it to C NULL â >> >> but apparently this doesnât apply to every pointer type. Should the >> documentation be updated? >> - Why does NBExternlObjectType not use optCoerceNilToNull? I donât want >> to use tricks to pass null if I donât have to. >> > > perhaps because it is not there? :) > So, lets add it. > > > Yay! :) I have zero assembly skillsâ¦ Iâve tried the following but then get > an âundefined label done18â error: > if coerceNilToNull is set *and* the argument is indeed nil, you want to jump over all other operations on this argument (see below). (and if coerceNilToNull is set, and the argument is not nil, you should still verify the class of the oop, this is missing in your code!) > > NBExternalObject>>pushAsPointer: gen > "push a pointer to handle value" > | asm proxy oop notNil done | > proxy := gen proxy. > asm := gen asm. > > <<<<<<<<<<<<<< new > > done := asm uniqueLabelName: 'doneâ. > > >>>>>>>>>>>>>>> new > > oop := gen reserveTemp. > loader emitLoad: gen to: oop. > > <<<<<<<<<<<<<< new > > "handle nils, if we care" > gen optCoerceNilToNull > ifTrue: [ > notNil := asm uniqueLabelName: 'notNil'. > proxy nilObject. > asm cmp: asm EAX with: oop; > jne: notNil; > xor: asm EAX with: asm EAX; > jmp: done; > label: notNil ] > ifFalse: [ > "we can skip class verification, if loader loads receiver, since nothing > to fear there" > loader isReceiver ifFalse: [ > self verifyClassOf: oop is: objectClass generator: gen. > ] ]. > > >>>>>>>>>>>>>>> new > proxy fetchPointer: (self handleIvarIndex) ofObject: oop. "handle ivar" > proxy varBytesFirstFieldOf: asm EAX. "handle value ptr" > "done label here:" asm label: done. > asm push: asm EAX. > gen releaseTemps: 1. > > > > - Related: there are instances where even non-pointer arguments are >> allowed to be NULL. Why does optCoerceNilToNull only work for pointer types? >> >> > like what? Suppose the argument type is a structure, e.g > "struct foo bar", that is non-pointer type. What in this case the NULL > value for the structure? > > > Ok, I think I understand. Are you saying that, since NULL is a macro, you > have no way of telling during runtime how NULL is defined? For a pointer it > is simply 0 or (void *) 0, but for a non pointer it depends on the context. > Is that about right? > > > >> Igor! Help! :) >> >> Cheers, >> Max >> >> > > > -- > Best regards, > Igor Stasenko. > > >

Nov. 16, 2014

Re: [Pharo-dev] How do I get a >512MB image with Pharo 3 on a mac?

by Sven Van Caekenberghe

I can run the following: Array streamContents: [ :out | 1 to: 20 do: [ :each | Transcript crShow: each asString. out nextPut: (ByteArray new: 16*1e6). Transcript crShow: ('Now using {1} Mb' format: { (Smalltalk vm memoryEnd / 1e6) floor }) ] ]. but not higher. > On 16 Nov 2014, at 23:48, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hi Paul, > > Maybe this older discussion can help: > > http://forum.world.st/Pharo-dev-Exploring-Heap-Size-Limits-td4696226.html#a… > > especially Phil's explorations. > > Regards, > > Sven > >> On 16 Nov 2014, at 21:49, Paul DeBruicker <pdebruic(a)gmail.com> wrote: >> >> Hi - >> >> I don't have a need for a big image. I just discovered I have no idea how to make one larger than ~512MB, and am curious. >> >> I was running into an out of memory error and discussing it in a thread on pharo-users list (http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi…) The details of that discussion aren't pertinent to this question. >> >> This script below ought to make an image that maxes out well above a GB but cannot cross 512MB without throwing an OutOfMemory error. >> >> >> | them | >> them := OrderedCollection new. >> [[them addLast: (ByteArray new: 16000000). >> Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush] repeat] >> on: OutOfMemory >> do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk garbageCollect]]. >> Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush. >> them := nil. >> Smalltalk garbageCollect. >> Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush >> >> >> In the transcript you will see an output of the image size as more ByteArrays are added to the OrderedCollection. On my laptop with 4GB of ram it always peaks at just over 500MB. >> >> Setting the memory from the command line (e.g. pharo-ui --memory 1000m Pharo.image) does not help. >> >> Is there a VM flag that I need to flip? >> >> Thanks >> >> Paul >

Nov. 16, 2014

Re: [Pharo-dev] How do I get a >512MB image with Pharo 3 on a mac?

by Sven Van Caekenberghe

Hi Paul, Maybe this older discussion can help: http://forum.world.st/Pharo-dev-Exploring-Heap-Size-Limits-td4696226.html#a… especially Phil's explorations. Regards, Sven > On 16 Nov 2014, at 21:49, Paul DeBruicker <pdebruic(a)gmail.com> wrote: > > Hi - > > I don't have a need for a big image. I just discovered I have no idea how to make one larger than ~512MB, and am curious. > > I was running into an out of memory error and discussing it in a thread on pharo-users list (http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi…) The details of that discussion aren't pertinent to this question. > > This script below ought to make an image that maxes out well above a GB but cannot cross 512MB without throwing an OutOfMemory error. > > > | them | > them := OrderedCollection new. > [[them addLast: (ByteArray new: 16000000). > Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush] repeat] > on: OutOfMemory > do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk garbageCollect]]. > Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush. > them := nil. > Smalltalk garbageCollect. > Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush > > > In the transcript you will see an output of the image size as more ByteArrays are added to the OrderedCollection. On my laptop with 4GB of ram it always peaks at just over 500MB. > > Setting the memory from the command line (e.g. pharo-ui --memory 1000m Pharo.image) does not help. > > Is there a VM flag that I need to flip? > > Thanks > > Paul

Nov. 16, 2014

How do I get a >512MB image with Pharo 3 on a mac?

by Paul DeBruicker

Hi - I don't have a need for a big image. I just discovered I have no idea how to make one larger than ~512MB, and am curious. I was running into an out of memory error and discussing it in a thread on pharo-users list (http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-fi…) The details of that discussion aren't pertinent to this question. This script below ought to make an image that maxes out well above a GB but cannot cross 512MB without throwing an OutOfMemory error. | them | them := OrderedCollection new. [[them addLast: (ByteArray new: 16000000). Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush] repeat] on: OutOfMemory do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk garbageCollect]]. Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush. them := nil. Smalltalk garbageCollect. Transcript cr; show: ((Smalltalk vm parameterAt: 3) / (1024*1024.0) printShowingDecimalPlaces: 1); flush In the transcript you will see an output of the image size as more ByteArrays are added to the OrderedCollection. On my laptop with 4GB of ram it always peaks at just over 500MB. Setting the memory from the command line (e.g. pharo-ui --memory 1000m Pharo.image) does not help. Is there a VM flag that I need to flip? Thanks Paul

Nov. 16, 2014

Re: [Pharo-dev] [NB] NativeBoost and optCoerceNilToNull

by Max Leske

> On 16.11.2014, at 19:35, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > > On 16 November 2014 19:11, Max Leske <maxleske(a)gmail.com <mailto:maxleske@gmail.com>> wrote: > >> On 15.11.2014, at 22:28, Igor Stasenko <siguctua(a)gmail.com <mailto:siguctua@gmail.com>> wrote: >> >> >> >> On 15 November 2014 15:53, Max Leske <maxleske(a)gmail.com <mailto:maxleske@gmail.com>> wrote: >> Once again Iâm struggling with NativeBoost. Iâm trying to use optCoerceNilToNull with this function: >> >> ^self call: #(LGitReturnCodeEnum git_commit_create( >> LGitId * theCommitId, >> LGitRepository repo, >> String update_ref, >> LGitSignature * theAuthor, >> LGitSignature * theCommitter, >> String message_encoding, >> String theMessage, >> LGitTree theTree, >> int parent_count, >> LGitCommit * parentsPointer)) options: #( optCoerceNilToNull ) >> >> The last argument can be NULL, per documentation, (no parents). >> LGitCommit is a subclass of NBExternalObject and apparently NBExternalObjctType>>pushAsPointer: does *not* use the optCoerceNilToNull option (the super implementation in NBExternalType>>pushAsPointer: however does). The upshot of course is that the call fails with âan instance of LGitCommit expectedâ. >> >> My questions: >> - The documentation says: >> >> "#optCoerceNilToNull" >> "passing nil as a pointer-type argument, converts it to C NULL â >> >> but apparently this doesnât apply to every pointer type. Should the documentation be updated? >> - Why does NBExternlObjectType not use optCoerceNilToNull? I donât want to use tricks to pass null if I donât have to. >> >> perhaps because it is not there? :) >> So, lets add it. > > Yay! :) I have zero assembly skillsâ¦ Iâve tried the following but then get an âundefined label done18â error: > > NBExternalObject>>pushAsPointer: gen > "push a pointer to handle value" > | asm proxy oop notNil done | > > proxy := gen proxy. > asm := gen asm. > > <<<<<<<<<<<<<< new > > done := asm uniqueLabelName: 'doneâ. > > >>>>>>>>>>>>>>> new > > oop := gen reserveTemp. > loader emitLoad: gen to: oop. > > <<<<<<<<<<<<<< new > > "handle nils, if we care" > gen optCoerceNilToNull > ifTrue: [ > notNil := asm uniqueLabelName: 'notNil'. > proxy nilObject. > asm cmp: asm EAX with: oop; > jne: notNil; > xor: asm EAX with: asm EAX; > jmp: done; > label: notNil ] > ifFalse: [ > "we can skip class verification, if loader loads receiver, since nothing to fear there" > loader isReceiver ifFalse: [ > self verifyClassOf: oop is: objectClass generator: gen. > ] ]. > > >>>>>>>>>>>>>>> new > > proxy fetchPointer: (self handleIvarIndex) ofObject: oop. "handle ivar" > proxy varBytesFirstFieldOf: asm EAX. "handle value ptr" > asm push: asm EAX. > > gen releaseTemps: 1. > > > please make an issue, so we can fix & integrate it. Here you go: https://pharo.fogbugz.com/f/cases/14479/Enable-nil-to-NULL-coercion-for-NBE… > >> >> - Related: there are instances where even non-pointer arguments are allowed to be NULL. Why does optCoerceNilToNull only work for pointer types? >> >> >> like what? Suppose the argument type is a structure, e.g >> "struct foo bar", that is non-pointer type. What in this case the NULL value for the structure? > > Ok, I think I understand. Are you saying that, since NULL is a macro, you have no way of telling during runtime how NULL is defined? For a pointer it is simply 0 or (void *) 0, but for a non pointer it depends on the context. Is that about right? > > > I am saying that there is only 'NULL pointer' defined, > i never heard anything about 'NULL Float', or 'NULL struct', or 'NULL union'. > This is why there's only nil->NULL coercion provided. >> >> Igor! Help! :) >> >> Cheers, >> Max >> >> >> >> >> -- >> Best regards, >> Igor Stasenko. > > > > -- > Best regards, > Igor Stasenko.

Nov. 16, 2014

Re: [Pharo-dev] [NB] Undocumented NativeBoost options

by Max Leske

Thanks Igor. > On 16.11.2014, at 20:04, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > > On 15 November 2014 12:37, Max Leske <maxleske(a)gmail.com <mailto:maxleske@gmail.com>> wrote: > I got a bit frustrated and ended up searching the source for â[\w]opt[[:upper]]â to find all NativeBoost options. Some of those options are in fact undocumented: > > optMayGC > - tells that during calling external function one may trigger GC. Therefore, if that option is on, the code generator will use a gate function for call, to safeguard against possible relocation of compiled method, which contains machine code (and thus could return from external function to wrong place). > > optProxyLabels > - used for debugging purposes, to see where the proxy function (and what exactly function) get called in generated code. Does not affects anything in resulting machine code. > > optCdecl > - use cdecl calling convention. This option are exposed through different public API > and not used directly > > optReturnNullAsNil > - for pointer return types and ExternalObject, coerce null return value to nil. > > optStdcall > - same as for optCdecl, except it is stdcall convention. > > optNonMovable > - tells the code generator that the generated code will never move (relocate) in memory. Used for generating code and placing it in external memory > > optReserveCallgateSpace > - not used > > optDebug > - halt at the end of code generation, to see what is generated. For debugging purposes. > > optNoAlignment > - ignore default stack alignment when generating a call to external function. > Can be used, if you absolutely sure that subject function doesn't requires stack alignment. Thus one can avoid emitting extra code for stack alignment => saving some CPU cycles. > > optEmitCall > - private. > > optNoCleanup > - do not emit code for cleaning stack after calling a function according to calling convention (not all conventions require stack cleanup, cdecl does). Basically, leaving stack pointer untouched after return from external call. > > optCheckFailOnEveryArgument > - check for the primitive failure after each argument coercion. > By default the check is generated only at single place, right before performing an external function call, after all arguments are coerced and pushed on stack. Useful for debugging, when it is hard to figure, which exactly argument fails the coercion. > > optStringOrNull > - allow passing nil for arguments of String type, effectively will push null pointer as argument > > WinUnicode > - Controls the coercion of Windows string types to be either single-byte or multi-byte. Related to types like LPCTSTR etc. See NBTString class and its sole class-side method. > > > > -- > Best regards, > Igor Stasenko.

Nov. 16, 2014