=
= signal
=
Unfortunately it doesn't fix it, or at least I get the same sympthoms. Sending SIGUSR1 prints this: SIGUSR1 Wed Jun 3 16:53:50 2015 /home/chous/toolbox/pharo-4.0/pharo-vm/pharo pharo VM version: 3.9-7 #1 Thu Apr 2 00:51:45 CEST 2015 gcc 4.6.3 [Production ITHB VM] Built from: NBCoInterpreter NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Apr 2 2015 With: NBCogit NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Apr 2 2015 Revision: https://github.com/pharo-project/pharo-vm.git Commit: 32d18ba0f2db9bee7f3bdbf16bdb24fe4801cfc5 Date: 2015-03-24 11:08:14 +0100 By: Esteban Lorenzano <estebanlm@gmail.com> Jenkins build #14904 Build host: Linux pharo-linux 3.2.0-31-generic-pae #50-Ubuntu SMP Fri Sep 7 16:39:45 UTC 2012 i686 i686 i386 GNU/Linux plugin path: /home/chous/toolbox/pharo-4.0/pharo-vm/ [default: /home/chous/toolbox/pharo-4.0/pharo-vm/] C stack backtrace & registers: eax 0xff981e94 ebx 0xff981db0 ecx 0xff981e48 edx 0xff981dfc edi 0xff981c80 esi 0xff981c80 ebp 0xff981d18 esp 0xff981d64 eip 0xff981f78 *[0xff981f78] /home/chous/toolbox/pharo/pharo-vm/pharo[0x80a33a2] /home/chous/toolbox/pharo/pharo-vm/pharo[0x80a3649] linux-gate.so.1(__kernel_rt_sigreturn+0x0)[0xf773acc0] /home/chous/toolbox/pharo/pharo-vm/pharo(signalSemaphoreWithIndex+0x28)[0x809d8c8] /home/chous/toolbox/pharo/pharo-vm/pharo[0x810868c] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf773acb0] /home/chous/toolbox/pharo/pharo-vm/pharo(signalSemaphoreWithIndex+0x5e)[0x809d8fe] /home/chous/toolbox/pharo/pharo-vm/pharo(aioPoll+0x22f)[0x809f0af] /home/chous/toolbox/pharo-4.0/pharo-vm/vm-display-X11(+0xe671)[0xf772a671] /home/chous/toolbox/pharo/pharo-vm/pharo(ioRelinquishProcessorForMicroseconds+0x17)[0x80a1887] /home/chous/toolbox/pharo/pharo-vm/pharo[0x80767fa] [0xb4a2fe0c] [0xb4a2d700] [0xb53b9382] [0xb4a2d648] [0x5b] All Smalltalk process stacks (active first): Process 0xb6d930c4 priority 10 0xff9ad450 M ProcessorScheduler class>idleProcess 0xb4d935c0: a(n) ProcessorScheduler class 0xff9ad470 I [] in ProcessorScheduler class>startUp 0xb4d935c0: a(n) ProcessorScheduler class 0xff9ad490 I [] in BlockClosure>newProcess 0xb6d92fe8: a(n) BlockClosure suspended processes Process 0xb68e1984 priority 50 0xff9a6490 M WeakArray class>finalizationProcess 0xb4d93790: a(n) WeakArray class 0xb69beb68 s [] in WeakArray class>restartFinalizationProcess 0xb68e1924 s [] in BlockClosure>newProcess Process 0xb5ced038 priority 80 0xff9af490 M DelayMicrosecondScheduler>runTimerEventLoop 0xb5bb6f9c: a(n) DelayMicrosecondScheduler 0xb6098314 s [] in DelayMicrosecondScheduler>startTimerEventLoop 0xb5cecfd8 s [] in BlockClosure>newProcess Process 0xb68ec880 priority 40 0xff9b2478 M [] in UnixOSProcessAccessor>(nil) 0xb60dc6d0: a(n) UnixOSProcessAccessor 0xff9b2490 M BlockClosure>repeat 0xb68ef2d4: a(n) BlockClosure 0xb68ef278 s [] in UnixOSProcessAccessor>(nil) 0xb68ec820 s [] in BlockClosure>newProcess Process 0xb6d92d78 priority 60 0xff98742c M InputEventFetcher>waitForInput 0xb5a09718: a(n) InputEventFetcher 0xff987450 M InputEventFetcher>eventLoop 0xb5a09718: a(n) InputEventFetcher 0xff987470 I [] in InputEventFetcher>installEventLoop 0xb5a09718: a(n) InputEventFetcher 0xff987490 I [] in BlockClosure>newProcess 0xb6d92c9c: a(n) BlockClosure Process 0xb6f25f94 priority 60 0xb6f25fcc s SmalltalkImage>lowSpaceWatcher 0xb71523e4 s [] in SmalltalkImage>installLowSpaceWatcher 0xb6f25f34 s [] in BlockClosure>newProcess Process 0xb73a4e7c priority 30 0xff99b470 M [] in AioEventHandler>handleExceptions:readEvents:writeEvents: 0xb73a49e4: a(n) AioEventHandler 0xff99b490 I [] in BlockClosure>newProcess 0xb73a4d90: a(n) BlockClosure Process 0xb6686c88 priority 40 0xffa073d0 M [] in Delay>wait 0xb73a63fc: a(n) Delay 0xffa073f0 M BlockClosure>ifCurtailed: 0xb73a6614: a(n) BlockClosure 0xffa0740c M Delay>wait 0xb73a63fc: a(n) Delay 0xffa07428 M PipeableOSProcess(PipeJunction)>outputOn: 0xb73a0d34: a(n) PipeableOSProcess 0xffa07444 M PipeableOSProcess(PipeJunction)>output 0xb73a0d34: a(n) PipeableOSProcess 0xffa0746c M [] in MCFileTreeGitRepository class>runOSProcessGitCommand:in: 0xb611fa88: a(n) MCFileTreeGitRepository class 0xffa0748c M BlockClosure>ensure: 0xb739d9dc: a(n) BlockClosure 0xff9e538c M MCFileTreeGitRepository class>runOSProcessGitCommand:in: 0xb611fa88: a(n) MCFileTreeGitRepository class 0xff9e53ac M MCFileTreeGitRepository class>runGitCommand:in: 0xb611fa88: a(n) MCFileTreeGitRepository class 0xff9e53cc M MCFileTreeGitRepository>gitCommand:in: 0xb612926c: a(n) MCFileTreeGitRepository 0xff9e53f4 M MCFileTreeGitRepository>gitVersionsForPackage: 0xb612926c: a(n) MCFileTreeGitRepository 0xff9e543c M [] in MCFileTreeGitRepository>loadAllFileNames 0xb612926c: a(n) MCFileTreeGitRepository 0xff9e5458 M FileSystemDirectoryEntry(Object)>in: 0xb71b3fe8: a(n) FileSystemDirectoryEntry 0xff9e548c M [] in MCFileTreeGitRepository>loadAllFileNames 0xb612926c: a(n) MCFileTreeGitRepository 0xffa04310 M BlockClosure>cull: 0xb71b4894: a(n) BlockClosure 0xffa04338 I [] in Job>run 0xb71b48b4: a(n) Job 0xffa04350 M BlockClosure>on:do: 0xb71b56b8: a(n) BlockClosure 0xffa0437c I [] in Job>run 0xb71b48b4: a(n) Job 0xffa0439c M BlockClosure>ensure: 0xb71b4980: a(n) BlockClosure 0xffa043c4 I Job>run 0xb71b48b4: a(n) Job 0xffa043e4 I MorphicUIManager(UIManager)>displayProgress:from:to:during: 0xb50a8790: a(n) MorphicUIManager 0xffa04414 I ByteString(String)>displayProgressFrom:to:during: 0xb61238d8: a(n) ByteString 0xffa04444 M MCFileTreeGitRepository>loadAllFileNames 0xb612926c: a(n) MCFileTreeGitRepository 0xffa04464 I MCFileTreeGitRepository>allFileNames 0xb612926c: a(n) MCFileTreeGitRepository 0xffa0448c M MCFileTreeGitRepository>goferVersionFrom: 0xb612926c: a(n) MCFileTreeGitRepository 0xff9e238c I MetacelloCachingGoferResolvedReference(GoferResolvedReference)>version 0xb71b3134: a(n) MetacelloCachingGoferResolvedReference 0xff9e23a4 M MetacelloCachingGoferResolvedReference>version 0xb71b3134: a(n) MetacelloCachingGoferResolvedReference 0xff9e23bc M [] in MetacelloFetchingMCSpecLoader>resolveDependencies:nearest:into: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9e23e0 M OrderedCollection>do: 0xb71b3234: a(n) OrderedCollection 0xff9e240c M [] in MetacelloFetchingMCSpecLoader>resolveDependencies:nearest:into: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9e2424 M BlockClosure>on:do: 0xb71b3334: a(n) BlockClosure 0xff9e244c M MetacelloFetchingMCSpecLoader>resolveDependencies:nearest:into: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9e2490 M [] in MetacelloFetchingMCSpecLoader>linearLoadPackageSpec:gofer: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9ae318 M MetacelloPharo30Platform(MetacelloPlatform)>do:displaying: 0xb50e8b94: a(n) MetacelloPharo30Platform 0xff9ae338 M MetacelloFetchingMCSpecLoader>linearLoadPackageSpec:gofer: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9ae358 M MetacelloPackageSpec>loadUsing:gofer: 0xb706be54: a(n) MetacelloPackageSpec 0xff9ae37c M [] in MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>linearLoadPackageSpecs:repositories: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9ae3a0 M OrderedCollection>do: 0xb70c807c: a(n) OrderedCollection 0xff9ae3c0 M MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>linearLoadPackageSpecs:repositories: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9ae3f0 I [] in MetacelloFetchingMCSpecLoader>linearLoadPackageSpecs:repositories: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xff9ae410 M BlockClosure>ensure: 0xb70c813c: a(n) BlockClosure 0xff9ae438 I MetacelloLoaderPolicy>pushLoadDirective:during: 0xb706cb7c: a(n) MetacelloLoaderPolicy 0xff9ae460 I MetacelloLoaderPolicy>pushLinearLoadDirectivesDuring:for: 0xb706cb7c: a(n) MetacelloLoaderPolicy 0xff9ae488 I MetacelloFetchingMCSpecLoader>linearLoadPackageSpecs:repositories: 0xb706d83c: a(n) MetacelloFetchingMCSpecLoader 0xb70c33c0 s MetacelloFetchingMCSpecLoader(MetacelloCommonMCSpecLoader)>load 0xb706d898 s MetacelloMCVersionSpecLoader>load 0xb71948d0 s MetacelloMCVersion>executeLoadFromArray: 0xb719492c s [] in MetacelloMCVersion>fetchRequiredFromArray: 0xb7194988 s [] in MetacelloPharo30Platform(MetacelloPlatform)>useStackCacheDuring:defaultDictionary: 0xb706d96c s BlockClosure>on:do: 0xb706d2d8 s MetacelloPharo30Platform(MetacelloPlatform)>useStackCacheDuring:defaultDictionary: 0xb706d258 s [] in MetacelloMCVersion>fetchRequiredFromArray: 0xb71949e4 s BlockClosure>ensure: 0xb706d15c s [] in MetacelloMCVersion>fetchRequiredFromArray: 0xb706d1e4 s MetacelloPharo30Platform(MetacelloPlatform)>do:displaying: 0xb706d0e4 s MetacelloMCVersion>fetchRequiredFromArray: 0xb706ccc0 s [] in MetacelloMCVersion>doLoadRequiredFromArray: 0xb715327c s BlockClosure>ensure: 0xb706cc34 s MetacelloMCVersion>doLoadRequiredFromArray: 0xb71532d8 s MetacelloMCVersion>load 0xb7153334 s UndefinedObject>(nil) 0xb7153390 s OpalCompiler>evaluate 0xb706ab30 s RubSmalltalkEditor>evaluate:andDo: 0xb706a7f4 s RubSmalltalkEditor>highlightEvaluateAndDo: 0xb7152edc s [] in GLMMorphicPharoPlaygroundRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb7152f38 s RubEditingArea(RubAbstractTextArea)>handleEdit: 0xb706a784 s [] in GLMMorphicPharoPlaygroundRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb7152f94 s WorldState>runStepMethodsIn: 0xb7152ff0 s WorldMorph>runStepMethods 0xb706a1cc s WorldState>doOneCycleNowFor: 0xb715304c s WorldState>doOneCycleFor: 0xb71530a8 s WorldMorph>doOneCycle 0xb6686f8c s [] in MorphicUIManager>spawnNewProcess 0xb6686c28 s [] in BlockClosure>newProcess Most recent primitives primCreatePipe new: at:put: at:put: basicNew basicNew: basicNew basicNew: primSQFileSetBlocking: basicNew: basicAt:put: basicNew: basicAt:put: at:put: basicNew primSigPipeNumber basicNew wait at:put: signal primForwardSignal:toSemaphore: wait at:put: signal primCreatePipe new: at:put: at:put: basicNew basicNew: basicNew basicNew: primSQFileSetNonBlocking: basicNew: basicAt:put: basicNew: basicAt:put: at:put: basicNew signal basicNew: basicAt:put: basicNew: basicAt:put: at:put: new: basicNew new: replaceFrom:to:with:startingAt: basicNew basicNew: primSQFileSetNonBlocking: basicNew stringHash:initialHash: primOSFileHandle: basicNew wait at:put: signal primAioEnable:forSemaphore:externalObject: basicNew objectAt: basicNew: stackp: basicNew primitiveResume wait wait signal wait signal primAioHandle:exceptionEvents:readEvents:writeEvents: signal basicNew: basicAt:put: primSQFileSetNonBlocking: basicNew: basicAt:put: basicNew: basicAt:put: at:put: basicNew basicNew wait signal primUTCMicrosecondsClock + + < primSignal:atUTCMicroseconds: wait signal wait wait relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: primUTCMicrosecondsClock + primSignal:atUTCMicroseconds: wait basicNew basicNew basicNew basicNew signal basicNew signal basicNew new: wait new: at:put: at:put: at:put: basicNew: at:put: basicNew: replaceFrom:to:with:startingAt: replaceFrom:to:with:startingAt: basicNew new: at:put: new: basicNew: replaceFrom:to:with:startingAt: replaceFrom:to:with:startingAt: at:put: basicNew: replaceFrom:to:with:startingAt: replaceFrom:to:with:startingAt: at:put: at:put: at:put: new: replaceFrom:to:with:startingAt: primSizeOfPointer new: at:put: at:put: at:put: primSizeOfPointer basicNew: basicNew at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: at:put: replaceFrom:to:with:startingAt: replaceFrom:to:with:startingAt: replaceFrom:to:with:startingAt: new: basicNew new: at:put: at:put: at:put: at:put: at:put: at:put: new: replaceFrom:to:with:startingAt: new: at:put: at:put: primGetCurrentWorkingDirectory basicNew: replaceFrom:to:with:startingAt: replaceFrom:to:with:startingAt: primForkExec:stdIn:stdOut:stdErr:argBuf:argOffsets:envBuf:envOffsets:workingDir: primGetPid primGetPid primGetPid basicNew basicNew wait at:put: signal wait shallowCopy new: replaceFrom:to:with:startingAt: signal wait replaceFrom:to:with:startingAt: at:put: signal primCloseNoError: primCloseNoError: primCloseNoError: signal basicNew: basicNew basicNew basicNew wait signal primUTCMicrosecondsClock + + < primSignal:atUTCMicroseconds: wait signal wait relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: basicNew: primRead:into:startingAt:count: basicNew signal wait basicNew: basicNew basicNew: replaceFrom:to:with:startingAt: replaceFrom:to:with:startingAt: signal basicNew signal basicNew new: wait signal wait signal primAioHandle:exceptionEvents:readEvents:writeEvents: signal wait relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: relinquishProcessorForMicroseconds: stack page bytes 4096 available headroom 3300 minimum unused headroom 2152 (SIGUSR1) 2015-06-03 14:15 GMT+02:00 David T. Lewis <lewis@mail.msen.com>:
On Wed, Jun 03, 2015 at 07:05:15AM +0200, Thierry Goubier wrote:
Hi Dave,
Le 03/06/2015 03:15, David T. Lewis a ?crit :
Hi Thierry and Jose,
I am reading this thread with interest and will help if I can.
I do have one idea that we have not tried before. I have a theory that this may be an intermittent problem caused by SIGCHLD signals (from the external OS process when it exits) being missed by the UnixOSProcessAccessor>>grimReaperProcess that handles them.
If this is happening, then I may be able to change grimReaperProcess to work around the problem.
When you see the OS deadlock condition, are you able tell if your Pharo VM process has subprocesses in the zombie state (indicating that grimReaperProcess did not clean them up)? The unix command "ps -axf | less" will let you look at the process tree and that may give us a clue if this is happening.
I found it very easy to reproduce and I do have a zombie children process to the pharo process.
Jose confirms this also (thanks).
Can you try filing in the attached UnixOSProcessAccessor>>grimReaperProcess and see if it helps? I do not know if it will make a difference, but the idea is to put a timeout on the semaphore that is waiting for signals from SIGCHLD. I am hoping that if these signals are sometimes being missed, then the timeout will allow the process to recover from the problem.
Interesting enough, the lock-up happens in a very specific place, a call to git branch, which is a very short command returning just a few characters (where all other commands have longuer outputs). Reducing the frequency of the calls to git branch by a bit of caching reduces the chances of a lock-up.
This is a good clue, and it may indicate a different kind of problem (so maybe I am looking in the wrong place). Ben's suggestion of adding a delay to the external process sounds like a good idea to help troubleshoot it.
Dave