July 2015 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] NativeBoost VM crash

by Max Leske

> On 23 Jul 2015, at 14:42, Holger Freyther <holger(a)freyther.de> wrote: > > Hi, > > > I have a class that calls RAND_bytes in libcrypto and I am (mis-?)using NativeBoost > for that. The function appears to work in general and I can make 1000000 calls in a > row. > > I am now invoking (RAND bytes: 4) asInteger from within and then get crashes if I > insert enough (10000) objects. Iâm using a Pharo3 image with todays Pharo-vm-mac-latest > and it is easily reproducible. > > I am not sure if that is before or after the C code has been called but the frame just > looks invalid (hence invalid). I had a breakpoint in the initial call to RAND_bytes and > looked at the âtrampolineâ, at the time it is crashing the frame is a different one and > the original is gone (disassembles to something different) . > > > Am I using NativeBoost correctly here? Do you have any hints? Is the code moving > or being garbage collected? Is there an easy way I could proof the theory? > > > kind regards > holger Have you tried using #optMayGC? That option will put the function into a safe memory region to prevent it from being moved during GC. Cheers, Max > > > Details > > * thread #1: tid = 0x120dae, 0x21a70140, queue = 'com.apple.main-thread', stop reason = signal SIGILL > frame #0: 0x21a70140 > -> 0x21a70140: negl %esp > 0x21a70142: addl -0xc(%ebp), %esp > 0x21a70145: movl 0xada30, %eax > 0x21a7014b: movl (%eax), %eax > > (lldb) register read > General Purpose Registers: > eax = 0x21a70110 > ebx = 0x000000e8 > ecx = 0x00000002 > edx = 0x0000000f > edi = 0x21a7010c > esi = 0x21a70131 > ebp = 0xbffacb38 > esp = 0x00000008 > ss = 0x00000023 > eflags = 0x00000202 > eip = 0x21a70140 > cs = 0x0000001b > ds = 0x00000023 > es = 0x00000023 > fs = 0x00000000 > gs = 0x0000000f > > (lldb) bt > * thread #1: tid = 0x120dae, 0x21a70140, queue = 'com.apple.main-thread', stop reason = signal SIGILL > * frame #0: 0x21a70140 > frame #1: 0x0009827b Pharo`primitiveNativeCall + 107 > frame #2: 0x1f400780 > > > (lldb) di -f > -> 0x21a70140: negl %esp > 0x21a70142: addl -0xc(%ebp), %esp > 0x21a70145: movl 0xada30, %eax > 0x21a7014b: movl (%eax), %eax > 0x21a7014d: movl %esp, -0x10(%ebp) > 0x21a70150: subl $0x4, %esp > 0x21a70153: andl $0xf, %esp > 0x21a70156: negl %esp > 0x21a70158: addl -0x10(%ebp), %esp > 0x21a7015b: pushl %eax > > > > > Object subclass: #RAND > instanceVariableNames: '' > classVariableNames: '' > poolDictionaries: '' > category: 'Hack'! > "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! > > RAND class > instanceVariableNames: ''! > > !RAND class methodsFor: 'as yet unclassified' stamp: 'HolgerHansPeterFreyther 7/23/2015 11:49'! > intRand: aByteArray size: aSize > <primitive: 'primitiveNativeCall' module: 'NativeBoostPlugin'> > > ^self nbCall: #(int RAND_bytes(byte* aByteArray, int aSize)) module: 'crypto'.! ! > > !RAND class methodsFor: 'as yet unclassified' stamp: 'HolgerHansPeterFreyther 7/23/2015 11:48'! > rand: numberOfBytes > | rand | > rand := ByteArray new: numberOfBytes. > ^(self intRand: rand size: rand size) = 1 > ifTrue: [ rand ] > ifFalse: [ nil ]! !

July 23, 2015

NativeBoost VM crash

by Holger Freyther

Hi, I have a class that calls RAND_bytes in libcrypto and I am (mis-?)using NativeBoost for that. The function appears to work in general and I can make 1000000 calls in a row. I am now invoking (RAND bytes: 4) asInteger from within and then get crashes if I insert enough (10000) objects. Iâm using a Pharo3 image with todays Pharo-vm-mac-latest and it is easily reproducible. I am not sure if that is before or after the C code has been called but the frame just looks invalid (hence invalid). I had a breakpoint in the initial call to RAND_bytes and looked at the âtrampolineâ, at the time it is crashing the frame is a different one and the original is gone (disassembles to something different) . Am I using NativeBoost correctly here? Do you have any hints? Is the code moving or being garbage collected? Is there an easy way I could proof the theory? kind regards holger Details * thread #1: tid = 0x120dae, 0x21a70140, queue = 'com.apple.main-thread', stop reason = signal SIGILL frame #0: 0x21a70140 -> 0x21a70140: negl %esp 0x21a70142: addl -0xc(%ebp), %esp 0x21a70145: movl 0xada30, %eax 0x21a7014b: movl (%eax), %eax (lldb) register read General Purpose Registers: eax = 0x21a70110 ebx = 0x000000e8 ecx = 0x00000002 edx = 0x0000000f edi = 0x21a7010c esi = 0x21a70131 ebp = 0xbffacb38 esp = 0x00000008 ss = 0x00000023 eflags = 0x00000202 eip = 0x21a70140 cs = 0x0000001b ds = 0x00000023 es = 0x00000023 fs = 0x00000000 gs = 0x0000000f (lldb) bt * thread #1: tid = 0x120dae, 0x21a70140, queue = 'com.apple.main-thread', stop reason = signal SIGILL * frame #0: 0x21a70140 frame #1: 0x0009827b Pharo`primitiveNativeCall + 107 frame #2: 0x1f400780 (lldb) di -f -> 0x21a70140: negl %esp 0x21a70142: addl -0xc(%ebp), %esp 0x21a70145: movl 0xada30, %eax 0x21a7014b: movl (%eax), %eax 0x21a7014d: movl %esp, -0x10(%ebp) 0x21a70150: subl $0x4, %esp 0x21a70153: andl $0xf, %esp 0x21a70156: negl %esp 0x21a70158: addl -0x10(%ebp), %esp 0x21a7015b: pushl %eax Object subclass: #RAND instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Hack'! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! RAND class instanceVariableNames: ''! !RAND class methodsFor: 'as yet unclassified' stamp: 'HolgerHansPeterFreyther 7/23/2015 11:49'! intRand: aByteArray size: aSize <primitive: 'primitiveNativeCall' module: 'NativeBoostPlugin'> ^self nbCall: #(int RAND_bytes(byte* aByteArray, int aSize)) module: 'crypto'.! ! !RAND class methodsFor: 'as yet unclassified' stamp: 'HolgerHansPeterFreyther 7/23/2015 11:48'! rand: numberOfBytes | rand | rand := ByteArray new: numberOfBytes. ^(self intRand: rand size: rand size) = 1 ifTrue: [ rand ] ifFalse: [ nil ]! !

July 23, 2015

Re: [Pharo-dev] Contributing to VoyageMongo (improving insertion/updating speed)

by Sven Van Caekenberghe

Good catch, that seems wrong to me, and as you showed it is way too slow. UUIDGenerator should certainly not be reinitialised each time a UUID is needed. There have been other discussions about UUID in the past. /dev/urandom is not available on all platforms anyway, so alternatives might be useful. Like Stef said, the MongoDB part is out of our scope. Please make an issue (about the #makeSeed itself). > On 23 Jul 2015, at 13:31, stepharo <stepharo(a)free.fr> wrote: > > Hi holger > > Esteban is on vacation and I do not who is another manager of the mongo repo. > > Stef > > Le 23/7/15 12:23, Holger Freyther a Ã©crit : >> Hi, >> >> I wanted to benchmark some look-ups with VoyageMongo and tried to insert >> some dummy objects and it was quite slow (pharo takes 99% CPU and mongo >> is idle). Using the nice profiler i noticed that VOMongoRepository>>#newVersion >> calls UUIDGenerator new makeSeed which will take quite long. >> >> >> [UUIDGenerator new makeSeed] bench '96.9 per second.â >> >> I am running this on OSX so it will end up in >>#makeUnixSeed which will open >> /dev/urandom. I needed cryptographic random numbers in another project and >> used NativeBoost to call into libcrypto of OpenSSL and use the RAND_bytes >> function to get them. >> >> >> [ (RAND rand: 4) asInteger ] bench '588,000 per second.â >> >> >> The quality of the random numbers should be comparable and now I wonder how >> I can help integrating such a change. >> >> From my point of view I can either patch Voyage-Mongo-Core to be able to >> use other random number sources. Or I could add another âseedâ source to >> the UUIDGenerator. What would be the preferred approach and how to move >> forward? Is NativeBoost something one could rely on in upcoming Pharo versions? >> >> For a simple insertion of a class with a couple (~5) fields the insertion time for >> 1000 items goes from ~10s to ~250ms. >> >> How can I move this forward? >> >> holger >> > >

July 23, 2015

Re: [Pharo-dev] More woden demos

by Alexandre Bergel

This is impressive Ronie! I return to Chile next week. We should plan about the future! Cheers Alexandre > On Jul 23, 2015, at 2:39 AM, Ronie Salgado <roniesalg(a)gmail.com> wrote: > > Hello, > > I have been working in making a exporter from blender into a custom JSON format for Woden with support for animations. Here I made a short video: https://www.youtube.com/watch?v=HQpaeDQ3TBM&feature=youtu.be > > I also made a new shader management system for Woden, so I took the opportunity for adding support for parallax occlusion mapping: https://www.youtube.com/watch?v=fFf7v37j8xg&feature=youtu.be > > Greetings, > Ronie -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

July 23, 2015

Re: [Pharo-dev] Contributing to VoyageMongo (improving insertion/updating speed)

by stepharo

Hi holger Esteban is on vacation and I do not who is another manager of the mongo repo. Stef Le 23/7/15 12:23, Holger Freyther a Ã©crit : > Hi, > > I wanted to benchmark some look-ups with VoyageMongo and tried to insert > some dummy objects and it was quite slow (pharo takes 99% CPU and mongo > is idle). Using the nice profiler i noticed that VOMongoRepository>>#newVersion > calls UUIDGenerator new makeSeed which will take quite long. > > > [UUIDGenerator new makeSeed] bench '96.9 per second.â > > I am running this on OSX so it will end up in >>#makeUnixSeed which will open > /dev/urandom. I needed cryptographic random numbers in another project and > used NativeBoost to call into libcrypto of OpenSSL and use the RAND_bytes > function to get them. > > > [ (RAND rand: 4) asInteger ] bench '588,000 per second.â > > > The quality of the random numbers should be comparable and now I wonder how > I can help integrating such a change. > > From my point of view I can either patch Voyage-Mongo-Core to be able to > use other random number sources. Or I could add another âseedâ source to > the UUIDGenerator. What would be the preferred approach and how to move > forward? Is NativeBoost something one could rely on in upcoming Pharo versions? > > For a simple insertion of a class with a couple (~5) fields the insertion time for > 1000 items goes from ~10s to ~250ms. > > How can I move this forward? > > holger >

July 23, 2015

Re: [Pharo-dev] [Pharo-users] MongoTalk: new place on smalltalkhub

by stepharo

Pay attention to open a bug entry so that we can track the enh and potentially harvest it for 40 and 50. Le 20/5/15 18:27, Norbert Hartl a Ã©crit : > Hmm, yes, it seems localHostName could be implemented better in a way > that works on all platforms. > Instead of using the dedicated primitives > (primHostNameSize/primHostNameResult:), it tries to lookup the > localHostAddress using DNS... > On OSX, primLocalAddress was changed to always return the loopback > address so one avoids ambiguities with multiple interfaces, thus, > 'localhost'. > I've commit a fixed version to the Pharo 4.0 inbox.

July 23, 2015

Contributing to VoyageMongo (improving insertion/updating speed)

by Holger Freyther

Hi, I wanted to benchmark some look-ups with VoyageMongo and tried to insert some dummy objects and it was quite slow (pharo takes 99% CPU and mongo is idle). Using the nice profiler i noticed that VOMongoRepository>>#newVersion calls UUIDGenerator new makeSeed which will take quite long. [UUIDGenerator new makeSeed] bench '96.9 per second.â I am running this on OSX so it will end up in >>#makeUnixSeed which will open /dev/urandom. I needed cryptographic random numbers in another project and used NativeBoost to call into libcrypto of OpenSSL and use the RAND_bytes function to get them. [ (RAND rand: 4) asInteger ] bench '588,000 per second.â The quality of the random numbers should be comparable and now I wonder how I can help integrating such a change. >From my point of view I can either patch Voyage-Mongo-Core to be able to use other random number sources. Or I could add another âseedâ source to the UUIDGenerator. What would be the preferred approach and how to move forward? Is NativeBoost something one could rely on in upcoming Pharo versions? For a simple insertion of a class with a couple (~5) fields the insertion time for 1000 items goes from ~10s to ~250ms. How can I move this forward? holger

July 23, 2015

Re: [Pharo-dev] More woden demos

by stepharo

Excellent. :) Ronie could not send mails when you introduce new features or refactor the code? Because we develop some demos for Thales and we want to make sure that we know when we have to check if they are not working anymore. We have a CI but we should develop some tests to see if it is working. Stef Le 23/7/15 02:39, Ronie Salgado a Ã©crit : > Hello, > > I have been working in making a exporter from blender into a custom > JSON format for Woden with support for animations. Here I made a short > video: https://www.youtube.com/watch?v=HQpaeDQ3TBM&feature=youtu.be > > I also made a new shader management system for Woden, so I took the > opportunity for adding support for parallax occlusion mapping: > https://www.youtube.com/watch?v=fFf7v37j8xg&feature=youtu.be > > Greetings, > Ronie

July 23, 2015

Seeking help to resolve startup error on CI automated build

by Jigyasa Grover

HiI am trying to build my first application in Pharo i.e. an offline text search application namely *searchQuick* which returns the list of file titles from the pre-loaded text files present in the resource folder when a string is searched for. You all are welcome to fork the code (https://github.com/jig08/sQuick_new) , try the infancy application and send in suggestions. Currently, I have put my application up on CI for automated build. I am currently getting a StartUp error on VERSION=development (40) The error is as follows: *The symbolic version '0.1-baseline' is not defined in ConfigurationOfsQuick_new for the current platform. Possible symbolic version values include: #(#bleedingEdge #development #stable)* I have checked ConfigurationOfsQuick_new and the corresponding method reads as follows:*------------------------------------------------------ development: spec <symbolicVersion: #'development'> spec for: #'common' version: '0.1-baseline'*.*------------------------------------------------------* It would be great if my mistake is pin-pointed and I could resolve it. Thanks and Regards Jigyasa

July 23, 2015

[MCGitHubRepository] Seeking better options to access resources folder from github repo

by Jigyasa Grover

Hi I am trying to build my first application in Pharo i.e. an offline text search application namely *searchQuick* which returns the list of file titles from the pre-loaded text files present in the resource folder when a string is searched for. You all are welcome to fork the code (https://github.com/jig08/sQuick_new) , try the infancy application and send in suggestions. So far, to access the resources folder from my github repo, I have been using the following code snippet: *----------------------------------------------------------------------------------------------------------------* *| resourceFolder resourcePath|* *MCGitHubRepository allSubInstancesDo: * * [ :repo |* * repo projectPath = 'jig08/sQuick_new' * * ifTrue: [ resourcePath := repo directory ] * * ]. * *resourceFolder := resourcePath / 'sQuick_new/resources'.* *----------------------------------------------------------------------------------------------------------------* I wanted to know a better option that could be used to access it, without checking all the entries present for a correct match i.e. not use *"**MCGitHubRepository allSubInstancesDo:"* Help Appreciated Thanks Jigyasa

July 23, 2015