- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] New methods for the String class

by HernÃ¡n Morales Durand

2014-02-25 8:49 GMT-03:00 phil(a)highoctane.be <phil(a)highoctane.be>: > Yes but... in Pharo strings are collections and a ton of things can be > done with collections already. > Other languages have more "basic" strings and are cutting them in all ways > possible but not as collections. > > things like detect, inject etc are not really in the repertoire of other > languages. Okay, there are chop, split, ... but these are pretty basic > things. > > Why not have a test suite with a test for demonstrating how to do these > things with the platform we do have? > > e.g. testChomp, testSplit, ...all put in a Tests-Strings-RubyStyle or > Tests-Strings-PythonStyle packages? > > And like there are Seaside-Pharo-XXX packages, why not a String-Ruby-XXXX > thing ? > > Yes, that is a really good idea. HernÃ¡n

Feb. 26, 2014

Re: [Pharo-dev] hashMultiply

by Martin McClure

On 02/25/2014 05:04 PM, Andres Valloud wrote: > >> When going to 64-bit, and with the new ObjectMemory scheme, I guess a >> couple of identity hashing functions will come under scrutiny. I'd think so. I haven't been following discussions of 64-bit design for Pharo. How many bits of identity hash are in each object, and how many bits is a SmallInteger (61, perhaps?) > > IMO it's not clear that SmallInteger>>identityHash should be implemented > that way. Finding a permutation of the small integers that also behaves > like a good quality hash function and evaluates quickly (in > significantly less time and complexity than, say, Bob Jenkins' lookup3) > would be a really interesting research project. I don't know if it's > possible. If no such thing exists, then getting at least some rough > idea of what's the minimum necessary complexity for such hash functions > would be valuable. Andres and I have an ongoing discussion on this topic. :-) I will say this: using SmallInteger>>hashMultiply as SmallInteger>>hash is *far* better than implementing SmallInteger>>hash as ^self. But Andres may be right that something else would be even more desirable. > > The implementation of hashMultiply, yes. Note however that > SmallInteger>>hash is ^self. >> >> So, 16384 * is the same as bitShift: 14 and it looks like done once, >> which may be better. And #bitShift *should* be faster. But test this. In VW, multiplication is quite a lot faster than bitShift: for some reason. Unless Andres has fixed this recently. :-) > > Keep in mind that the speed at which hash values are calculated is only > part of the story. If the hash function quality is not great, or the > hashed collection implementation is not efficient and induces collisions > or other extra work, improving the efficiency of the hash functions > won't do much. I think it's mentioned in the hash book (I'd have to > check), but once I made a hash function 5x times slower to get better > quality and the result was that a report that was taking 30 minutes took > 90 seconds instead (and hashing was gone from the profiler output). I wholeheartedly agree with this! The quality of the hash function is much more important than the speed. Also, remember that the #hash method is only *half* of the hash function. The other half is in the hashed collection (usually \\ tableSize). Those two must coordinate well to form a good hash function or you can get poor results. And the collections that are theoretically fastest with a good hash function (open addressing with linear probing) seem to be the most sensitive to the quality of the hash function. > > If this is a primitive instead, then you can also avoid the overflow > into large integers and do the math with (basically) > > mov eax, smallInteger > shr eax, numberOfTagBits > mul eax, 1664525 "the multiplication that throws out the high bits" > shl eax, 4 "throw out bits 29-32" > shr eax, 4 > lea eax, [eax * 2^numberOfTagBits + smallIntegerTagBits] Right. In Pharo, you might use NativeBoost to implement this, as you noted. Regards, -Martin

Feb. 26, 2014

Re: [Pharo-dev] easier start with Phobos

by Alexandre Bergel

I do not have much time to play with this, but can you give a one-click ready-to-use example? Alexandre On Feb 25, 2014, at 8:07 PM, Pavel Krivanek <pavel.krivanek(a)gmail.com> wrote: > If you want do develop applications in Pharo with native user interface, you can use Phobos framework based on Mozilla XULRunner client. However to start with it required to download a bundle for your platform and make some modifications in configuration files. > > To make the entry point easier I created a configuration of Phobos that you can download directly in the Configuration Browser in Pharo 3.0. During the download it will ask you whether you want to run PhobosDemo application. Answer yes and it will download required files for you including XULRunner for your platform into the working directory (on Linux it expects xulrunner is already preinstalled). Mainly on Windows make sure that Pharo is enabled on your firewall because it needs to start a server (default port is 46528). Then it should show a window with rich demo of Phobos capabilities. You can play with sources of the PhobosDemo package. > > To start to write your own application firstly stop the demo: > > PhobosDemo allInstancesDo: #stop. > > Then make own subclass of PhobosApplication, create own subclass of PhobosComponent and specify it in your application class as mainClass. Star the server. > > app := MyPhobosApp new. > app start. > > ..and restart the client. It is useful to run the client with the argument -jsconsole to show errors on the JavaScript side. On MacOS you need to modify the file PhobosDemo.app/Contents/MacOS/start.sh > > Please let me know if something will went wrong. > > Cheers, > -- Pavel -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

Feb. 26, 2014

Re: [Pharo-dev] hashMultiply

by Andres Valloud

Hello... On 2/25/14 1:17 , phil(a)highoctane.be wrote: > I am currently reading through the Hashing in Smalltalk book > (http://www.lulu.com/shop/andres-valloud/hashing-in-smalltalk-theory-and-pra…) > and, my head hurting notwithstanding, there are indeed a ton of gems in > this system. As he mentions, doing the exercises brings a lot of extra :-) :) thank you. > When going to 64-bit, and with the new ObjectMemory scheme, I guess a > couple of identity hashing functions will come under scrutiny. > > e.g. > > SmallInteger>>hashMultiply > | low | > > low := self bitAnd: 16383. > ^(16r260D * low + ((16r260D * (self bitShift: -14) + (16r0065 * low) > bitAnd: 16383) * 16384)) > bitAnd: 16r0FFFFFFF > > > which will need some more bits. IMO it's not clear that SmallInteger>>identityHash should be implemented that way. Finding a permutation of the small integers that also behaves like a good quality hash function and evaluates quickly (in significantly less time and complexity than, say, Bob Jenkins' lookup3) would be a really interesting research project. I don't know if it's possible. If no such thing exists, then getting at least some rough idea of what's the minimum necessary complexity for such hash functions would be valuable. Looking at hashMultiply as a non-identity hash function, one would start having problems when significantly more than 2^28 objects are stored in a single hashed collection. 2^28 objects with e.g. 12 bytes per header and a minimum of one instance variable (so the hash value isn't a instance-constant) stored in a hashed collection requires more than 4gb, so that is clearly a 64 bit image problem. In 64 bits, 2^28 objects with e.g. 16 bytes per header and a minimum of one instance variable each is already 6gb, and a minimum of 8gb with the hashed collection itself. Because of these figures, I'd think improving the implementation of hashed collections takes priority over adding more non-identity hash function bits (as long as the existing hash values are of good quality). Did you look at the Hash Analysis Tool I wrote? It's in the Cincom public Store repository. It comes in two bundles: Hash Analysis Tool, and Hash Analysis Tool - Extensions. With everything loaded, the tool comes with 300+ hash functions and 100+ data sets out of the box. The code is MIT. > I had a look at how it was done in VisualWorks; The implementation of hashMultiply, yes. Note however that SmallInteger>>hash is ^self. > hashMultiply > "Multiply the receiver by 16r0019660D mod 2^28 > without using large integer arithmetic for speed. > The constant is a generator of the multiplicative > subgroup of Z_2^30, see Knuth's TAOCP vol 2." > <primitive: 1747> > | low14Bits | > low14Bits := self bitAnd: 16r3FFF. > ^16384 > * (16r260D * (self bitShift: -14) + (16r0065 * low14Bits) bitAnd: 16r3FFF) > + (16r260D * low14Bits) bitAnd: 16rFFFFFFF > > The hashing book version has: > > multiplication > "Computes self times 1664525 mod 2^38 while avoiding overflow into a > large integer by making the multiplication into two 14 bits chunks. Do > not use any division or modulo operation." > | lowBits highBits| > > lowBits := self bitAnd: 16r3FFF. > highBits := self bitShift: -14. > ^(lowBits * 16r260D) > + (((lowBits * 16r0065) bitAnd: 16r3FFF) bitShift: 14) > + (((highBits * 16r260D) bitAnd: 16r3FFF) bitShift: 14) > bitAnd: 16rFFFFFFF > > So, 16384 * is the same as bitShift: 14 and it looks like done once, > which may be better. It should be a primitive (or otherwise optimized somehow). At some point though that hash function was implemented for e.g. ByteArray in Squeak, I thought at that point the multiplication step was also implemented as a primitive? > Also VW marks it as a primitive, which Pharo does not. In VW it is also a translated primitive, i.e. it's executed directly in the JIT without calling C. Keep in mind that the speed at which hash values are calculated is only part of the story. If the hash function quality is not great, or the hashed collection implementation is not efficient and induces collisions or other extra work, improving the efficiency of the hash functions won't do much. I think it's mentioned in the hash book (I'd have to check), but once I made a hash function 5x times slower to get better quality and the result was that a report that was taking 30 minutes took 90 seconds instead (and hashing was gone from the profiler output). > Would we gain > some speed doing that? hashMultiply is used a lof for identity hashes. > > Bytecode has quite some work to do: > > 37 <70> self > 38 <20> pushConstant: 16383 > 39 <BE> send: bitAnd: > 40 <68> popIntoTemp: 0 > 41 <21> pushConstant: 9741 > 42 <10> pushTemp: 0 > 43 <B8> send: * > 44 <21> pushConstant: 9741 > 45 <70> self > 46 <22> pushConstant: -14 > 47 <BC> send: bitShift: > 48 <B8> send: * > 49 <23> pushConstant: 101 > 50 <10> pushTemp: 0 > 51 <B8> send: * > 52 <B0> send: + > 53 <20> pushConstant: 16383 > 54 <BE> send: bitAnd: > 55 <24> pushConstant: 16384 > 56 <B8> send: * > 57 <B0> send: + > 58 <25> pushConstant: 268435455 > 59 <BE> send: bitAnd: > 60 <7C> returnTop If this is a primitive instead, then you can also avoid the overflow into large integers and do the math with (basically) mov eax, smallInteger shr eax, numberOfTagBits mul eax, 1664525 "the multiplication that throws out the high bits" shl eax, 4 "throw out bits 29-32" shr eax, 4 lea eax, [eax * 2^numberOfTagBits + smallIntegerTagBits] Please excuse trivial omissions in the above, it's written only for the sake of illustration (e.g. it looks like the 3 last instructions can be combined into two... lea followed by shr). Also, did you see the latency of integer multiplication instructions in modern x86 processors?... > I ran some experiments timing things. > > It looks like that replacing 16384 * by bitShift:14 leads to a small > gain, bitShift (primitive 17) being faster than * (primitive 9) Keep in mind those operations still have to check for overflow into large integers. In this case, large integers are not necessary. Andres. > The bytecode is identical, except send: bitShift instead of send: * > > multiplication3 > | low | > > low := self bitAnd: 16383. > ^(16r260D * low + ((16r260D * (self bitShift: -14) + (16r0065 * low) > bitAnd: 16383) bitShift: 14)) > bitAnd: 16r0FFFFFFF > > > > [500000 timesRepeat: [ 15000 hashMultiply ]] timeToRun 12 > [500000 timesRepeat: [ 15000 multiplication ]] timeToRun 41 (worse) > [500000 timesRepeat: [ 15000 multiplication3 ]] timeToRun 10 (better) > > It looks like correct for SmallInteger minVal to: SmallInteger maxVal > > Now, VW gives: [500000 timesRepeat: [ 15000 hashMultiply ]] timeToRun > 1.149 milliseconds > > Definitely worth investigating the primitive thing, or some NB Asm as > this is used about everywhere (Collections etc). > > Toughts? > > Phil

Feb. 26, 2014

easier start with Phobos

by Pavel Krivanek

If you want do develop applications in Pharo with native user interface, you can use Phobos framework based on Mozilla XULRunner client. However to start with it required to download a bundle for your platform and make some modifications in configuration files. To make the entry point easier I created a configuration of Phobos that you can download directly in the Configuration Browser in Pharo 3.0. During the download it will ask you whether you want to run PhobosDemo application. Answer yes and it will download required files for you including XULRunnerfor your platform into the working directory (on Linux it expects xulrunner is already preinstalled). Mainly on Windows make sure that Pharois enabled on your firewall because it needs to start a server (default port is 46528). Then it should show a window with rich demo of Phobos capabilities. You can play with sources of the PhobosDemo package. To start to write your own application firstly stop the demo: PhobosDemo allInstancesDo: #stop. Then make own subclass of PhobosApplication, create own subclass of PhobosComponent and specify it in your application class as mainClass. Star the server. app := MyPhobosApp new. app start. ..and restart the client. It is useful to run the client with the argument -jsconsole to show errors on the JavaScript side. On MacOS you need to modify the file PhobosDemo.app/Contents/MacOS/start.sh Please let me know if something will went wrong. Cheers, -- Pavel

Feb. 26, 2014

Re: [Pharo-dev] Pharo 3.0 "Sorry that name is already in use"

by Martin McClure

On 02/25/2014 12:27 PM, Pharo4Stef wrote: > probably that a changeset was created and it tried to reload a cs with the same name. Ah, so probably the changeset that had been created but not completed when the first update was interrupted, I'm guessing? That makes sense. Not crucial, of course, but it would be nice to avoid presenting the user with such vague errors -- either suppress the error or present enough information to understand it. Regards, -Martin > > On 25 Feb 2014, at 20:39, Martin McClure <martin(a)hand2mouse.com> wrote: > >> This is a bug report of sorts. >> >> I had a Pharo 3.0 image that was last updated about a month ago. I >> started a Software Update, and it was pulling in updates, when I needed >> to sleep my (Linux) laptop. When I re-opened the laptop, there was a >> walkback there, which I didn't find surprising since I'd interrupted the >> update's network connection by sleeping the laptop. It was some kind of >> "can't load" error; I don't remember the details. >> >> Anyway, I abandoned that, and chose "Software update" again. Almost >> immediately I got a little info box in the lower left that said "Sorry >> that name is already in use". This box stayed there until the update was >> complete. There was no other context given, so I have no idea what it >> was complaining about. >>

Feb. 25, 2014

Re: [Pharo-dev] Pharo contribution CI: Email on failed build

by Norbert Hartl

Am 25.02.2014 um 21:19 schrieb Esteban A. Maringolo <emaringolo(a)gmail.com>: > 2014-02-25 12:53 GMT-03:00 Norbert Hartl <norbert(a)hartl.name>: >> I was logged in but as this is a foreign service I forgot that I can configure it myself and assumed the opposite :) Just another embarrassing moment. > > Embarrasing for you or not, it helped me to learn about such feature > in Jenkins. ;) > The embarrassing moment wasnât about the knowledge of that feature because I use it on my own jenkins server. It was about not seeing that configure link :) Anyway, Iâm glad someone learned something. Norbert

Feb. 25, 2014

Re: [Pharo-dev] Pharo 3.0 "Sorry that name is already in use"

by Pharo4Stef

probably that a changeset was created and it tried to reload a cs with the same name. On 25 Feb 2014, at 20:39, Martin McClure <martin(a)hand2mouse.com> wrote: > This is a bug report of sorts. > > I had a Pharo 3.0 image that was last updated about a month ago. I > started a Software Update, and it was pulling in updates, when I needed > to sleep my (Linux) laptop. When I re-opened the laptop, there was a > walkback there, which I didn't find surprising since I'd interrupted the > update's network connection by sleeping the laptop. It was some kind of > "can't load" error; I don't remember the details. > > Anyway, I abandoned that, and chose "Software update" again. Almost > immediately I got a little info box in the lower left that said "Sorry > that name is already in use". This box stayed there until the update was > complete. There was no other context given, so I have no idea what it > was complaining about. > > Regards, > > -Martin >

Feb. 25, 2014

Re: [Pharo-dev] Pharo contribution CI: Email on failed build

by Esteban A. Maringolo

2014-02-25 12:53 GMT-03:00 Norbert Hartl <norbert(a)hartl.name>: > I was logged in but as this is a foreign service I forgot that I can configure it myself and assumed the opposite :) Just another embarrassing moment. Embarrasing for you or not, it helped me to learn about such feature in Jenkins. ;) Thanks! Esteban A. Maringolo

Feb. 25, 2014

Re: [Pharo-dev] image traffic

by Pharo4Stef

On 25 Feb 2014, at 17:48, Christophe Demarey <Christophe.Demarey(a)inria.fr> wrote: > > Le 25 fÃ©vr. 2014 Ã 17:37, Norbert Hartl a Ã©crit : > >> If you look at the hosts list than you can see that 75% of the overall traffic is generated by cloud-gw.inria.fr. The CI jobs download the vm with curl. >> >> What brings me to the next point. The âHostsâ section in those states might break a data protection law. I donât know how they are in france but in germany this would be illegal. We are not allowed to store IP addresses for a longer period. And making them public is really asking for trouble. > > I think we have some equivalent laws. > For another project, we had to anonymize awstats results. > Maybe raw statistics should be kept private and only use numbers for marketing. +1 > > Christophe. >

Feb. 25, 2014