May 2017 - Pharo-dev - lists.pharo.org

[pharo-project/pharo-core] e64cb1: 60474

by GitHub

Branch: refs/heads/6.0 Home: https://github.com/pharo-project/pharo-core Commit: e64cb12453542258736da30c4fbf0cf9042c92e6 https://github.com/pharo-project/pharo-core/commit/e64cb12453542258736da30c… Author: Jenkins Build Server <board(a)pharo-project.org> Date: 2017-05-02 (Tue, 02 May 2017) Changed paths: M Collections-Abstract.package/SequenceableCollection.class/instance/enumerating/groupsOf_atATimeCollect_.st M Collections-Arithmetic.package/extension/Collection/instance/average.st M Kernel.package/SmallInteger.class/instance/printing/printString.st R ScriptLoader60.package/ScriptLoader.class/instance/pharo - scripts/script60473.st A ScriptLoader60.package/ScriptLoader.class/instance/pharo - scripts/script60474.st R ScriptLoader60.package/ScriptLoader.class/instance/pharo - updates/update60473.st A ScriptLoader60.package/ScriptLoader.class/instance/pharo - updates/update60474.st M ScriptLoader60.package/ScriptLoader.class/instance/public/commentForCurrentUpdate.st A Shout.package/SHTextStylerST80.class/class/accessing/styleTable.st A Shout.package/SHTextStylerST80.class/class/accessing/styleTable_.st A Shout.package/SHTextStylerST80.class/class/attributes/attributeArrayForColor_emphasis_font_.st A Shout.package/SHTextStylerST80.class/class/attributes/attributesFor_pixelHeight_.st A Shout.package/SHTextStylerST80.class/class/attributes/initialTextAttributesForPixelHeight_.st A Shout.package/SHTextStylerST80.class/class/attributes/textAttributesByPixelHeight.st R Shout.package/SHTextStylerST80.class/class/style table/attributeArrayForColor_emphasis_font_.st R Shout.package/SHTextStylerST80.class/class/style table/attributesFor_pixelHeight_.st R Shout.package/SHTextStylerST80.class/class/style table/blueStyleTable.st R Shout.package/SHTextStylerST80.class/class/style table/darkStyleTable.st R Shout.package/SHTextStylerST80.class/class/style table/defaultStyleTable.st R Shout.package/SHTextStylerST80.class/class/style table/initialTextAttributesForPixelHeight_.st R Shout.package/SHTextStylerST80.class/class/style table/solarizedStyleTable.st R Shout.package/SHTextStylerST80.class/class/style table/styleTable.st R Shout.package/SHTextStylerST80.class/class/style table/styleTable_.st R Shout.package/SHTextStylerST80.class/class/style table/tangoStyleTable.st R Shout.package/SHTextStylerST80.class/class/style table/textAttributesByPixelHeight.st R Shout.package/SHTextStylerST80.class/class/style table/vintageStyleTable.st A Shout.package/SHTextStylerST80.class/class/styles/blueStyleTable.st A Shout.package/SHTextStylerST80.class/class/styles/darkStyleTable.st A Shout.package/SHTextStylerST80.class/class/styles/defaultStyleTable.st A Shout.package/SHTextStylerST80.class/class/styles/solarizedStyleTable.st A Shout.package/SHTextStylerST80.class/class/styles/tangoStyleTable.st A Shout.package/SHTextStylerST80.class/class/styles/vintageStyleTable.st A UnifiedFFI.package/FFIMethodRegistry.class/class/system startup/resetAll.st M UnifiedFFI.package/FFIMethodRegistry.class/class/system startup/shutDown_.st A UnifiedFFI.package/FFIMethodRegistry.class/class/system startup/startUp_.st Log Message: ----------- 60474 19982 Better method comments https://pharo.fogbugz.com/f/cases/19982 19925 Document that SmallInteger>>printString breaks the #printOn: rule https://pharo.fogbugz.com/f/cases/19925 19999 Improve SHTextStylerST80 class side protocols https://pharo.fogbugz.com/f/cases/19999 19990 UFFI not reset method cache at image startup https://pharo.fogbugz.com/f/cases/19990 http://files.pharo.org/image/60/60474.zip

May 2, 2017

Estebanâs list of blocking issues for release Pharo 6.0 (updated to May 02)

by Esteban Lorenzano

Hello, This are the things I believe still needs to be solved to be able to release Pharo 6.0: VM: ==== - libgit2: Still not there. I decided to migrate to libgit2 0.25.1 (which we need to do it anyway) because that solves automagically all compilation problems. Of course, that needs work on the image :( Image: ===== - NEW: migrate iceberg to 0.25.1 (not as much work as you may think). - iceberg still needs to go inside the image, Iâm running to release version 0.4 (maybe this week?) and then we can integrate. FIXED SINCE LAST REPORT: ==== - vm: make all tests pass. This is working now in 32bits which was what we attempt to accomplish. - vm: there was a problem with latest VM and primitiveFindSubstring that needs to be fixed. I adapted our compilation to use OSVM primitive instead ours. This makes all tests passes, but it can happen that something needs still to be fixed on VM. Next: ---- - move latest vm to stable. - integrate changes on libgit2 I think we will be ready to release around next week (or course, there is still a long list of things to accomplish before). RELEASE CHECKLIST: ---- - Prepare release notes Image: - add welcome browser - generate PharoV60.sources - move SystemVersion - prepare new ScriptLoader70 package (in theory, this will not needed anymore but better to have it as a backup :P) VM: - Promote new stable WebSite: - Update index - Update download page Infrastructure: - ZeroConf: Open new 7.0 development - SmalltalkCI: Open new 7.0 development - Create platform stable versions - move all CI projects to Pharo 7.0 Pub: - announce post - announce mail - post to YC, Reddit, etc. cheers! Esteban

May 2, 2017

Re: [Pharo-dev] Smalltalk Internet Browser

by askoh

Craig: Thanks for the reply. I think I see your point. WebAssembly lives in the browser and is supposed to be almost as fast as native Assembly. So the Smalltalk VMs, which are mostly C, can be compiled to WebAssembly to run inside browsers everywhere and be fast too. Wow. Is WebAssembly released yet? When can we expect such a VM running a standard Smalltalk? What are the pros and cons compared to native apps? All the best, Aik-Siong Koh Craig Latta wrote > Hi Aik-Siong-- > >> I want to go to a web page, select a body of text and have Smalltalk >> code analyze the text in some way and return the answer in the >> browser or in the Smalltalk app or file output. There should be >> two-way communication between app and browser. I would like to do >> live debugging too. > > Ah, it sounds like you want something that would work with > arbitrary websites you happen to visit. I would do that by making a > SqueakJS-powered extension for an existing web browser (e.g., a Chrome > extension). The JavaScript bridge in SqueakJS gives you full two-way > livecoding access to the host browser's JS engine, and I'd probably use > it for the Smalltalk app's UI (driving some JS front-end library, like > React). And there's no reason the object memory can't be Pharo. > > At this point, I think it makes much more sense to run Smalltalk in > a web browser rather than building a Smalltalk web browser. I find > SqueakJS easily fast enough for useful things, like the Cog app > streaming Eliot mentioned. WebAssembly will enable web browsers to run > Cog directly (although subject to web security constraints), making > JavaScript completely optional. > > Yes, the web platform is full of astounding accidental complexity, > as Stephen pointed out. Still, there's a lot of great work, community > energy, and reach in which Smalltalk can now participate, and I'm > enjoying it. (Thanks again, Bert, Dan and all!) > > > -C > > -- > Craig Latta > Black Page Digital > Amsterdam :: San Francisco > craig@ > +31 6 2757 7177 (SMS ok) > + 1 415 287 3547 (no SMS) -- View this message in context: http://forum.world.st/Smalltalk-Internet-Browser-tp4944879p4945093.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.

May 2, 2017

Re: [Pharo-dev] First commit

by Evan Donahue

Thanks. I have resubmitted and it seems to have worked. If anyone wants to let me know if it looks like I did everything right, I can hopefully move on to tackling other things. Thanks, Evan -- View this message in context: http://forum.world.st/First-commit-tp4944984p4945091.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.

May 2, 2017

Re: [Pharo-dev] [Vm-dev] [squeak-dev] Byte & String collection hash performance; a modest proposal for change.

by Martin McClure

On 05/01/2017 11:55 AM, Eliot Miranda wrote: > > > On Mon, May 1, 2017 at 10:37 AM, Tony Garnock-Jones <tonyg(a)ccs.neu.edu > <mailto:tonyg@ccs.neu.edu>> wrote: > > On 5/1/17 1:26 PM, Levente Uzonyi wrote: > > I presume that a general purpose in-image solution would be more > > complex. String already has too many subclasses (6 in Squeak), while at > > the same time other kind of new subclasses would be welcome too, e.g. > > Strings with 2-byte characters. > > Since these properties are orthogonal, there would be many new > > subclasses to cover all cases. > > A classic motivating case for Traits, right? > > > Alas no. The problem with String's subclasses is that they're binary > objects. They have no inst vars in which one can cache a hash. They're > juts a flat vector of bytes. If one added a trait to them the system > would fail because there's no way to add an inst var to a binary object > in the current Smalltalk object representation. hence Levente's very > clever idea of hiding the hash in a hidden header word. > That is a clever idea. I wouldn't actually *implement* that solution though. It would add considerable complexity, considering that strings are mutable and their hashes can change. And the problem that this would solve is hypothetical isn't it? At least I haven't seen anyone pointing to any actual applications that need to hash very large strings. Regards, -Martin

May 2, 2017

Re: [Pharo-dev] Smalltalk Internet Browser

by Craig Latta

Hi Aik-Siong-- > I want to go to a web page, select a body of text and have Smalltalk > code analyze the text in some way and return the answer in the > browser or in the Smalltalk app or file output. There should be > two-way communication between app and browser. I would like to do > live debugging too. Ah, it sounds like you want something that would work with arbitrary websites you happen to visit. I would do that by making a SqueakJS-powered extension for an existing web browser (e.g., a Chrome extension). The JavaScript bridge in SqueakJS gives you full two-way livecoding access to the host browser's JS engine, and I'd probably use it for the Smalltalk app's UI (driving some JS front-end library, like React). And there's no reason the object memory can't be Pharo. At this point, I think it makes much more sense to run Smalltalk in a web browser rather than building a Smalltalk web browser. I find SqueakJS easily fast enough for useful things, like the Cog app streaming Eliot mentioned. WebAssembly will enable web browsers to run Cog directly (although subject to web security constraints), making JavaScript completely optional. Yes, the web platform is full of astounding accidental complexity, as Stephen pointed out. Still, there's a lot of great work, community energy, and reach in which Smalltalk can now participate, and I'm enjoying it. (Thanks again, Bert, Dan and all!) -C -- Craig Latta Black Page Digital Amsterdam :: San Francisco craig(a)blackpagedigital.com +31 6 2757 7177 (SMS ok) + 1 415 287 3547 (no SMS)

May 1, 2017

Re: [Pharo-dev] Meanwhile in FogBugz: issue 20000

by Tudor Girba

+1 Doru > On Apr 28, 2017, at 12:47 PM, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: > > Tx torsten for these kind words. > > You got quoted :) > https://pharoweekly.wordpress.com/2017/04/28/20000-issues-mark-in-fogbugz/ > > On Thu, Apr 27, 2017 at 5:49 PM, Torsten Bergmann <astares(a)gmx.de> wrote: > Hi, > > our community today hit the 20000 issues mark in FogBugz. > > https://pharo.fogbugz.com/f/cases/20000 > > And a good change to once again say THANK YOU, especially to the ones who care most on > > - updating the bug tracker > - fixing issues > - reviewing issues > - integrating issues > - doing also the boring tasks in caring about the bugtracker > - pushing Pharo in the right direction > > Just wow and impressive! Pharo 6 will be another nice milestone in a lively open source > Smalltalk system that we all wanted to see becoming a reality! > > Thx > T. > > BTW: > The issue 20000 is about clickable URLs in Help Browser. Currently we copy them to clipboard - as we do > not have web browser support in the default image. I guess it is too late to push this package into the > image due to the freeze. > > So this slices nonetheless tries to improve the situation by checking if the unified WebBrowser support package > (which is available since a year [1]) is loaded or not. If it is loaded it opens a real web browser. > > So when Pharo 6 is released and someone may lament about not being able for the URLs to really work > we can point him to load the "WebBrowser" package and he should be fine. > > [1] http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/2016-April/12072… > > -- www.tudorgirba.com www.feenk.com "Presenting is storytelling."

May 1, 2017

Re: [Pharo-dev] [squeak-dev] [Vm-dev] Byte & String collection hash performance; a modest proposal for change.

by Eliot Miranda

On Mon, May 1, 2017 at 10:37 AM, Tony Garnock-Jones <tonyg(a)ccs.neu.edu> wrote: > On 5/1/17 1:26 PM, Levente Uzonyi wrote: > > I presume that a general purpose in-image solution would be more > > complex. String already has too many subclasses (6 in Squeak), while at > > the same time other kind of new subclasses would be welcome too, e.g. > > Strings with 2-byte characters. > > Since these properties are orthogonal, there would be many new > > subclasses to cover all cases. > > A classic motivating case for Traits, right? > Alas no. The problem with String's subclasses is that they're binary objects. They have no inst vars in which one can cache a hash. They're juts a flat vector of bytes. If one added a trait to them the system would fail because there's no way to add an inst var to a binary object in the current Smalltalk object representation. hence Levente's very clever idea of hiding the hash in a hidden header word. > Tony > > -- _,,,^..^,,,_ best, Eliot

May 1, 2017

Re: [Pharo-dev] [Vm-dev] [squeak-dev] Byte & String collection hash performance; a modest proposal for change.

by David T. Lewis

Does it need to be done in the VM? Why not make a class LargeString with instance variables aString and myCalculatedHashValueForTheString. That way you can cache the hash value and calculate it any way you want. I know vey little about hashing, just wondering if this kind of thing can be handled more easily in the image. Dave > > On Mon, 1 May 2017, Levente Uzonyi wrote: > >> Well, I had started to write a reply, but I had to postpone it. >> I mostly agree with your suggestions. >> >> One thing that can be done about large strings is to cache the >> calculated >> hash value in the larger strings. Currently the object representation >> changes when the string contains 255 or more characters. In that case an >> additional 64 bit field is added to the object header to store its >> length. >> If we were to use the upper 28+1 bits of that field to cache the hash, >> there would still be 35-bits to encode length, which would be enough to >> represent strings up to 8 GiB. > > Well, we can keep the whole range minus one bit, but then we can't store > the hash for strings larger than 8 GiB. > > Levente > >> But this would require further VM changes (e.g. at:put: would have to >> flush the cache). >> >> Levente >> >> On Mon, 1 May 2017, Martin McClure wrote: >> >>> I see no replies to this on any of the three lists it was sent to, so I >>> guess I'll chime in. >>> >>> tl;dr: >>> Making a primitive for #hashMultiply, probably a good idea, in some >>> form, since doing it in Smalltalk is awkward. >>> Only hashing every N-th character of large strings, probably a very bad >>> idea. Performance might well get worse, the complexity does not seem >>> justified, and it would open a sizeable security hole. >>> >>> More verbiage below for those interested. >>> >>> Regards, >>> -Martin >>> >>> On 04/18/2017 07:09 PM, Eliot Miranda wrote: >>>> Hi All, >>>> >>>> the hash algorithm used for ByteString in Squeak and Pharo is good >>>> for "small" strings and overkill for large strings. >>> >>> Why do you say it's overkill for large strings? Are there applications >>> with large strings that are being negatively impacted by the current >>> algorithm? Which ones, and impacted how? >>> >>>> It is important in many applications to get well distributed string >>>> hashes, especially over the range of strings that constitute things >>>> like method names, URLs, etc. Consequently, the current algorithm >>>> includes every character in a string. This works very well for >>>> "small" strings and results in very slow hashes (and hence long >>>> latencies, because the hash is an uninterruptible primitive) for large >>>> strings, where large may be several megabytes. >>> >>> A simple solution for the uninterruptable primitive is to not make it a >>> primitive. Make #hashMultiply a primitive (since this particular kind >>> of >>> numeric modulo computation is really painful in Smalltalk), and do the >>> rest in a loop in Smalltalk. It sounds like you've done the >>> #hashMultiply primitive already. >>> >>> If the overhead of calling a primitive for each character proves to be >>> too much, even with the faster primitive calling methodologies you >>> talked about in the "Cog Primitive Performance" thread on the Vm-dev >>> list, a more complex primitive could take a range of bytes, so large >>> strings would be done in batches, solving the latency problem. >>> >>>> >>>> Let's look at the basic hash algorithm. >>> [...] >>>> >>>> In looking at this I've added a primitive for hashMultiply; primitive >>>> #159 implements precisely self * 1664525 bitAnd: 16r0FFFFFFF for >>>> SmallInteger and LargePositiveInteger receivers, as fast as possible >>>> in the Cog JIT. With this machinery in place it's instructive to >>>> compare the cost of the primitive against the non-primitive Smalltalk >>>> code. >>>> >>>> First let me introduce a set of replacement hash functions, newHashN. >>>> These hash all characters in strings up to a certain size, and then no >>>> more than that number for larger strings. Here are newHash64 and >>>> newHash2048, which use pure Smalltalk, including an inlined >>>> hashMultiply written to avoid SmallInteger overflow. Also measured >>>> are the obvious variants newHash128, newHash256, newHash512 & >>>> mewHash1024. >>>> >>>> >>> [...] >>>> So the idea here is to step through the string by 1 for strings sizes >>>> up to N - 1, and by greater than 1 for strings of size >= N, limiting >>>> the maximum number of characters sampled to between N // 2 and N - 1. >>> >>> The history of computing is littered with the bones of those who have >>> tried this kind of thing. It doesn't end well. Yes, you get a faster >>> hash function. And then you find, for sets of data that you or your >>> users actually want to use, that you get collisions like crazy, and >>> much >>> worse overall performance than you started with. >>> >>> Sure, it works OK for the sets of data that the designer *tested*, and >>> probably for the sets of data that they *anticipated*. But real-world >>> data is tricky. It includes data sets where the characters that differ >>> are the ones that the hash thinks are unimportant, and there goes your >>> performance, by orders of magnitude. For instance, early versions of >>> Java used a limited number of characters to hash strings. One of the >>> biggest compatibility-breaking changes they were forced to make in >>> later >>> Java versions was to consider *all* characters in hashing. It turned >>> out >>> that it was very common to hash URLs, and many distinct URLs had most >>> of >>> their characters in common. >>> >>> And you don't always get to choose your data -- sometimes you have an >>> opponent who is actively looking to create collisions as a >>> denial-of-service attack. There was a fair-sized kerfluffle about this >>> a >>> few years ago -- most web scripting languages made it too easy to mount >>> this kind of attack. >>> >> https://arstechnica.com/business/2011/12/huge-portions-of-web-vulnerable-to… >>> "...an attacker can degenerate the hash table by sending lots of >>> colliding keys. ...making it possible to exhaust hours of CPU time >>> using >>> a single HTTP request." >>> To guard against this kind of attack you need a randomized element in >>> your hash (not a bad idea for Smalltalk, actually, and pretty easy -- >>> mixing in the identity hash of the collection might be sufficient) or a >>> cryptographic hash (not worth the computational expense for most >>> purposes). However, even adding a randomized element would not prevent >>> this kind of attack if you predictably completely ignore some >>> characters >>> of the input string. That just makes it *so* easy to generate data that >>> validates, and is not equal, but causes collisions. >>> >>> So really, for general-purpose use (i.e. what's built into the >>> language) >>> hash *every* character of *all* strings. If someone finds that this is >>> a >>> performance problem in a real-world situation, it can be addressed in >>> an >>> application-specific way. >> >> >

May 1, 2017

Re: [Pharo-dev] [Vm-dev] Byte & String collection hash performance; a modest proposal for change.

by Martin McClure

On 05/01/2017 02:24 AM, Andrei Chis wrote: > > performance, by orders of magnitude. For instance, early versions of > Java used a limited number of characters to hash strings. One of the > biggest compatibility-breaking changes they were forced to make in > later > Java versions was to consider *all* characters in hashing. It > turned out > that it was very common to hash URLs, and many distinct URLs had > most of > their characters in common. > > > Was curious about this and found the actual issues [1] if anybody else > is interested. > > [1] http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4045622 > Thanks for digging this up, Andrei! I was working from memory, and had forgotten some of the details. Yeah, they had to change the language spec, which was a Big Deal at the time. (and Josh Bloch and Guy Steele, no less!) Regards, -Martin

May 1, 2017