- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Smalltalk Internet Browser

by Craig Latta

Hi Aik-Siong-- > I want to go to a web page, select a body of text and have Smalltalk > code analyze the text in some way and return the answer in the > browser or in the Smalltalk app or file output. There should be > two-way communication between app and browser. I would like to do > live debugging too. Ah, it sounds like you want something that would work with arbitrary websites you happen to visit. I would do that by making a SqueakJS-powered extension for an existing web browser (e.g., a Chrome extension). The JavaScript bridge in SqueakJS gives you full two-way livecoding access to the host browser's JS engine, and I'd probably use it for the Smalltalk app's UI (driving some JS front-end library, like React). And there's no reason the object memory can't be Pharo. At this point, I think it makes much more sense to run Smalltalk in a web browser rather than building a Smalltalk web browser. I find SqueakJS easily fast enough for useful things, like the Cog app streaming Eliot mentioned. WebAssembly will enable web browsers to run Cog directly (although subject to web security constraints), making JavaScript completely optional. Yes, the web platform is full of astounding accidental complexity, as Stephen pointed out. Still, there's a lot of great work, community energy, and reach in which Smalltalk can now participate, and I'm enjoying it. (Thanks again, Bert, Dan and all!) -C -- Craig Latta Black Page Digital Amsterdam :: San Francisco craig(a)blackpagedigital.com +31 6 2757 7177 (SMS ok) + 1 415 287 3547 (no SMS)

May 1, 2017

Re: [Pharo-dev] Meanwhile in FogBugz: issue 20000

by Tudor Girba

+1 Doru > On Apr 28, 2017, at 12:47 PM, Stephane Ducasse <stepharo.self(a)gmail.com> wrote: > > Tx torsten for these kind words. > > You got quoted :) > https://pharoweekly.wordpress.com/2017/04/28/20000-issues-mark-in-fogbugz/ > > On Thu, Apr 27, 2017 at 5:49 PM, Torsten Bergmann <astares(a)gmx.de> wrote: > Hi, > > our community today hit the 20000 issues mark in FogBugz. > > https://pharo.fogbugz.com/f/cases/20000 > > And a good change to once again say THANK YOU, especially to the ones who care most on > > - updating the bug tracker > - fixing issues > - reviewing issues > - integrating issues > - doing also the boring tasks in caring about the bugtracker > - pushing Pharo in the right direction > > Just wow and impressive! Pharo 6 will be another nice milestone in a lively open source > Smalltalk system that we all wanted to see becoming a reality! > > Thx > T. > > BTW: > The issue 20000 is about clickable URLs in Help Browser. Currently we copy them to clipboard - as we do > not have web browser support in the default image. I guess it is too late to push this package into the > image due to the freeze. > > So this slices nonetheless tries to improve the situation by checking if the unified WebBrowser support package > (which is available since a year [1]) is loaded or not. If it is loaded it opens a real web browser. > > So when Pharo 6 is released and someone may lament about not being able for the URLs to really work > we can point him to load the "WebBrowser" package and he should be fine. > > [1] http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/2016-April/12072… > > -- www.tudorgirba.com www.feenk.com "Presenting is storytelling."

May 1, 2017

Re: [Pharo-dev] [squeak-dev] [Vm-dev] Byte & String collection hash performance; a modest proposal for change.

by Eliot Miranda

On Mon, May 1, 2017 at 10:37 AM, Tony Garnock-Jones <tonyg(a)ccs.neu.edu> wrote: > On 5/1/17 1:26 PM, Levente Uzonyi wrote: > > I presume that a general purpose in-image solution would be more > > complex. String already has too many subclasses (6 in Squeak), while at > > the same time other kind of new subclasses would be welcome too, e.g. > > Strings with 2-byte characters. > > Since these properties are orthogonal, there would be many new > > subclasses to cover all cases. > > A classic motivating case for Traits, right? > Alas no. The problem with String's subclasses is that they're binary objects. They have no inst vars in which one can cache a hash. They're juts a flat vector of bytes. If one added a trait to them the system would fail because there's no way to add an inst var to a binary object in the current Smalltalk object representation. hence Levente's very clever idea of hiding the hash in a hidden header word. > Tony > > -- _,,,^..^,,,_ best, Eliot

May 1, 2017

Re: [Pharo-dev] [Vm-dev] [squeak-dev] Byte & String collection hash performance; a modest proposal for change.

by David T. Lewis

Does it need to be done in the VM? Why not make a class LargeString with instance variables aString and myCalculatedHashValueForTheString. That way you can cache the hash value and calculate it any way you want. I know vey little about hashing, just wondering if this kind of thing can be handled more easily in the image. Dave > > On Mon, 1 May 2017, Levente Uzonyi wrote: > >> Well, I had started to write a reply, but I had to postpone it. >> I mostly agree with your suggestions. >> >> One thing that can be done about large strings is to cache the >> calculated >> hash value in the larger strings. Currently the object representation >> changes when the string contains 255 or more characters. In that case an >> additional 64 bit field is added to the object header to store its >> length. >> If we were to use the upper 28+1 bits of that field to cache the hash, >> there would still be 35-bits to encode length, which would be enough to >> represent strings up to 8 GiB. > > Well, we can keep the whole range minus one bit, but then we can't store > the hash for strings larger than 8 GiB. > > Levente > >> But this would require further VM changes (e.g. at:put: would have to >> flush the cache). >> >> Levente >> >> On Mon, 1 May 2017, Martin McClure wrote: >> >>> I see no replies to this on any of the three lists it was sent to, so I >>> guess I'll chime in. >>> >>> tl;dr: >>> Making a primitive for #hashMultiply, probably a good idea, in some >>> form, since doing it in Smalltalk is awkward. >>> Only hashing every N-th character of large strings, probably a very bad >>> idea. Performance might well get worse, the complexity does not seem >>> justified, and it would open a sizeable security hole. >>> >>> More verbiage below for those interested. >>> >>> Regards, >>> -Martin >>> >>> On 04/18/2017 07:09 PM, Eliot Miranda wrote: >>>> Hi All, >>>> >>>> the hash algorithm used for ByteString in Squeak and Pharo is good >>>> for "small" strings and overkill for large strings. >>> >>> Why do you say it's overkill for large strings? Are there applications >>> with large strings that are being negatively impacted by the current >>> algorithm? Which ones, and impacted how? >>> >>>> It is important in many applications to get well distributed string >>>> hashes, especially over the range of strings that constitute things >>>> like method names, URLs, etc. Consequently, the current algorithm >>>> includes every character in a string. This works very well for >>>> "small" strings and results in very slow hashes (and hence long >>>> latencies, because the hash is an uninterruptible primitive) for large >>>> strings, where large may be several megabytes. >>> >>> A simple solution for the uninterruptable primitive is to not make it a >>> primitive. Make #hashMultiply a primitive (since this particular kind >>> of >>> numeric modulo computation is really painful in Smalltalk), and do the >>> rest in a loop in Smalltalk. It sounds like you've done the >>> #hashMultiply primitive already. >>> >>> If the overhead of calling a primitive for each character proves to be >>> too much, even with the faster primitive calling methodologies you >>> talked about in the "Cog Primitive Performance" thread on the Vm-dev >>> list, a more complex primitive could take a range of bytes, so large >>> strings would be done in batches, solving the latency problem. >>> >>>> >>>> Let's look at the basic hash algorithm. >>> [...] >>>> >>>> In looking at this I've added a primitive for hashMultiply; primitive >>>> #159 implements precisely self * 1664525 bitAnd: 16r0FFFFFFF for >>>> SmallInteger and LargePositiveInteger receivers, as fast as possible >>>> in the Cog JIT. With this machinery in place it's instructive to >>>> compare the cost of the primitive against the non-primitive Smalltalk >>>> code. >>>> >>>> First let me introduce a set of replacement hash functions, newHashN. >>>> These hash all characters in strings up to a certain size, and then no >>>> more than that number for larger strings. Here are newHash64 and >>>> newHash2048, which use pure Smalltalk, including an inlined >>>> hashMultiply written to avoid SmallInteger overflow. Also measured >>>> are the obvious variants newHash128, newHash256, newHash512 & >>>> mewHash1024. >>>> >>>> >>> [...] >>>> So the idea here is to step through the string by 1 for strings sizes >>>> up to N - 1, and by greater than 1 for strings of size >= N, limiting >>>> the maximum number of characters sampled to between N // 2 and N - 1. >>> >>> The history of computing is littered with the bones of those who have >>> tried this kind of thing. It doesn't end well. Yes, you get a faster >>> hash function. And then you find, for sets of data that you or your >>> users actually want to use, that you get collisions like crazy, and >>> much >>> worse overall performance than you started with. >>> >>> Sure, it works OK for the sets of data that the designer *tested*, and >>> probably for the sets of data that they *anticipated*. But real-world >>> data is tricky. It includes data sets where the characters that differ >>> are the ones that the hash thinks are unimportant, and there goes your >>> performance, by orders of magnitude. For instance, early versions of >>> Java used a limited number of characters to hash strings. One of the >>> biggest compatibility-breaking changes they were forced to make in >>> later >>> Java versions was to consider *all* characters in hashing. It turned >>> out >>> that it was very common to hash URLs, and many distinct URLs had most >>> of >>> their characters in common. >>> >>> And you don't always get to choose your data -- sometimes you have an >>> opponent who is actively looking to create collisions as a >>> denial-of-service attack. There was a fair-sized kerfluffle about this >>> a >>> few years ago -- most web scripting languages made it too easy to mount >>> this kind of attack. >>> >> https://arstechnica.com/business/2011/12/huge-portions-of-web-vulnerable-to… >>> "...an attacker can degenerate the hash table by sending lots of >>> colliding keys. ...making it possible to exhaust hours of CPU time >>> using >>> a single HTTP request." >>> To guard against this kind of attack you need a randomized element in >>> your hash (not a bad idea for Smalltalk, actually, and pretty easy -- >>> mixing in the identity hash of the collection might be sufficient) or a >>> cryptographic hash (not worth the computational expense for most >>> purposes). However, even adding a randomized element would not prevent >>> this kind of attack if you predictably completely ignore some >>> characters >>> of the input string. That just makes it *so* easy to generate data that >>> validates, and is not equal, but causes collisions. >>> >>> So really, for general-purpose use (i.e. what's built into the >>> language) >>> hash *every* character of *all* strings. If someone finds that this is >>> a >>> performance problem in a real-world situation, it can be addressed in >>> an >>> application-specific way. >> >> >

May 1, 2017

Re: [Pharo-dev] [Vm-dev] Byte & String collection hash performance; a modest proposal for change.

by Martin McClure

On 05/01/2017 02:24 AM, Andrei Chis wrote: > > performance, by orders of magnitude. For instance, early versions of > Java used a limited number of characters to hash strings. One of the > biggest compatibility-breaking changes they were forced to make in > later > Java versions was to consider *all* characters in hashing. It > turned out > that it was very common to hash URLs, and many distinct URLs had > most of > their characters in common. > > > Was curious about this and found the actual issues [1] if anybody else > is interested. > > [1] http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4045622 > Thanks for digging this up, Andrei! I was working from memory, and had forgotten some of the details. Yeah, they had to change the language spec, which was a Big Deal at the time. (and Josh Bloch and Guy Steele, no less!) Regards, -Martin

May 1, 2017

Re: [Pharo-dev] First commit

by Ben Coman

On Mon, May 1, 2017 at 2:25 AM, Evan Donahue <emdonahu(a)gmail.com> wrote: > Hello, > > I have just tried to submit my first patch to the main Pharo inbox. I > submitted a patch for issue > > https://pharo.fogbugz.com/f/cases/17848 > > because it basically amounted to changing an error message and seemed like > a good place to start figuring out how the submission process works. > > I saved my slice to the Pharo60Inbox, but when I typed my commit message > and hit save, > I presume you mean, after you typed your commit message, you hit <Accept>. > I did not see any confirmation. How can I tell if my slice was submitted > or not? > It seems like it did not save. I opened Pharo60Inbox and filtering on 17848 shows nothing. You should have got up a window titled something like "Version: SLICE-Issue-NNNN-Blah-blah-blah-BenComan.1 Can you see it in your package cache? If its there, you can select it and choose <Copy> to copy it to Pharo60Inbox (but <Copy> requires you to have added your password to the Pharo60Inbox repo defintion). cheers -ben

May 1, 2017

About glorp

by Stephane Ducasse

Hi I loaded glorp following this Metacello new smalltalkhubUser: 'DBXTalk' project: 'Garage'; configuration: 'GarageGlorp'; version: '0.2'; load. And the tests are telling me NativePostgresDriver not found.... :( Do we have to have a database set to run the Glorp tests ? Stef

May 1, 2017

Re: [Pharo-dev] [Pharo-users] Process scheduling

by Ben Coman

On Mon, May 1, 2017 at 4:42 AM, <raffaello.giulietti(a)lifeware.ch> wrote: > On 30/04/17 17:01, Ben Coman wrote: > > > > > > On Sun, Apr 30, 2017 at 10:11 PM, <raffaello.giulietti(a)lifeware.ch > > <mailto:raffaello.giulietti@lifeware.ch>> wrote: > > > > Hi, > > > > I can't find a description on how ProcessorScheduler in Pharo (i.e., > > Cog) is supposed to do context switches in absence of other > > synchronization points. > > > > Is it following the BlueBook spec? Just to recall that model, the VM, > > when about to execute the current bytecode instruction of the current > > Process, checks whether a strictly higher priority Process is ready > to > > run and if so, switches to (one of) the highest priority among them. > > > > Thanks in advance for clarifications. > > > > > > This topic is probably a bit advanced pharo-users list. > > You'll likely get better answers in vm-dev list of pharo-dev list. > > Can you post any follow ups there? > > > > ProcessScheduler doesn't have much to do with context switching, > > except providing the #yield primitive for a user to invoke manually. > > As you noted, the actual context switching happens in the VM. > > > > The charts here [1] might be useful to get an overview of the execution > > loop. > > Keep in mind though Its a very rough view of a novice exploring the Cog > > execution engine > > and may not be entirely accurate. But it will give you a starting point. > > > > For another example, look at > > CoInterpreterPrimitives>>primitiveEnterCriticalSection > > in VMMaker (http://source.squeak.org/VMMaker) > > > > cheers -ben > > Hi Ben, > > of course, I can post the question on the pharo-dev mailing list and > certainly will if nobody on this one can answer my question. > > However, Process is a Pharo user level class, I was responding to your literal question about ProcessScheduler and what part of the system does the context switching. Now I'm guessing your interest is more about user-level multi-threaded behaviour of Processes, so I hunted down some discussions you may find be interesting * http://forum.world.st/The-Trunk-Kernel-mt-1009-mcz-td4887889.html (search for preemptionYields) * http://forum.world.st/threading-in-Pharo-td4750534.html * http://forum.world.st/threading-in-Pharo-td4750534.html * http://forum.world.st/VM-Maker-VMMaker-oscog-eem-942-mcz-td4790993.html However I didn't quite follow where you said... "in absence of other synchronization points." Do you mean without suspend/resuming Processes or waiting/signalling Semaphore and something else? I'm not sure if this is what your looking for, but in summary... Processes at the same priority are scheduled cooperatively. they must deliberately yield or sleep to be put at the back of their priority's run-queue.before another process of same priority can run. Higher priority processes pre-empt lower priority processes only when they are woken (i.e. signalled or resumed). The pre-empted process stays at the front of its priority's run-queue so that the higher-priority-preemption is transparent to the same-priority-cooperative scheduling. not something private to > the implementation of the vm. Any programmer should know what to expect > and what to not expect when different Processes are started, and how > they behave and interact. It's part of the general knowledge of the > Smalltalk programming model, on par with Object, Class and Behavior. > So, before I switch to the pharo-dev list, let's wait and see... > > BTW, it seems that reference [1], containing the charts you mention, > could not find its way in your reply. Can you please make it available? > > Whoops. [1] http://forum.world.st/CogVM-Execution-Flow-td4900675.html#a4900676 -------------------- Now a follow up question of my own, for [pharo-dev]. The Process comment says "When any one of several paths of control can be advanced, the single instance of ProcessorScheduler named Processor **determines** which one will actually be advanced partly using the value of priority." Did I misunderstand how this works? I thought determination of which process next ran occurred in the VM by StackInterpreter>>wakeHighestPriority and ProcessorScheduler seems more a place to just get values associated scheduling. cheers -ben

May 1, 2017

Re: [Pharo-dev] [Vm-dev] Byte & String collection hash performance; a modest proposal for change.

by Andrei Chis

On Mon, May 1, 2017 at 9:09 AM, Martin McClure <martin(a)hand2mouse.com> wrote: > > I see no replies to this on any of the three lists it was sent to, so I > guess I'll chime in. > > tl;dr: > Making a primitive for #hashMultiply, probably a good idea, in some > form, since doing it in Smalltalk is awkward. > Only hashing every N-th character of large strings, probably a very bad > idea. Performance might well get worse, the complexity does not seem > justified, and it would open a sizeable security hole. > > More verbiage below for those interested. > > Regards, > -Martin > > On 04/18/2017 07:09 PM, Eliot Miranda wrote: > > Hi All, > > > > the hash algorithm used for ByteString in Squeak and Pharo is good > > for "small" strings and overkill for large strings. > > Why do you say it's overkill for large strings? Are there applications > with large strings that are being negatively impacted by the current > algorithm? Which ones, and impacted how? > > > It is important in many applications to get well distributed string > > hashes, especially over the range of strings that constitute things > > like method names, URLs, etc. Consequently, the current algorithm > > includes every character in a string. This works very well for > > "small" strings and results in very slow hashes (and hence long > > latencies, because the hash is an uninterruptible primitive) for large > > strings, where large may be several megabytes. > > A simple solution for the uninterruptable primitive is to not make it a > primitive. Make #hashMultiply a primitive (since this particular kind of > numeric modulo computation is really painful in Smalltalk), and do the > rest in a loop in Smalltalk. It sounds like you've done the > #hashMultiply primitive already. > > If the overhead of calling a primitive for each character proves to be > too much, even with the faster primitive calling methodologies you > talked about in the "Cog Primitive Performance" thread on the Vm-dev > list, a more complex primitive could take a range of bytes, so large > strings would be done in batches, solving the latency problem. > > > > > Let's look at the basic hash algorithm. > [...] > > > > In looking at this I've added a primitive for hashMultiply; primitive > > #159 implements precisely self * 1664525 bitAnd: 16r0FFFFFFF for > > SmallInteger and LargePositiveInteger receivers, as fast as possible > > in the Cog JIT. With this machinery in place it's instructive to > > compare the cost of the primitive against the non-primitive Smalltalk > > code. > > > > First let me introduce a set of replacement hash functions, newHashN. > > These hash all characters in strings up to a certain size, and then no > > more than that number for larger strings. Here are newHash64 and > > newHash2048, which use pure Smalltalk, including an inlined > > hashMultiply written to avoid SmallInteger overflow. Also measured > > are the obvious variants newHash128, newHash256, newHash512 & > mewHash1024. > > > > > [...] > > So the idea here is to step through the string by 1 for strings sizes > > up to N - 1, and by greater than 1 for strings of size >= N, limiting > > the maximum number of characters sampled to between N // 2 and N - 1. > > The history of computing is littered with the bones of those who have > tried this kind of thing. It doesn't end well. Yes, you get a faster > hash function. And then you find, for sets of data that you or your > users actually want to use, that you get collisions like crazy, and much > worse overall performance than you started with. > > Sure, it works OK for the sets of data that the designer *tested*, and > probably for the sets of data that they *anticipated*. But real-world > data is tricky. It includes data sets where the characters that differ > are the ones that the hash thinks are unimportant, and there goes your > performance, by orders of magnitude. For instance, early versions of > Java used a limited number of characters to hash strings. One of the > biggest compatibility-breaking changes they were forced to make in later > Java versions was to consider *all* characters in hashing. It turned out > that it was very common to hash URLs, and many distinct URLs had most of > their characters in common. > Was curious about this and found the actual issues [1] if anybody else is interested. [1] http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4045622 Cheers, Andrei > > And you don't always get to choose your data -- sometimes you have an > opponent who is actively looking to create collisions as a > denial-of-service attack. There was a fair-sized kerfluffle about this a > few years ago -- most web scripting languages made it too easy to mount > this kind of attack. > https://arstechnica.com/business/2011/12/huge- > portions-of-web-vulnerable-to-hashing-denial-of-service-attack/ > "...an attacker can degenerate the hash table by sending lots of > colliding keys. ...making it possible to exhaust hours of CPU time using > a single HTTP request." > To guard against this kind of attack you need a randomized element in > your hash (not a bad idea for Smalltalk, actually, and pretty easy -- > mixing in the identity hash of the collection might be sufficient) or a > cryptographic hash (not worth the computational expense for most > purposes). However, even adding a randomized element would not prevent > this kind of attack if you predictably completely ignore some characters > of the input string. That just makes it *so* easy to generate data that > validates, and is not equal, but causes collisions. > > So really, for general-purpose use (i.e. what's built into the language) > hash *every* character of *all* strings. If someone finds that this is a > performance problem in a real-world situation, it can be addressed in an > application-specific way. > >

May 1, 2017

Re: [Pharo-dev] First commit

by Stephane Ducasse

if you open Pharo60Inbox you should be able to see it. On Sun, Apr 30, 2017 at 8:25 PM, Evan Donahue <emdonahu(a)gmail.com> wrote: > Hello, > > I have just tried to submit my first patch to the main Pharo inbox. I > submitted a patch for issue > > https://pharo.fogbugz.com/f/cases/17848 > > because it basically amounted to changing an error message and seemed like > a good place to start figuring out how the submission process works. > > I saved my slice to the Pharo60Inbox, but when I typed my commit message > and hit save, I did not see any confirmation. How can I tell if my slice > was submitted or not? > > Thank you, > Evan >

May 1, 2017