May 2017 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] First commit

by Ben Coman

On Mon, May 1, 2017 at 2:25 AM, Evan Donahue <emdonahu(a)gmail.com> wrote: > Hello, > > I have just tried to submit my first patch to the main Pharo inbox. I > submitted a patch for issue > > https://pharo.fogbugz.com/f/cases/17848 > > because it basically amounted to changing an error message and seemed like > a good place to start figuring out how the submission process works. > > I saved my slice to the Pharo60Inbox, but when I typed my commit message > and hit save, > I presume you mean, after you typed your commit message, you hit <Accept>. > I did not see any confirmation. How can I tell if my slice was submitted > or not? > It seems like it did not save. I opened Pharo60Inbox and filtering on 17848 shows nothing. You should have got up a window titled something like "Version: SLICE-Issue-NNNN-Blah-blah-blah-BenComan.1 Can you see it in your package cache? If its there, you can select it and choose <Copy> to copy it to Pharo60Inbox (but <Copy> requires you to have added your password to the Pharo60Inbox repo defintion). cheers -ben

May 1, 2017

About glorp

by Stephane Ducasse

Hi I loaded glorp following this Metacello new smalltalkhubUser: 'DBXTalk' project: 'Garage'; configuration: 'GarageGlorp'; version: '0.2'; load. And the tests are telling me NativePostgresDriver not found.... :( Do we have to have a database set to run the Glorp tests ? Stef

May 1, 2017

Re: [Pharo-dev] [Pharo-users] Process scheduling

by Ben Coman

On Mon, May 1, 2017 at 4:42 AM, <raffaello.giulietti(a)lifeware.ch> wrote: > On 30/04/17 17:01, Ben Coman wrote: > > > > > > On Sun, Apr 30, 2017 at 10:11 PM, <raffaello.giulietti(a)lifeware.ch > > <mailto:raffaello.giulietti@lifeware.ch>> wrote: > > > > Hi, > > > > I can't find a description on how ProcessorScheduler in Pharo (i.e., > > Cog) is supposed to do context switches in absence of other > > synchronization points. > > > > Is it following the BlueBook spec? Just to recall that model, the VM, > > when about to execute the current bytecode instruction of the current > > Process, checks whether a strictly higher priority Process is ready > to > > run and if so, switches to (one of) the highest priority among them. > > > > Thanks in advance for clarifications. > > > > > > This topic is probably a bit advanced pharo-users list. > > You'll likely get better answers in vm-dev list of pharo-dev list. > > Can you post any follow ups there? > > > > ProcessScheduler doesn't have much to do with context switching, > > except providing the #yield primitive for a user to invoke manually. > > As you noted, the actual context switching happens in the VM. > > > > The charts here [1] might be useful to get an overview of the execution > > loop. > > Keep in mind though Its a very rough view of a novice exploring the Cog > > execution engine > > and may not be entirely accurate. But it will give you a starting point. > > > > For another example, look at > > CoInterpreterPrimitives>>primitiveEnterCriticalSection > > in VMMaker (http://source.squeak.org/VMMaker) > > > > cheers -ben > > Hi Ben, > > of course, I can post the question on the pharo-dev mailing list and > certainly will if nobody on this one can answer my question. > > However, Process is a Pharo user level class, I was responding to your literal question about ProcessScheduler and what part of the system does the context switching. Now I'm guessing your interest is more about user-level multi-threaded behaviour of Processes, so I hunted down some discussions you may find be interesting * http://forum.world.st/The-Trunk-Kernel-mt-1009-mcz-td4887889.html (search for preemptionYields) * http://forum.world.st/threading-in-Pharo-td4750534.html * http://forum.world.st/threading-in-Pharo-td4750534.html * http://forum.world.st/VM-Maker-VMMaker-oscog-eem-942-mcz-td4790993.html However I didn't quite follow where you said... "in absence of other synchronization points." Do you mean without suspend/resuming Processes or waiting/signalling Semaphore and something else? I'm not sure if this is what your looking for, but in summary... Processes at the same priority are scheduled cooperatively. they must deliberately yield or sleep to be put at the back of their priority's run-queue.before another process of same priority can run. Higher priority processes pre-empt lower priority processes only when they are woken (i.e. signalled or resumed). The pre-empted process stays at the front of its priority's run-queue so that the higher-priority-preemption is transparent to the same-priority-cooperative scheduling. not something private to > the implementation of the vm. Any programmer should know what to expect > and what to not expect when different Processes are started, and how > they behave and interact. It's part of the general knowledge of the > Smalltalk programming model, on par with Object, Class and Behavior. > So, before I switch to the pharo-dev list, let's wait and see... > > BTW, it seems that reference [1], containing the charts you mention, > could not find its way in your reply. Can you please make it available? > > Whoops. [1] http://forum.world.st/CogVM-Execution-Flow-td4900675.html#a4900676 -------------------- Now a follow up question of my own, for [pharo-dev]. The Process comment says "When any one of several paths of control can be advanced, the single instance of ProcessorScheduler named Processor **determines** which one will actually be advanced partly using the value of priority." Did I misunderstand how this works? I thought determination of which process next ran occurred in the VM by StackInterpreter>>wakeHighestPriority and ProcessorScheduler seems more a place to just get values associated scheduling. cheers -ben

May 1, 2017

Re: [Pharo-dev] [Vm-dev] Byte & String collection hash performance; a modest proposal for change.

by Andrei Chis

On Mon, May 1, 2017 at 9:09 AM, Martin McClure <martin(a)hand2mouse.com> wrote: > > I see no replies to this on any of the three lists it was sent to, so I > guess I'll chime in. > > tl;dr: > Making a primitive for #hashMultiply, probably a good idea, in some > form, since doing it in Smalltalk is awkward. > Only hashing every N-th character of large strings, probably a very bad > idea. Performance might well get worse, the complexity does not seem > justified, and it would open a sizeable security hole. > > More verbiage below for those interested. > > Regards, > -Martin > > On 04/18/2017 07:09 PM, Eliot Miranda wrote: > > Hi All, > > > > the hash algorithm used for ByteString in Squeak and Pharo is good > > for "small" strings and overkill for large strings. > > Why do you say it's overkill for large strings? Are there applications > with large strings that are being negatively impacted by the current > algorithm? Which ones, and impacted how? > > > It is important in many applications to get well distributed string > > hashes, especially over the range of strings that constitute things > > like method names, URLs, etc. Consequently, the current algorithm > > includes every character in a string. This works very well for > > "small" strings and results in very slow hashes (and hence long > > latencies, because the hash is an uninterruptible primitive) for large > > strings, where large may be several megabytes. > > A simple solution for the uninterruptable primitive is to not make it a > primitive. Make #hashMultiply a primitive (since this particular kind of > numeric modulo computation is really painful in Smalltalk), and do the > rest in a loop in Smalltalk. It sounds like you've done the > #hashMultiply primitive already. > > If the overhead of calling a primitive for each character proves to be > too much, even with the faster primitive calling methodologies you > talked about in the "Cog Primitive Performance" thread on the Vm-dev > list, a more complex primitive could take a range of bytes, so large > strings would be done in batches, solving the latency problem. > > > > > Let's look at the basic hash algorithm. > [...] > > > > In looking at this I've added a primitive for hashMultiply; primitive > > #159 implements precisely self * 1664525 bitAnd: 16r0FFFFFFF for > > SmallInteger and LargePositiveInteger receivers, as fast as possible > > in the Cog JIT. With this machinery in place it's instructive to > > compare the cost of the primitive against the non-primitive Smalltalk > > code. > > > > First let me introduce a set of replacement hash functions, newHashN. > > These hash all characters in strings up to a certain size, and then no > > more than that number for larger strings. Here are newHash64 and > > newHash2048, which use pure Smalltalk, including an inlined > > hashMultiply written to avoid SmallInteger overflow. Also measured > > are the obvious variants newHash128, newHash256, newHash512 & > mewHash1024. > > > > > [...] > > So the idea here is to step through the string by 1 for strings sizes > > up to N - 1, and by greater than 1 for strings of size >= N, limiting > > the maximum number of characters sampled to between N // 2 and N - 1. > > The history of computing is littered with the bones of those who have > tried this kind of thing. It doesn't end well. Yes, you get a faster > hash function. And then you find, for sets of data that you or your > users actually want to use, that you get collisions like crazy, and much > worse overall performance than you started with. > > Sure, it works OK for the sets of data that the designer *tested*, and > probably for the sets of data that they *anticipated*. But real-world > data is tricky. It includes data sets where the characters that differ > are the ones that the hash thinks are unimportant, and there goes your > performance, by orders of magnitude. For instance, early versions of > Java used a limited number of characters to hash strings. One of the > biggest compatibility-breaking changes they were forced to make in later > Java versions was to consider *all* characters in hashing. It turned out > that it was very common to hash URLs, and many distinct URLs had most of > their characters in common. > Was curious about this and found the actual issues [1] if anybody else is interested. [1] http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4045622 Cheers, Andrei > > And you don't always get to choose your data -- sometimes you have an > opponent who is actively looking to create collisions as a > denial-of-service attack. There was a fair-sized kerfluffle about this a > few years ago -- most web scripting languages made it too easy to mount > this kind of attack. > https://arstechnica.com/business/2011/12/huge- > portions-of-web-vulnerable-to-hashing-denial-of-service-attack/ > "...an attacker can degenerate the hash table by sending lots of > colliding keys. ...making it possible to exhaust hours of CPU time using > a single HTTP request." > To guard against this kind of attack you need a randomized element in > your hash (not a bad idea for Smalltalk, actually, and pretty easy -- > mixing in the identity hash of the collection might be sufficient) or a > cryptographic hash (not worth the computational expense for most > purposes). However, even adding a randomized element would not prevent > this kind of attack if you predictably completely ignore some characters > of the input string. That just makes it *so* easy to generate data that > validates, and is not equal, but causes collisions. > > So really, for general-purpose use (i.e. what's built into the language) > hash *every* character of *all* strings. If someone finds that this is a > performance problem in a real-world situation, it can be addressed in an > application-specific way. > >

May 1, 2017

Re: [Pharo-dev] First commit

by Stephane Ducasse

if you open Pharo60Inbox you should be able to see it. On Sun, Apr 30, 2017 at 8:25 PM, Evan Donahue <emdonahu(a)gmail.com> wrote: > Hello, > > I have just tried to submit my first patch to the main Pharo inbox. I > submitted a patch for issue > > https://pharo.fogbugz.com/f/cases/17848 > > because it basically amounted to changing an error message and seemed like > a good place to start figuring out how the submission process works. > > I saved my slice to the Pharo60Inbox, but when I typed my commit message > and hit save, I did not see any confirmation. How can I tell if my slice > was submitted or not? > > Thank you, > Evan >

May 1, 2017

Re: [Pharo-dev] Byte & String collection hash performance; a modest proposal for change.

by Martin McClure

I see no replies to this on any of the three lists it was sent to, so I guess I'll chime in. tl;dr: Making a primitive for #hashMultiply, probably a good idea, in some form, since doing it in Smalltalk is awkward. Only hashing every N-th character of large strings, probably a very bad idea. Performance might well get worse, the complexity does not seem justified, and it would open a sizeable security hole. More verbiage below for those interested. Regards, -Martin On 04/18/2017 07:09 PM, Eliot Miranda wrote: > Hi All, > > the hash algorithm used for ByteString in Squeak and Pharo is good > for "small" strings and overkill for large strings. Why do you say it's overkill for large strings? Are there applications with large strings that are being negatively impacted by the current algorithm? Which ones, and impacted how? > It is important in many applications to get well distributed string > hashes, especially over the range of strings that constitute things > like method names, URLs, etc. Consequently, the current algorithm > includes every character in a string. This works very well for > "small" strings and results in very slow hashes (and hence long > latencies, because the hash is an uninterruptible primitive) for large > strings, where large may be several megabytes. A simple solution for the uninterruptable primitive is to not make it a primitive. Make #hashMultiply a primitive (since this particular kind of numeric modulo computation is really painful in Smalltalk), and do the rest in a loop in Smalltalk. It sounds like you've done the #hashMultiply primitive already. If the overhead of calling a primitive for each character proves to be too much, even with the faster primitive calling methodologies you talked about in the "Cog Primitive Performance" thread on the Vm-dev list, a more complex primitive could take a range of bytes, so large strings would be done in batches, solving the latency problem. > > Let's look at the basic hash algorithm. [...] > > In looking at this I've added a primitive for hashMultiply; primitive > #159 implements precisely self * 1664525 bitAnd: 16r0FFFFFFF for > SmallInteger and LargePositiveInteger receivers, as fast as possible > in the Cog JIT. With this machinery in place it's instructive to > compare the cost of the primitive against the non-primitive Smalltalk > code. > > First let me introduce a set of replacement hash functions, newHashN. > These hash all characters in strings up to a certain size, and then no > more than that number for larger strings. Here are newHash64 and > newHash2048, which use pure Smalltalk, including an inlined > hashMultiply written to avoid SmallInteger overflow. Also measured > are the obvious variants newHash128, newHash256, newHash512 & mewHash1024. > > [...] > So the idea here is to step through the string by 1 for strings sizes > up to N - 1, and by greater than 1 for strings of size >= N, limiting > the maximum number of characters sampled to between N // 2 and N - 1. The history of computing is littered with the bones of those who have tried this kind of thing. It doesn't end well. Yes, you get a faster hash function. And then you find, for sets of data that you or your users actually want to use, that you get collisions like crazy, and much worse overall performance than you started with. Sure, it works OK for the sets of data that the designer *tested*, and probably for the sets of data that they *anticipated*. But real-world data is tricky. It includes data sets where the characters that differ are the ones that the hash thinks are unimportant, and there goes your performance, by orders of magnitude. For instance, early versions of Java used a limited number of characters to hash strings. One of the biggest compatibility-breaking changes they were forced to make in later Java versions was to consider *all* characters in hashing. It turned out that it was very common to hash URLs, and many distinct URLs had most of their characters in common. And you don't always get to choose your data -- sometimes you have an opponent who is actively looking to create collisions as a denial-of-service attack. There was a fair-sized kerfluffle about this a few years ago -- most web scripting languages made it too easy to mount this kind of attack. https://arstechnica.com/business/2011/12/huge-portions-of-web-vulnerable-to… "...an attacker can degenerate the hash table by sending lots of colliding keys. ...making it possible to exhaust hours of CPU time using a single HTTP request." To guard against this kind of attack you need a randomized element in your hash (not a bad idea for Smalltalk, actually, and pretty easy -- mixing in the identity hash of the collection might be sufficient) or a cryptographic hash (not worth the computational expense for most purposes). However, even adding a randomized element would not prevent this kind of attack if you predictably completely ignore some characters of the input string. That just makes it *so* easy to generate data that validates, and is not equal, but causes collisions. So really, for general-purpose use (i.e. what's built into the language) hash *every* character of *all* strings. If someone finds that this is a performance problem in a real-world situation, it can be addressed in an application-specific way.

May 1, 2017

Esteban's ChangeLog week of 24 April 2017

by estebanlm＠gmail.com

Hello! This is my weekly ChangeLog, from 24 April 2017 to 30 April 2017. You can see it in a better format by going here: http://log.smallworks.eu/web/search?from=24/4/2017&to=30/4/2017 ChangeLog ========= 27 April 2017: -------------- * = VERY IMPORTANT We moved iceberg repositories to: [https://github.com/pharo-vcs/iceberg](https://github.com/pharo-vcs/iceberg) So, now to install you will need: ---- Metacello new baseline: 'Iceberg'; repository: 'github://pharo-vcs/iceberg:dev-0.4'; load. ---- I also uploaded a Configuration to the [catalog](http://catalog.pharo.org), so you can load while doing the transition. * I figured out that we will need it, so I'm working on a cherry-pick system for [iceberg](http://github.com/npasserini/iceberg). Still nothing to show but stay tuned. Also, I'm working on translate [libgit2 bindings](http://github.com/npasserini/libgit2-pharo-bindings) to version 0.25.1, because of compilation issues with 0.23 and because we will need it anyway. * I spent some time fixing 'once again' the VM build to allow +libgit2+ module to load. This time it was just a stupid cache issue... latest VM will work now. 24 April 2017: -------------- * ... and I spent the rest of the day recovering the vm tests. So far I have running +linux+ and +macOS+. Windows tests are have still 3 errors, I will end this tomorrow. * I just spend 2hs fixing a corruption on pharo [site](http://pharo.org). Is becoming annoying, but there is a problem in certain cases were +VOLazyProxy+ throws an exception which provoques a semaphore to stay in +wait+ status and then keeping conections to the database. And well, that ends... not in the best of the ways. cheers! Esteban

May 1, 2017