- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Byte & String collection hash performance; a modest proposal for change.

by Martin McClure

I see no replies to this on any of the three lists it was sent to, so I guess I'll chime in. tl;dr: Making a primitive for #hashMultiply, probably a good idea, in some form, since doing it in Smalltalk is awkward. Only hashing every N-th character of large strings, probably a very bad idea. Performance might well get worse, the complexity does not seem justified, and it would open a sizeable security hole. More verbiage below for those interested. Regards, -Martin On 04/18/2017 07:09 PM, Eliot Miranda wrote: > Hi All, > > the hash algorithm used for ByteString in Squeak and Pharo is good > for "small" strings and overkill for large strings. Why do you say it's overkill for large strings? Are there applications with large strings that are being negatively impacted by the current algorithm? Which ones, and impacted how? > It is important in many applications to get well distributed string > hashes, especially over the range of strings that constitute things > like method names, URLs, etc. Consequently, the current algorithm > includes every character in a string. This works very well for > "small" strings and results in very slow hashes (and hence long > latencies, because the hash is an uninterruptible primitive) for large > strings, where large may be several megabytes. A simple solution for the uninterruptable primitive is to not make it a primitive. Make #hashMultiply a primitive (since this particular kind of numeric modulo computation is really painful in Smalltalk), and do the rest in a loop in Smalltalk. It sounds like you've done the #hashMultiply primitive already. If the overhead of calling a primitive for each character proves to be too much, even with the faster primitive calling methodologies you talked about in the "Cog Primitive Performance" thread on the Vm-dev list, a more complex primitive could take a range of bytes, so large strings would be done in batches, solving the latency problem. > > Let's look at the basic hash algorithm. [...] > > In looking at this I've added a primitive for hashMultiply; primitive > #159 implements precisely self * 1664525 bitAnd: 16r0FFFFFFF for > SmallInteger and LargePositiveInteger receivers, as fast as possible > in the Cog JIT. With this machinery in place it's instructive to > compare the cost of the primitive against the non-primitive Smalltalk > code. > > First let me introduce a set of replacement hash functions, newHashN. > These hash all characters in strings up to a certain size, and then no > more than that number for larger strings. Here are newHash64 and > newHash2048, which use pure Smalltalk, including an inlined > hashMultiply written to avoid SmallInteger overflow. Also measured > are the obvious variants newHash128, newHash256, newHash512 & mewHash1024. > > [...] > So the idea here is to step through the string by 1 for strings sizes > up to N - 1, and by greater than 1 for strings of size >= N, limiting > the maximum number of characters sampled to between N // 2 and N - 1. The history of computing is littered with the bones of those who have tried this kind of thing. It doesn't end well. Yes, you get a faster hash function. And then you find, for sets of data that you or your users actually want to use, that you get collisions like crazy, and much worse overall performance than you started with. Sure, it works OK for the sets of data that the designer *tested*, and probably for the sets of data that they *anticipated*. But real-world data is tricky. It includes data sets where the characters that differ are the ones that the hash thinks are unimportant, and there goes your performance, by orders of magnitude. For instance, early versions of Java used a limited number of characters to hash strings. One of the biggest compatibility-breaking changes they were forced to make in later Java versions was to consider *all* characters in hashing. It turned out that it was very common to hash URLs, and many distinct URLs had most of their characters in common. And you don't always get to choose your data -- sometimes you have an opponent who is actively looking to create collisions as a denial-of-service attack. There was a fair-sized kerfluffle about this a few years ago -- most web scripting languages made it too easy to mount this kind of attack. https://arstechnica.com/business/2011/12/huge-portions-of-web-vulnerable-to… "...an attacker can degenerate the hash table by sending lots of colliding keys. ...making it possible to exhaust hours of CPU time using a single HTTP request." To guard against this kind of attack you need a randomized element in your hash (not a bad idea for Smalltalk, actually, and pretty easy -- mixing in the identity hash of the collection might be sufficient) or a cryptographic hash (not worth the computational expense for most purposes). However, even adding a randomized element would not prevent this kind of attack if you predictably completely ignore some characters of the input string. That just makes it *so* easy to generate data that validates, and is not equal, but causes collisions. So really, for general-purpose use (i.e. what's built into the language) hash *every* character of *all* strings. If someone finds that this is a performance problem in a real-world situation, it can be addressed in an application-specific way.

May 1, 2017

Esteban's ChangeLog week of 24 April 2017

by estebanlm＠gmail.com

Hello! This is my weekly ChangeLog, from 24 April 2017 to 30 April 2017. You can see it in a better format by going here: http://log.smallworks.eu/web/search?from=24/4/2017&to=30/4/2017 ChangeLog ========= 27 April 2017: -------------- * = VERY IMPORTANT We moved iceberg repositories to: [https://github.com/pharo-vcs/iceberg](https://github.com/pharo-vcs/iceberg) So, now to install you will need: ---- Metacello new baseline: 'Iceberg'; repository: 'github://pharo-vcs/iceberg:dev-0.4'; load. ---- I also uploaded a Configuration to the [catalog](http://catalog.pharo.org), so you can load while doing the transition. * I figured out that we will need it, so I'm working on a cherry-pick system for [iceberg](http://github.com/npasserini/iceberg). Still nothing to show but stay tuned. Also, I'm working on translate [libgit2 bindings](http://github.com/npasserini/libgit2-pharo-bindings) to version 0.25.1, because of compilation issues with 0.23 and because we will need it anyway. * I spent some time fixing 'once again' the VM build to allow +libgit2+ module to load. This time it was just a stupid cache issue... latest VM will work now. 24 April 2017: -------------- * ... and I spent the rest of the day recovering the vm tests. So far I have running +linux+ and +macOS+. Windows tests are have still 3 errors, I will end this tomorrow. * I just spend 2hs fixing a corruption on pharo [site](http://pharo.org). Is becoming annoying, but there is a problem in certain cases were +VOLazyProxy+ throws an exception which provoques a semaphore to stay in +wait+ status and then keeping conections to the database. And well, that ends... not in the best of the ways. cheers! Esteban

May 1, 2017

Re: [Pharo-dev] Problem with FileSystem in WIndows

by Alistair Grant

On Sat, Apr 29, 2017 at 10:15:15PM -0300, Casimiro de Almeida Barreto wrote: > Good night all. > > I?ve had trouble in windows concerning FileSystem. > > ((FileSystem disk root) / ?C:\temp\e8720bb4-b90a-0d00-9b1f-008709e5552b.txt?) > delete > > Fails as if the file didn?t exist. Message: > > FileDoesNotExist: Path / ?C:\temp\e8720bb4-b90a-0d00-9b1f-008709e5552b.txt? > > And I wonder what?s wrong. Because it works in linux and MacOS. > > Trasncript show: ((FileSYstem disk root) / ?C:\temp\ > e8720bb4-b90a-0d00-9b1f-008709e5552b.txt?) > > Returns: > > File @ C:\temp\e8720bb4-b90a-0d00-9b1f-008709e5552b.txt\ > > And I wonder where the last \ came from and if is it that is messing all and > how to fix things. This is partially related to the problem I reported in "FileReference / and Parent" in that the filename isn't being parsed properly. See: http://forum.world.st/FileReference-and-parent-td4941066.html Note that the patch I suggest in the email thread doesn't solve the problem either. The trailing slash in the example above is because the path isn't being parsed, the windows file store thinks that there is only a single segment, which means the drive, so must be a directory. As a workaround, if you have a complete path string, try using #asFileReference, e.g.: 'C:\temp\e8720bb4-b90a-0d00-9b1f-008709e5552b.txt' asFileReference exists Cheers, Alistair

April 30, 2017

RES: RES: RES: Problem with FileSystem in WIndows

by Casimiro de Almeida Barreto

Thanks for the tip Sven. -----Mensagem original----- De: Pharo-dev [mailto:pharo-dev-bounces@lists.pharo.org] Em nome de Sven Van Caekenberghe Enviada em: domingo, 30 de abril de 2017 17:14 Para: Pharo Development List <pharo-dev(a)lists.pharo.org> Assunto: Re: [Pharo-dev] RES: RES: Problem with FileSystem in WIndows I have a feeling that you are working on the wrong level, consider: file := FileReference newTempFilePrefix: 'data' suffix: '.txt'. file writeStreamDo: [ :out | out << 'Casimiro de Almeida Barreto' ]. file contents. file ensureDelete. file exists. The #newTempFilePrefix:suffix: method uses 'FileLocator temp' as target directory, which will be different on different platforms. The trick is to use these FileLocator locations, try 'FileLocator supportedOrigins', to have an abstract, cross platform starting point (except for the Windows drives, obviously). Then you construct directories using messages. Remember, the whole image, file browser, file dialogs, they all work cross platform, so you can do so too. --- Este email foi escaneado pelo Avast antivÃrus. https://www.avast.com/antivirus

April 30, 2017

Re: [Pharo-dev] Smalltalk Internet Browser

by askoh

Yes. It is a novel way of remote desktop. Perhaps Craig can tell if it can do the following. I want to go to a web page, select a body of text and have Smalltalk code analyze the text in some way and return the answer in the browser or in the Smalltalk app or file output. There should be two-way communication between app and browser. I would like to do live debugging too. Aik-Siong Koh Hi Askoh, are you also aware of Snowglobe https://thiscontext.com/2016/10/31/app-streaming-with-snowglobe/ ? This allows one to use a standard Smalltalk image running on the native Cog VM to render in a web browser. _,,,^..^,,,_ (phone) -- View this message in context: http://forum.world.st/Smalltalk-Internet-Browser-tp4944879p4944998.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.

April 30, 2017

Re: [Pharo-dev] RES: RES: Problem with FileSystem in WIndows

by Sven Van Caekenberghe

I have a feeling that you are working on the wrong level, consider: file := FileReference newTempFilePrefix: 'data' suffix: '.txt'. file writeStreamDo: [ :out | out << 'Casimiro de Almeida Barreto' ]. file contents. file ensureDelete. file exists. The #newTempFilePrefix:suffix: method uses 'FileLocator temp' as target directory, which will be different on different platforms. The trick is to use these FileLocator locations, try 'FileLocator supportedOrigins', to have an abstract, cross platform starting point (except for the Windows drives, obviously). Then you construct directories using messages. Remember, the whole image, file browser, file dialogs, they all work cross platform, so you can do so too. > On 30 Apr 2017, at 21:37, Casimiro de Almeida Barreto <casimiro.barreto(a)gmail.com> wrote: > >> Clear example follows. By the way â/â and â\â are not interchangeableâ¦ in Windows FileSystem requires â\â to work properly: >> >> >> >> | fn l fileStream | >> >> fn := 'temp\',UUIDGenerator next asString,'.txt'. >> >> Transcript clear; show: fn; cr. >> >> fileStream := ((FileSystem root) / fn) writeStream. >> >> fileStream >> >> ifNil: [ Transcript show: 'Fail to create writeStream'; cr ] >> >> ifNotNil: [ Transcript show: 'Opened' ]. >> >> fileStream nextPutAll: 'Casimiro de Almeida Barreto'. >> >> fileStream close. >> >> fileStream := ((FileSystem root) / fn) readStream. >> >> l := fileStream upToEnd. >> >> Transcript show: l; cr. >> >> ((FileSystem root) / fn) delete. >> > > This example creates a file named c:\temp\380cafd3-c80a-0d00-b117-0a8a049bf78d.txt and I was able to write âCasimiro de Almeida Barretoâ to it. And then open the file in the read mode and read âCasimiro de Almeida Barretoâ from it. Butâ¦ ((FileSystem root) / fn) delete returns a: âFileDoesNotExist: Path /âtemp\380cafd3-c80a-0d00-b117-0a8a049bf78d.txtâ. Apparently the problem is at WindowsStore(DiskStore) delete: > > <image001.jpg> > > Livre de vÃrus. www.avast.com.

April 30, 2017

Re: [Pharo-dev] Smalltalk Internet Browser

by Eliot Miranda

Hi Askoh, are you also aware of Snowglobe https://thiscontext.com/2016/10/31/app-streaming-with-snowglobe/ ? This allows one to use a standard Smalltalk image running on the native Cog VM to render in a web browser. _,,,^..^,,,_ (phone) > On Apr 28, 2017, at 7:11 PM, askoh <askoh(a)askoh.com> wrote: > > Being connected to the internet is going to be a necessity for any piece of > software in the immediate future. So every Smalltalk development environment > or application should have that capability as default. To push that envelop, > every image should have a Smalltalk native Internet Browser. By developing > that Internet Browser, we are demonstrating the power of Smalltalk. There is > no reason why Smalltalk cannot be as intimate to the World Wide Web as > Javascript. We can have the IDE in Smalltalk, Internet Browser in Smalltalk > and a WWW language in Smalltalk. As is uniquely Smalltalk, every bit of code > is visible, modifiable and debuggable. This supped up environment will grow > and grow to be a full OS. > > How can we gather the relevant Smalltalk code and necessary plugins to make > an Internet Browser? I remember Croquet had internet browser and video > player. What is missing? > > All the best, > Aik-Siong Koh > > > > -- > View this message in context: http://forum.world.st/Smalltalk-Internet-Browser-tp4944879.html > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. >

April 30, 2017

Re: [Pharo-dev] Smalltalk Internet Browser

by askoh

Thanks everyone for your input. I am aware of Amber - worthy effort. I am more interested in Smalltalk IDE or app having a Browser. I am thinking more of Smalltalk calling Chromium Embedded Framework (CEF) via APIs. The browser is in direct two-way communication with the Smalltalk app. The complete API may be huge, but we can start with 1, 10, 100 APIs and still do useful stuff. Smalltalk missed out to Java because it did not embrace the internet. Let's not continue to miss out. Someone mentioned Pharo should be good with data analysis. Well, I think a lot, if not all, that data is going to come from the internet. Hence an embedded browser is a great helper. Python rode on C/C++. Ruby rode on Rails and on Web. What can Smalltalk ride on? VisualWorks Appex seems to be an attempt to ride on Javascript. VW Appex manages both Smalltalk and Javascript in the same image. Javascript edits in Appex are reflected to the Internet Browser when it is refreshed. Most interestingly, Javascript edits in the Internet Browser Dev Tools are picked up by Appex and put into the image. What do you think of this strategy? Aik-Siong Koh nacho wrote > Did you tried Amber? > Is Smalltalk on the browser. > I believe that what you mean is embedding the V8 engine into Pharo. But > man, thats a huge work. Really... > my 2c. > > > *Lic. Ignacio Sniechowski, MBA* > *Prosavic SRL* > â* (5411) 4542-6712* > ð±* (54911) 6749-4721* > > > > > > > > > > > > > > > > > > > > > On Sat, Apr 29, 2017 at 7:53 PM, askoh [via Smalltalk] < > ml+s1294792n4944919h82@.nabble >> wrote: > >> We means all communities of Smalltalk environments. Pharo is of course >> the >> biggest, hence my post here first. I am verifying the soundness of the >> idea >> of SIB and then do a student project on it. I am from the VisualWork >> community. I look forward to porting freeCAD: 3D CAD with Motion >> Simulation >> to Pharo when it has 64bit version on Windows. >> >> Aik-Siong Koh >> >> Denis Kudriashov wrote >> 2017-04-29 19:00 GMT+02:00 askoh <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=4944919&i=0>>: >> >> > We start out by embedding Chrome inside the IDE. Make them communicate >> > well. >> > We then port bit by bit to Smalltalk based on needs and priorities. >> > >> >> I am wondering, when you said "we" what you mean? Did you found team to >> work on such project? >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://forum.world.st/Smalltalk-Internet-Browser-tp4944879p4944919.html >> To start a new topic under Pharo Smalltalk Developers, email >> > ml+s1294792n1294837h40@.nabble >> To unsubscribe from Pharo Smalltalk Developers, click here >> <http://forum.world.st/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1294837&code=MDgwMG5hY2hvQGdtYWlsLmNvbXwxMjk0ODM3fC0xOTAxMTExODEy> >> . >> NAML >> <http://forum.world.st/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> -- View this message in context: http://forum.world.st/Smalltalk-Internet-Browser-tp4944879p4944991.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.

April 30, 2017

RES: RES: Problem with FileSystem in WIndows

by Casimiro de Almeida Barreto

Clear example follows. By the way â/â and â\â are not interchangeableâ¦ in Windows FileSystem requires â\â to work properly: | fn l fileStream | fn := 'temp\',UUIDGenerator next asString,'.txt'. Transcript clear; show: fn; cr. fileStream := ((FileSystem root) / fn) writeStream. fileStream ifNil: [ Transcript show: 'Fail to create writeStream'; cr ] ifNotNil: [ Transcript show: 'Opened' ]. fileStream nextPutAll: 'Casimiro de Almeida Barreto'. fileStream close. fileStream := ((FileSystem root) / fn) readStream. l := fileStream upToEnd. Transcript show: l; cr. ((FileSystem root) / fn) delete. This example creates a file named c:\temp\380cafd3-c80a-0d00-b117-0a8a049bf78d.txt and I was able to write âCasimiro de Almeida Barretoâ to it. And then open the file in the read mode and read âCasimiro de Almeida Barretoâ from it. Butâ¦ ((FileSystem root) / fn) delete returns a: âFileDoesNotExist: Path /âtemp\380cafd3-c80a-0d00-b117-0a8a049bf78d.txtâ. Apparently the problem is at WindowsStore(DiskStore) delete: --- Este email foi escaneado pelo Avast antivÃrus. https://www.avast.com/antivirus

April 30, 2017

Re: [Pharo-dev] First commit

by Juraj Kubelka

Hi, -- Juraj Kubelka > El 30-04-2017, a las 15:25, Evan Donahue <emdonahu(a)gmail.com> escribiÃ³: > > Hello, > > I have just tried to submit my first patch to the main Pharo inbox. I submitted a patch for issue > > https://pharo.fogbugz.com/f/cases/17848 > > because it basically amounted to changing an error message and seemed like a good place to start figuring out how the submission process works. > > I saved my slice to the Pharo60Inbox, but when I typed my commit message and hit save, I did not see any confirmation. How can I tell if my slice was submitted or not? I am not in front of my computer, but you can open the repository and search for the SLICE. Than you should set resolve in FogBugz. Juraj > > Thank you, > Evan

April 30, 2017