January 2018 - Pharo-dev - lists.pharo.org

FFI Struct Argument Pass By Value Fails on Mac 64 bit
by Todd Blanchard March 6, 2018

March 6, 2018

I've been trying to track this down for a couple weeks now. I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd. I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct. I've carved out the wee bit of clang to reproduce this in a tiny library. The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side. The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c I'm stuck. The FFI64 plugin is way beyond my comprehension. // microclang.c typedef struct { const void *ptr_data[2]; unsigned int_data; } CXSourceLocation; /** * \brief Identifies a half-open character range in the source code. * * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the * starting and end locations from a source range, respectively. */ typedef struct { const void *ptr_data[2]; unsigned begin_int_data; unsigned end_int_data; } CXSourceRange; const char* first = "first_pointer"; const char* second = "second_pointer"; // return a fake range with non zero data CXSourceRange clang_getArbitraryRange() { CXSourceRange range = {0}; range.ptr_data[0] = (void*)first; range.ptr_data[1] = (void*)second; range.begin_int_data = 17; range.end_int_data = 24; return range; } // Actual clang function - range is always zero'd here despite having values in the image CXSourceLocation clang_getRangeStart(CXSourceRange range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; return Result; } CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, range.begin_int_data }; return Result; }

4 13

Best VM for the Mac>
by Sean P. DeNigris March 4, 2018

March 4, 2018

What is the best VM right now for the Mac? Where/how would one get it? I recently downloaded a fresh Launcher from CI which comes with CoInterpreter VMMaker.oscog-eem.2254 uuid: 4f2c2cce-f4a2-469a-93f1-97ed941df0ad Jul 20 2017, but get.pharo.../vm70 downloads CoInterpreter VMMaker.oscog-eem.2265 uuid: 76b62109-629a-4c39-9641-67b53321df9a Aug 27 2017â¦ ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

6 18

Iceberg workflow
by Ben Coman March 3, 2018

March 3, 2018

I'm still struggling with Iceberg workflow for a personal repo. World menu > Iceberg Clicked <Clone Repository> Remote URL <-- git@github.com:bencoman/libclang-pharo-bindings.git Clicked <Create Repository> So now I have a new row... Name = libclang-pharo-bindings Current Branch = master Loaded version = No package loaded Status = Not loaded but nothing loaded into Image, which seems expected from Iceberg Operations Overview... * http://forum.world.st/Iceberg-operations-overview-td5036218.html (even though "Clone" is not shown there, I guess its the same as "Checkout") but I can't work out how to load it. In the filesystem I see the repo has been checked out, under... /home/ben/Pharo/images/latest-64-02/pharo-local/iceberg/bencoman/libclang-pharo-bindings Now from Iceberg Operations Overview, I guess I need to either load, merge, reload or pull. These all are apparently exposed in the UI. Right-clicking on the repo, the only one I see is "Pull incoming commits" and... Pharo quietly exits, with no debug log. Restarting to gather more info, after selecting the repo, the [Packages] tab is empty (maybe that is normal). Then I try Right-click>Refresh Repository with no effect. Right-click > Synchronize Repository takes me to three tabs with empty list views. The [Update] tab has a <Pull> button which also causes Pharo to exit. Versions... Ubuntu 16.04-64bit Pharo-7.0+alpha.build.300.sha.f81891fbde76d6a92e2827229293cb377d3b2c1c (64 Bit) Unix built on Aug 27 2017 20:00:40 Compiler: 4.6.3 VMMaker versionString VM: 201708271955 cheers -ben

6 16

Random corrupted data when copying from very large byte array
by Cyrille Delaunay Feb. 16, 2018

Feb. 16, 2018

Hi everyone, I just added a new bug entry for an issue we are experimenting since some times: https://pharo.fogbugz.com/f/cases/20982/Random-corrupted-data-when-copying-… Here is the description: *History:* This issue has been spotted after experimenting strange behavior with seaside upload. After uploading a big file from a web browser, the modeled file generated within pharo image begins with 4 unexpected bytes. This issue occurs randomly: sometimes the first 4 bytes are right. Sometimes the first 4 bytes are wrong. This issue only occurs with Pharo 6. This issue occurs for all platforms (Mac, Ubuntu, Windows) *Steps to reproduce:* I have been able to set up a small scenario that highlight the issue. Download Pharo 6.1 on my Mac (Sierra 10.12.6): https://pharo.org/web/download Then, iterate over this process till spotting the issue: => start the pharo image => execute this piece of code in a playground 1: 2: 3: 4: 5: 6: ZnServer startDefaultOn: 1701.ZnServer default maximumEntitySize: 80* 1024 * 1024.'/Users/cdelaunay/myzip.zip' asFileReference writeStreamDo: [ :out |out binary; nextPutAll: #[80 75 3 4 10 0 0 0 0 0 125 83 67 73 0 0 0 0 0 0].18202065 timesRepeat: [ out nextPut: 0 ]]. => Open a web browser page on: http://localhost:1701/form-test-3 => Upload the file zip file previously generated ('myzip.zip'). => If the web page displays: "contents=000000000a00 <https://pharo.kilnhg.com/Search?search=000000000a00>..." (or anything unexpected), THIS IS THE ISSUE ! => If the web page displays: "contents=504b03040a00 <https://pharo.kilnhg.com/Search?search=504b03040a00>..", the upload worked fine. You can close the image (without saving) *Debugging:* Bob Arning has been able to reproduce the issue with my scenario. He dived into the code involved during this process, till reaching some "basic" methods where he saw the issue occuring. Here are the conclusion till there: => A corruption occurs while reading an input stream with method ZnUtils class>>readUpToEnd:limit: The first 4 bytes may be altered randomely. => The first 4 bytes are initially correctly written to an outputStream. But, the first 4 bytes of this outputStream gets altered (corrupted), sometimes when the inner byte array grows OR when performing the final "outputStream contents" => Here is a piece of code that reproduce the issue (still randomely. stopping an restarting the image may change the behavior) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: test4"self test4" | species bufferSize buffer totalRead outputStream answer inputStream ba byte1 | ba := ByteArray new: 18202085. ba atAllPut: 99. 1 to: 20 do: [ :i | ba at: i put: (#[80 75 3 4 10 7 7 7 7 7 125 83 67 73 7 7 7 7 7 7] at: i) ]. inputStream := ba readStream. bufferSize := 16384. species := ByteArray. buffer := species new: bufferSize. totalRead := 0. outputStream := nil. [ inputStream atEnd ] whileFalse: [ | readCount | readCount := inputStream readInto: buffer startingAt: 1 count: bufferSize. totalRead = 0 ifTrue: [ byte1 := buffer first. ]. totalRead := totalRead + readCount. outputStream ifNil: [ inputStream atEnd ifTrue: [ ^ buffer copyFrom: 1 to: readCount ] ifFalse: [ outputStream := (species new: bufferSize) writeStream ] ]. outputStream next: readCount putAll: buffer startingAt: 1. byte1 = outputStream contents first ifFalse: [ self halt ]. ]. answer := outputStream ifNil: [ species new ] ifNotNil: [ outputStream contents ]. byte1 = answer first ifFalse: [ self halt ]. ^answer *suspicions* This issue appeared with Pharo 6. Some people suggested that it could be a vm issue, and to try my little scenario with the last available vm. I am not sure where to find the last available vm. I did the test using these elements: https://files.pharo.org/image/60/latest.zip https://files.pharo.org/get-files/70/pharo-mac-latest.zip/ <https://files.pharo.org/get-files/70/> The issue is still present -- Cyrille Delaunay

6 15

OpalCompiler evaluate speed
by Nicolas Cellier Feb. 15, 2018

Feb. 15, 2018

I have an ArbitraryPrecisionFloatTests doing an exhaustive test for printing and reevaluating all positve half precision float. That's about 2^15 or approximately 32k loop which evaluate snippets like (ArbitraryPrecisionFloat readFrom: '1.123' readStream numBits: 10) The test was naively written with Compiler evaluate: and was using the legacy Compiler. If I rewrite self class compiler evaluate: the test times out. Let's see what increase is necessary: [ ArbitraryPrecisionFloatTest new testPrintAndEvaluate ] timeToRun. -> 3s with legacy Compiler -> 14s with OpalCompiler It's not unexpected that intermediate representation (IR) reification has a cost, but here the 4.5x is a bit too much... This test did account for 1/4 of total test duration already (3s out of 12s). With Opal, the total test duration doubles... (14s out of 23s) So let's analyze the hot spot with: MessageTally spyOn: [ ArbitraryPrecisionFloatTest new testPrintAndEvaluate ]. (I didn't use AndreasSystemProfiler becuase outputs seems a bit garbbled, no matter since the primitives do not account that much, a MessageTally will do the job) I first see a hot spot which does not seem that necessary: | |24.6% {3447ms} RBMethodNode(RBProgramNode)>>formattedCode >From the comments I understand that AST-based stuff requires a pattern (DoIt) and an explicit return (^), but this expensive formatting seems too much for just evaluating. i think that we should change that. Then comes: | |20.7% {2902ms} RBMethodNode>>generate: which is split in two halves, ATS->IR and IR->bytecode | | |9.3% {1299ms} RBMethodNode>>generateIR | | | |11.4% {1596ms} IRMethod>>generate: But then I see this cost a 2nd time which also leave room for progress: | |10.9% {1529ms} RBMethodNode>>generateIR | | |12.9% {1814ms} IRMethod>>generate: The first is in RBMethodNode>>generateWithSource, the second in OpalCompiler>>compile Last comes the parse time (sourceCode -> AST) | 13.2% {1858ms} OpalCompiler>>parse Along with semantic analysis | 6.0% {837ms} OpalCompiler>>doSemanticAnalysis ----------------------------------- For comparison, the legacy Compiler decomposes into: | |61.5% {2223ms} Parser>>parse:class:category:noPattern:context:notifying:ifFail: which more or less covers parse time + semantic analysis time. That means that Opal does a fair work for this stage. Then, the direct AST->byteCode phase is: | 16.9% {609ms} MethodNode>>generate IR costs almost a 5x on this phase, but we know it's the price to pay for the additional features that it potentially offers. If only we would do it once... And that's all for the legacy one... -------------------------------------- This little exercize shows that a 2x acceleration of OpalCompiler evaluate seems achievable: - simplify the uselessely expensive formatted code - generate bytecodes once, not twice Then it will be a bit more 2x slower than legacy, which is a better trade for yet to come superior features potentially brought by Opal. It would be interesting to carry same analysis on method compilation

6 12

Weird ZnClient benchmarking result
by Sven Van Caekenberghe Feb. 6, 2018

Feb. 6, 2018

Hi, While benchmarking ZnClient I encountered something weird and counter intuitive that I cannot explain. This was on macOS running Pharo 7 and 6. First you need to setup a fast local HTTP server, can be anything, for example: $ brew install nginx $ brew services start nginx $ curl http://localhost:8080 In Pharo, the benchmark code looks as follows: ZnClient new in: [ :client | [ client get: 'http://localhost:8080' ] bench ]. On my machine this gives about 2500 req/s. ZnClient uses object logging: it emits special event objects that contain information regarding what happens (full details and timing information) using an Announcer. Subscribers can then decide what to do with this detailed information. This mechanism is always active, but there is an option to turn it totally off for maximum performance: ZnClient new in: [ :client | client loggingOff. [ client get: 'http://localhost:8080' ] bench ]. This gives about 3000 req/s, a non-trivial + 500 req/s improvement. Now comes the weird thing. GT contains a custom view for Announcers, inspect: ZnLogEvent announcer. And select the Announcements tab. Leave this window open. Now, the first expression (with full logging enabled and an active subscriber with a UI) is way FASTER than first, it yields about 7000 to 8000 req/s. Since the benchmark runs in the main UI thread, no actual drawing happens until the expression finishes. Running both expression inside a Time profiler show how in the first case significant time (actually *most* time ;-) is spent in the logging: In the second case, with logging disabled, that changes to: One has to take into account that the request has to be processed by the server, so from the client's perspective, most time will always be spent waiting for a reply, #readResponse. With the logging enabled, probably less time is spent waiting, since the client does a bit more work. Still what should be slower is more than 2 times as fast ! I have a feeling that this has to do with scheduling (probably interacting with network IO), as going deep into the last #readResponse above, I see: But I still cannot understand why doing apparently more work results in faster throughput. Can anyone explain this behaviour ? Thx, Sven

6 8

nil suspend
by Aliaksei Syrel Feb. 5, 2018

Feb. 5, 2018

Hi, I was looking at nil (already funny) and found a very interesting method: suspend "Kills off processes that didn't terminate properly" "Display reverse; reverse." "<-- So we can catch the suspend bug" Processor terminateActive It means that the following code is probably the shortest way to kill your image (save first!) nil suspend It terminates a UI process if evaluated from anywhere using CMD+d :D Cheers, Alex

5 4

[ANN] Updating Consultants Listing
by Marcus Denker Feb. 2, 2018

Feb. 2, 2018

Hi, We have http://consultants.pharo.org <http://consultants.pharo.org/> I want to change it to - not be a forward to files.pharo.org <http://files.pharo.org/> - be auto-generated from the https://association.pharo.org <https://association.pharo.org/> But we should update it already before that is finished. So: If you want to be listed, please send your infos in the form: PharoConsultant { #name : Your name', #email : your(a)email.com', #website : 'http://your-website.com', #id : 'yourname (currently not used for anything)', #location : 'City, Country of your business', #languages : 'what languages do you speak', #industryExperienceSince : 1994, #pharoExperienceSince : 2008, #areasOfExpertise : 'what are your areas of expertise. E.g. business apllications, user interface, legacy, ...', #smalltalkExperienceSince : 2002 } (or open a pull request at: https://github.com/pharo-project/pharo-project-consultants/ <https://github.com/pharo-project/pharo-project-consultants/>) Marcus

3 4

[Launcher][Win]Cannot download the VM from Pharo
by Vincent.Blondeau＠lamresearch.com Feb. 1, 2018

Feb. 1, 2018

Hi, With the Pharo launcher, the VM are automatically downloaded from the files.pharo.org server. However, it seems that the download of some VMs doesn't work at least under Windows 10. For the P7 image, the download stops at 38% (~2.5Mo) with the error "ConnectionClosed: Cannot Read Data", and, I obtain the same error with this cmd: ZnEasy get: 'http://files.pharo.org/get-files/70/pharo-win-stable.zip'. However, I can fully download the image through the Internet Browser. The download with Pharo is working: ZnEasy get: 'https://files.pharo.org/'. give me a 200 OK response. Thanks in advance for your help, Cheers, Vincent

5 7

FFI without the Pharo sources
by Vincent.Blondeau＠lamresearch.com Jan. 31, 2018

Jan. 31, 2018

Hi, I am working to be able to create some standalone apps in Pharo. And for this, I would like to remove of the .sources and .changes. But, by removing them, the args names are reset to arg1, arg2, arg3, ... and the FFI cannot be used anymore. Does someone (Esteban?) have a solution that I could implement to fix this issue? Thanks! Vincent

7 17