November 2017 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo TechTalk 21 Nov: Discord Demo

by Marcus Denker

Hi, I added links to the TechTalk archive: http://pharo.org/TechTalk > On 21 Nov 2017, at 20:02, Juraj Kubelka <juraj.kubelka(a)icloud.com> wrote: > > Hi, > > the TechTalk record is available at the same link: https://www.youtube.com/watch?v=33kXsOiP6wA <https://www.youtube.com/watch?v=33kXsOiP6wA> > and includes outline to simplify navigation. > > Cheers, > Juraj > > > TechTalk Outline: > - 01:58 The beginning of the talk > - 04:30 Webhook > - 04:33 How to Create Webhook > - 05:39 Webhook Examples > - 10:33 Webhook Use Case: Script of the Day from Nautilus Code Browser > - 18:39 Webhook Use Case: Server Problem Notification > - 22:45 Bot App (chatbot) > - 24:47 How to Create a Bot App > - 28:28 Bot App Examples > - 33:17 Bot Use Case: Source Code Expertise > - 41:40 Standard User Client > - 42:57 User Client Example > - 45:06 User Client Use Case: Asking Directly from Pharo Playground > - 47:06 User Client Use Case: Receiving Questions and Answering in Pharo > - 50:50 Final Thoughts About Discord Integration in Inspector and Debugger > - 52:44 Discussion > >> On Nov 21, 2017, at 12:54, Juraj Kubelka <juraj.kubelka(a)icloud.com <mailto:juraj.kubelka@icloud.com>> wrote: >> >> Hi! >> >> The TechTalk starts in about 10 minutes. Join us on Discord, the techtalk channel. >> >> Cheers, >> Juraj >> >> >>> On Nov 21, 2017, at 10:11, Marcus Denker <marcus.denker(a)inria.fr <mailto:marcus.denker@inria.fr>> wrote: >>> >>> The link to the live stream is this: >>> >>> https://www.youtube.com/watch?v=33kXsOiP6wA <https://www.youtube.com/watch?v=33kXsOiP6wA> >>> >>> It start in a bit less than 3 hours. >>> >>> Marcus >>> >>>> On 18 Nov 2017, at 09:13, Marcus Denker <marcus.denker(a)inria.fr <mailto:marcus.denker@inria.fr>> wrote: >>>> >>>> Pharo TechTalk: Discord Demo >>>> When? 21 Nov 2017 5:00 PM - 7:00 PM (UTC+01:00) >>>> >>>> Topic: "Discord communication Demoâ, how to script discord from Pharo. >>>> >>>> https://association.pharo.org/event-2642665 <https://association.pharo.org/event-2642665> >>> >> >

Nov. 22, 2017

Iceberg operations overview

by Pavel Krivanek

Hi, during brainstorming of the Iceberg UI simplification we created a document that shows what the current Iceberg operations do. It should help you to clarify the role of the working copy etc. See the attachment. Note: merge uses the reference commit of the loaded version as the base. Cheers, -- Pavel

Nov. 22, 2017

Re: [Pharo-dev] [Vm-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

by Todd Blanchard

Playground code: LibCClang startFromRange:(LibCClang defaultRange) it worked if the resulting struct is not all zeroes. A tiny C program is the equivalent. int main() { if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) printf("That failed\n"); else printf("That worked\n") return 0; } > On Nov 21, 2017, at 11:18 PM, Ben Coman <btc(a)openinworld.com> wrote: > > Hi Todd, > > Could you advise the the Playground code to use to test this? > And maybe a C program that links to the microclang shared library that does the equivalent test? > > cheers -ben > > On 22 November 2017 at 13:38, Todd Blanchard <tblanchard(a)mac.com <mailto:tblanchard@mac.com>> wrote: > > I've been trying to track this down for a couple weeks now. > > I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd. > > I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct. > > I've carved out the wee bit of clang to reproduce this in a tiny library. > > The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side. > > The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c > > I'm stuck. The FFI64 plugin is way beyond my comprehension. > > // microclang.c > > typedef struct { > const void *ptr_data[2]; > unsigned int_data; > } CXSourceLocation; > > /** > * \brief Identifies a half-open character range in the source code. > * > * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the > * starting and end locations from a source range, respectively. > */ > typedef struct { > const void *ptr_data[2]; > unsigned begin_int_data; > unsigned end_int_data; > } CXSourceRange; > > const char* first = "first_pointer"; > const char* second = "second_pointer"; > > // return a fake range with non zero data > CXSourceRange clang_getArbitraryRange() > { > CXSourceRange range = {0}; > range.ptr_data[0] = (void*)first; > range.ptr_data[1] = (void*)second; > range.begin_int_data = 17; > range.end_int_data = 24; > return range; > } > > // Actual clang function - range is always zero'd here despite having values in the image > CXSourceLocation clang_getRangeStart(CXSourceRange range) { > // Special decoding for CXSourceLocations for CXLoadedDiagnostics. > if ((uintptr_t)range.ptr_data[0] & 0x1) { > CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; > return Result; > } > > CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, > range.begin_int_data }; > return Result; > } > > > > >

Nov. 22, 2017

Re: [Pharo-dev] [Vm-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

by Ben Coman

Hi Todd, Could you advise the the Playground code to use to test this? And maybe a C program that links to the microclang shared library that does the equivalent test? cheers -ben On 22 November 2017 at 13:38, Todd Blanchard <tblanchard(a)mac.com> wrote: > > I've been trying to track this down for a couple weeks now. > > I have concluded that structs passed by value to functions on the 64 bit > VM are not properly populated. The struct's memory is all zero'd. > > I found this while trying to work with LibClang and found that functions > that fetched code locations from code ranges always returned invalid zero'd > locations. After spending some time with lldb I have traced the problem > into the native code and found that the argument is not correct. > > I've carved out the wee bit of clang to reproduce this in a tiny library. > > The gist of it is below and the entire file is included. Basically the > struct passed to the function clang_getRangeStart is zero'd memory > regardless of the data I send from the image side. > > The build command I used on sierra is clang -shared -undefined > dynamic_lookup -o microclang.dylib microclang.c > > I'm stuck. The FFI64 plugin is way beyond my comprehension. > > // microclang.c > > typedef struct { > const void *ptr_data[2]; > unsigned int_data; > } CXSourceLocation; > > /** > * \brief Identifies a half-open character range in the source code. > * > * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the > * starting and end locations from a source range, respectively. > */ > typedef struct { > const void *ptr_data[2]; > unsigned begin_int_data; > unsigned end_int_data; > } CXSourceRange; > > const char* first = "first_pointer"; > const char* second = "second_pointer"; > > // return a fake range with non zero data > CXSourceRange clang_getArbitraryRange() > { > CXSourceRange range = {0}; > range.ptr_data[0] = (void*)first; > range.ptr_data[1] = (void*)second; > range.begin_int_data = 17; > range.end_int_data = 24; > return range; > } > > // Actual clang function - range is always zero'd here despite having > values in the image > CXSourceLocation clang_getRangeStart(CXSourceRange range) { > // Special decoding for CXSourceLocations for CXLoadedDiagnostics. > if ((uintptr_t)range.ptr_data[0] & 0x1) { > CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; > return Result; > } > > > CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, > range.begin_int_data }; > return Result; > } > > > > >

Nov. 22, 2017

FFI Struct Argument Pass By Value Fails on Mac 64 bit

by Todd Blanchard

I've been trying to track this down for a couple weeks now. I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd. I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct. I've carved out the wee bit of clang to reproduce this in a tiny library. The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side. The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c I'm stuck. The FFI64 plugin is way beyond my comprehension. // microclang.c typedef struct { const void *ptr_data[2]; unsigned int_data; } CXSourceLocation; /** * \brief Identifies a half-open character range in the source code. * * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the * starting and end locations from a source range, respectively. */ typedef struct { const void *ptr_data[2]; unsigned begin_int_data; unsigned end_int_data; } CXSourceRange; const char* first = "first_pointer"; const char* second = "second_pointer"; // return a fake range with non zero data CXSourceRange clang_getArbitraryRange() { CXSourceRange range = {0}; range.ptr_data[0] = (void*)first; range.ptr_data[1] = (void*)second; range.begin_int_data = 17; range.end_int_data = 24; return range; } // Actual clang function - range is always zero'd here despite having values in the image CXSourceLocation clang_getRangeStart(CXSourceRange range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; return Result; } CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, range.begin_int_data }; return Result; }

Nov. 22, 2017

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo TechTalk 21 Nov: Discord Demo

by Sean P. DeNigris

Juraj Kubelka wrote > the TechTalk record is available at the same link: > https://www.youtube.com/watch?v=33kXsOiP6wA Thanks! The quality seems to top out at 480p, which is pretty blurry. It makes the code nearly unreadable. Any way to bump the resolution on the next one? ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Nov. 22, 2017

Re: [Pharo-dev] OpalCompiler evaluate speed

by Ben Coman

On 22 November 2017 at 05:49, Nicolas Cellier < nicolas.cellier.aka.nice(a)gmail.com> wrote: > > > 2017-11-21 14:19 GMT+01:00 Nicolas Cellier <nicolas.cellier.aka.nice@ > gmail.com>: > >> I have an ArbitraryPrecisionFloatTests doing an exhaustive test for >> printing and reevaluating all positve half precision float. >> >> That's about 2^15 or approximately 32k loop which evaluate snippets like >> >> (ArbitraryPrecisionFloat readFrom: '1.123' readStream numBits: 10) >> >> The test was naively written with Compiler evaluate: and was using the >> legacy Compiler. >> >> If I rewrite self class compiler evaluate: the test times out. >> Let's see what increase is necessary: >> >> [ ArbitraryPrecisionFloatTest new testPrintAndEvaluate ] timeToRun. >> -> 3s with legacy Compiler >> -> 14s with OpalCompiler >> >> It's not unexpected that intermediate representation (IR) reification has >> a cost, but here the 4.5x is a bit too much... >> This test did account for 1/4 of total test duration already (3s out of >> 12s). >> With Opal, the total test duration doubles... (14s out of 23s) >> >> So let's analyze the hot spot with: >> >> MessageTally spyOn: [ ArbitraryPrecisionFloatTest new >> testPrintAndEvaluate ]. >> >> (I didn't use AndreasSystemProfiler becuase outputs seems a bit garbbled, >> no matter since the primitives do not account that much, a MessageTally >> will do the job) >> >> I first see a hot spot which does not seem that necessary: >> >> | |24.6% {3447ms} RBMethodNode(RBProgramNode)>>formattedCode >> >> From the comments I understand that AST-based stuff requires a pattern >> (DoIt) and an explicit return (^), but this expensive formatting seems too >> much for just evaluating. i think that we should change that. >> >> Then comes: >> >> | |20.7% {2902ms} RBMethodNode>>generate: >> >> which is split in two halves, ATS->IR and IR->bytecode >> >> | | |9.3% {1299ms} RBMethodNode>>generateIR >> >> | | | |11.4% {1596ms} IRMethod>>generate: >> >> But then I see this cost a 2nd time which also leave room for progress: >> >> | |10.9% {1529ms} RBMethodNode>>generateIR >> >> | | |12.9% {1814ms} IRMethod>>generate: >> >> The first is in RBMethodNode>>generateWithSource, the second in >> OpalCompiler>>compile >> >> Last comes the parse time (sourceCode -> AST) >> >> | 13.2% {1858ms} OpalCompiler>>parse >> >> Along with semantic analysis >> >> | 6.0% {837ms} OpalCompiler>>doSemanticAnalysis >> >> ----------------------------------- >> >> For comparison, the legacy Compiler decomposes into: >> >> | |61.5% {2223ms} Parser>>parse:class:category:n >> oPattern:context:notifying:ifFail: >> >> which more or less covers parse time + semantic analysis time. >> That means that Opal does a fair work for this stage. >> >> Then, the direct AST->byteCode phase is: >> >> | 16.9% {609ms} MethodNode>>generate >> >> IR costs almost a 5x on this phase, but we know it's the price to pay for >> the additional features that it potentially offers. If only we would do it >> once... >> >> And that's all for the legacy one... >> >> -------------------------------------- >> >> This little exercize shows that a 2x acceleration of OpalCompiler >> evaluate seems achievable: >> - simplify the uselessely expensive formatted code >> - generate bytecodes once, not twice >> >> Then it will be a bit more 2x slower than legacy, which is a better trade >> for yet to come superior features potentially brought by Opal. >> >> It would be interesting to carry same analysis on method compilation >> > > Digging further here is what I find: > > compile sends generate: and answer a CompiledMethod > translate sends compile but throw the CompiledMethod away, and just answer > the AST. > Most senders of translate will also generate: (thus we generate: twice > quite often, loosing a 2x factor in compilation). > > A 2x gain is a huge gain when installing big code bases, especially if the > custom is to throw image away and reconstruct. > No matter if a bot does the job, it does it for twice many watts and at > the end, we're waiting for the bot. > > However, before changing anything, further clarification is required: > translate does one more thing, it catches ReparseAfterSourceEditing and > retry compilation (once). > So my question: are there some cases when generate: will cause > ReparseAfterSourceEditing? > I don't know the full answer about other cases, but I can provide the background why ReparseAfterSourceEditing was added. IIRC, a few years ago with the move to an AST based system, there was a problem with syntax highlighting where the AST referenced its original source which caused highlighting offsets when reference to source modified in the editor. Trying to work backwards from modified source to update all AST elements source-location proved an intractable problem. The workaround I found was to move only in a forward direction regenerating AST from source every keystroke. Performance was acceptable so this became the permanent solution. I don't have access to an image to check, but you should find ReparseAfterSourceEditing raised only in one location near editor #changed: Maybe this should activate only for interactively modified code, and disabled/bypassed for bulk loading code. For testing purposes commenting it out should not harm the system, just produce visual artifacts in syntax highlighting. > That could happen in generation phase if some byte code limit is exceeded, > and an interactive handling corrects code... > I did not see any such condition, but code base is huge... > At worst, the impact should only be a temporary visual artifact. Corrected on the next keystroke. (unless ReparseAfterSourceEditing has been adopted for other than original purpose, but I'd guess not) cheers -ben

Nov. 21, 2017

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo TechTalk 21 Nov: Discord Demo

by Dimitris Chloupis

Well done :) Now you can make a discord client inside the Pharo image if you want. On Tue, 21 Nov 2017 at 21:03, Juraj Kubelka <juraj.kubelka(a)icloud.com> wrote: > Hi, > > the TechTalk record is available at the same link: > https://www.youtube.com/watch?v=33kXsOiP6wA > and includes outline to simplify navigation. > > Cheers, > Juraj > > > TechTalk Outline: > - 01:58 The beginning of the talk > - 04:30 Webhook > - 04:33 How to Create Webhook > - 05:39 Webhook Examples > - 10:33 Webhook Use Case: Script of the Day from Nautilus Code Browser > - 18:39 Webhook Use Case: Server Problem Notification > - 22:45 Bot App (chatbot) > - 24:47 How to Create a Bot App > - 28:28 Bot App Examples > - 33:17 Bot Use Case: Source Code Expertise > - 41:40 Standard User Client > - 42:57 User Client Example > - 45:06 User Client Use Case: Asking Directly from Pharo Playground > - 47:06 User Client Use Case: Receiving Questions and Answering in Pharo > - 50:50 Final Thoughts About Discord Integration in Inspector and > Debugger > - 52:44 Discussion > > On Nov 21, 2017, at 12:54, Juraj Kubelka <juraj.kubelka(a)icloud.com> wrote: > > Hi! > > The TechTalk starts in about 10 minutes. Join us on Discord, the techtalk > channel. > > Cheers, > Juraj > > > On Nov 21, 2017, at 10:11, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > The link to the live stream is this: > > https://www.youtube.com/watch?v=33kXsOiP6wA > > It start in a bit less than 3 hours. > > Marcus > > On 18 Nov 2017, at 09:13, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > Pharo TechTalk: Discord Demo > When? 21 Nov 2017 5:00 PM - 7:00 PM (UTC+01:00) > > Topic: "Discord communication Demoâ, how to script discord from Pharo. > > https://association.pharo.org/event-2642665 > > > > >

Nov. 21, 2017

Re: [Pharo-dev] OpalCompiler evaluate speed

by Nicolas Cellier

2017-11-21 14:19 GMT+01:00 Nicolas Cellier < nicolas.cellier.aka.nice(a)gmail.com>: > I have an ArbitraryPrecisionFloatTests doing an exhaustive test for > printing and reevaluating all positve half precision float. > > That's about 2^15 or approximately 32k loop which evaluate snippets like > > (ArbitraryPrecisionFloat readFrom: '1.123' readStream numBits: 10) > > The test was naively written with Compiler evaluate: and was using the > legacy Compiler. > > If I rewrite self class compiler evaluate: the test times out. > Let's see what increase is necessary: > > [ ArbitraryPrecisionFloatTest new testPrintAndEvaluate ] timeToRun. > -> 3s with legacy Compiler > -> 14s with OpalCompiler > > It's not unexpected that intermediate representation (IR) reification has > a cost, but here the 4.5x is a bit too much... > This test did account for 1/4 of total test duration already (3s out of > 12s). > With Opal, the total test duration doubles... (14s out of 23s) > > So let's analyze the hot spot with: > > MessageTally spyOn: [ ArbitraryPrecisionFloatTest new > testPrintAndEvaluate ]. > > (I didn't use AndreasSystemProfiler becuase outputs seems a bit garbbled, > no matter since the primitives do not account that much, a MessageTally > will do the job) > > I first see a hot spot which does not seem that necessary: > > | |24.6% {3447ms} RBMethodNode(RBProgramNode)>>formattedCode > > From the comments I understand that AST-based stuff requires a pattern > (DoIt) and an explicit return (^), but this expensive formatting seems too > much for just evaluating. i think that we should change that. > > Then comes: > > | |20.7% {2902ms} RBMethodNode>>generate: > > which is split in two halves, ATS->IR and IR->bytecode > > | | |9.3% {1299ms} RBMethodNode>>generateIR > > | | | |11.4% {1596ms} IRMethod>>generate: > > But then I see this cost a 2nd time which also leave room for progress: > > | |10.9% {1529ms} RBMethodNode>>generateIR > > | | |12.9% {1814ms} IRMethod>>generate: > > The first is in RBMethodNode>>generateWithSource, the second in > OpalCompiler>>compile > > Last comes the parse time (sourceCode -> AST) > > | 13.2% {1858ms} OpalCompiler>>parse > > Along with semantic analysis > > | 6.0% {837ms} OpalCompiler>>doSemanticAnalysis > > ----------------------------------- > > For comparison, the legacy Compiler decomposes into: > > | |61.5% {2223ms} Parser>>parse:class:category: > noPattern:context:notifying:ifFail: > > which more or less covers parse time + semantic analysis time. > That means that Opal does a fair work for this stage. > > Then, the direct AST->byteCode phase is: > > | 16.9% {609ms} MethodNode>>generate > > IR costs almost a 5x on this phase, but we know it's the price to pay for > the additional features that it potentially offers. If only we would do it > once... > > And that's all for the legacy one... > > -------------------------------------- > > This little exercize shows that a 2x acceleration of OpalCompiler evaluate > seems achievable: > - simplify the uselessely expensive formatted code > - generate bytecodes once, not twice > > Then it will be a bit more 2x slower than legacy, which is a better trade > for yet to come superior features potentially brought by Opal. > > It would be interesting to carry same analysis on method compilation > Digging further here is what I find: compile sends generate: and answer a CompiledMethod translate sends compile but throw the CompiledMethod away, and just answer the AST. Most senders of translate will also generate: (thus we generate: twice quite often, loosing a 2x factor in compilation). A 2x gain is a huge gain when installing big code bases, especially if the custom is to throw image away and reconstruct. No matter if a bot does the job, it does it for twice many watts and at the end, we're waiting for the bot. However, before changing anything, further clarification is required: translate does one more thing, it catches ReparseAfterSourceEditing and retry compilation (once). So my question: are there some cases when generate: will cause ReparseAfterSourceEditing? That could happen in generation phase if some byte code limit is exceeded, and an interactive handling corrects code... I did not see any such condition, but code base is huge...

Nov. 21, 2017

Re: [Pharo-dev] Pharo 6.1 64 bits segmentation fault loading code on Linux

by Thierry Goubier

Le 21/11/2017 Ã 21:00, Gabriel Cotelli a Ã©critÂ : > I can't find a latest 6.1 vm so I tried with the Pharo 7 Latest VM (wget > -O- get.pharo.org/64/vmTLatest70 <http://get.pharo.org/64/vmTLatest70> | > bash) and the 6.1 image but crashes with a segmentation fault also. I also have a segfault with vmTLatest70. At the moment, I use cog_linux64x64_pharo.cog.spur_201711061254, and this one doesn't segfault. Otherwise, you can use the snap version of Pharo. This one doesn't segfault. You may have to install snap on your linux mint, and the snap has effect when you're calling external commands from Pharo. Thierry > > On Tue, Nov 21, 2017 at 12:03 PM, Thierry Goubier > <thierry.goubier(a)gmail.com <mailto:thierry.goubier@gmail.com>> wrote: > > Hi Gabriel, > > I had such segfaultsÂ with the stable 6.1/64bits on Linux. I solved > them by ensuring the use of a more recent version than the stable one. > > Thierry > > > 2017-11-21 15:56 GMT+01:00 Gabriel Cotelli <g.cotelli(a)gmail.com > <mailto:g.cotelli@gmail.com>>: > > I've created the following issue: > > https://pharo.fogbugz.com/f/cases/20737/Segmentation-fault-trying-to-load-c… > <https://pharo.fogbugz.com/f/cases/20737/Segmentation-fault-trying-to-load-c…> > I can reproduce it easily so if you need some more info let me know. > > Regards, > Gabriel > > >

Nov. 21, 2017