December 2019 - Pharo-dev - lists.pharo.org

Are people using the Pharo VM builds from BinTray?

by David T. Lewis

Hi, There is an automated CI build for opensmalltalk VMs with artifacts for Pharo, Squeak, and Newspeak at https://bintray.com/opensmalltalk/vm/cog/. I suspect that the Pharo builds may be redundant with the official PharoVM distributions. Is anyone using the Pharo VMs from BinTray? Otherwise if it's not a problem for any Pharo users, I think the CI proprietors would like to turn off some of their unnecessary builds to save resources. My understanding is that the builds can easily be turned back on as needed in the future. Fabio asked Marcel who asked me, so I am asking you :-) Thanks, Dave

Dec. 12, 2019

[Pharo 8.0] Build #1069: Announcements: Small Cleanup

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1069 was: SUCCESS. The Pull Request #5367 was integrated: "Announcements: Small Cleanup" Pull request url: https://github.com/pharo-project/pharo/pull/5367 Issue Url: https://github.com/pharo-project/pharo/issues/AnnouncementsClean Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Dec. 12, 2019

[Pharo 8.0] Build #1068: [easyReview] slightly better DNU message

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1068 was: SUCCESS. The Pull Request #5361 was integrated: "[easyReview] slightly better DNU message" Pull request url: https://github.com/pharo-project/pharo/pull/5361 Issue Url: https://github.com/pharo-project/pharo/issues/typo Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Dec. 12, 2019

[Pharo 8.0] Build #1067: Demeter's law: fix two cases

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1067 was: FAILURE. The Pull Request #5366 was integrated: "Demeter's law: fix two cases" Pull request url: https://github.com/pharo-project/pharo/pull/5366 Issue Url: https://github.com/pharo-project/pharo/issues/Demeterslawfix Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Dec. 12, 2019

[Pharo 8.0] Build #1066: Change url in GoverApiTest >>testRepositoryErrors

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1066 was: SUCCESS. The Pull Request #5363 was integrated: "Change url in GoverApiTest >>testRepositoryErrors" Pull request url: https://github.com/pharo-project/pharo/pull/5363 Issue Url: https://github.com/pharo-project/pharo/issues/GoferApiTestServerChange Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Dec. 12, 2019

Re: [Pharo-dev] [Vm-dev] Debugging GCC code generation

by Ben Coman

On Thu, 12 Dec 2019 at 04:03, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > > Hi Pablo, > > On Wed, Dec 11, 2019 at 11:33 AM tesonep(a)gmail.com <tesonep(a)gmail.com> > wrote: > >> Hi Nicolas, >> thanks for the comment, you are right the problem is the bad >> original code. But my opinion is that it just is not reporting the >> situation correctly, generating a warning or non-optimizing the code >> looks more like a expected behavior. Because as I have said, using a >> constant as index in the last statement generates a meaningful warning >> and the non-optimizated version of the function. >> >> And again as you said, the only thing to learn about all this is that >> we should not write crappy code. >> > > Hang on. "crappy code" is too strong. Back in tre ANSI days it was not > undefined behavior, and in fact I think this only because undefined > behavior because C compilers were faced with lots of code where variables > were typed int but were being used on 64-bit machines. Had the C world > used long as its default type then there would have been no need to make > casting the address of a long into the address of an int undefined > behavior. So this isn't crappy code (if you think of what it means in > memory on a 64-bit little endnan machine it is perfectly sensible). It > /is/ code that is inappropriate given C99. > > So yes, we have to bow to the C gods and generate C99 compliant code, but > what we were doing here wasn't crappy, it was merely code that became > undefined behavior so that C compilers could generate more efficient code > for 64-bit systems in the presence of code that contains lots of integer > variables declared as int. So let's just say that we have to generate C99 > compliant code. You'll notice that the Smalltalk code in the original does > exactly what you suggest Pablo. > So maybe interesting to see what happens if C89 is specified... https://renenyffenegger.ch/notes/development/languages/C-C-plus-plus/GCC/op… Possibly useful sanitizer... https://developers.redhat.com/blog/2014/10/16/gcc-undefined-behavior-saniti… cheers -ben

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Nicolas Cellier

It's the tribute to pay for optimization. With unbounded pointer and pointer aliasing, the C compiler is in the worse world for optimizing. As long as a FORTRAN compiler will beat a C compiler in generated code efficiency, the C guys will have to amend the standard ;) So we have plenty of new rules (C89 is 30 years ago, not exactly as new as we pretend), the restrict keyword, etc... Le mer. 11 dÃ©c. 2019 Ã 21:14, Eliot Miranda <eliot.miranda(a)gmail.com> a Ã©crit : > > > On Wed, Dec 11, 2019 at 12:03 PM Nicolas Cellier < > nicolas.cellier.aka.nice(a)gmail.com> wrote: > >> Yes, >> But we have to replace natural crappy code (split a long in 2 ints) that >> was once legal, by an even more crappy code (memcpy), so all in all, it's a >> crappy art. >> > > :-) Indeed. Personally I liked it when C was a portable assembler. > That's what it's fit for and that's what it should be good at. Trying to > pretend it's Pascal is, um, pretentious? > > >> >> Le mer. 11 dÃ©c. 2019 Ã 19:30, tesonep(a)gmail.com <tesonep(a)gmail.com> a >> Ã©crit : >> >>> Hi Nicolas, >>> thanks for the comment, you are right the problem is the bad >>> original code. But my opinion is that it just is not reporting the >>> situation correctly, generating a warning or non-optimizing the code >>> looks more like a expected behavior. Because as I have said, using a >>> constant as index in the last statement generates a meaningful warning >>> and the non-optimizated version of the function. >>> >>> And again as you said, the only thing to learn about all this is that >>> we should not write crappy code. >>> >>> On Wed, Dec 11, 2019 at 7:11 PM Nicolas Cellier >>> <nicolas.cellier.aka.nice(a)gmail.com> wrote: >>> > >>> > Of course, when I say "your" code, it's the code you have shown, and >>> probably "our" (VMMaker) code ;) >>> > >>> > Le mer. 11 dÃ©c. 2019 Ã 19:05, Nicolas Cellier < >>> nicolas.cellier.aka.nice(a)gmail.com> a Ã©crit : >>> >> >>> >> Hi Pablo (again), >>> >> no, not a bug. >>> >> >>> >> The problem is in the source code. The compiler has the right to >>> presume that your code is exempt of UB, because you cannot depend on UB >>> (obviously). >>> >> So it can eliminate all code which corresponds to UB. >>> >> >>> >> The compiler has the right to assume that a pointer to an int cannot >>> point to a long (UB). >>> >> So modifying a long cannot have any sort of impact on the content of >>> the int pointer. >>> >> So the compiler can decouple both path return int content and assign >>> long. >>> >> But assigning the long has no effect, so the code can be suppressed >>> altogether. >>> >> >>> >> Le mer. 11 dÃ©c. 2019 Ã 18:54, tesonep(a)gmail.com <tesonep(a)gmail.com> >>> a Ã©crit : >>> >>> >>> >>> Hi Aliaksei, >>> >>> to me it looks like a bug of GCC optimization. Basically, it is >>> >>> assuming that the x variable is used but never read or its value is >>> >>> never used. Also it assumes the same of the i variable, as we are >>> only >>> >>> accessing indirectly to the memory where it locates (the code is even >>> >>> assuming that the variable exists, but it can be optimize out as in >>> >>> this scenario). Even though, the original C code is valid C code, we >>> >>> are not helping the compiler writing code like that. So I have >>> >>> rewritten the code in a way that does not use indirect memory access >>> >>> to the stack space. >>> >>> >>> >>> One thing more that makes me think is a bug, if you use an int >>> >>> constant as the index and not a parameter, the error does not occur >>> >>> (the code is not badly optimized) and there is a warning about the >>> >>> not-so-great access to the stack. >>> >>> >>> >>> On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> >>> wrote: >>> >>> > >>> >>> > Hi Pablo, >>> >>> > >>> >>> > Wow! Thank you for the detective story :) >>> >>> > >>> >>> > Do I understand correctly that the original code causes undefined >>> behavior and therefore can be changed (or even removed) by the compiler? >>> >>> > (because it returns something that is referencing memory on the >>> stack) >>> >>> > >>> >>> > Please keep posting similar things in future! It is very educative >>> :) >>> >>> > >>> >>> > Cheers, >>> >>> > Alex >>> >>> > >>> >>> > >>> >>> > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> >>> wrote: >>> >>> >> >>> >>> >> Hi, >>> >>> >> this mail is related to Pharo because it is knowledge I found >>> >>> >> debugging the build of the VM, but the rest is to document it and >>> >>> >> perhaps someone will found it interesting (also I couldn't find it >>> >>> >> easily using Google). Sorry for the long mail! >>> >>> >> >>> >>> >> The problem >>> >>> >> ========== >>> >>> >> >>> >>> >> The following code does not produce good code in 8.3 when using >>> optimizations: >>> >>> >> >>> >>> >> long __attribute__ ((noinline)) myFunc(long i, int index){ >>> >>> >> long v; >>> >>> >> long x = i >> 3; >>> >>> >> >>> >>> >> v = x; >>> >>> >> return ((int*)(&v))[index]; >>> >>> >> } >>> >>> >> >>> >>> >> #include <stdio.h> >>> >>> >> >>> >>> >> int main(){ >>> >>> >> >>> >>> >> long i; >>> >>> >> int x; >>> >>> >> >>> >>> >> scanf("%ld", &i); >>> >>> >> scanf("%d", &x); >>> >>> >> >>> >>> >> printf("%ld",myFunc(i,x)); >>> >>> >> } >>> >>> >> >>> >>> >> Basically, with -02, it generates the following code: >>> >>> >> >>> >>> >> myFunc: >>> >>> >> movslq %esi, %rsi >>> >>> >> movslq -8(%rsp,%rsi,4), %rax >>> >>> >> ret >>> >>> >> >>> >>> >> And with -01 it generates the following code: >>> >>> >> >>> >>> >> myFunc: >>> >>> >> sarq $3, %rdi >>> >>> >> movq %rdi, -8(%rsp) >>> >>> >> movslq %esi, %rsi >>> >>> >> movslq -8(%rsp,%rsi,4), %rax >>> >>> >> ret >>> >>> >> >>> >>> >> As you can see, the more optimized version is losing the bit >>> shift (or >>> >>> >> any other operation). >>> >>> >> To detect the problem it was not easy, basically, I have used the >>> >>> >> Pharo Tests to detect the failing function and then to understand >>> the >>> >>> >> generation of code in GCC, we need to debug its generation. >>> >>> >> >>> >>> >> The first snippet is generated using: >>> >>> >> >>> >>> >> gcc -O2 prb.c -S -o prb.s -Wall >>> >>> >> >>> >>> >> and the second using: >>> >>> >> >>> >>> >> gcc -O1 prb.c -S -o prb.s -Wall >>> >>> >> >>> >>> >> The GCC pipeline has different steps, I have centered my self in >>> the >>> >>> >> tree optimizations. >>> >>> >> GCC dumps each of the intermediate states of the compilation, >>> working >>> >>> >> with C like trees. For example to get the optimized version of the >>> >>> >> program, before generating the Assembler we have to do: >>> >>> >> >>> >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout >>> >>> >> >>> >>> >> This will generate: >>> >>> >> >>> >>> >> __attribute__((noinline)) >>> >>> >> myFunc (long int i, int index) >>> >>> >> { >>> >>> >> long int v; >>> >>> >> long unsigned int _1; >>> >>> >> long unsigned int _2; >>> >>> >> int * _3; >>> >>> >> int _4; >>> >>> >> long int _8; >>> >>> >> >>> >>> >> <bb 2> [local count: 1073741825]: >>> >>> >> _1 = (long unsigned int) index_7(D); >>> >>> >> _2 = _1 * 4; >>> >>> >> _3 = &v + _2; >>> >>> >> _4 = *_3; >>> >>> >> _8 = (long int) _4; >>> >>> >> v ={v} {CLOBBER}; >>> >>> >> return _8; >>> >>> >> >>> >>> >> } >>> >>> >> >>> >>> >> This is the optimized SSA (static single assign) version of the >>> >>> >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) >>> >>> >> As you can see in this version the code is already optimized, and >>> our >>> >>> >> operations not correctly generated. We expect to see something >>> like: >>> >>> >> >>> >>> >> __attribute__((noinline)) >>> >>> >> myFunc (long int i, int index) >>> >>> >> { >>> >>> >> long int x; >>> >>> >> long int v; >>> >>> >> long unsigned int _1; >>> >>> >> long unsigned int _2; >>> >>> >> int * _3; >>> >>> >> int _4; >>> >>> >> long int _10; >>> >>> >> >>> >>> >> <bb 2> [local count: 1073741825]: >>> >>> >> x_6 = i_5(D) >> 3; >>> >>> >> v = x_6; >>> >>> >> _1 = (long unsigned int) index_9(D); >>> >>> >> _2 = _1 * 4; >>> >>> >> _3 = &v + _2; >>> >>> >> _4 = *_3; >>> >>> >> _10 = (long int) _4; >>> >>> >> v ={v} {CLOBBER}; >>> >>> >> return _10; >>> >>> >> >>> >>> >> } >>> >>> >> >>> >>> >> In the first SSA version, we are lacking the shift operation: >>> >>> >> >>> >>> >> x_6 = i_5(D) >> 3; >>> >>> >> v = x_6; >>> >>> >> >>> >>> >> So, we need to see in which of the optimizations and >>> transformations >>> >>> >> our code is lost: >>> >>> >> To see all the steps we should execute: >>> >>> >> >>> >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all >>> >>> >> >>> >>> >> This will generate a lot, really a lot, of files in the same >>> directory. >>> >>> >> They have the name: prb.c.xxx.yyyy >>> >>> >> Where xxx is the number of the step, and yyyy is the name of the >>> step. >>> >>> >> Each of the files contains the result of applying the changes, so >>> what >>> >>> >> I have done is looking in binary search where my code was lost. >>> >>> >> >>> >>> >> I have found that the problem was in the first dead code >>> elimination >>> >>> >> step (prb.c.041t.cddce1) >>> >>> >> GCC does not like the crap code that we are writing, as we are >>> copying >>> >>> >> a stack variable and then accessing directly to the memory: >>> >>> >> >>> >>> >> v = x; >>> >>> >> return ((int*)(&v))[index]; >>> >>> >> >>> >>> >> So, basically, it was assuming that we were only using v and not >>> x, so >>> >>> >> all the code modifying x is described (this optimization is called >>> >>> >> "dead store code deletion"). Basically tries to remove all the >>> code >>> >>> >> that is affecting stack (temporary) variables that are never used. >>> >>> >> >>> >>> >> I am not sure if this is a bug in GCC, so I will report it. But I >>> have >>> >>> >> fixed the problem writing the code in a proper way. >>> >>> >> >>> >>> >> Sorry again for the long mail, I wanted to store the knowledge and >>> >>> >> propagate it before I eventually will flush it. I like these >>> things, >>> >>> >> but I am not debugging the GCC optimization pipeline all the days. >>> >>> >> >>> >>> >> -- >>> >>> >> Pablo Tesone. >>> >>> >> tesonep(a)gmail.com >>> >>> >> >>> >>> >>> >>> >>> >>> -- >>> >>> Pablo Tesone. >>> >>> tesonep(a)gmail.com >>> >>> >>> >>> >>> -- >>> Pablo Tesone. >>> tesonep(a)gmail.com >>> >>> > > -- > _,,,^..^,,,_ > best, Eliot >

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Eliot Miranda

On Wed, Dec 11, 2019 at 12:03 PM Nicolas Cellier < nicolas.cellier.aka.nice(a)gmail.com> wrote: > Yes, > But we have to replace natural crappy code (split a long in 2 ints) that > was once legal, by an even more crappy code (memcpy), so all in all, it's a > crappy art. > :-) Indeed. Personally I liked it when C was a portable assembler. That's what it's fit for and that's what it should be good at. Trying to pretend it's Pascal is, um, pretentious? > > Le mer. 11 dÃ©c. 2019 Ã 19:30, tesonep(a)gmail.com <tesonep(a)gmail.com> a > Ã©crit : > >> Hi Nicolas, >> thanks for the comment, you are right the problem is the bad >> original code. But my opinion is that it just is not reporting the >> situation correctly, generating a warning or non-optimizing the code >> looks more like a expected behavior. Because as I have said, using a >> constant as index in the last statement generates a meaningful warning >> and the non-optimizated version of the function. >> >> And again as you said, the only thing to learn about all this is that >> we should not write crappy code. >> >> On Wed, Dec 11, 2019 at 7:11 PM Nicolas Cellier >> <nicolas.cellier.aka.nice(a)gmail.com> wrote: >> > >> > Of course, when I say "your" code, it's the code you have shown, and >> probably "our" (VMMaker) code ;) >> > >> > Le mer. 11 dÃ©c. 2019 Ã 19:05, Nicolas Cellier < >> nicolas.cellier.aka.nice(a)gmail.com> a Ã©crit : >> >> >> >> Hi Pablo (again), >> >> no, not a bug. >> >> >> >> The problem is in the source code. The compiler has the right to >> presume that your code is exempt of UB, because you cannot depend on UB >> (obviously). >> >> So it can eliminate all code which corresponds to UB. >> >> >> >> The compiler has the right to assume that a pointer to an int cannot >> point to a long (UB). >> >> So modifying a long cannot have any sort of impact on the content of >> the int pointer. >> >> So the compiler can decouple both path return int content and assign >> long. >> >> But assigning the long has no effect, so the code can be suppressed >> altogether. >> >> >> >> Le mer. 11 dÃ©c. 2019 Ã 18:54, tesonep(a)gmail.com <tesonep(a)gmail.com> a >> Ã©crit : >> >>> >> >>> Hi Aliaksei, >> >>> to me it looks like a bug of GCC optimization. Basically, it is >> >>> assuming that the x variable is used but never read or its value is >> >>> never used. Also it assumes the same of the i variable, as we are only >> >>> accessing indirectly to the memory where it locates (the code is even >> >>> assuming that the variable exists, but it can be optimize out as in >> >>> this scenario). Even though, the original C code is valid C code, we >> >>> are not helping the compiler writing code like that. So I have >> >>> rewritten the code in a way that does not use indirect memory access >> >>> to the stack space. >> >>> >> >>> One thing more that makes me think is a bug, if you use an int >> >>> constant as the index and not a parameter, the error does not occur >> >>> (the code is not badly optimized) and there is a warning about the >> >>> not-so-great access to the stack. >> >>> >> >>> On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> >> wrote: >> >>> > >> >>> > Hi Pablo, >> >>> > >> >>> > Wow! Thank you for the detective story :) >> >>> > >> >>> > Do I understand correctly that the original code causes undefined >> behavior and therefore can be changed (or even removed) by the compiler? >> >>> > (because it returns something that is referencing memory on the >> stack) >> >>> > >> >>> > Please keep posting similar things in future! It is very educative >> :) >> >>> > >> >>> > Cheers, >> >>> > Alex >> >>> > >> >>> > >> >>> > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> >> wrote: >> >>> >> >> >>> >> Hi, >> >>> >> this mail is related to Pharo because it is knowledge I found >> >>> >> debugging the build of the VM, but the rest is to document it and >> >>> >> perhaps someone will found it interesting (also I couldn't find it >> >>> >> easily using Google). Sorry for the long mail! >> >>> >> >> >>> >> The problem >> >>> >> ========== >> >>> >> >> >>> >> The following code does not produce good code in 8.3 when using >> optimizations: >> >>> >> >> >>> >> long __attribute__ ((noinline)) myFunc(long i, int index){ >> >>> >> long v; >> >>> >> long x = i >> 3; >> >>> >> >> >>> >> v = x; >> >>> >> return ((int*)(&v))[index]; >> >>> >> } >> >>> >> >> >>> >> #include <stdio.h> >> >>> >> >> >>> >> int main(){ >> >>> >> >> >>> >> long i; >> >>> >> int x; >> >>> >> >> >>> >> scanf("%ld", &i); >> >>> >> scanf("%d", &x); >> >>> >> >> >>> >> printf("%ld",myFunc(i,x)); >> >>> >> } >> >>> >> >> >>> >> Basically, with -02, it generates the following code: >> >>> >> >> >>> >> myFunc: >> >>> >> movslq %esi, %rsi >> >>> >> movslq -8(%rsp,%rsi,4), %rax >> >>> >> ret >> >>> >> >> >>> >> And with -01 it generates the following code: >> >>> >> >> >>> >> myFunc: >> >>> >> sarq $3, %rdi >> >>> >> movq %rdi, -8(%rsp) >> >>> >> movslq %esi, %rsi >> >>> >> movslq -8(%rsp,%rsi,4), %rax >> >>> >> ret >> >>> >> >> >>> >> As you can see, the more optimized version is losing the bit shift >> (or >> >>> >> any other operation). >> >>> >> To detect the problem it was not easy, basically, I have used the >> >>> >> Pharo Tests to detect the failing function and then to understand >> the >> >>> >> generation of code in GCC, we need to debug its generation. >> >>> >> >> >>> >> The first snippet is generated using: >> >>> >> >> >>> >> gcc -O2 prb.c -S -o prb.s -Wall >> >>> >> >> >>> >> and the second using: >> >>> >> >> >>> >> gcc -O1 prb.c -S -o prb.s -Wall >> >>> >> >> >>> >> The GCC pipeline has different steps, I have centered my self in >> the >> >>> >> tree optimizations. >> >>> >> GCC dumps each of the intermediate states of the compilation, >> working >> >>> >> with C like trees. For example to get the optimized version of the >> >>> >> program, before generating the Assembler we have to do: >> >>> >> >> >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout >> >>> >> >> >>> >> This will generate: >> >>> >> >> >>> >> __attribute__((noinline)) >> >>> >> myFunc (long int i, int index) >> >>> >> { >> >>> >> long int v; >> >>> >> long unsigned int _1; >> >>> >> long unsigned int _2; >> >>> >> int * _3; >> >>> >> int _4; >> >>> >> long int _8; >> >>> >> >> >>> >> <bb 2> [local count: 1073741825]: >> >>> >> _1 = (long unsigned int) index_7(D); >> >>> >> _2 = _1 * 4; >> >>> >> _3 = &v + _2; >> >>> >> _4 = *_3; >> >>> >> _8 = (long int) _4; >> >>> >> v ={v} {CLOBBER}; >> >>> >> return _8; >> >>> >> >> >>> >> } >> >>> >> >> >>> >> This is the optimized SSA (static single assign) version of the >> >>> >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) >> >>> >> As you can see in this version the code is already optimized, and >> our >> >>> >> operations not correctly generated. We expect to see something >> like: >> >>> >> >> >>> >> __attribute__((noinline)) >> >>> >> myFunc (long int i, int index) >> >>> >> { >> >>> >> long int x; >> >>> >> long int v; >> >>> >> long unsigned int _1; >> >>> >> long unsigned int _2; >> >>> >> int * _3; >> >>> >> int _4; >> >>> >> long int _10; >> >>> >> >> >>> >> <bb 2> [local count: 1073741825]: >> >>> >> x_6 = i_5(D) >> 3; >> >>> >> v = x_6; >> >>> >> _1 = (long unsigned int) index_9(D); >> >>> >> _2 = _1 * 4; >> >>> >> _3 = &v + _2; >> >>> >> _4 = *_3; >> >>> >> _10 = (long int) _4; >> >>> >> v ={v} {CLOBBER}; >> >>> >> return _10; >> >>> >> >> >>> >> } >> >>> >> >> >>> >> In the first SSA version, we are lacking the shift operation: >> >>> >> >> >>> >> x_6 = i_5(D) >> 3; >> >>> >> v = x_6; >> >>> >> >> >>> >> So, we need to see in which of the optimizations and >> transformations >> >>> >> our code is lost: >> >>> >> To see all the steps we should execute: >> >>> >> >> >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all >> >>> >> >> >>> >> This will generate a lot, really a lot, of files in the same >> directory. >> >>> >> They have the name: prb.c.xxx.yyyy >> >>> >> Where xxx is the number of the step, and yyyy is the name of the >> step. >> >>> >> Each of the files contains the result of applying the changes, so >> what >> >>> >> I have done is looking in binary search where my code was lost. >> >>> >> >> >>> >> I have found that the problem was in the first dead code >> elimination >> >>> >> step (prb.c.041t.cddce1) >> >>> >> GCC does not like the crap code that we are writing, as we are >> copying >> >>> >> a stack variable and then accessing directly to the memory: >> >>> >> >> >>> >> v = x; >> >>> >> return ((int*)(&v))[index]; >> >>> >> >> >>> >> So, basically, it was assuming that we were only using v and not >> x, so >> >>> >> all the code modifying x is described (this optimization is called >> >>> >> "dead store code deletion"). Basically tries to remove all the code >> >>> >> that is affecting stack (temporary) variables that are never used. >> >>> >> >> >>> >> I am not sure if this is a bug in GCC, so I will report it. But I >> have >> >>> >> fixed the problem writing the code in a proper way. >> >>> >> >> >>> >> Sorry again for the long mail, I wanted to store the knowledge and >> >>> >> propagate it before I eventually will flush it. I like these >> things, >> >>> >> but I am not debugging the GCC optimization pipeline all the days. >> >>> >> >> >>> >> -- >> >>> >> Pablo Tesone. >> >>> >> tesonep(a)gmail.com >> >>> >> >> >>> >> >>> >> >>> -- >> >>> Pablo Tesone. >> >>> tesonep(a)gmail.com >> >>> >> >> >> -- >> Pablo Tesone. >> tesonep(a)gmail.com >> >> -- _,,,^..^,,,_ best, Eliot

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Eliot Miranda

Hi Pablo, On Wed, Dec 11, 2019 at 11:33 AM tesonep(a)gmail.com <tesonep(a)gmail.com> wrote: > Hi Nicolas, > thanks for the comment, you are right the problem is the bad > original code. But my opinion is that it just is not reporting the > situation correctly, generating a warning or non-optimizing the code > looks more like a expected behavior. Because as I have said, using a > constant as index in the last statement generates a meaningful warning > and the non-optimizated version of the function. > > And again as you said, the only thing to learn about all this is that > we should not write crappy code. > Hang on. "crappy code" is too strong. Back in tre ANSI days it was not undefined behavior, and in fact I think this only because undefined behavior because C compilers were faced with lots of code where variables were typed int but were being used on 64-bit machines. Had the C world used long as its default type then there would have been no need to make casting the address of a long into the address of an int undefined behavior. So this isn't crappy code (if you think of what it means in memory on a 64-bit little endnan machine it is perfectly sensible). It /is/ code that is inappropriate given C99. So yes, we have to bow to the C gods and generate C99 compliant code, but what we were doing here wasn't crappy, it was merely code that became undefined behavior so that C compilers could generate more efficient code for 64-bit systems in the presence of code that contains lots of integer variables declared as int. So let's just say that we have to generate C99 compliant code. You'll notice that the Smalltalk code in the original does exactly what you suggest Pablo. fetchLong32: fieldIndex ofFloatObject: oop "index by word size, and return a pointer as long as the word size" | bits | (self isImmediateFloat: oop) ifFalse: [^self fetchLong32: fieldIndex ofObject: oop]. bits := self smallFloatBitsOf: oop. ^self cCode: [(self cCoerceSimple: (self addressOf: bits) to: #'int *') at: fieldIndex] inSmalltalk: [self flag: #endian. fieldIndex = 0 ifTrue: [bits bitAnd: 16rFFFFFFFF] ifFalse: [bits >> 32]] > On Wed, Dec 11, 2019 at 7:11 PM Nicolas Cellier > <nicolas.cellier.aka.nice(a)gmail.com> wrote: > > > > Of course, when I say "your" code, it's the code you have shown, and > probably "our" (VMMaker) code ;) > > > > Le mer. 11 dÃ©c. 2019 Ã 19:05, Nicolas Cellier < > nicolas.cellier.aka.nice(a)gmail.com> a Ã©crit : > >> > >> Hi Pablo (again), > >> no, not a bug. > >> > >> The problem is in the source code. The compiler has the right to > presume that your code is exempt of UB, because you cannot depend on UB > (obviously). > >> So it can eliminate all code which corresponds to UB. > >> > >> The compiler has the right to assume that a pointer to an int cannot > point to a long (UB). > >> So modifying a long cannot have any sort of impact on the content of > the int pointer. > >> So the compiler can decouple both path return int content and assign > long. > >> But assigning the long has no effect, so the code can be suppressed > altogether. > >> > >> Le mer. 11 dÃ©c. 2019 Ã 18:54, tesonep(a)gmail.com <tesonep(a)gmail.com> a > Ã©crit : > >>> > >>> Hi Aliaksei, > >>> to me it looks like a bug of GCC optimization. Basically, it is > >>> assuming that the x variable is used but never read or its value is > >>> never used. Also it assumes the same of the i variable, as we are only > >>> accessing indirectly to the memory where it locates (the code is even > >>> assuming that the variable exists, but it can be optimize out as in > >>> this scenario). Even though, the original C code is valid C code, we > >>> are not helping the compiler writing code like that. So I have > >>> rewritten the code in a way that does not use indirect memory access > >>> to the stack space. > >>> > >>> One thing more that makes me think is a bug, if you use an int > >>> constant as the index and not a parameter, the error does not occur > >>> (the code is not badly optimized) and there is a warning about the > >>> not-so-great access to the stack. > >>> > >>> On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> > wrote: > >>> > > >>> > Hi Pablo, > >>> > > >>> > Wow! Thank you for the detective story :) > >>> > > >>> > Do I understand correctly that the original code causes undefined > behavior and therefore can be changed (or even removed) by the compiler? > >>> > (because it returns something that is referencing memory on the > stack) > >>> > > >>> > Please keep posting similar things in future! It is very educative :) > >>> > > >>> > Cheers, > >>> > Alex > >>> > > >>> > > >>> > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> > wrote: > >>> >> > >>> >> Hi, > >>> >> this mail is related to Pharo because it is knowledge I found > >>> >> debugging the build of the VM, but the rest is to document it and > >>> >> perhaps someone will found it interesting (also I couldn't find it > >>> >> easily using Google). Sorry for the long mail! > >>> >> > >>> >> The problem > >>> >> ========== > >>> >> > >>> >> The following code does not produce good code in 8.3 when using > optimizations: > >>> >> > >>> >> long __attribute__ ((noinline)) myFunc(long i, int index){ > >>> >> long v; > >>> >> long x = i >> 3; > >>> >> > >>> >> v = x; > >>> >> return ((int*)(&v))[index]; > >>> >> } > >>> >> > >>> >> #include <stdio.h> > >>> >> > >>> >> int main(){ > >>> >> > >>> >> long i; > >>> >> int x; > >>> >> > >>> >> scanf("%ld", &i); > >>> >> scanf("%d", &x); > >>> >> > >>> >> printf("%ld",myFunc(i,x)); > >>> >> } > >>> >> > >>> >> Basically, with -02, it generates the following code: > >>> >> > >>> >> myFunc: > >>> >> movslq %esi, %rsi > >>> >> movslq -8(%rsp,%rsi,4), %rax > >>> >> ret > >>> >> > >>> >> And with -01 it generates the following code: > >>> >> > >>> >> myFunc: > >>> >> sarq $3, %rdi > >>> >> movq %rdi, -8(%rsp) > >>> >> movslq %esi, %rsi > >>> >> movslq -8(%rsp,%rsi,4), %rax > >>> >> ret > >>> >> > >>> >> As you can see, the more optimized version is losing the bit shift > (or > >>> >> any other operation). > >>> >> To detect the problem it was not easy, basically, I have used the > >>> >> Pharo Tests to detect the failing function and then to understand > the > >>> >> generation of code in GCC, we need to debug its generation. > >>> >> > >>> >> The first snippet is generated using: > >>> >> > >>> >> gcc -O2 prb.c -S -o prb.s -Wall > >>> >> > >>> >> and the second using: > >>> >> > >>> >> gcc -O1 prb.c -S -o prb.s -Wall > >>> >> > >>> >> The GCC pipeline has different steps, I have centered my self in the > >>> >> tree optimizations. > >>> >> GCC dumps each of the intermediate states of the compilation, > working > >>> >> with C like trees. For example to get the optimized version of the > >>> >> program, before generating the Assembler we have to do: > >>> >> > >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout > >>> >> > >>> >> This will generate: > >>> >> > >>> >> __attribute__((noinline)) > >>> >> myFunc (long int i, int index) > >>> >> { > >>> >> long int v; > >>> >> long unsigned int _1; > >>> >> long unsigned int _2; > >>> >> int * _3; > >>> >> int _4; > >>> >> long int _8; > >>> >> > >>> >> <bb 2> [local count: 1073741825]: > >>> >> _1 = (long unsigned int) index_7(D); > >>> >> _2 = _1 * 4; > >>> >> _3 = &v + _2; > >>> >> _4 = *_3; > >>> >> _8 = (long int) _4; > >>> >> v ={v} {CLOBBER}; > >>> >> return _8; > >>> >> > >>> >> } > >>> >> > >>> >> This is the optimized SSA (static single assign) version of the > >>> >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) > >>> >> As you can see in this version the code is already optimized, and > our > >>> >> operations not correctly generated. We expect to see something like: > >>> >> > >>> >> __attribute__((noinline)) > >>> >> myFunc (long int i, int index) > >>> >> { > >>> >> long int x; > >>> >> long int v; > >>> >> long unsigned int _1; > >>> >> long unsigned int _2; > >>> >> int * _3; > >>> >> int _4; > >>> >> long int _10; > >>> >> > >>> >> <bb 2> [local count: 1073741825]: > >>> >> x_6 = i_5(D) >> 3; > >>> >> v = x_6; > >>> >> _1 = (long unsigned int) index_9(D); > >>> >> _2 = _1 * 4; > >>> >> _3 = &v + _2; > >>> >> _4 = *_3; > >>> >> _10 = (long int) _4; > >>> >> v ={v} {CLOBBER}; > >>> >> return _10; > >>> >> > >>> >> } > >>> >> > >>> >> In the first SSA version, we are lacking the shift operation: > >>> >> > >>> >> x_6 = i_5(D) >> 3; > >>> >> v = x_6; > >>> >> > >>> >> So, we need to see in which of the optimizations and transformations > >>> >> our code is lost: > >>> >> To see all the steps we should execute: > >>> >> > >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all > >>> >> > >>> >> This will generate a lot, really a lot, of files in the same > directory. > >>> >> They have the name: prb.c.xxx.yyyy > >>> >> Where xxx is the number of the step, and yyyy is the name of the > step. > >>> >> Each of the files contains the result of applying the changes, so > what > >>> >> I have done is looking in binary search where my code was lost. > >>> >> > >>> >> I have found that the problem was in the first dead code elimination > >>> >> step (prb.c.041t.cddce1) > >>> >> GCC does not like the crap code that we are writing, as we are > copying > >>> >> a stack variable and then accessing directly to the memory: > >>> >> > >>> >> v = x; > >>> >> return ((int*)(&v))[index]; > >>> >> > >>> >> So, basically, it was assuming that we were only using v and not x, > so > >>> >> all the code modifying x is described (this optimization is called > >>> >> "dead store code deletion"). Basically tries to remove all the code > >>> >> that is affecting stack (temporary) variables that are never used. > >>> >> > >>> >> I am not sure if this is a bug in GCC, so I will report it. But I > have > >>> >> fixed the problem writing the code in a proper way. > >>> >> > >>> >> Sorry again for the long mail, I wanted to store the knowledge and > >>> >> propagate it before I eventually will flush it. I like these things, > >>> >> but I am not debugging the GCC optimization pipeline all the days. > >>> >> > >>> >> -- > >>> >> Pablo Tesone. > >>> >> tesonep(a)gmail.com > >>> >> > >>> > >>> > >>> -- > >>> Pablo Tesone. > >>> tesonep(a)gmail.com > >>> > > > -- > Pablo Tesone. > tesonep(a)gmail.com > > -- _,,,^..^,,,_ best, Eliot

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Nicolas Cellier

Yes, But we have to replace natural crappy code (split a long in 2 ints) that was once legal, by an even more crappy code (memcpy), so all in all, it's a crappy art. Le mer. 11 dÃ©c. 2019 Ã 19:30, tesonep(a)gmail.com <tesonep(a)gmail.com> a Ã©crit : > Hi Nicolas, > thanks for the comment, you are right the problem is the bad > original code. But my opinion is that it just is not reporting the > situation correctly, generating a warning or non-optimizing the code > looks more like a expected behavior. Because as I have said, using a > constant as index in the last statement generates a meaningful warning > and the non-optimizated version of the function. > > And again as you said, the only thing to learn about all this is that > we should not write crappy code. > > On Wed, Dec 11, 2019 at 7:11 PM Nicolas Cellier > <nicolas.cellier.aka.nice(a)gmail.com> wrote: > > > > Of course, when I say "your" code, it's the code you have shown, and > probably "our" (VMMaker) code ;) > > > > Le mer. 11 dÃ©c. 2019 Ã 19:05, Nicolas Cellier < > nicolas.cellier.aka.nice(a)gmail.com> a Ã©crit : > >> > >> Hi Pablo (again), > >> no, not a bug. > >> > >> The problem is in the source code. The compiler has the right to > presume that your code is exempt of UB, because you cannot depend on UB > (obviously). > >> So it can eliminate all code which corresponds to UB. > >> > >> The compiler has the right to assume that a pointer to an int cannot > point to a long (UB). > >> So modifying a long cannot have any sort of impact on the content of > the int pointer. > >> So the compiler can decouple both path return int content and assign > long. > >> But assigning the long has no effect, so the code can be suppressed > altogether. > >> > >> Le mer. 11 dÃ©c. 2019 Ã 18:54, tesonep(a)gmail.com <tesonep(a)gmail.com> a > Ã©crit : > >>> > >>> Hi Aliaksei, > >>> to me it looks like a bug of GCC optimization. Basically, it is > >>> assuming that the x variable is used but never read or its value is > >>> never used. Also it assumes the same of the i variable, as we are only > >>> accessing indirectly to the memory where it locates (the code is even > >>> assuming that the variable exists, but it can be optimize out as in > >>> this scenario). Even though, the original C code is valid C code, we > >>> are not helping the compiler writing code like that. So I have > >>> rewritten the code in a way that does not use indirect memory access > >>> to the stack space. > >>> > >>> One thing more that makes me think is a bug, if you use an int > >>> constant as the index and not a parameter, the error does not occur > >>> (the code is not badly optimized) and there is a warning about the > >>> not-so-great access to the stack. > >>> > >>> On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> > wrote: > >>> > > >>> > Hi Pablo, > >>> > > >>> > Wow! Thank you for the detective story :) > >>> > > >>> > Do I understand correctly that the original code causes undefined > behavior and therefore can be changed (or even removed) by the compiler? > >>> > (because it returns something that is referencing memory on the > stack) > >>> > > >>> > Please keep posting similar things in future! It is very educative :) > >>> > > >>> > Cheers, > >>> > Alex > >>> > > >>> > > >>> > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> > wrote: > >>> >> > >>> >> Hi, > >>> >> this mail is related to Pharo because it is knowledge I found > >>> >> debugging the build of the VM, but the rest is to document it and > >>> >> perhaps someone will found it interesting (also I couldn't find it > >>> >> easily using Google). Sorry for the long mail! > >>> >> > >>> >> The problem > >>> >> ========== > >>> >> > >>> >> The following code does not produce good code in 8.3 when using > optimizations: > >>> >> > >>> >> long __attribute__ ((noinline)) myFunc(long i, int index){ > >>> >> long v; > >>> >> long x = i >> 3; > >>> >> > >>> >> v = x; > >>> >> return ((int*)(&v))[index]; > >>> >> } > >>> >> > >>> >> #include <stdio.h> > >>> >> > >>> >> int main(){ > >>> >> > >>> >> long i; > >>> >> int x; > >>> >> > >>> >> scanf("%ld", &i); > >>> >> scanf("%d", &x); > >>> >> > >>> >> printf("%ld",myFunc(i,x)); > >>> >> } > >>> >> > >>> >> Basically, with -02, it generates the following code: > >>> >> > >>> >> myFunc: > >>> >> movslq %esi, %rsi > >>> >> movslq -8(%rsp,%rsi,4), %rax > >>> >> ret > >>> >> > >>> >> And with -01 it generates the following code: > >>> >> > >>> >> myFunc: > >>> >> sarq $3, %rdi > >>> >> movq %rdi, -8(%rsp) > >>> >> movslq %esi, %rsi > >>> >> movslq -8(%rsp,%rsi,4), %rax > >>> >> ret > >>> >> > >>> >> As you can see, the more optimized version is losing the bit shift > (or > >>> >> any other operation). > >>> >> To detect the problem it was not easy, basically, I have used the > >>> >> Pharo Tests to detect the failing function and then to understand > the > >>> >> generation of code in GCC, we need to debug its generation. > >>> >> > >>> >> The first snippet is generated using: > >>> >> > >>> >> gcc -O2 prb.c -S -o prb.s -Wall > >>> >> > >>> >> and the second using: > >>> >> > >>> >> gcc -O1 prb.c -S -o prb.s -Wall > >>> >> > >>> >> The GCC pipeline has different steps, I have centered my self in the > >>> >> tree optimizations. > >>> >> GCC dumps each of the intermediate states of the compilation, > working > >>> >> with C like trees. For example to get the optimized version of the > >>> >> program, before generating the Assembler we have to do: > >>> >> > >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout > >>> >> > >>> >> This will generate: > >>> >> > >>> >> __attribute__((noinline)) > >>> >> myFunc (long int i, int index) > >>> >> { > >>> >> long int v; > >>> >> long unsigned int _1; > >>> >> long unsigned int _2; > >>> >> int * _3; > >>> >> int _4; > >>> >> long int _8; > >>> >> > >>> >> <bb 2> [local count: 1073741825]: > >>> >> _1 = (long unsigned int) index_7(D); > >>> >> _2 = _1 * 4; > >>> >> _3 = &v + _2; > >>> >> _4 = *_3; > >>> >> _8 = (long int) _4; > >>> >> v ={v} {CLOBBER}; > >>> >> return _8; > >>> >> > >>> >> } > >>> >> > >>> >> This is the optimized SSA (static single assign) version of the > >>> >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) > >>> >> As you can see in this version the code is already optimized, and > our > >>> >> operations not correctly generated. We expect to see something like: > >>> >> > >>> >> __attribute__((noinline)) > >>> >> myFunc (long int i, int index) > >>> >> { > >>> >> long int x; > >>> >> long int v; > >>> >> long unsigned int _1; > >>> >> long unsigned int _2; > >>> >> int * _3; > >>> >> int _4; > >>> >> long int _10; > >>> >> > >>> >> <bb 2> [local count: 1073741825]: > >>> >> x_6 = i_5(D) >> 3; > >>> >> v = x_6; > >>> >> _1 = (long unsigned int) index_9(D); > >>> >> _2 = _1 * 4; > >>> >> _3 = &v + _2; > >>> >> _4 = *_3; > >>> >> _10 = (long int) _4; > >>> >> v ={v} {CLOBBER}; > >>> >> return _10; > >>> >> > >>> >> } > >>> >> > >>> >> In the first SSA version, we are lacking the shift operation: > >>> >> > >>> >> x_6 = i_5(D) >> 3; > >>> >> v = x_6; > >>> >> > >>> >> So, we need to see in which of the optimizations and transformations > >>> >> our code is lost: > >>> >> To see all the steps we should execute: > >>> >> > >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all > >>> >> > >>> >> This will generate a lot, really a lot, of files in the same > directory. > >>> >> They have the name: prb.c.xxx.yyyy > >>> >> Where xxx is the number of the step, and yyyy is the name of the > step. > >>> >> Each of the files contains the result of applying the changes, so > what > >>> >> I have done is looking in binary search where my code was lost. > >>> >> > >>> >> I have found that the problem was in the first dead code elimination > >>> >> step (prb.c.041t.cddce1) > >>> >> GCC does not like the crap code that we are writing, as we are > copying > >>> >> a stack variable and then accessing directly to the memory: > >>> >> > >>> >> v = x; > >>> >> return ((int*)(&v))[index]; > >>> >> > >>> >> So, basically, it was assuming that we were only using v and not x, > so > >>> >> all the code modifying x is described (this optimization is called > >>> >> "dead store code deletion"). Basically tries to remove all the code > >>> >> that is affecting stack (temporary) variables that are never used. > >>> >> > >>> >> I am not sure if this is a bug in GCC, so I will report it. But I > have > >>> >> fixed the problem writing the code in a proper way. > >>> >> > >>> >> Sorry again for the long mail, I wanted to store the knowledge and > >>> >> propagate it before I eventually will flush it. I like these things, > >>> >> but I am not debugging the GCC optimization pipeline all the days. > >>> >> > >>> >> -- > >>> >> Pablo Tesone. > >>> >> tesonep(a)gmail.com > >>> >> > >>> > >>> > >>> -- > >>> Pablo Tesone. > >>> tesonep(a)gmail.com > >>> > > > -- > Pablo Tesone. > tesonep(a)gmail.com > >

Dec. 11, 2019