December 2019 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Debugging GCC code generation

by tesonep＠gmail.com

Hi Nicolas, thanks for the comment, you are right the problem is the bad original code. But my opinion is that it just is not reporting the situation correctly, generating a warning or non-optimizing the code looks more like a expected behavior. Because as I have said, using a constant as index in the last statement generates a meaningful warning and the non-optimizated version of the function. And again as you said, the only thing to learn about all this is that we should not write crappy code. On Wed, Dec 11, 2019 at 7:11 PM Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> wrote: > > Of course, when I say "your" code, it's the code you have shown, and probably "our" (VMMaker) code ;) > > Le mer. 11 dÃ©c. 2019 Ã 19:05, Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com> a Ã©crit : >> >> Hi Pablo (again), >> no, not a bug. >> >> The problem is in the source code. The compiler has the right to presume that your code is exempt of UB, because you cannot depend on UB (obviously). >> So it can eliminate all code which corresponds to UB. >> >> The compiler has the right to assume that a pointer to an int cannot point to a long (UB). >> So modifying a long cannot have any sort of impact on the content of the int pointer. >> So the compiler can decouple both path return int content and assign long. >> But assigning the long has no effect, so the code can be suppressed altogether. >> >> Le mer. 11 dÃ©c. 2019 Ã 18:54, tesonep(a)gmail.com <tesonep(a)gmail.com> a Ã©crit : >>> >>> Hi Aliaksei, >>> to me it looks like a bug of GCC optimization. Basically, it is >>> assuming that the x variable is used but never read or its value is >>> never used. Also it assumes the same of the i variable, as we are only >>> accessing indirectly to the memory where it locates (the code is even >>> assuming that the variable exists, but it can be optimize out as in >>> this scenario). Even though, the original C code is valid C code, we >>> are not helping the compiler writing code like that. So I have >>> rewritten the code in a way that does not use indirect memory access >>> to the stack space. >>> >>> One thing more that makes me think is a bug, if you use an int >>> constant as the index and not a parameter, the error does not occur >>> (the code is not badly optimized) and there is a warning about the >>> not-so-great access to the stack. >>> >>> On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> wrote: >>> > >>> > Hi Pablo, >>> > >>> > Wow! Thank you for the detective story :) >>> > >>> > Do I understand correctly that the original code causes undefined behavior and therefore can be changed (or even removed) by the compiler? >>> > (because it returns something that is referencing memory on the stack) >>> > >>> > Please keep posting similar things in future! It is very educative :) >>> > >>> > Cheers, >>> > Alex >>> > >>> > >>> > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> wrote: >>> >> >>> >> Hi, >>> >> this mail is related to Pharo because it is knowledge I found >>> >> debugging the build of the VM, but the rest is to document it and >>> >> perhaps someone will found it interesting (also I couldn't find it >>> >> easily using Google). Sorry for the long mail! >>> >> >>> >> The problem >>> >> ========== >>> >> >>> >> The following code does not produce good code in 8.3 when using optimizations: >>> >> >>> >> long __attribute__ ((noinline)) myFunc(long i, int index){ >>> >> long v; >>> >> long x = i >> 3; >>> >> >>> >> v = x; >>> >> return ((int*)(&v))[index]; >>> >> } >>> >> >>> >> #include <stdio.h> >>> >> >>> >> int main(){ >>> >> >>> >> long i; >>> >> int x; >>> >> >>> >> scanf("%ld", &i); >>> >> scanf("%d", &x); >>> >> >>> >> printf("%ld",myFunc(i,x)); >>> >> } >>> >> >>> >> Basically, with -02, it generates the following code: >>> >> >>> >> myFunc: >>> >> movslq %esi, %rsi >>> >> movslq -8(%rsp,%rsi,4), %rax >>> >> ret >>> >> >>> >> And with -01 it generates the following code: >>> >> >>> >> myFunc: >>> >> sarq $3, %rdi >>> >> movq %rdi, -8(%rsp) >>> >> movslq %esi, %rsi >>> >> movslq -8(%rsp,%rsi,4), %rax >>> >> ret >>> >> >>> >> As you can see, the more optimized version is losing the bit shift (or >>> >> any other operation). >>> >> To detect the problem it was not easy, basically, I have used the >>> >> Pharo Tests to detect the failing function and then to understand the >>> >> generation of code in GCC, we need to debug its generation. >>> >> >>> >> The first snippet is generated using: >>> >> >>> >> gcc -O2 prb.c -S -o prb.s -Wall >>> >> >>> >> and the second using: >>> >> >>> >> gcc -O1 prb.c -S -o prb.s -Wall >>> >> >>> >> The GCC pipeline has different steps, I have centered my self in the >>> >> tree optimizations. >>> >> GCC dumps each of the intermediate states of the compilation, working >>> >> with C like trees. For example to get the optimized version of the >>> >> program, before generating the Assembler we have to do: >>> >> >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout >>> >> >>> >> This will generate: >>> >> >>> >> __attribute__((noinline)) >>> >> myFunc (long int i, int index) >>> >> { >>> >> long int v; >>> >> long unsigned int _1; >>> >> long unsigned int _2; >>> >> int * _3; >>> >> int _4; >>> >> long int _8; >>> >> >>> >> <bb 2> [local count: 1073741825]: >>> >> _1 = (long unsigned int) index_7(D); >>> >> _2 = _1 * 4; >>> >> _3 = &v + _2; >>> >> _4 = *_3; >>> >> _8 = (long int) _4; >>> >> v ={v} {CLOBBER}; >>> >> return _8; >>> >> >>> >> } >>> >> >>> >> This is the optimized SSA (static single assign) version of the >>> >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) >>> >> As you can see in this version the code is already optimized, and our >>> >> operations not correctly generated. We expect to see something like: >>> >> >>> >> __attribute__((noinline)) >>> >> myFunc (long int i, int index) >>> >> { >>> >> long int x; >>> >> long int v; >>> >> long unsigned int _1; >>> >> long unsigned int _2; >>> >> int * _3; >>> >> int _4; >>> >> long int _10; >>> >> >>> >> <bb 2> [local count: 1073741825]: >>> >> x_6 = i_5(D) >> 3; >>> >> v = x_6; >>> >> _1 = (long unsigned int) index_9(D); >>> >> _2 = _1 * 4; >>> >> _3 = &v + _2; >>> >> _4 = *_3; >>> >> _10 = (long int) _4; >>> >> v ={v} {CLOBBER}; >>> >> return _10; >>> >> >>> >> } >>> >> >>> >> In the first SSA version, we are lacking the shift operation: >>> >> >>> >> x_6 = i_5(D) >> 3; >>> >> v = x_6; >>> >> >>> >> So, we need to see in which of the optimizations and transformations >>> >> our code is lost: >>> >> To see all the steps we should execute: >>> >> >>> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all >>> >> >>> >> This will generate a lot, really a lot, of files in the same directory. >>> >> They have the name: prb.c.xxx.yyyy >>> >> Where xxx is the number of the step, and yyyy is the name of the step. >>> >> Each of the files contains the result of applying the changes, so what >>> >> I have done is looking in binary search where my code was lost. >>> >> >>> >> I have found that the problem was in the first dead code elimination >>> >> step (prb.c.041t.cddce1) >>> >> GCC does not like the crap code that we are writing, as we are copying >>> >> a stack variable and then accessing directly to the memory: >>> >> >>> >> v = x; >>> >> return ((int*)(&v))[index]; >>> >> >>> >> So, basically, it was assuming that we were only using v and not x, so >>> >> all the code modifying x is described (this optimization is called >>> >> "dead store code deletion"). Basically tries to remove all the code >>> >> that is affecting stack (temporary) variables that are never used. >>> >> >>> >> I am not sure if this is a bug in GCC, so I will report it. But I have >>> >> fixed the problem writing the code in a proper way. >>> >> >>> >> Sorry again for the long mail, I wanted to store the knowledge and >>> >> propagate it before I eventually will flush it. I like these things, >>> >> but I am not debugging the GCC optimization pipeline all the days. >>> >> >>> >> -- >>> >> Pablo Tesone. >>> >> tesonep(a)gmail.com >>> >> >>> >>> >>> -- >>> Pablo Tesone. >>> tesonep(a)gmail.com >>> -- Pablo Tesone. tesonep(a)gmail.com

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Nicolas Cellier

Of course, when I say "your" code, it's the code you have shown, and probably "our" (VMMaker) code ;) Le mer. 11 dÃ©c. 2019 Ã 19:05, Nicolas Cellier < nicolas.cellier.aka.nice(a)gmail.com> a Ã©crit : > Hi Pablo (again), > no, not a bug. > > The problem is in the source code. The compiler has the right to presume > that your code is exempt of UB, because you cannot depend on UB (obviously). > So it can eliminate all code which corresponds to UB. > > The compiler has the right to assume that a pointer to an int cannot point > to a long (UB). > So modifying a long cannot have any sort of impact on the content of the > int pointer. > So the compiler can decouple both path return int content and assign long. > But assigning the long has no effect, so the code can be suppressed > altogether. > > Le mer. 11 dÃ©c. 2019 Ã 18:54, tesonep(a)gmail.com <tesonep(a)gmail.com> a > Ã©crit : > >> Hi Aliaksei, >> to me it looks like a bug of GCC optimization. Basically, it is >> assuming that the x variable is used but never read or its value is >> never used. Also it assumes the same of the i variable, as we are only >> accessing indirectly to the memory where it locates (the code is even >> assuming that the variable exists, but it can be optimize out as in >> this scenario). Even though, the original C code is valid C code, we >> are not helping the compiler writing code like that. So I have >> rewritten the code in a way that does not use indirect memory access >> to the stack space. >> >> One thing more that makes me think is a bug, if you use an int >> constant as the index and not a parameter, the error does not occur >> (the code is not badly optimized) and there is a warning about the >> not-so-great access to the stack. >> >> On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> >> wrote: >> > >> > Hi Pablo, >> > >> > Wow! Thank you for the detective story :) >> > >> > Do I understand correctly that the original code causes undefined >> behavior and therefore can be changed (or even removed) by the compiler? >> > (because it returns something that is referencing memory on the stack) >> > >> > Please keep posting similar things in future! It is very educative :) >> > >> > Cheers, >> > Alex >> > >> > >> > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> >> wrote: >> >> >> >> Hi, >> >> this mail is related to Pharo because it is knowledge I found >> >> debugging the build of the VM, but the rest is to document it and >> >> perhaps someone will found it interesting (also I couldn't find it >> >> easily using Google). Sorry for the long mail! >> >> >> >> The problem >> >> ========== >> >> >> >> The following code does not produce good code in 8.3 when using >> optimizations: >> >> >> >> long __attribute__ ((noinline)) myFunc(long i, int index){ >> >> long v; >> >> long x = i >> 3; >> >> >> >> v = x; >> >> return ((int*)(&v))[index]; >> >> } >> >> >> >> #include <stdio.h> >> >> >> >> int main(){ >> >> >> >> long i; >> >> int x; >> >> >> >> scanf("%ld", &i); >> >> scanf("%d", &x); >> >> >> >> printf("%ld",myFunc(i,x)); >> >> } >> >> >> >> Basically, with -02, it generates the following code: >> >> >> >> myFunc: >> >> movslq %esi, %rsi >> >> movslq -8(%rsp,%rsi,4), %rax >> >> ret >> >> >> >> And with -01 it generates the following code: >> >> >> >> myFunc: >> >> sarq $3, %rdi >> >> movq %rdi, -8(%rsp) >> >> movslq %esi, %rsi >> >> movslq -8(%rsp,%rsi,4), %rax >> >> ret >> >> >> >> As you can see, the more optimized version is losing the bit shift (or >> >> any other operation). >> >> To detect the problem it was not easy, basically, I have used the >> >> Pharo Tests to detect the failing function and then to understand the >> >> generation of code in GCC, we need to debug its generation. >> >> >> >> The first snippet is generated using: >> >> >> >> gcc -O2 prb.c -S -o prb.s -Wall >> >> >> >> and the second using: >> >> >> >> gcc -O1 prb.c -S -o prb.s -Wall >> >> >> >> The GCC pipeline has different steps, I have centered my self in the >> >> tree optimizations. >> >> GCC dumps each of the intermediate states of the compilation, working >> >> with C like trees. For example to get the optimized version of the >> >> program, before generating the Assembler we have to do: >> >> >> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout >> >> >> >> This will generate: >> >> >> >> __attribute__((noinline)) >> >> myFunc (long int i, int index) >> >> { >> >> long int v; >> >> long unsigned int _1; >> >> long unsigned int _2; >> >> int * _3; >> >> int _4; >> >> long int _8; >> >> >> >> <bb 2> [local count: 1073741825]: >> >> _1 = (long unsigned int) index_7(D); >> >> _2 = _1 * 4; >> >> _3 = &v + _2; >> >> _4 = *_3; >> >> _8 = (long int) _4; >> >> v ={v} {CLOBBER}; >> >> return _8; >> >> >> >> } >> >> >> >> This is the optimized SSA (static single assign) version of the >> >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) >> >> As you can see in this version the code is already optimized, and our >> >> operations not correctly generated. We expect to see something like: >> >> >> >> __attribute__((noinline)) >> >> myFunc (long int i, int index) >> >> { >> >> long int x; >> >> long int v; >> >> long unsigned int _1; >> >> long unsigned int _2; >> >> int * _3; >> >> int _4; >> >> long int _10; >> >> >> >> <bb 2> [local count: 1073741825]: >> >> x_6 = i_5(D) >> 3; >> >> v = x_6; >> >> _1 = (long unsigned int) index_9(D); >> >> _2 = _1 * 4; >> >> _3 = &v + _2; >> >> _4 = *_3; >> >> _10 = (long int) _4; >> >> v ={v} {CLOBBER}; >> >> return _10; >> >> >> >> } >> >> >> >> In the first SSA version, we are lacking the shift operation: >> >> >> >> x_6 = i_5(D) >> 3; >> >> v = x_6; >> >> >> >> So, we need to see in which of the optimizations and transformations >> >> our code is lost: >> >> To see all the steps we should execute: >> >> >> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all >> >> >> >> This will generate a lot, really a lot, of files in the same directory. >> >> They have the name: prb.c.xxx.yyyy >> >> Where xxx is the number of the step, and yyyy is the name of the step. >> >> Each of the files contains the result of applying the changes, so what >> >> I have done is looking in binary search where my code was lost. >> >> >> >> I have found that the problem was in the first dead code elimination >> >> step (prb.c.041t.cddce1) >> >> GCC does not like the crap code that we are writing, as we are copying >> >> a stack variable and then accessing directly to the memory: >> >> >> >> v = x; >> >> return ((int*)(&v))[index]; >> >> >> >> So, basically, it was assuming that we were only using v and not x, so >> >> all the code modifying x is described (this optimization is called >> >> "dead store code deletion"). Basically tries to remove all the code >> >> that is affecting stack (temporary) variables that are never used. >> >> >> >> I am not sure if this is a bug in GCC, so I will report it. But I have >> >> fixed the problem writing the code in a proper way. >> >> >> >> Sorry again for the long mail, I wanted to store the knowledge and >> >> propagate it before I eventually will flush it. I like these things, >> >> but I am not debugging the GCC optimization pipeline all the days. >> >> >> >> -- >> >> Pablo Tesone. >> >> tesonep(a)gmail.com >> >> >> >> >> -- >> Pablo Tesone. >> tesonep(a)gmail.com >> >>

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Nicolas Cellier

Hi Pablo (again), no, not a bug. The problem is in the source code. The compiler has the right to presume that your code is exempt of UB, because you cannot depend on UB (obviously). So it can eliminate all code which corresponds to UB. The compiler has the right to assume that a pointer to an int cannot point to a long (UB). So modifying a long cannot have any sort of impact on the content of the int pointer. So the compiler can decouple both path return int content and assign long. But assigning the long has no effect, so the code can be suppressed altogether. Le mer. 11 dÃ©c. 2019 Ã 18:54, tesonep(a)gmail.com <tesonep(a)gmail.com> a Ã©crit : > Hi Aliaksei, > to me it looks like a bug of GCC optimization. Basically, it is > assuming that the x variable is used but never read or its value is > never used. Also it assumes the same of the i variable, as we are only > accessing indirectly to the memory where it locates (the code is even > assuming that the variable exists, but it can be optimize out as in > this scenario). Even though, the original C code is valid C code, we > are not helping the compiler writing code like that. So I have > rewritten the code in a way that does not use indirect memory access > to the stack space. > > One thing more that makes me think is a bug, if you use an int > constant as the index and not a parameter, the error does not occur > (the code is not badly optimized) and there is a warning about the > not-so-great access to the stack. > > On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> > wrote: > > > > Hi Pablo, > > > > Wow! Thank you for the detective story :) > > > > Do I understand correctly that the original code causes undefined > behavior and therefore can be changed (or even removed) by the compiler? > > (because it returns something that is referencing memory on the stack) > > > > Please keep posting similar things in future! It is very educative :) > > > > Cheers, > > Alex > > > > > > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> > wrote: > >> > >> Hi, > >> this mail is related to Pharo because it is knowledge I found > >> debugging the build of the VM, but the rest is to document it and > >> perhaps someone will found it interesting (also I couldn't find it > >> easily using Google). Sorry for the long mail! > >> > >> The problem > >> ========== > >> > >> The following code does not produce good code in 8.3 when using > optimizations: > >> > >> long __attribute__ ((noinline)) myFunc(long i, int index){ > >> long v; > >> long x = i >> 3; > >> > >> v = x; > >> return ((int*)(&v))[index]; > >> } > >> > >> #include <stdio.h> > >> > >> int main(){ > >> > >> long i; > >> int x; > >> > >> scanf("%ld", &i); > >> scanf("%d", &x); > >> > >> printf("%ld",myFunc(i,x)); > >> } > >> > >> Basically, with -02, it generates the following code: > >> > >> myFunc: > >> movslq %esi, %rsi > >> movslq -8(%rsp,%rsi,4), %rax > >> ret > >> > >> And with -01 it generates the following code: > >> > >> myFunc: > >> sarq $3, %rdi > >> movq %rdi, -8(%rsp) > >> movslq %esi, %rsi > >> movslq -8(%rsp,%rsi,4), %rax > >> ret > >> > >> As you can see, the more optimized version is losing the bit shift (or > >> any other operation). > >> To detect the problem it was not easy, basically, I have used the > >> Pharo Tests to detect the failing function and then to understand the > >> generation of code in GCC, we need to debug its generation. > >> > >> The first snippet is generated using: > >> > >> gcc -O2 prb.c -S -o prb.s -Wall > >> > >> and the second using: > >> > >> gcc -O1 prb.c -S -o prb.s -Wall > >> > >> The GCC pipeline has different steps, I have centered my self in the > >> tree optimizations. > >> GCC dumps each of the intermediate states of the compilation, working > >> with C like trees. For example to get the optimized version of the > >> program, before generating the Assembler we have to do: > >> > >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout > >> > >> This will generate: > >> > >> __attribute__((noinline)) > >> myFunc (long int i, int index) > >> { > >> long int v; > >> long unsigned int _1; > >> long unsigned int _2; > >> int * _3; > >> int _4; > >> long int _8; > >> > >> <bb 2> [local count: 1073741825]: > >> _1 = (long unsigned int) index_7(D); > >> _2 = _1 * 4; > >> _3 = &v + _2; > >> _4 = *_3; > >> _8 = (long int) _4; > >> v ={v} {CLOBBER}; > >> return _8; > >> > >> } > >> > >> This is the optimized SSA (static single assign) version of the > >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) > >> As you can see in this version the code is already optimized, and our > >> operations not correctly generated. We expect to see something like: > >> > >> __attribute__((noinline)) > >> myFunc (long int i, int index) > >> { > >> long int x; > >> long int v; > >> long unsigned int _1; > >> long unsigned int _2; > >> int * _3; > >> int _4; > >> long int _10; > >> > >> <bb 2> [local count: 1073741825]: > >> x_6 = i_5(D) >> 3; > >> v = x_6; > >> _1 = (long unsigned int) index_9(D); > >> _2 = _1 * 4; > >> _3 = &v + _2; > >> _4 = *_3; > >> _10 = (long int) _4; > >> v ={v} {CLOBBER}; > >> return _10; > >> > >> } > >> > >> In the first SSA version, we are lacking the shift operation: > >> > >> x_6 = i_5(D) >> 3; > >> v = x_6; > >> > >> So, we need to see in which of the optimizations and transformations > >> our code is lost: > >> To see all the steps we should execute: > >> > >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all > >> > >> This will generate a lot, really a lot, of files in the same directory. > >> They have the name: prb.c.xxx.yyyy > >> Where xxx is the number of the step, and yyyy is the name of the step. > >> Each of the files contains the result of applying the changes, so what > >> I have done is looking in binary search where my code was lost. > >> > >> I have found that the problem was in the first dead code elimination > >> step (prb.c.041t.cddce1) > >> GCC does not like the crap code that we are writing, as we are copying > >> a stack variable and then accessing directly to the memory: > >> > >> v = x; > >> return ((int*)(&v))[index]; > >> > >> So, basically, it was assuming that we were only using v and not x, so > >> all the code modifying x is described (this optimization is called > >> "dead store code deletion"). Basically tries to remove all the code > >> that is affecting stack (temporary) variables that are never used. > >> > >> I am not sure if this is a bug in GCC, so I will report it. But I have > >> fixed the problem writing the code in a proper way. > >> > >> Sorry again for the long mail, I wanted to store the knowledge and > >> propagate it before I eventually will flush it. I like these things, > >> but I am not debugging the GCC optimization pipeline all the days. > >> > >> -- > >> Pablo Tesone. > >> tesonep(a)gmail.com > >> > > > -- > Pablo Tesone. > tesonep(a)gmail.com > >

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Nicolas Cellier

Hi Pablo, pointer aliasing is considered UB indeed and accessing a pointer outside of allocated bounds also is UB the recommended way to perform a between one pointer type and another (reinterpret_cast) is to use memcpy, so it should be: long v; int i[sizeof(long)/sizeof(int)]; memcpy( i , &v, sizeof(v) ); return i[index]; Normally, the compiler -O2 is smart enough to not allocate memory for i[2] and can perform all ops thru registers Le mer. 11 dÃ©c. 2019 Ã 18:01, Aliaksei Syrel <alex.syrel(a)gmail.com> a Ã©crit : > Hi Pablo, > > Wow! Thank you for the detective story :) > > Do I understand correctly that the original code causes undefined behavior > and therefore can be changed (or even removed) by the compiler? > (because it returns something that is referencing memory on the stack) > > Please keep posting similar things in future! It is very educative :) > > Cheers, > Alex > > > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> wrote: > >> Hi, >> this mail is related to Pharo because it is knowledge I found >> debugging the build of the VM, but the rest is to document it and >> perhaps someone will found it interesting (also I couldn't find it >> easily using Google). Sorry for the long mail! >> >> The problem >> ========== >> >> The following code does not produce good code in 8.3 when using >> optimizations: >> >> long __attribute__ ((noinline)) myFunc(long i, int index){ >> long v; >> long x = i >> 3; >> >> v = x; >> return ((int*)(&v))[index]; >> } >> >> #include <stdio.h> >> >> int main(){ >> >> long i; >> int x; >> >> scanf("%ld", &i); >> scanf("%d", &x); >> >> printf("%ld",myFunc(i,x)); >> } >> >> Basically, with -02, it generates the following code: >> >> myFunc: >> movslq %esi, %rsi >> movslq -8(%rsp,%rsi,4), %rax >> ret >> >> And with -01 it generates the following code: >> >> myFunc: >> sarq $3, %rdi >> movq %rdi, -8(%rsp) >> movslq %esi, %rsi >> movslq -8(%rsp,%rsi,4), %rax >> ret >> >> As you can see, the more optimized version is losing the bit shift (or >> any other operation). >> To detect the problem it was not easy, basically, I have used the >> Pharo Tests to detect the failing function and then to understand the >> generation of code in GCC, we need to debug its generation. >> >> The first snippet is generated using: >> >> gcc -O2 prb.c -S -o prb.s -Wall >> >> and the second using: >> >> gcc -O1 prb.c -S -o prb.s -Wall >> >> The GCC pipeline has different steps, I have centered my self in the >> tree optimizations. >> GCC dumps each of the intermediate states of the compilation, working >> with C like trees. For example to get the optimized version of the >> program, before generating the Assembler we have to do: >> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout >> >> This will generate: >> >> __attribute__((noinline)) >> myFunc (long int i, int index) >> { >> long int v; >> long unsigned int _1; >> long unsigned int _2; >> int * _3; >> int _4; >> long int _8; >> >> <bb 2> [local count: 1073741825]: >> _1 = (long unsigned int) index_7(D); >> _2 = _1 * 4; >> _3 = &v + _2; >> _4 = *_3; >> _8 = (long int) _4; >> v ={v} {CLOBBER}; >> return _8; >> >> } >> >> This is the optimized SSA (static single assign) version of the >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) >> As you can see in this version the code is already optimized, and our >> operations not correctly generated. We expect to see something like: >> >> __attribute__((noinline)) >> myFunc (long int i, int index) >> { >> long int x; >> long int v; >> long unsigned int _1; >> long unsigned int _2; >> int * _3; >> int _4; >> long int _10; >> >> <bb 2> [local count: 1073741825]: >> x_6 = i_5(D) >> 3; >> v = x_6; >> _1 = (long unsigned int) index_9(D); >> _2 = _1 * 4; >> _3 = &v + _2; >> _4 = *_3; >> _10 = (long int) _4; >> v ={v} {CLOBBER}; >> return _10; >> >> } >> >> In the first SSA version, we are lacking the shift operation: >> >> x_6 = i_5(D) >> 3; >> v = x_6; >> >> So, we need to see in which of the optimizations and transformations >> our code is lost: >> To see all the steps we should execute: >> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all >> >> This will generate a lot, really a lot, of files in the same directory. >> They have the name: prb.c.xxx.yyyy >> Where xxx is the number of the step, and yyyy is the name of the step. >> Each of the files contains the result of applying the changes, so what >> I have done is looking in binary search where my code was lost. >> >> I have found that the problem was in the first dead code elimination >> step (prb.c.041t.cddce1) >> GCC does not like the crap code that we are writing, as we are copying >> a stack variable and then accessing directly to the memory: >> >> v = x; >> return ((int*)(&v))[index]; >> >> So, basically, it was assuming that we were only using v and not x, so >> all the code modifying x is described (this optimization is called >> "dead store code deletion"). Basically tries to remove all the code >> that is affecting stack (temporary) variables that are never used. >> >> I am not sure if this is a bug in GCC, so I will report it. But I have >> fixed the problem writing the code in a proper way. >> >> Sorry again for the long mail, I wanted to store the knowledge and >> propagate it before I eventually will flush it. I like these things, >> but I am not debugging the GCC optimization pipeline all the days. >> >> -- >> Pablo Tesone. >> tesonep(a)gmail.com >> >>

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by tesonep＠gmail.com

Hi Aliaksei, to me it looks like a bug of GCC optimization. Basically, it is assuming that the x variable is used but never read or its value is never used. Also it assumes the same of the i variable, as we are only accessing indirectly to the memory where it locates (the code is even assuming that the variable exists, but it can be optimize out as in this scenario). Even though, the original C code is valid C code, we are not helping the compiler writing code like that. So I have rewritten the code in a way that does not use indirect memory access to the stack space. One thing more that makes me think is a bug, if you use an int constant as the index and not a parameter, the error does not occur (the code is not badly optimized) and there is a warning about the not-so-great access to the stack. On Wed, Dec 11, 2019 at 6:01 PM Aliaksei Syrel <alex.syrel(a)gmail.com> wrote: > > Hi Pablo, > > Wow! Thank you for the detective story :) > > Do I understand correctly that the original code causes undefined behavior and therefore can be changed (or even removed) by the compiler? > (because it returns something that is referencing memory on the stack) > > Please keep posting similar things in future! It is very educative :) > > Cheers, > Alex > > > On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> wrote: >> >> Hi, >> this mail is related to Pharo because it is knowledge I found >> debugging the build of the VM, but the rest is to document it and >> perhaps someone will found it interesting (also I couldn't find it >> easily using Google). Sorry for the long mail! >> >> The problem >> ========== >> >> The following code does not produce good code in 8.3 when using optimizations: >> >> long __attribute__ ((noinline)) myFunc(long i, int index){ >> long v; >> long x = i >> 3; >> >> v = x; >> return ((int*)(&v))[index]; >> } >> >> #include <stdio.h> >> >> int main(){ >> >> long i; >> int x; >> >> scanf("%ld", &i); >> scanf("%d", &x); >> >> printf("%ld",myFunc(i,x)); >> } >> >> Basically, with -02, it generates the following code: >> >> myFunc: >> movslq %esi, %rsi >> movslq -8(%rsp,%rsi,4), %rax >> ret >> >> And with -01 it generates the following code: >> >> myFunc: >> sarq $3, %rdi >> movq %rdi, -8(%rsp) >> movslq %esi, %rsi >> movslq -8(%rsp,%rsi,4), %rax >> ret >> >> As you can see, the more optimized version is losing the bit shift (or >> any other operation). >> To detect the problem it was not easy, basically, I have used the >> Pharo Tests to detect the failing function and then to understand the >> generation of code in GCC, we need to debug its generation. >> >> The first snippet is generated using: >> >> gcc -O2 prb.c -S -o prb.s -Wall >> >> and the second using: >> >> gcc -O1 prb.c -S -o prb.s -Wall >> >> The GCC pipeline has different steps, I have centered my self in the >> tree optimizations. >> GCC dumps each of the intermediate states of the compilation, working >> with C like trees. For example to get the optimized version of the >> program, before generating the Assembler we have to do: >> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout >> >> This will generate: >> >> __attribute__((noinline)) >> myFunc (long int i, int index) >> { >> long int v; >> long unsigned int _1; >> long unsigned int _2; >> int * _3; >> int _4; >> long int _8; >> >> <bb 2> [local count: 1073741825]: >> _1 = (long unsigned int) index_7(D); >> _2 = _1 * 4; >> _3 = &v + _2; >> _4 = *_3; >> _8 = (long int) _4; >> v ={v} {CLOBBER}; >> return _8; >> >> } >> >> This is the optimized SSA (static single assign) version of the >> function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) >> As you can see in this version the code is already optimized, and our >> operations not correctly generated. We expect to see something like: >> >> __attribute__((noinline)) >> myFunc (long int i, int index) >> { >> long int x; >> long int v; >> long unsigned int _1; >> long unsigned int _2; >> int * _3; >> int _4; >> long int _10; >> >> <bb 2> [local count: 1073741825]: >> x_6 = i_5(D) >> 3; >> v = x_6; >> _1 = (long unsigned int) index_9(D); >> _2 = _1 * 4; >> _3 = &v + _2; >> _4 = *_3; >> _10 = (long int) _4; >> v ={v} {CLOBBER}; >> return _10; >> >> } >> >> In the first SSA version, we are lacking the shift operation: >> >> x_6 = i_5(D) >> 3; >> v = x_6; >> >> So, we need to see in which of the optimizations and transformations >> our code is lost: >> To see all the steps we should execute: >> >> gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all >> >> This will generate a lot, really a lot, of files in the same directory. >> They have the name: prb.c.xxx.yyyy >> Where xxx is the number of the step, and yyyy is the name of the step. >> Each of the files contains the result of applying the changes, so what >> I have done is looking in binary search where my code was lost. >> >> I have found that the problem was in the first dead code elimination >> step (prb.c.041t.cddce1) >> GCC does not like the crap code that we are writing, as we are copying >> a stack variable and then accessing directly to the memory: >> >> v = x; >> return ((int*)(&v))[index]; >> >> So, basically, it was assuming that we were only using v and not x, so >> all the code modifying x is described (this optimization is called >> "dead store code deletion"). Basically tries to remove all the code >> that is affecting stack (temporary) variables that are never used. >> >> I am not sure if this is a bug in GCC, so I will report it. But I have >> fixed the problem writing the code in a proper way. >> >> Sorry again for the long mail, I wanted to store the knowledge and >> propagate it before I eventually will flush it. I like these things, >> but I am not debugging the GCC optimization pipeline all the days. >> >> -- >> Pablo Tesone. >> tesonep(a)gmail.com >> -- Pablo Tesone. tesonep(a)gmail.com

Dec. 11, 2019

Re: [Pharo-dev] Debugging GCC code generation

by Aliaksei Syrel

Hi Pablo, Wow! Thank you for the detective story :) Do I understand correctly that the original code causes undefined behavior and therefore can be changed (or even removed) by the compiler? (because it returns something that is referencing memory on the stack) Please keep posting similar things in future! It is very educative :) Cheers, Alex On Wed, 11 Dec 2019 at 17:35, tesonep(a)gmail.com <tesonep(a)gmail.com> wrote: > Hi, > this mail is related to Pharo because it is knowledge I found > debugging the build of the VM, but the rest is to document it and > perhaps someone will found it interesting (also I couldn't find it > easily using Google). Sorry for the long mail! > > The problem > ========== > > The following code does not produce good code in 8.3 when using > optimizations: > > long __attribute__ ((noinline)) myFunc(long i, int index){ > long v; > long x = i >> 3; > > v = x; > return ((int*)(&v))[index]; > } > > #include <stdio.h> > > int main(){ > > long i; > int x; > > scanf("%ld", &i); > scanf("%d", &x); > > printf("%ld",myFunc(i,x)); > } > > Basically, with -02, it generates the following code: > > myFunc: > movslq %esi, %rsi > movslq -8(%rsp,%rsi,4), %rax > ret > > And with -01 it generates the following code: > > myFunc: > sarq $3, %rdi > movq %rdi, -8(%rsp) > movslq %esi, %rsi > movslq -8(%rsp,%rsi,4), %rax > ret > > As you can see, the more optimized version is losing the bit shift (or > any other operation). > To detect the problem it was not easy, basically, I have used the > Pharo Tests to detect the failing function and then to understand the > generation of code in GCC, we need to debug its generation. > > The first snippet is generated using: > > gcc -O2 prb.c -S -o prb.s -Wall > > and the second using: > > gcc -O1 prb.c -S -o prb.s -Wall > > The GCC pipeline has different steps, I have centered my self in the > tree optimizations. > GCC dumps each of the intermediate states of the compilation, working > with C like trees. For example to get the optimized version of the > program, before generating the Assembler we have to do: > > gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout > > This will generate: > > __attribute__((noinline)) > myFunc (long int i, int index) > { > long int v; > long unsigned int _1; > long unsigned int _2; > int * _3; > int _4; > long int _8; > > <bb 2> [local count: 1073741825]: > _1 = (long unsigned int) index_7(D); > _2 = _1 * 4; > _3 = &v + _2; > _4 = *_3; > _8 = (long int) _4; > v ={v} {CLOBBER}; > return _8; > > } > > This is the optimized SSA (static single assign) version of the > function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) > As you can see in this version the code is already optimized, and our > operations not correctly generated. We expect to see something like: > > __attribute__((noinline)) > myFunc (long int i, int index) > { > long int x; > long int v; > long unsigned int _1; > long unsigned int _2; > int * _3; > int _4; > long int _10; > > <bb 2> [local count: 1073741825]: > x_6 = i_5(D) >> 3; > v = x_6; > _1 = (long unsigned int) index_9(D); > _2 = _1 * 4; > _3 = &v + _2; > _4 = *_3; > _10 = (long int) _4; > v ={v} {CLOBBER}; > return _10; > > } > > In the first SSA version, we are lacking the shift operation: > > x_6 = i_5(D) >> 3; > v = x_6; > > So, we need to see in which of the optimizations and transformations > our code is lost: > To see all the steps we should execute: > > gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all > > This will generate a lot, really a lot, of files in the same directory. > They have the name: prb.c.xxx.yyyy > Where xxx is the number of the step, and yyyy is the name of the step. > Each of the files contains the result of applying the changes, so what > I have done is looking in binary search where my code was lost. > > I have found that the problem was in the first dead code elimination > step (prb.c.041t.cddce1) > GCC does not like the crap code that we are writing, as we are copying > a stack variable and then accessing directly to the memory: > > v = x; > return ((int*)(&v))[index]; > > So, basically, it was assuming that we were only using v and not x, so > all the code modifying x is described (this optimization is called > "dead store code deletion"). Basically tries to remove all the code > that is affecting stack (temporary) variables that are never used. > > I am not sure if this is a bug in GCC, so I will report it. But I have > fixed the problem writing the code in a proper way. > > Sorry again for the long mail, I wanted to store the knowledge and > propagate it before I eventually will flush it. I like these things, > but I am not debugging the GCC optimization pipeline all the days. > > -- > Pablo Tesone. > tesonep(a)gmail.com > >

Dec. 11, 2019

Debugging GCC code generation

by tesonep＠gmail.com

Hi, this mail is related to Pharo because it is knowledge I found debugging the build of the VM, but the rest is to document it and perhaps someone will found it interesting (also I couldn't find it easily using Google). Sorry for the long mail! The problem ========== The following code does not produce good code in 8.3 when using optimizations: long __attribute__ ((noinline)) myFunc(long i, int index){ long v; long x = i >> 3; v = x; return ((int*)(&v))[index]; } #include <stdio.h> int main(){ long i; int x; scanf("%ld", &i); scanf("%d", &x); printf("%ld",myFunc(i,x)); } Basically, with -02, it generates the following code: myFunc: movslq %esi, %rsi movslq -8(%rsp,%rsi,4), %rax ret And with -01 it generates the following code: myFunc: sarq $3, %rdi movq %rdi, -8(%rsp) movslq %esi, %rsi movslq -8(%rsp,%rsi,4), %rax ret As you can see, the more optimized version is losing the bit shift (or any other operation). To detect the problem it was not easy, basically, I have used the Pharo Tests to detect the failing function and then to understand the generation of code in GCC, we need to debug its generation. The first snippet is generated using: gcc -O2 prb.c -S -o prb.s -Wall and the second using: gcc -O1 prb.c -S -o prb.s -Wall The GCC pipeline has different steps, I have centered my self in the tree optimizations. GCC dumps each of the intermediate states of the compilation, working with C like trees. For example to get the optimized version of the program, before generating the Assembler we have to do: gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-optimized=/dev/stdout This will generate: __attribute__((noinline)) myFunc (long int i, int index) { long int v; long unsigned int _1; long unsigned int _2; int * _3; int _4; long int _8; <bb 2> [local count: 1073741825]: _1 = (long unsigned int) index_7(D); _2 = _1 * 4; _3 = &v + _2; _4 = *_3; _8 = (long int) _4; v ={v} {CLOBBER}; return _8; } This is the optimized SSA (static single assign) version of the function (https://gcc.gnu.org/onlinedocs/gccint/SSA.html) As you can see in this version the code is already optimized, and our operations not correctly generated. We expect to see something like: __attribute__((noinline)) myFunc (long int i, int index) { long int x; long int v; long unsigned int _1; long unsigned int _2; int * _3; int _4; long int _10; <bb 2> [local count: 1073741825]: x_6 = i_5(D) >> 3; v = x_6; _1 = (long unsigned int) index_9(D); _2 = _1 * 4; _3 = &v + _2; _4 = *_3; _10 = (long int) _4; v ={v} {CLOBBER}; return _10; } In the first SSA version, we are lacking the shift operation: x_6 = i_5(D) >> 3; v = x_6; So, we need to see in which of the optimizations and transformations our code is lost: To see all the steps we should execute: gcc -O2 prb.c -S -o prb.s -Wall -fdump-tree-all This will generate a lot, really a lot, of files in the same directory. They have the name: prb.c.xxx.yyyy Where xxx is the number of the step, and yyyy is the name of the step. Each of the files contains the result of applying the changes, so what I have done is looking in binary search where my code was lost. I have found that the problem was in the first dead code elimination step (prb.c.041t.cddce1) GCC does not like the crap code that we are writing, as we are copying a stack variable and then accessing directly to the memory: v = x; return ((int*)(&v))[index]; So, basically, it was assuming that we were only using v and not x, so all the code modifying x is described (this optimization is called "dead store code deletion"). Basically tries to remove all the code that is affecting stack (temporary) variables that are never used. I am not sure if this is a bug in GCC, so I will report it. But I have fixed the problem writing the code in a proper way. Sorry again for the long mail, I wanted to store the knowledge and propagate it before I eventually will flush it. I like these things, but I am not debugging the GCC optimization pipeline all the days. -- Pablo Tesone. tesonep(a)gmail.com

Dec. 11, 2019

[Pharo 8.0] Build #1065: Revert "[Perfs] x10 on method compiling with #deferFlushDuring:"

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1065 was: SUCCESS. The Pull Request #5359 was integrated: "Revert "[Perfs] x10 on method compiling with #deferFlushDuring:"" Pull request url: https://github.com/pharo-project/pharo/pull/5359 Issue Url: https://github.com/pharo-project/pharo/issues/revert Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Dec. 11, 2019

Re: [Pharo-dev] Willow Bootstrap & JQueryUI loading errors

by Gabriel Cotelli

Sanjay, I've tried it on a Windows machine with a fresh Pharo installation and it works fine. The v12 branch of Willow is pointing to the same commit that v12.1.0 tag so this shouldn't be a problem. Can you describe in more detail the error that you're getting and the steps to reproduce from a fresh Pharo install? I know that the loading in Windows is really painful specially for projects in filetree format like Seaside. If you don't need the tests you can load the "Deployment" group that pulls less parts of Seaside onto the image so it should be faster. You can also use the #willow channel in the Pharo discord server or in Buenos Aires Smalltalk Slack (ask me for an invitation if you want) to ask questions. Regards Gabriel On Wed, Dec 11, 2019 at 7:09 AM Sanjay Minni <sm(a)planage.com> wrote: > I get an error when trying to load Willow-bootstrap and willow-JQueryUI > using > the Metacello script > > (prior to this I have loaded Willow 12.1.0 first in Pharo 8 / Windows 10) > > maybe this has something to do with the following line in > setUpDependencies: > baseline: 'Willow' with: [ spec repository: > 'github://ba-st/Willow:v12/source' ] > > if I pull in BaselineOfWillow-Bootstrap / JqueryUI thru iceberg, then edit > Willow:v12 to Willow:12.1.0, then I can successfully load thru iceberg > right-click metacello(default) > > except that it takes too long - seems it goes thru all the packages incl > seaside etc each time > > can the Baseline(s) be fixed in the appropriate way and can the loading be > optimised > > thanks > Sanjay > > > > > > ----- > cheers, > Sanjay > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html > >

Dec. 11, 2019

Re: [Pharo-dev] code loading performance

by Marcus Denker

Maybe the best is to revert for now https://github.com/pharo-project/pharo/pull/5359 <https://github.com/pharo-project/pharo/pull/5359> > On 11 Dec 2019, at 14:39, George Ganea <georgeganea(a)gmail.com> wrote: > > Hi Vincent, > > In the end we tried it after it was merged :) Our CI uses the latest Pharo 8 image. > Unfortunately we did not see an improvement, and even worse, it seems to break one of our builds that is downstream from GToolkit. We got all sorts of errors that seemed to be related to incomplete method code. > Two examples below. > > Syntax Error on line 2: ''']'' expected' > ======================================== > 1: privateState > 2: ^ privateState ifNil: [ p > > groupedByIntervalType: aTemporalIntervalType fromTimestamp: startTimestamp toTimestamp: endTimestamp > ^ self > groupedByIntervalType: aTemporalIntervalType > inGroupOfType: self temporalGroupType > from: (aTemporalIntervalT')' expected -> > > Weâve now switched our CI to a fixed version image that is prior to this merge. > Even though the code weâre loading is in a private repo, please let me know if there is anything I can do to help debug this. > > Thank you, > George > >> On 6 Dec 2019, at 19:05, George Ganea <georgeganea(a)gmail.com <mailto:georgeganea@gmail.com>> wrote: >> >> Hi Vincent, >> >> I will definitely try it out and report back. >> >> Thank you! >> George >>> Hi Georges, >>> >>> There have been quite some improvements those last weeks on the performance of loading classes and methods. But we are still waiting for the https://github.com/pharo-project/pharo/pull/5292 <https://github.com/pharo-project/pharo/pull/5292> to be integrated. >>> >>> And you have to encapsulate the loading code into: SourceFiles deferFlushDuring: [...] and use the latest pharo 8.0 image. >>> >>> You can give a try with this and tell us how it goes! >>> >>> Cheers, >>> Vincent >>> >>> -----Original Message----- >>> From: Pharo-dev On Behalf Of George Ganea >>> Sent: Friday, 6 December 2019 17:08 >>> To: pharo-dev at lists.pharo.org <http://lists.pharo.org/mailman/listinfo/pharo-dev_lists.pharo.org> >>> Cc: Chis Vasile Andrei <dedal18 at yahoo.com <http://lists.pharo.org/mailman/listinfo/pharo-dev_lists.pharo.org>> >>> Subject: [Pharo-dev] code loading performance >>> >>> Hi all, >>> >>> Currently loading GToolkit takes quite some time (arount 18 minutes at the best of times) we were wondering if thereâs been any attempts at improving code loading times in Metacello/Monticello. Or mabye there are some ideas on how one might start doing something like this? >>> >>> Cheers, >>> George >> >

Dec. 11, 2019