New stable VM for Pharo8?
Hi all, Esteban, In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on. While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them. Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra. Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair ( https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... ). We have tested that particular commit ( a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8? Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :) Ideas? Thanks, Guille and Pablo
Thanks guille and pablo for you dedication for Pharo, I love you :) Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra? Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>).
We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
On Tue, Apr 2, 2019 at 2:11 PM ducasse <stepharo@netcourrier.com> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)
Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?
I don't know exactly on what versions it is not working because I have Windows10 and Mojave. But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...). AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX. In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen. I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation. In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair ( https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... ).
We have tested that particular commit ( a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Hi, For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help): -metal Use metal if the Metal detection code works. -opengl Use OpenGL. The previous default. -core-graphics Use the older CoreGraphics based backend. -headless Null without rendering This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument. On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem. Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess. Best regards, Ronie El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (< guillermopolito@gmail.com>) escribió:
On Tue, Apr 2, 2019 at 2:11 PM ducasse <stepharo@netcourrier.com> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)
Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?
I don't know exactly on what versions it is not working because I have Windows10 and Mojave. But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).
AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX. In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.
I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation. In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair ( https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... ).
We have tested that particular commit ( a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
--
Guille Polito
Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille
CRIStAL - UMR 9189
French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*
*Web:* *http://guillep.github.io* <http://guillep.github.io>
*Phone: *+33 06 52 70 66 13
Thanks ronie for the explanation. I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :( May be Pharo should have a branch so that we can promote a Pharo stable VM. I feel uncomfortable to have to remember which commits is the stable one for us because how can we progress there? I do not see how we can jump over commits. And my concern is how can we improve the time you and Pablo and Guille are spending handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle regressions and unsure that we can blessed a VM a stable for real. We should discuss it during PharoDays consortium meeting and the next consortium online meeting. Stef
On 2 Apr 2019, at 16:56, Ronie Salgado <roniesalg@gmail.com> wrote:
Hi,
For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):
-metal Use metal if the Metal detection code works. -opengl Use OpenGL. The previous default. -core-graphics Use the older CoreGraphics based backend. -headless Null without rendering
This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.
On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.
Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.
Best regards, Ronie
El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>>) escribió:
On Tue, Apr 2, 2019 at 2:11 PM ducasse <stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>> wrote: Thanks guille and pablo for you dedication for Pharo, I love you :)
Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?
I don't know exactly on what versions it is not working because I have Windows10 and Mojave. But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).
AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX. In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.
I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation. In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>).
We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
--
Guille Polito Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr <http://www.cnrs.fr/>
Web: http://guillep.github.io <http://guillep.github.io/> Phone: +33 06 52 70 66 13
Hi,
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :(
The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution. My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it. The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame. Best regards, Ronie El mar., 2 abr. 2019 a las 12:28, ducasse (<stepharo@netcourrier.com>) escribió:
Thanks ronie for the explanation. I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(
May be Pharo should have a branch so that we can promote a Pharo stable VM. I feel uncomfortable to have to remember which commits is the stable one for us because how can we progress there? I do not see how we can jump over commits.
And my concern is how can we improve the time you and Pablo and Guille are spending handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle regressions and unsure that we can blessed a VM a stable for real.
We should discuss it during PharoDays consortium meeting and the next consortium online meeting.
Stef
On 2 Apr 2019, at 16:56, Ronie Salgado <roniesalg@gmail.com> wrote:
Hi,
For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):
-metal Use metal if the Metal detection code works. -opengl Use OpenGL. The previous default. -core-graphics Use the older CoreGraphics based backend. -headless Null without rendering
This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.
On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.
Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.
Best regards, Ronie
El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (< guillermopolito@gmail.com>) escribió:
On Tue, Apr 2, 2019 at 2:11 PM ducasse <stepharo@netcourrier.com> wrote:
Thanks guille and pablo for you dedication for Pharo, I love you :)
Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?
I don't know exactly on what versions it is not working because I have Windows10 and Mojave. But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).
AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX. In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.
I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation. In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair ( https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... ).
We have tested that particular commit ( a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
--
Guille Polito Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr/>*
*Web:* *http://guillep.github.io* <http://guillep.github.io/> *Phone: *+33 06 52 70 66 13
Ok thanks ronie. This is why the headless VM is on P8 roadmap. I think that Esteban will start to work on it in May. But he can confirm his agenda. Stef
On 2 Apr 2019, at 17:46, Ronie Salgado <roniesalg@gmail.com> wrote:
Hi,
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.
My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.
The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.
Best regards, Ronie
El mar., 2 abr. 2019 a las 12:28, ducasse (<stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>>) escribió: Thanks ronie for the explanation. I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(
May be Pharo should have a branch so that we can promote a Pharo stable VM. I feel uncomfortable to have to remember which commits is the stable one for us because how can we progress there? I do not see how we can jump over commits.
And my concern is how can we improve the time you and Pablo and Guille are spending handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle regressions and unsure that we can blessed a VM a stable for real.
We should discuss it during PharoDays consortium meeting and the next consortium online meeting.
Stef
On 2 Apr 2019, at 16:56, Ronie Salgado <roniesalg@gmail.com <mailto:roniesalg@gmail.com>> wrote:
Hi,
For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):
-metal Use metal if the Metal detection code works. -opengl Use OpenGL. The previous default. -core-graphics Use the older CoreGraphics based backend. -headless Null without rendering
This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.
On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.
Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.
Best regards, Ronie
El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>>) escribió:
On Tue, Apr 2, 2019 at 2:11 PM ducasse <stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>> wrote: Thanks guille and pablo for you dedication for Pharo, I love you :)
Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?
I don't know exactly on what versions it is not working because I have Windows10 and Mojave. But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).
AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX. In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.
I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation. In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>).
We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
--
Guille Polito Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr <http://www.cnrs.fr/>
Web: http://guillep.github.io <http://guillep.github.io/> Phone: +33 06 52 70 66 13
On 2 Apr 2019, at 17:53, ducasse <stepharo@netcourrier.com> wrote:
Ok thanks ronie. This is why the headless VM is on P8 roadmap. I think that Esteban will start to work on it in May. But he can confirm his agenda.
Well, indirectly we are already working on it :) Pablo will concentrate on it in may (while I continue working on Spec2 until we have a beta). I wanted to work on it myself, but this time distributions fits better :) Esteban
Stef
On 2 Apr 2019, at 17:46, Ronie Salgado <roniesalg@gmail.com <mailto:roniesalg@gmail.com>> wrote:
Hi,
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.
My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.
The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.
Best regards, Ronie
El mar., 2 abr. 2019 a las 12:28, ducasse (<stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>>) escribió: Thanks ronie for the explanation. I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(
May be Pharo should have a branch so that we can promote a Pharo stable VM. I feel uncomfortable to have to remember which commits is the stable one for us because how can we progress there? I do not see how we can jump over commits.
And my concern is how can we improve the time you and Pablo and Guille are spending handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle regressions and unsure that we can blessed a VM a stable for real.
We should discuss it during PharoDays consortium meeting and the next consortium online meeting.
Stef
On 2 Apr 2019, at 16:56, Ronie Salgado <roniesalg@gmail.com <mailto:roniesalg@gmail.com>> wrote:
Hi,
For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):
-metal Use metal if the Metal detection code works. -opengl Use OpenGL. The previous default. -core-graphics Use the older CoreGraphics based backend. -headless Null without rendering
This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.
On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.
Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.
Best regards, Ronie
El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>>) escribió:
On Tue, Apr 2, 2019 at 2:11 PM ducasse <stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>> wrote: Thanks guille and pablo for you dedication for Pharo, I love you :)
Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?
I don't know exactly on what versions it is not working because I have Windows10 and Mojave. But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).
AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX. In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.
I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation. In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>).
We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
--
Guille Polito Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr <http://www.cnrs.fr/>
Web: http://guillep.github.io <http://guillep.github.io/> Phone: +33 06 52 70 66 13
On 2 Apr 2019, at 20:30, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On 2 Apr 2019, at 17:53, ducasse <stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>> wrote:
Ok thanks ronie. This is why the headless VM is on P8 roadmap. I think that Esteban will start to work on it in May. But he can confirm his agenda.
Well, indirectly we are already working on it :) Pablo will concentrate on it in may (while I continue working on Spec2 until we have a beta).
And many thanks to Schmidt on Lifeware, because of them I can say âPablo will do itâ instead saying âI will see how I multiply myself to work on itâ :D Esteban
I wanted to work on it myself, but this time distributions fits better :)
Esteban
Stef
On 2 Apr 2019, at 17:46, Ronie Salgado <roniesalg@gmail.com <mailto:roniesalg@gmail.com>> wrote:
Hi,
I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( The plugin itself is optional, but it depends on code being exposed from the VM windowing system in order to blit a final rendered rectangle on top the VM window. Since Apple is deprecating OpenGL, then supporting Metal (or another Apple supported alternative) is a necessity. I am more of a fan of the no VM window solution.
My biggest problem with Metal is on testing with older versions of OS X, and with older versions of the hardware itself. It works perfectly on my Macbook air Early 2015 (The VM, and Woden). But it does not work for example on Alexandre Bergel older desktop machine (black windows with Woden). Tomorrow I will be debugging Woden on Alexandre machine, and I will take some time on trying to debug this VM problem, if I can reproduce it.
The way of debugging a Metal application is by running it with the XCode debugger, enabling GPU debugging, and capturing some frames in the debugger that which allows to inspect the draw calls and the resources that are used for generating the frame.
Best regards, Ronie
El mar., 2 abr. 2019 a las 12:28, ducasse (<stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>>) escribió: Thanks ronie for the explanation. I thought that plugin was a plugin, i.e., optional but it looks like it is not. :( When I read your mail, it looks like we have to problem with a plugin we do not use. Even better :(
May be Pharo should have a branch so that we can promote a Pharo stable VM. I feel uncomfortable to have to remember which commits is the stable one for us because how can we progress there? I do not see how we can jump over commits.
And my concern is how can we improve the time you and Pablo and Guille are spending handling such case. This is ok to look for a bug and fix, now we will have to put in place a process to handle regressions and unsure that we can blessed a VM a stable for real.
We should discuss it during PharoDays consortium meeting and the next consortium online meeting.
Stef
On 2 Apr 2019, at 16:56, Ronie Salgado <roniesalg@gmail.com <mailto:roniesalg@gmail.com>> wrote:
Hi,
For a job I have to adapt old Squeak code (B3DAcceleratorPlugin) to newer graphics APIs (Metal and D3D11). The VM window has to use the same rendering interface as the plugin for rendering. Recently for trying to solve the issue with the older machines I implemented support selecting the older renderers with a command line option, and I added code for trying to detect whether the current Metal implementation is valid, or not. These command line options are the followings (can be obtained by using -help):
-metal Use metal if the Metal detection code works. -opengl Use OpenGL. The previous default. -core-graphics Use the older CoreGraphics based backend. -headless Null without rendering
This command line option has to be the first command line option passed to the VM program. This is because the normal command line parsing is done after creating the VM window. As a workaround I am parsing the command line option on the main() function as the first command line argument.
On my Mac machine I have Mojave installed. If someone could give pointers on how to setup a VM with the older version, or a way to emulate the older version I could be able to debug the problem.
Perhaps as a workaround, for Pharo we should disable the Metal support (Remove the -DUSE_METAL from the makefile), and maybe go for the CoreGraphics backend, which seems to be better tested and I think does not give conflict problems when trying to use OpenGL from the image side. In Pharo we do not use the B3DAcceleratorPlugin, which in my opinion is a complete mess.
Best regards, Ronie
El mar., 2 abr. 2019 a las 10:58, Guillermo Polito (<guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>>) escribió:
On Tue, Apr 2, 2019 at 2:11 PM ducasse <stepharo@netcourrier.com <mailto:stepharo@netcourrier.com>> wrote: Thanks guille and pablo for you dedication for Pharo, I love you :)
Now just that I understand, does it mean that the latest open smalltalk vm does not run on anything else that Mojave and High Sierra?
I don't know exactly on what versions it is not working because I have Windows10 and Mojave. But I know that it's not working for Pablo on OSX Sierra (which dates from 2016...).
AFAIU, Ronie has updated the VM's code to render on Metal, which made that at some point the VM was not compiling for latest versions of OSX. In these last weeks Ronie has also updated the code to make it compile in older machines, but still in those versions that not support Metal the main window just renders a black screen.
I don't know what are the following steps regarding this, I put Ronie in cc, he may clarify a bit the situation. In the meanwhile, using the build I say in the email above may suffice us all I think ^^.
Stef (back writing research proposalâ¦..)
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>).
We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
--
Guille Polito Research Engineer
Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr <http://www.cnrs.fr/>
Web: http://guillep.github.io <http://guillep.github.io/> Phone: +33 06 52 70 66 13
Which reminds me https://www.amazon.com/Scientific-Progress-Goes-Boink-Collection/dp/08362187... <https://www.amazon.com/Scientific-Progress-Goes-Boink-Collection/dp/08362187...> is about calvin creating a machine to create clones so that they clean his room. And after the clones are getting as lazy as him and they fight and the machine explodes :)
Just bumping, I've marked with Pablo the VM a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...> as stable for Pharo8. This should fix the directory issues and the primitive ones. For those using the Pharo launcher: - if you use 64 bits, use the VM manager to update your VMs (remove the old 80 VM) [image: image.png] [image: image.png] - If you re using 32 bits, you'll have to wait a bit more. I've found an issue in the middle, fix in progress here (with a test, yay!) https://github.com/pharo-project/pharo/pull/3169 Guille and Pablo On Tue, Apr 2, 2019 at 8:53 PM ducasse <stepharo@netcourrier.com> wrote:
Which reminds me https://www.amazon.com/Scientific-Progress-Goes-Boink-Collection/dp/08362187...
is about calvin creating a machine to create clones so that they clean his room. And after the clones are getting as lazy as him and they fight and the machine explodes :)
-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
Hi Guille, On Tue, 2 Apr 2019 at 13:40, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...).
We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Cyril has already reported this issue in Pharo 8, so I agree that an updated VM for Pharo 8 would be nice. As you say, this doesn't affect Pharo 7 (or earlier). There is another, similar, issue that does affect Pharo 7 and earlier: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/368 which is resolved by: commit e9fee6facb61c1163e820673bd8734b59a831535 Author: AlistairGrant <akgrant0710@gmail.com> Date: Wed Feb 6 08:23:32 2019 +0100 FilePlugin: reopen the directory when jumping around In dir_Lookup, when jumping around, i.e. index ~= lastIndex+1, reopen the directory rather than rewinding. This is required as the open directory caches entries (at least with CIFS mounted file system), and if files are deleted between calls the wrong answer will be returned. Issue: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/368 This issue doesn't affect Pharo 8 since dir_Lookup() isn't used anymore. I'm only aware of this being reported once, and the issue has probably been there since 2010, or maybe even earlier. It is worthwhile being aware of it, but probably not enough to force a new VM for Pharo 7 immediately. Cheers, Alistair
Hi,
On 2 Apr 2019, at 13:40, Guillermo Polito <guillermopolito@gmail.com> wrote:
Hi all, Esteban,
In the last sprint with Pablo and Pierre we have detected several misbehaviours with files. The issue was particular reproducible when trying to run iceberg tests: the VM kind of ran out of slots for opening files. The strangest thing is that the leaks seemed to happen when deleting directories, so we had a look at the changes done to iterate directories, get file attributes and so on.
While on the FileReference code everything seemed ok, we have observerd with `lsof` that still several directories remained open even after deleting them.
Latest VM fixes this issue but contains another one related to Metal rendering, which prevents rendering in OSX versions < than High Sierra.
Tracking the issue a bit, this seems to be a bug in the FileAttributes plugin that was fixed by this PR by alistair (https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28... <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>).
We have tested that particular commit (a838346b1a67712cc28298534dafbd0c26ea34fb <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/a838346b1a67712cc28...>) and this one contains Alistair's fixes but not yet the new rendering using Metal. We have been testing that VM since yesterday and it seems stable, what do you think about marking it stable for Pharo8
I will do it tomorrow morning :) Esteban
?
Regarding Pharo7, Pharo7's stable VM is the same as Pharo8's, so it contains the same issue. However, since on the image side the code does not use the problematic primitives, we do not think we need to change the VM for Pharo7 (yet). However, there is seemingly a fix for directories containing blanks, maybe Alistair can give us some input here? :)
Ideas?
Thanks, Guille and Pablo
participants (5)
-
Alistair Grant -
ducasse -
Esteban Lorenzano -
Guillermo Polito -
Ronie Salgado