Fwd: [Vm-dev] why GT forked the VM and how to kill our fork
_,,,^..^,,,_ (phone) Begin forwarded message:
From: Eliot Miranda <eliot.miranda@gmail.com> Date: April 26, 2020 at 10:39:35 AM PDT To: Open Smalltalk Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org> Subject: Re: [Vm-dev] why GT forked the VM and how to kill our fork
Hi Doru,
On Apr 26, 2020, at 12:57 AM, Tudor Girba <tudor@tudorgirba.com> wrote:
 Hi,
Currently, at feenk we have feenkcom/opensmalltalk-vm: https://github.com/feenkcom/opensmalltalk-vm
This is a small fork of the headless branch from pharo-project/opensmalltalk-vm that appeared out of practical necessities, but that we would like to avoid having. This post briefly describes the changes in the feenkcom/opensmalltalk-vm repo and the functionality those changes provide for Glamorous Toolkit.
For Glamorous Toolkit we aimed for the following functionality: ⢠Open the GUI natively and have native display quality (GUI opened through FFI calls) ⢠Have a Glamorous Toolkit app for Mac OS that works as any other apps for Mac OS ⢠Create end-user applications that are fully customisable (executable name, menus, etc) ⢠Use Github actions for doing all compilations of external libraries and the vm instead of Travis CI. ⢠Have Iceberg running in native windows (which requires nested FFI callbacks)
There has been work on these issues in both OpenSmalltalk/opensmalltalk-vm and pharo-project/opensmalltalk-vm but they were not entirely addressed. We needed to have something reliable a few months ago, and forking and doing some quick changes made that possible.
Ideally we want to be able to run Glamorous Toolkit on both OpenSmalltalk/opensmalltalk-vm and pharo-project/opensmalltalk-vm.
To have native GUIs we relied on Ronie Salgadoâs work on the headless vm and started with pharo-project/opensmalltalk-vm - headless branch: https://github.com/pharo-project/opensmalltalk-vm/tree/headless That provided a solution for opening the GUI from the image through FFI calls. Currently we use Glutin (a library for OpenGL context creation, written in pure Rust) and this made it possible to run the entire Glamorous Toolkit inside a callback.
On macOS when running an app, even a notarized one, the OS warns the user that the app is downloaded from the internet, and the user needs to confirm that they agree. Once the user agrees the app should automatically start. This is not currently possible with Pharo apps (for example PharoLaunched.app) and users have to again run the app manually after giving permission. Also Gatekeeper in macOS runs applications downloaded from zips in a randomized read-only DMG. We do not want this behaviour as users not copying Glamorous Toolkit to the Applications folder on macOS would then experience incorrect application behaviour.
To create end-user applications we also need to fully customize the executable name (what the user sees in the Task Runner/Activity monitor), icons, native menus. Part of this work is already integrated in the pharo-project/opensmalltalk-vm - headless branch (Customizing the OS X icons, Brand the VM executable and package).
Since last year Github offers Github Actions similar to Travis. We found it much easier to use than Travis for external libraries and the vm. Also we get to manage the code and the builds in the same place. This work is already integrated in the pharo-project/opensmalltalk-vm - headless branch (Build the VM under GitHub actions: https://github.com/pharo-project/opensmalltalk-vm/pull/56).
The issues related to running Iceberg is a bit more technical. By moving to the headless vm we are running the entire image computation inside a callback from Glutin (https://github.com/rust-windowing/glutin/). When using Iceberg we get nested callbacks which we could not get to work using Alien. Instead we are using the ThreadedFFI Plugin and running all callback from Iceberg and Glutin using the Threaded FFI plugin (https://github.com/pharo-project/threadedFFI-Plugin). Currently we have a small fork of this plugin (feenkcom/threadedFFI-Plugin) and we also ship a custom plugin with the VM to fix a race condition due to having two copies of the callback stack (a pull request is here: https://github.com/pharo-project/threadedFFI-Plugin/pull/17).
One detail here is that Alien provides the callback machinery used alongside ThreadedFFIPlugin. The Alien call-out machinery is an old hack used only by Newspeak, that could be eliminated in non-Newspeak builds. But AFAIA Alienâs callback machinery is *the* callback machinery everywhere for the OpenSmalltalk vm.
While not specific to our environment, openssl1.0 is no longer supported, and we are seeing users who are unable to run Pharo due to version conflicts, as reported in https://github.com/pharo-project/opensmalltalk-vm/issues/62.
To sum up, a fork was the easiest way to get all this running. Now some changes are already in the pharo-project/opensmalltalk-vm - headless branch. What we are still missing are the changes that get the VM to work nicely with Mac OS and a bug fix in ThreadedFFI.
We would also love it to have all these changes integrated in OpenSmalltalk/opensmalltalk-vm in the headless vm. This requires additional coordination as the required changes are somewhat deeper.
Can it be done with pull requests? Should we try and firm a team of volunteers, eg one from feenk and one from OpenSmalltalk to try and merge in the feenk code?
I would volunteer but I am way too busy getting the Terf system working. TFir e able, the ARMv8 JIT has been working for months but Iâve had no time to release it because of Terf and other payed work (got to pay the bills).
Please let us know you would prefer to coordinate.
Any volunteers?
Cheers, Tudor, on behalf of the feenk team
-- feenk.com
"The coherence of a trip is given by the clearness of the goal."
Hi all, I want to add a little more details, event that the mail of Doru was much more than clear. I can tell for more details that those changes have been integrated in December, thanks to Github it is easy to check the changes in the PRs 42, 50, 53, 56, 57 and 58 that Ronie has proposed and that have been integrated (https://github.com/pharo-project/opensmalltalk-vm/pulls?q=is%3Apr+is%3Aclose...). There is a single PR that Alistair have open, that I need to do some work to be able to implement it. As the C code generator requires a way of adding the EXPORT declaration in the generated function that requires so, so it is not needed to do by hand. Also, the changes proposed to the Queue-based ThreadedFFI plugin have been integrated (https://github.com/pharo-project/threadedFFI-Plugin/pulls?q=is%3Apr+is%3Aclo...). Thanks Ronie and Alistair for the collaborations. The remaining PR is not integrated as it removes a tool that we consider useful for debugging the incorrect return order of Callbacks. We have added checks in the plugin and in the image side to detect the attempts to return in an incorrect order the callbacks avoiding corruptions in the C stack. Just another clarification, that maybe Alistair can explain more, they are using the queue-based solution as it can handle better concurrent calls to Smalltalk code without the need to be inside a callout to C. As this is a known limitation of the scheme where the active Smalltalk process is hijacked to execute the callback. We are more than thankfully to the contribution of Doru, Ronie, Alistair and all the team of Feenk to the improvement of Pharo and the VM. They are doing an excellent job, and it is not only these contributions but also the work they have done to migrate the Smalltalk code of the VM into Git preserving the history, without that it would have been imposible to get to the state where we are. Cheers, Pablo On Mon, Apr 27, 2020 at 6:01 PM Eliot Miranda <eliot.miranda@gmail.com> wrote:
_,,,^..^,,,_ (phone)
Begin forwarded message:
From: Eliot Miranda <eliot.miranda@gmail.com> Date: April 26, 2020 at 10:39:35 AM PDT To: Open Smalltalk Virtual Machine Development Discussion <vm-dev@lists.squeakfoundation.org> Subject: Re: [Vm-dev] why GT forked the VM and how to kill our fork
Hi Doru,
On Apr 26, 2020, at 12:57 AM, Tudor Girba <tudor@tudorgirba.com> wrote:

Hi,
Currently, at feenk we have feenkcom/opensmalltalk-vm:
https://github.com/feenkcom/opensmalltalk-vm
This is a small fork of the headless branch from pharo-project/opensmalltalk-vm that appeared out of practical necessities, but that we would like to avoid having. This post briefly describes the changes in the feenkcom/opensmalltalk-vm repo and the functionality those changes provide for Glamorous Toolkit.
For Glamorous Toolkit we aimed for the following functionality:
⢠Open the GUI natively and have native display quality (GUI opened through FFI calls)
⢠Have a Glamorous Toolkit app for Mac OS that works as any other apps for Mac OS
⢠Create end-user applications that are fully customisable (executable name, menus, etc)
⢠Use Github actions for doing all compilations of external libraries and the vm instead of Travis CI.
⢠Have Iceberg running in native windows (which requires nested FFI callbacks)
There has been work on these issues in both OpenSmalltalk/opensmalltalk-vm and pharo-project/opensmalltalk-vm but they were not entirely addressed. We needed to have something reliable a few months ago, and forking and doing some quick changes made that possible.
Ideally we want to be able to run Glamorous Toolkit on both OpenSmalltalk/opensmalltalk-vm and pharo-project/opensmalltalk-vm.
To have native GUIs we relied on Ronie Salgadoâs work on the headless vm and started with pharo-project/opensmalltalk-vm - headless branch:
https://github.com/pharo-project/opensmalltalk-vm/tree/headless
That provided a solution for opening the GUI from the image through FFI calls. Currently we use Glutin (a library for OpenGL context creation, written in pure Rust) and this made it possible to run the entire Glamorous Toolkit inside a callback.
On macOS when running an app, even a notarized one, the OS warns the user that the app is downloaded from the internet, and the user needs to confirm that they agree. Once the user agrees the app should automatically start. This is not currently possible with Pharo apps (for example PharoLaunched.app) and users have to again run the app manually after giving permission. Also Gatekeeper in macOS runs applications downloaded from zips in a randomized read-only DMG. We do not want this behaviour as users not copying Glamorous Toolkit to the Applications folder on macOS would then experience incorrect application behaviour.
To create end-user applications we also need to fully customize the executable name (what the user sees in the Task Runner/Activity monitor), icons, native menus. Part of this work is already integrated in the pharo-project/opensmalltalk-vm - headless branch (Customizing the OS X icons, Brand the VM executable and package).
Since last year Github offers Github Actions similar to Travis. We found it much easier to use than Travis for external libraries and the vm. Also we get to manage the code and the builds in the same place. This work is already integrated in the pharo-project/opensmalltalk-vm - headless branch (Build the VM under GitHub actions: https://github.com/pharo-project/opensmalltalk-vm/pull/56).
The issues related to running Iceberg is a bit more technical. By moving to the headless vm we are running the entire image computation inside a callback from Glutin (https://github.com/rust-windowing/glutin/). When using Iceberg we get nested callbacks which we could not get to work using Alien. Instead we are using the ThreadedFFI Plugin and running all callback from Iceberg and Glutin using the Threaded FFI plugin (https://github.com/pharo-project/threadedFFI-Plugin). Currently we have a small fork of this plugin (feenkcom/threadedFFI-Plugin) and we also ship a custom plugin with the VM to fix a race condition due to having two copies of the callback stack (a pull request is here: https://github.com/pharo-project/threadedFFI-Plugin/pull/17).
One detail here is that Alien provides the callback machinery used alongside ThreadedFFIPlugin. The Alien call-out machinery is an old hack used only by Newspeak, that could be eliminated in non-Newspeak builds. But AFAIA Alienâs callback machinery is *the* callback machinery everywhere for the OpenSmalltalk vm.
While not specific to our environment, openssl1.0 is no longer supported, and we are seeing users who are unable to run Pharo due to version conflicts, as reported in https://github.com/pharo-project/opensmalltalk-vm/issues/62.
To sum up, a fork was the easiest way to get all this running. Now some changes are already in the pharo-project/opensmalltalk-vm - headless branch. What we are still missing are the changes that get the VM to work nicely with Mac OS and a bug fix in ThreadedFFI.
We would also love it to have all these changes integrated in OpenSmalltalk/opensmalltalk-vm in the headless vm. This requires additional coordination as the required changes are somewhat deeper.
Can it be done with pull requests? Should we try and firm a team of volunteers, eg one from feenk and one from OpenSmalltalk to try and merge in the feenk code?
I would volunteer but I am way too busy getting the Terf system working. TFir e able, the ARMv8 JIT has been working for months but Iâve had no time to release it because of Terf and other payed work (got to pay the bills).
Please let us know you would prefer to coordinate.
Any volunteers?
Cheers,
Tudor, on behalf of the feenk team
--
feenk.com
"The coherence of a trip is given by the clearness of the goal."
-- Pablo Tesone. tesonep@gmail.com
participants (2)
-
Eliot Miranda -
tesonep＠gmail.com