[ANN] CPPBridge: One Ring to rule them ALL
*https://youtu.be/pI4PR3XaX6Q <https://youtu.be/pI4PR3XaX6Q>* *What is it ?* CPPBridge is a library that allows Pharo to share memory with any C/C++ application. Opening the door not only to IPC and data sharing but also even complete control of C++ code from Pharo and vice versa. *How to install ?* In a few hours it should be available from Package Browser, if not you can always fetch it with Metacello from here because it comes with a Baseline https://github.com/kilon/CPPBridge *Why bother making such a library ? * In my saga to find a way to use Pharo as a scripting language for Unreal Game Engine, I had two options a) Build Unreal as a Library and use Pharo UFFI to launch and control it b) Embed Pharo inside the Unreal Executable (this is what every other scripting language uses for controlling Unreal) Option (a) was a no go, because Unreal is huge , complex and uses its own custom made build tools, making a DLL for Pharo or an army of DLLs out of the question Option (b) Embeding Pharo inside an executable is impossible and implementing it also insanely complex Naturally my mind went first into sockets which is what I use to make Pharo able to use Python libraries. However sockets have proven too slow for the insanely fast loops of Unreal. *What are the advantages ?* 1) *No need to move data around.* Sharing memory means you don't have to move data around, you can use directly the shared memory 2)* Extend the Pharo image beyond Pharo.* Shared memory is mapped into a file means that you can do with C++ what you can do with Pharo image, save the live state directly to a binary file. That means we no longer need to worry about sessions and reinitializing C/C++ data since memory mapped file acts as an extension of the Pharo image. 3) *Blazing fast. *Memory mapping is a function that comes directly from the OS kernel and as such it has to be extremely fast. Memory mapping is also what is used for dynamically linked shared libraries an extremely important feature for any application including Pharo that heavily depends on (see Cairo for Athens). So its a very mature , well tested method. 4) *No extra libraries needed* to be installed, CPPBridge uses OS libraries to work out of the box 5) *Low level handling of memory.* Direct access to memory you can even manipulate the memory byte by byte 6)* Memory efficient.* Memory mapping excels at large data, the bigger the better. Can take full advantage of your entire free memory and not waste a byte. That means also that can be used to optimise Pharo memory, since you could compress your Pharo objects to bytes and mapped file will store the live state. 7) *Tons of Languages. *Because memory mapping is a core functionality for every OS out there, pretty much every programming language supports it. CPPBridge currently supports only C/C++ but all languages can be supported giving access to Pharo to any library for any programming language. Sky is the limit 8) *Self Documented. *CPPBridge is small, simple and with large class comment and comments for every method. YouTube video tutorial also available and linked on top. 9) *Works both ways*, C/C++ and Pharo can access and modify the shared memory. Making it possible for C/C++ to use Pharo libraries and Pharo to use C/C++ libraries. 10) Experiments have proven that it improves sex life... if it does not please file a bug report ;) *What are the disadvantages ?* 1) *Candy Crash Saga*. Dare do something incorrectly and Pharo will crash. CPPBridge can easily point to wrong address if you are not aware of what you doing. 2) *C++/C* . If you think you can avoid learning C/C++ and that this is a magic solution , think again. The C/C++ application must be modified to include shared memory mapping for CPPBridge to work . 3) *Local only*. Unlike sockets, Shared Memory works only on the same machine so no remote execution and manipulation of code like in my socket bridge to Python 4) *UFFI still No1 option*. No replacement for UFFI it actually depends on it to work , so if you can turn your C/C++ code into a DLL that should be your first option. *Roadmap * Currently CPPBridge works only on MacOS , most likely on Linux too (because it uses the Unix architecture) but I will have to test it. Windows is coming next ASAP, since its my No1 platform for creating commercial games. Maybe establish a small protocol of communication via the Shared Memory , JSON looks like a good universal format *Thanks* Big thanks to Eliot for inspiring me and Esteban for helping me figure out things.
JSON? -- But you were talking about speed. Once you put JSON as a base for IPC, then say goodbye to speed. Because JSON parsing/writing will eat like 90% of time even if you would use sockets. So, why not using some kind of binary protocol? Unless you wanna use JSON for initial handshaking exchange. Then it is quite reasonable. But all of that still won't free you from inventing some kind of contract(s) which would allow parties to agree who writes where and/or who writes, while other waits then reads etc. On 9 November 2016 at 00:06, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
*https://youtu.be/pI4PR3XaX6Q <https://youtu.be/pI4PR3XaX6Q>*
*What is it ?*
CPPBridge is a library that allows Pharo to share memory with any C/C++ application. Opening the door not only to IPC and data sharing but also even complete control of C++ code from Pharo and vice versa.
*How to install ?*
In a few hours it should be available from Package Browser, if not you can always fetch it with Metacello from here because it comes with a Baseline
https://github.com/kilon/CPPBridge
*Why bother making such a library ? *
In my saga to find a way to use Pharo as a scripting language for Unreal Game Engine, I had two options
a) Build Unreal as a Library and use Pharo UFFI to launch and control it b) Embed Pharo inside the Unreal Executable (this is what every other scripting language uses for controlling Unreal)
Option (a) was a no go, because Unreal is huge , complex and uses its own custom made build tools, making a DLL for Pharo or an army of DLLs out of the question
Option (b) Embeding Pharo inside an executable is impossible and implementing it also insanely complex
Naturally my mind went first into sockets which is what I use to make Pharo able to use Python libraries. However sockets have proven too slow for the insanely fast loops of Unreal.
*What are the advantages ?*
1) *No need to move data around.* Sharing memory means you don't have to move data around, you can use directly the shared memory
2)* Extend the Pharo image beyond Pharo.* Shared memory is mapped into a file means that you can do with C++ what you can do with Pharo image, save the live state directly to a binary file. That means we no longer need to worry about sessions and reinitializing C/C++ data since memory mapped file acts as an extension of the Pharo image.
3) *Blazing fast. *Memory mapping is a function that comes directly from the OS kernel and as such it has to be extremely fast. Memory mapping is also what is used for dynamically linked shared libraries an extremely important feature for any application including Pharo that heavily depends on (see Cairo for Athens). So its a very mature , well tested method.
4) *No extra libraries needed* to be installed, CPPBridge uses OS libraries to work out of the box
5) *Low level handling of memory.* Direct access to memory you can even manipulate the memory byte by byte
6)* Memory efficient.* Memory mapping excels at large data, the bigger the better. Can take full advantage of your entire free memory and not waste a byte. That means also that can be used to optimise Pharo memory, since you could compress your Pharo objects to bytes and mapped file will store the live state.
7) *Tons of Languages. *Because memory mapping is a core functionality for every OS out there, pretty much every programming language supports it. CPPBridge currently supports only C/C++ but all languages can be supported giving access to Pharo to any library for any programming language. Sky is the limit
8) *Self Documented. *CPPBridge is small, simple and with large class comment and comments for every method. YouTube video tutorial also available and linked on top.
9) *Works both ways*, C/C++ and Pharo can access and modify the shared memory. Making it possible for C/C++ to use Pharo libraries and Pharo to use C/C++ libraries.
10) Experiments have proven that it improves sex life... if it does not please file a bug report ;)
*What are the disadvantages ?*
1) *Candy Crash Saga*. Dare do something incorrectly and Pharo will crash. CPPBridge can easily point to wrong address if you are not aware of what you doing.
2) *C++/C* . If you think you can avoid learning C/C++ and that this is a magic solution , think again. The C/C++ application must be modified to include shared memory mapping for CPPBridge to work .
3) *Local only*. Unlike sockets, Shared Memory works only on the same machine so no remote execution and manipulation of code like in my socket bridge to Python
4) *UFFI still No1 option*. No replacement for UFFI it actually depends on it to work , so if you can turn your C/C++ code into a DLL that should be your first option.
*Roadmap *
Currently CPPBridge works only on MacOS , most likely on Linux too (because it uses the Unix architecture) but I will have to test it.
Windows is coming next ASAP, since its my No1 platform for creating commercial games.
Maybe establish a small protocol of communication via the Shared Memory , JSON looks like a good universal format
*Thanks*
Big thanks to Eliot for inspiring me and Esteban for helping me figure out things.
-- Best regards, Igor Stasenko.
I have only played with c structs and it looks like they do the job fine . If json parsing is slow especially in Pharo then it's out of the question. Parsing must be kept close to zero because I will be running some very tight loops of 100 frames per second and there is no way that I will let Pharo take its time as the whole idea of using the fastest way to IPC was to keep Pharo in sync with unreal . Heavy lifting will be done only by Unreal. Pharo will be used just for logic. In any case there is a ton of testing and profiling needed to be done . So these are just the first steps. On Wed, 9 Nov 2016 at 02:58, Igor Stasenko <siguctua@gmail.com> wrote:
JSON? -- But you were talking about speed. Once you put JSON as a base for IPC, then say goodbye to speed.
Because JSON parsing/writing will eat like 90% of time even if you would use sockets. So, why not using some kind of binary protocol? Unless you wanna use JSON for initial handshaking exchange. Then it is quite reasonable.
But all of that still won't free you from inventing some kind of contract(s) which would allow parties to agree who writes where and/or who writes, while other waits then reads etc.
On 9 November 2016 at 00:06, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
*https://youtu.be/pI4PR3XaX6Q <https://youtu.be/pI4PR3XaX6Q>*
*What is it ?*
CPPBridge is a library that allows Pharo to share memory with any C/C++ application. Opening the door not only to IPC and data sharing but also even complete control of C++ code from Pharo and vice versa.
*How to install ?*
In a few hours it should be available from Package Browser, if not you can always fetch it with Metacello from here because it comes with a Baseline
https://github.com/kilon/CPPBridge
*Why bother making such a library ? *
In my saga to find a way to use Pharo as a scripting language for Unreal Game Engine, I had two options
a) Build Unreal as a Library and use Pharo UFFI to launch and control it b) Embed Pharo inside the Unreal Executable (this is what every other scripting language uses for controlling Unreal)
Option (a) was a no go, because Unreal is huge , complex and uses its own custom made build tools, making a DLL for Pharo or an army of DLLs out of the question
Option (b) Embeding Pharo inside an executable is impossible and implementing it also insanely complex
Naturally my mind went first into sockets which is what I use to make Pharo able to use Python libraries. However sockets have proven too slow for the insanely fast loops of Unreal.
*What are the advantages ?*
1) *No need to move data around.* Sharing memory means you don't have to move data around, you can use directly the shared memory
2)* Extend the Pharo image beyond Pharo.* Shared memory is mapped into a file means that you can do with C++ what you can do with Pharo image, save the live state directly to a binary file. That means we no longer need to worry about sessions and reinitializing C/C++ data since memory mapped file acts as an extension of the Pharo image.
3) *Blazing fast. *Memory mapping is a function that comes directly from the OS kernel and as such it has to be extremely fast. Memory mapping is also what is used for dynamically linked shared libraries an extremely important feature for any application including Pharo that heavily depends on (see Cairo for Athens). So its a very mature , well tested method.
4) *No extra libraries needed* to be installed, CPPBridge uses OS libraries to work out of the box
5) *Low level handling of memory.* Direct access to memory you can even manipulate the memory byte by byte
6)* Memory efficient.* Memory mapping excels at large data, the bigger the better. Can take full advantage of your entire free memory and not waste a byte. That means also that can be used to optimise Pharo memory, since you could compress your Pharo objects to bytes and mapped file will store the live state.
7) *Tons of Languages. *Because memory mapping is a core functionality for every OS out there, pretty much every programming language supports it. CPPBridge currently supports only C/C++ but all languages can be supported giving access to Pharo to any library for any programming language. Sky is the limit
8) *Self Documented. *CPPBridge is small, simple and with large class comment and comments for every method. YouTube video tutorial also available and linked on top.
9) *Works both ways*, C/C++ and Pharo can access and modify the shared memory. Making it possible for C/C++ to use Pharo libraries and Pharo to use C/C++ libraries.
10) Experiments have proven that it improves sex life... if it does not please file a bug report ;)
*What are the disadvantages ?*
1) *Candy Crash Saga*. Dare do something incorrectly and Pharo will crash. CPPBridge can easily point to wrong address if you are not aware of what you doing.
2) *C++/C* . If you think you can avoid learning C/C++ and that this is a magic solution , think again. The C/C++ application must be modified to include shared memory mapping for CPPBridge to work .
3) *Local only*. Unlike sockets, Shared Memory works only on the same machine so no remote execution and manipulation of code like in my socket bridge to Python
4) *UFFI still No1 option*. No replacement for UFFI it actually depends on it to work , so if you can turn your C/C++ code into a DLL that should be your first option.
*Roadmap *
Currently CPPBridge works only on MacOS , most likely on Linux too (because it uses the Unix architecture) but I will have to test it.
Windows is coming next ASAP, since its my No1 platform for creating commercial games.
Maybe establish a small protocol of communication via the Shared Memory , JSON looks like a good universal format
*Thanks*
Big thanks to Eliot for inspiring me and Esteban for helping me figure out things.
-- Best regards, Igor Stasenko.
On 9 November 2016 at 02:09, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
I have only played with c structs and it looks like they do the job fine . If json parsing is slow especially in Pharo then it's out of the question. Parsing must be kept close to zero because I will be running some very tight loops of 100 frames per second and there is no way that I will let Pharo take its time as the whole idea of using the fastest way to IPC was to keep Pharo in sync with unreal . Heavy lifting will be done only by Unreal. Pharo will be used just for logic.
Well, it depends what you want: if you want to give away a complete/general solution for IPC with Pharo via shared memory, then you should create some kind of library (both C++ and Pharo) which would allow users to setup the bridge configuration in the way they want, leaving the application-specific exchange to the hands of users.. And if you just want to make own data exchange for own purpose, then you are basically done.
In any case there is a ton of testing and profiling needed to be done . So these are just the first steps. On Wed, 9 Nov 2016 at 02:58, Igor Stasenko <siguctua@gmail.com> wrote:
JSON? -- But you were talking about speed. Once you put JSON as a base for IPC, then say goodbye to speed.
Because JSON parsing/writing will eat like 90% of time even if you would use sockets. So, why not using some kind of binary protocol? Unless you wanna use JSON for initial handshaking exchange. Then it is quite reasonable.
But all of that still won't free you from inventing some kind of contract(s) which would allow parties to agree who writes where and/or who writes, while other waits then reads etc.
On 9 November 2016 at 00:06, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
*https://youtu.be/pI4PR3XaX6Q <https://youtu.be/pI4PR3XaX6Q>*
*What is it ?*
CPPBridge is a library that allows Pharo to share memory with any C/C++ application. Opening the door not only to IPC and data sharing but also even complete control of C++ code from Pharo and vice versa.
*How to install ?*
In a few hours it should be available from Package Browser, if not you can always fetch it with Metacello from here because it comes with a Baseline
https://github.com/kilon/CPPBridge
*Why bother making such a library ? *
In my saga to find a way to use Pharo as a scripting language for Unreal Game Engine, I had two options
a) Build Unreal as a Library and use Pharo UFFI to launch and control it b) Embed Pharo inside the Unreal Executable (this is what every other scripting language uses for controlling Unreal)
Option (a) was a no go, because Unreal is huge , complex and uses its own custom made build tools, making a DLL for Pharo or an army of DLLs out of the question
Option (b) Embeding Pharo inside an executable is impossible and implementing it also insanely complex
Naturally my mind went first into sockets which is what I use to make Pharo able to use Python libraries. However sockets have proven too slow for the insanely fast loops of Unreal.
*What are the advantages ?*
1) *No need to move data around.* Sharing memory means you don't have to move data around, you can use directly the shared memory
2)* Extend the Pharo image beyond Pharo.* Shared memory is mapped into a file means that you can do with C++ what you can do with Pharo image, save the live state directly to a binary file. That means we no longer need to worry about sessions and reinitializing C/C++ data since memory mapped file acts as an extension of the Pharo image.
3) *Blazing fast. *Memory mapping is a function that comes directly from the OS kernel and as such it has to be extremely fast. Memory mapping is also what is used for dynamically linked shared libraries an extremely important feature for any application including Pharo that heavily depends on (see Cairo for Athens). So its a very mature , well tested method.
4) *No extra libraries needed* to be installed, CPPBridge uses OS libraries to work out of the box
5) *Low level handling of memory.* Direct access to memory you can even manipulate the memory byte by byte
6)* Memory efficient.* Memory mapping excels at large data, the bigger the better. Can take full advantage of your entire free memory and not waste a byte. That means also that can be used to optimise Pharo memory, since you could compress your Pharo objects to bytes and mapped file will store the live state.
7) *Tons of Languages. *Because memory mapping is a core functionality for every OS out there, pretty much every programming language supports it. CPPBridge currently supports only C/C++ but all languages can be supported giving access to Pharo to any library for any programming language. Sky is the limit
8) *Self Documented. *CPPBridge is small, simple and with large class comment and comments for every method. YouTube video tutorial also available and linked on top.
9) *Works both ways*, C/C++ and Pharo can access and modify the shared memory. Making it possible for C/C++ to use Pharo libraries and Pharo to use C/C++ libraries.
10) Experiments have proven that it improves sex life... if it does not please file a bug report ;)
*What are the disadvantages ?*
1) *Candy Crash Saga*. Dare do something incorrectly and Pharo will crash. CPPBridge can easily point to wrong address if you are not aware of what you doing.
2) *C++/C* . If you think you can avoid learning C/C++ and that this is a magic solution , think again. The C/C++ application must be modified to include shared memory mapping for CPPBridge to work .
3) *Local only*. Unlike sockets, Shared Memory works only on the same machine so no remote execution and manipulation of code like in my socket bridge to Python
4) *UFFI still No1 option*. No replacement for UFFI it actually depends on it to work , so if you can turn your C/C++ code into a DLL that should be your first option.
*Roadmap *
Currently CPPBridge works only on MacOS , most likely on Linux too (because it uses the Unix architecture) but I will have to test it.
Windows is coming next ASAP, since its my No1 platform for creating commercial games.
Maybe establish a small protocol of communication via the Shared Memory , JSON looks like a good universal format
*Thanks*
Big thanks to Eliot for inspiring me and Esteban for helping me figure out things.
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
I am the only potential customer of CPPBridge because a) I am the only C++ coder AFAIK b) Using DLLs (dynamically linked shared libraries) is still by far the best solution because using the UFFI you can do everything from Pharo. Compared to CPPBridge which you will need to code in C++. So as you can imagine making a general library for IPC communication via shared memory is not my goal. The Unreal part will be a separate library that will depend on this one. I have not touched that one yet but I will soon. UnrealBridge library will become my most important Pharo project because the goal is to start selling games using Pharo and Unreal. IPC Protocol wise , I don't think synchronisation will be an issue because Unreal will be too fast for Pharo anyway and its ok if Pharo lags behind even a few dozens of milliseconds which is a matter of losing a couple of frames. I may implement a hybrid synchronisation for when Pharo lag becomes too severe but that will have and see in practice. Something that interest me is "stealing" your idea for NBOpenGL where you made an automatic generator for the library that used OpenGL headers. This is something that could benefit me a lot because Unreal contains over 10.000 methods and wrapping them one by one is not something I am looking forward to doing. This means generating both C++ and Pharo code. Of course if I feel that what I implement with UnrealBridge is general purpose enough I can port it back to CPPBridge. In any case I must say it was a lot of fun accomplishing this goal and cant wait to dive deeper into IPC magic, making Pharo the nerve centre, the brain , of my coding environment is a role that I think Pharo would excel at. It will also allow me to use Pharo as much as possible and provide a super powerful game / graphics engine to the Pharo community. Shared memory is that just the means to that goal. Moose could also a play role later on for analysing my game code both in Pharo and C++ and use Roassal for some nifty visualizations though I could use Unreal as backend to Roassal for some nice 3d visualisations. So ton of ideas, tons of fun and a ton of time to accomplish them. PS: I have to say you have been inspiration for my IPC saga, you did the unthinkable (at least to me) and brought Assembly to Pharo, I never had a use for Assembly but nonetheless you taught me that Pharo potential is more massive than I imagined, so thank you :) On Wed, Nov 9, 2016 at 3:21 AM Igor Stasenko <siguctua@gmail.com> wrote:
On 9 November 2016 at 02:09, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
I have only played with c structs and it looks like they do the job fine . If json parsing is slow especially in Pharo then it's out of the question. Parsing must be kept close to zero because I will be running some very tight loops of 100 frames per second and there is no way that I will let Pharo take its time as the whole idea of using the fastest way to IPC was to keep Pharo in sync with unreal . Heavy lifting will be done only by Unreal. Pharo will be used just for logic.
Well, it depends what you want: if you want to give away a complete/general solution for IPC with Pharo via shared memory, then you should create some kind of library (both C++ and Pharo) which would allow users to setup the bridge configuration in the way they want, leaving the application-specific exchange to the hands of users.. And if you just want to make own data exchange for own purpose, then you are basically done.
In any case there is a ton of testing and profiling needed to be done . So these are just the first steps. On Wed, 9 Nov 2016 at 02:58, Igor Stasenko <siguctua@gmail.com> wrote:
JSON? -- But you were talking about speed. Once you put JSON as a base for IPC, then say goodbye to speed.
Because JSON parsing/writing will eat like 90% of time even if you would use sockets. So, why not using some kind of binary protocol? Unless you wanna use JSON for initial handshaking exchange. Then it is quite reasonable.
But all of that still won't free you from inventing some kind of contract(s) which would allow parties to agree who writes where and/or who writes, while other waits then reads etc.
On 9 November 2016 at 00:06, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
*https://youtu.be/pI4PR3XaX6Q <https://youtu.be/pI4PR3XaX6Q>*
*What is it ?*
CPPBridge is a library that allows Pharo to share memory with any C/C++ application. Opening the door not only to IPC and data sharing but also even complete control of C++ code from Pharo and vice versa.
*How to install ?*
In a few hours it should be available from Package Browser, if not you can always fetch it with Metacello from here because it comes with a Baseline
https://github.com/kilon/CPPBridge
*Why bother making such a library ? *
In my saga to find a way to use Pharo as a scripting language for Unreal Game Engine, I had two options
a) Build Unreal as a Library and use Pharo UFFI to launch and control it b) Embed Pharo inside the Unreal Executable (this is what every other scripting language uses for controlling Unreal)
Option (a) was a no go, because Unreal is huge , complex and uses its own custom made build tools, making a DLL for Pharo or an army of DLLs out of the question
Option (b) Embeding Pharo inside an executable is impossible and implementing it also insanely complex
Naturally my mind went first into sockets which is what I use to make Pharo able to use Python libraries. However sockets have proven too slow for the insanely fast loops of Unreal.
*What are the advantages ?*
1) *No need to move data around.* Sharing memory means you don't have to move data around, you can use directly the shared memory
2)* Extend the Pharo image beyond Pharo.* Shared memory is mapped into a file means that you can do with C++ what you can do with Pharo image, save the live state directly to a binary file. That means we no longer need to worry about sessions and reinitializing C/C++ data since memory mapped file acts as an extension of the Pharo image.
3) *Blazing fast. *Memory mapping is a function that comes directly from the OS kernel and as such it has to be extremely fast. Memory mapping is also what is used for dynamically linked shared libraries an extremely important feature for any application including Pharo that heavily depends on (see Cairo for Athens). So its a very mature , well tested method.
4) *No extra libraries needed* to be installed, CPPBridge uses OS libraries to work out of the box
5) *Low level handling of memory.* Direct access to memory you can even manipulate the memory byte by byte
6)* Memory efficient.* Memory mapping excels at large data, the bigger the better. Can take full advantage of your entire free memory and not waste a byte. That means also that can be used to optimise Pharo memory, since you could compress your Pharo objects to bytes and mapped file will store the live state.
7) *Tons of Languages. *Because memory mapping is a core functionality for every OS out there, pretty much every programming language supports it. CPPBridge currently supports only C/C++ but all languages can be supported giving access to Pharo to any library for any programming language. Sky is the limit
8) *Self Documented. *CPPBridge is small, simple and with large class comment and comments for every method. YouTube video tutorial also available and linked on top.
9) *Works both ways*, C/C++ and Pharo can access and modify the shared memory. Making it possible for C/C++ to use Pharo libraries and Pharo to use C/C++ libraries.
10) Experiments have proven that it improves sex life... if it does not please file a bug report ;)
*What are the disadvantages ?*
1) *Candy Crash Saga*. Dare do something incorrectly and Pharo will crash. CPPBridge can easily point to wrong address if you are not aware of what you doing.
2) *C++/C* . If you think you can avoid learning C/C++ and that this is a magic solution , think again. The C/C++ application must be modified to include shared memory mapping for CPPBridge to work .
3) *Local only*. Unlike sockets, Shared Memory works only on the same machine so no remote execution and manipulation of code like in my socket bridge to Python
4) *UFFI still No1 option*. No replacement for UFFI it actually depends on it to work , so if you can turn your C/C++ code into a DLL that should be your first option.
*Roadmap *
Currently CPPBridge works only on MacOS , most likely on Linux too (because it uses the Unix architecture) but I will have to test it.
Windows is coming next ASAP, since its my No1 platform for creating commercial games.
Maybe establish a small protocol of communication via the Shared Memory , JSON looks like a good universal format
*Thanks*
Big thanks to Eliot for inspiring me and Esteban for helping me figure out things.
-- Best regards, Igor Stasenko.
-- Best regards, Igor Stasenko.
On 9 November 2016 at 08:41, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
I am the only potential customer of CPPBridge because
a) I am the only C++ coder AFAIK b) Using DLLs (dynamically linked shared libraries) is still by far the best solution because using the UFFI you can do everything from Pharo. Compared to CPPBridge which you will need to code in C++.
So as you can imagine making a general library for IPC communication via shared memory is not my goal.
The Unreal part will be a separate library that will depend on this one. I have not touched that one yet but I will soon.
UnrealBridge library will become my most important Pharo project because the goal is to start selling games using Pharo and Unreal.
IPC Protocol wise , I don't think synchronisation will be an issue because Unreal will be too fast for Pharo anyway and its ok if Pharo lags behind even a few dozens of milliseconds which is a matter of losing a couple of frames. I may implement a hybrid synchronisation for when Pharo lag becomes too severe but that will have and see in practice.
Something that interest me is "stealing" your idea for NBOpenGL where you made an automatic generator for the library that used OpenGL headers. This is something that could benefit me a lot because Unreal contains over 10.000 methods and wrapping them one by one is not something I am looking forward to doing. This means generating both C++ and Pharo code.
Well, a while before that, i wrote own lobotomized smalltalk implementation in C and started generating bindings to Ogre3D engine. I even had certain success with it, but then i started grand-rewriting of VM and abandoned it. That was before i switched to Squeak, then Pharo. So, your incentives quite familiar to me :) I was quite fun journey, and i learned a lot while doing it.. especially about smalltalk. Of course if I feel that what I implement with UnrealBridge is general
purpose enough I can port it back to CPPBridge.
In any case I must say it was a lot of fun accomplishing this goal and cant wait to dive deeper into IPC magic, making Pharo the nerve centre, the brain , of my coding environment is a role that I think Pharo would excel at. It will also allow me to use Pharo as much as possible and provide a super powerful game / graphics engine to the Pharo community. Shared memory is that just the means to that goal.
Moose could also a play role later on for analysing my game code both in Pharo and C++ and use Roassal for some nifty visualizations though I could use Unreal as backend to Roassal for some nice 3d visualisations.
So ton of ideas, tons of fun and a ton of time to accomplish them.
PS: I have to say you have been inspiration for my IPC saga, you did the unthinkable (at least to me) and brought Assembly to Pharo, I never had a use for Assembly but nonetheless you taught me that Pharo potential is more massive than I imagined, so thank you :)
I don't understand why people find assembly scary or mind-boggling? I dived into assembly few months after i learned my first programming language, it was Zilog 8-bit CPU. A marvelous gem :) And this was always fascinating to feel that you can control the computer's behaviour down to a tiniest detail. We were even researching what certain i/o ports and interrupts were responsible for by setting different bits/bytes here and there and see what happen. Because if you don't understand something down to the tiniest detail - you cannot be sure that what you doing will work, or work optimally. So i find it frustrating that most of programmers don't know and not even thinking about touching assembly. Because it very simple, straightforward and megalomaniac-rewarding :) -- Best regards, Igor Stasenko.
Well, a while before that, i wrote own lobotomized smalltalk implementation in C and started generating bindings to Ogre3D engine. I even had certain success with it, but then i started grand-rewriting of VM and abandoned it. That was before i switched to Squeak, then Pharo. So, your incentives quite familiar to me :) I was quite fun journey, and i learned a lot while doing it.. especially about smalltalk.
I have no interest into implementing Smalltalk at C side , neither my purpose is education, quite the contrary I try to find way to utilize Pharo the best way possible without compromising on performance
I don't understand why people find assembly scary or mind-boggling? I dived into assembly few months after i learned my first programming language, it was Zilog 8-bit CPU. A marvelous gem :) And this was always fascinating to feel that you can control the computer's behaviour down to a tiniest detail. We were even researching what certain i/o ports and interrupts were responsible for by setting different bits/bytes here and there and see what happen. Because if you don't understand something down to the tiniest detail - you cannot be sure that what you doing will work, or work optimally. So i find it frustrating that most of programmers don't know and not even thinking about touching assembly. Because it very simple, straightforward and megalomaniac-rewarding :) -- Best regards, Igor Stasenko.
Assembly is hard, just compare its hello world with the hello world of other progamming languages. Its just insane. The thing is that because we started on early , we experienced assembly when it used to be fairly simple. I started with an Amstrad CPC 6128 with 128 kb of ram and 4 mhz CPU, it uses Z80 assembly which is fairly simple even when compared to Pharo. Modern Assembly however have grown on complexity because it depend on the complexity of the hardware. Personally I don even like to call it a programming language , just beautified machine code. I agree though Assembly is a lot of fun and a great source of knowledge.
On 10 November 2016 at 11:36, Dimitris Chloupis <kilon.alios@gmail.com> wrote:
Well, a while before that, i wrote own lobotomized smalltalk implementation in C and started generating bindings to Ogre3D engine. I even had certain success with it, but then i started grand-rewriting of VM and abandoned it. That was before i switched to Squeak, then Pharo. So, your incentives quite familiar to me :) I was quite fun journey, and i learned a lot while doing it.. especially about smalltalk.
I have no interest into implementing Smalltalk at C side , neither my purpose is education, quite the contrary I try to find way to utilize Pharo the best way possible without compromising on performance
I don't understand why people find assembly scary or mind-boggling? I dived into assembly few months after i learned my first programming language, it was Zilog 8-bit CPU. A marvelous gem :) And this was always fascinating to feel that you can control the computer's behaviour down to a tiniest detail. We were even researching what certain i/o ports and interrupts were responsible for by setting different bits/bytes here and there and see what happen. Because if you don't understand something down to the tiniest detail - you cannot be sure that what you doing will work, or work optimally. So i find it frustrating that most of programmers don't know and not even thinking about touching assembly. Because it very simple, straightforward and megalomaniac-rewarding :) -- Best regards, Igor Stasenko.
Assembly is hard, just compare its hello world with the hello world of other progamming languages. Its just insane.
you are looking at wrong angle. To print 'hello world' in most of languages is a matter of single and simple expression. Which usually considered an elementary operation in this language. It just happens that in assembler the elementary operation is different, but it doesn't means that you cannot look at a single expression (any instruction) and cannot understand what it does.. e.g. - it prints 'hello world' or - it sets the register value both quite simple, elementary and easy to understand, isn't? Just different levels of abstraction.
The thing is that because we started on early , we experienced assembly when it used to be fairly simple. I started with an Amstrad CPC 6128 with 128 kb of ram and 4 mhz CPU, it uses Z80 assembly which is fairly simple even when compared to Pharo.
Modern Assembly however have grown on complexity because it depend on the complexity of the hardware. Personally I don even like to call it a programming language , just beautified machine code.
i was never considered assemble as a language, it is indeed just a set of CPU instructions. the early assembler compilers is a thin wrappers with slight syntactic sugar here and there.. but not much different to direct instructions. so i would not call it programming language.. since it usually abstracts nothing from direct machine code.
I agree though Assembly is a lot of fun and a great source of knowledge.
-- Best regards, Igor Stasenko.
Great news for windows users , CPP is ported to Windows. Again like linux and macos it has its own class called "CPPBridgeWin" where all Windows methods are located and comes with two examples with reading and writing to the shared memory. In your github-cache of your pharo folder inside the CPP subfolder you will find two subfolders one called simply CPP which shows the Windows Visual C++ version of writing and create the shared memory and a CPPReceive subfolder that contains the Windows Visual C++ version for reading from the shared memory . Both come with their own project files so they can be opened easily inside Visual Studio Community edition which is free to download and use. Both are tested on latest 64 bit Windows 10. One thing I forgot to mention and this applies to the whole CPP project is that C++ is completely optional. That means that shared memory could be shared instead of between a C++ application and Pharo application , instead we can have multiple pharo images executing at the same time sharing the same exact memory. This is useful for those of you who work with very big data and you dont like the idea of it being garbage collected or copied in all your images running at the same time. This way you save memory , you have access to manual memory management with all its pitfalls (yes a wrong pointer will crash your pharo application in an instant) and advantages (it does not get any faster than shared memory with manual memory management, this is the method your OS uses to do pretty much everything data orientated and for loading DLLs). Because CPP supports file mapping that means the data is auto saved into the file , one file for all pharo images running at the same time , this way you can store the live state like you do with a regular pharo image and feel safe that you will never need to worry about forgeting to save the file like you do with pharo images. Essentially that means you can completely bypass C++ and do it all in just pharo code. As always everything is commented and documented . --- ÎÏ ÏÏ Ïο e-mail ελÎγÏθηκε για ιοÏÏ Î±ÏÏ Ïο ÏÏÏγÏαμμα Avast antivirus. https://www.avast.com/antivirus
Hi. It would be nice to see simple code (not in video) how to create memory and read/write data to it. As I understand you create bindings to OS shared memory functions. What else CPPBridge provides? Why not separate bindings to another project? because it could be useful without any C++ relation. Best regards, Denis 2016-11-09 0:06 GMT+01:00 Dimitris Chloupis <kilon.alios@gmail.com>:
*https://youtu.be/pI4PR3XaX6Q <https://youtu.be/pI4PR3XaX6Q>*
*What is it ?*
CPPBridge is a library that allows Pharo to share memory with any C/C++ application. Opening the door not only to IPC and data sharing but also even complete control of C++ code from Pharo and vice versa.
*How to install ?*
In a few hours it should be available from Package Browser, if not you can always fetch it with Metacello from here because it comes with a Baseline
https://github.com/kilon/CPPBridge
*Why bother making such a library ? *
In my saga to find a way to use Pharo as a scripting language for Unreal Game Engine, I had two options
a) Build Unreal as a Library and use Pharo UFFI to launch and control it b) Embed Pharo inside the Unreal Executable (this is what every other scripting language uses for controlling Unreal)
Option (a) was a no go, because Unreal is huge , complex and uses its own custom made build tools, making a DLL for Pharo or an army of DLLs out of the question
Option (b) Embeding Pharo inside an executable is impossible and implementing it also insanely complex
Naturally my mind went first into sockets which is what I use to make Pharo able to use Python libraries. However sockets have proven too slow for the insanely fast loops of Unreal.
*What are the advantages ?*
1) *No need to move data around.* Sharing memory means you don't have to move data around, you can use directly the shared memory
2)* Extend the Pharo image beyond Pharo.* Shared memory is mapped into a file means that you can do with C++ what you can do with Pharo image, save the live state directly to a binary file. That means we no longer need to worry about sessions and reinitializing C/C++ data since memory mapped file acts as an extension of the Pharo image.
3) *Blazing fast. *Memory mapping is a function that comes directly from the OS kernel and as such it has to be extremely fast. Memory mapping is also what is used for dynamically linked shared libraries an extremely important feature for any application including Pharo that heavily depends on (see Cairo for Athens). So its a very mature , well tested method.
4) *No extra libraries needed* to be installed, CPPBridge uses OS libraries to work out of the box
5) *Low level handling of memory.* Direct access to memory you can even manipulate the memory byte by byte
6)* Memory efficient.* Memory mapping excels at large data, the bigger the better. Can take full advantage of your entire free memory and not waste a byte. That means also that can be used to optimise Pharo memory, since you could compress your Pharo objects to bytes and mapped file will store the live state.
7) *Tons of Languages. *Because memory mapping is a core functionality for every OS out there, pretty much every programming language supports it. CPPBridge currently supports only C/C++ but all languages can be supported giving access to Pharo to any library for any programming language. Sky is the limit
8) *Self Documented. *CPPBridge is small, simple and with large class comment and comments for every method. YouTube video tutorial also available and linked on top.
9) *Works both ways*, C/C++ and Pharo can access and modify the shared memory. Making it possible for C/C++ to use Pharo libraries and Pharo to use C/C++ libraries.
10) Experiments have proven that it improves sex life... if it does not please file a bug report ;)
*What are the disadvantages ?*
1) *Candy Crash Saga*. Dare do something incorrectly and Pharo will crash. CPPBridge can easily point to wrong address if you are not aware of what you doing.
2) *C++/C* . If you think you can avoid learning C/C++ and that this is a magic solution , think again. The C/C++ application must be modified to include shared memory mapping for CPPBridge to work .
3) *Local only*. Unlike sockets, Shared Memory works only on the same machine so no remote execution and manipulation of code like in my socket bridge to Python
4) *UFFI still No1 option*. No replacement for UFFI it actually depends on it to work , so if you can turn your C/C++ code into a DLL that should be your first option.
*Roadmap *
Currently CPPBridge works only on MacOS , most likely on Linux too (because it uses the Unix architecture) but I will have to test it.
Windows is coming next ASAP, since its my No1 platform for creating commercial games.
Maybe establish a small protocol of communication via the Shared Memory , JSON looks like a good universal format
*Thanks*
Big thanks to Eliot for inspiring me and Esteban for helping me figure out things.
"Hi. It would be nice to see simple code (not in video) how to create memory and read/write data to it." If you get the Project it comes with an example how to use in the class side of the CPPBridge Class and its in methods retrieveSharedValueStep1 and retrieveSharedValueStep2, first one open the files , access the shared memory and pick the value from shared memory, the second unmaps the file from memory and closes the file (access to shared memory remains , the file is is just no longer auto-saved by the OS). All methods are documented and class comes with a comment. Yes basically what I have done is to extend the LibC class because every function I am using is from LibC aka as C standard library that comes included in MacOS and Linux. Windows is doing its own thing , I have not ported to windows yet, though if you have something that install LibC (part of GCC) on Windows probably it works there too. But I will make a native port to Windows as well because its crucial for my gamers to play my games out of the box with no additional install of libraries or what else. One thing I did not make clear, the file opened plays a minor role for the shared memory its just used to store a dumb of the memory just like we do with the Pharo image (though in the case of Pharo image is bytecode) . As such file I/O is never involved in accessing the data instead it gives direct access to the memory , hence why its blazing fast. I don't have any issue to add this to LibC of UFFI but this decision is to Esteban, I just did not want to put more things in the Pharo image that people wont use since we try to decrease the size of the image. CPPBridge currently does not provide anything else, however in truth it does not need to. The moment you share date, that means you share state and you can even control code execution. Unfortunately C/C++ is not a dynamic language like Python, with Python all I had to do was to pass strings to python eval() and let python execute whatever command I wanted, but with C++ these things will have to be precompiled in order to work. Most likely I will be adding in the near future classes and method of convenience for using C++ libraries/code from Pharo and Pharo libraries/code from C++. Possible will add also Python support. Other language that can be supported are C# (and any other language running on .NET or Mono related) , Java (again any other language running on JavaVM ), Ruby , Pearl, D, J, Julia, R , so pretty much any language out there. C/C++ obviously is optional. I am pasting bellow the source of the two methods I mentioned, code can be also viewed here https://github.com/kilon/CPPBridge/blob/master/CPPBridge.package/CPPBridge.c... and here https://github.com/kilon/CPPBridge/blob/master/CPPBridge.package/CPPBridge.c... --------------------------------------------------------- retrieveSharedValueStep1 <example> "This method is an example that retrieves a struct from a shared memory section, in order for this example to work you need you first copy paste the contents of the Example Souce Code of the C++ file in the comment section of this class to a C++ source code file and compile it a run then replace the path of in this code of CPPBridge openFile: with the correct path of the bin that the C++ files has created , in order for this to work also you need to execute the C++ example first so it creates and file and share the memory. After executing this method you can execute retrieveSharedValueStep2 to unmap and close the memory mapped file (keeps sharing the memory it just does not store it to the file)" | instance fdNumber lseek mmapPointer data struct | instance := CPPBridge new. fdNumber := CPPBridge openFile: '/Users/kilon/git/Pharo/Atlas/mmapped.bin' flags: (O_RDWR) . "Warning !!! You must change the path to the file that is located in your hard drive. The file should be at the same location you built atlas-server.cpp which is responsible for creating the file." lseek := CPPBridge lSeek_fd: fdNumber range:3999 value:0. mmapPointer := CPPBridge mmap_adress: 0 fileSize:4000 flag1: (PROT_READ | PROT_WRITE )flag2: MAP_SHARED fd: fdNumber offset: 0 . struct := CPPStruct pointTo: (mmapPointer getHandle ). data :={ instance. fdNumber . lseek. mmapPointer . struct}. data inspect. ExampleDATA := data. ^data " This is an alternative array that contains only the two values of the C Struct : 1) data = char[3000] this is where we store the string 2) count = int this is where we store the size of the string struct := {(mmapPointer getHandle copyFrom: 1 to:3000 )asString . (mmapPointer getHandle integerAt: 3001 size:4 signed: false)}. mmapPointer is the pointer that points to the first byte of the shared memory. getHandle gives us the memory adress that the pointer points to copyFrom:1 to:3000 copies byte from byte 0 (remember C counts from 0 , Pharo counts from 1) to byte 3000 because the string we store is stored as a char array of 3000 elements, each element is a char, each char is 1 byte in leght and represents a single character of the string. on the other hand integerAt: 3001 size: 4 signed: false returns us the value count memeber of the C struct . its an integer in position 3001 because our string is a char[3000] and the size is 4 bytes because its an C int, signed false because we use no negative values because it does not make sense for a string to have negative length." ------------------------------------------ retrieveSharedValueStep2 <example> "this one is to be triggered after retrieveSharedValueStep1 when we are finished with the shared memory and we want to erase it and close the memory mapped file ExampleDATA = { CPPBridgeinstance . fd . lseek . mmap . struct}" CPPBridge munmap_data: (ExampleDATA at: 4) filesize: 4000. CPPBridge closeFile: (ExampleDATA at: 2).
2016-11-09 11:41 GMT+01:00 Dimitris Chloupis <kilon.alios@gmail.com>:
"Hi.
It would be nice to see simple code (not in video) how to create memory and read/write data to it."
If you get the Project it comes with an example how to use in the class side of the CPPBridge Class and its in methods retrieveSharedValueStep1 and retrieveSharedValueStep2
I just a lazy guy. Thank's for pointing code here.
I would not call you lazy, its normal to want to see documentation before installing something and at least an example. Actually thats a good idea, I will add a few more comments in the example and use it on repo's front page. On Wed, Nov 9, 2016 at 12:56 PM Denis Kudriashov <dionisiydk@gmail.com> wrote:
2016-11-09 11:41 GMT+01:00 Dimitris Chloupis <kilon.alios@gmail.com>:
"Hi.
It would be nice to see simple code (not in video) how to create memory and read/write data to it."
If you get the Project it comes with an example how to use in the class side of the CPPBridge Class and its in methods retrieveSharedValueStep1 and retrieveSharedValueStep2
I just a lazy guy. Thank's for pointing code here.
Ok documentation has improved a lot , it should be more obvious now https://kilon.github.io/CPPBridge/ Any comment , critic more than welcomed :) On Wed, Nov 9, 2016 at 1:10 PM Dimitris Chloupis <kilon.alios@gmail.com> wrote:
I would not call you lazy, its normal to want to see documentation before installing something and at least an example. Actually thats a good idea, I will add a few more comments in the example and use it on repo's front page.
On Wed, Nov 9, 2016 at 12:56 PM Denis Kudriashov <dionisiydk@gmail.com> wrote:
2016-11-09 11:41 GMT+01:00 Dimitris Chloupis <kilon.alios@gmail.com>:
"Hi.
It would be nice to see simple code (not in video) how to create memory and read/write data to it."
If you get the Project it comes with an example how to use in the class side of the CPPBridge Class and its in methods retrieveSharedValueStep1 and retrieveSharedValueStep2
I just a lazy guy. Thank's for pointing code here.
participants (3)
-
Denis Kudriashov -
Dimitris Chloupis -
Igor Stasenko