Touch Events VM changes
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change. I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)? Cheers, Jeff -- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
Hi, I agree that we need a unified event model. Now⦠our vision is to solve that in image side, not in vm (and vm should send the raw events âas isâ). The reason of why we want it in image is easy: image means pharo, and it means more hands (or eyes) over it. Then is easier to maintain :) Anyway, we are slowly working on it (See Igor & Ronnie work around OSWindow). But is probably too slow :) Esteban On 11 Mar 2014, at 11:37, J.F. Rick <self@je77.com> wrote:
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)?
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
Definitely agreed on the strategy of processing the raw events on the image side as that will be much more flexible. But, we still need the VM to send the raw events. I'll check on the vm-dev list. I think I'm already subscribed to it. Cheers, Jeff On Tue, Mar 11, 2014 at 12:03 PM, Esteban Lorenzano <estebanlm@gmail.com>wrote:
Hi,
I agree that we need a unified event model. Now... our vision is to solve that in image side, not in vm (and vm should send the raw events "as is"). The reason of why we want it in image is easy: image means pharo, and it means more hands (or eyes) over it. Then is easier to maintain :) Anyway, we are slowly working on it (See Igor & Ronnie work around OSWindow). But is probably too slow :)
Esteban
On 11 Mar 2014, at 11:37, J.F. Rick <self@je77.com> wrote:
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)?
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
I've got a Windows 8 laptop with a touch screen. So, I now have to look how the events are provided. Phil On Tue, Mar 11, 2014 at 12:03 PM, Esteban Lorenzano <estebanlm@gmail.com>wrote:
Hi,
I agree that we need a unified event model. Now... our vision is to solve that in image side, not in vm (and vm should send the raw events "as is"). The reason of why we want it in image is easy: image means pharo, and it means more hands (or eyes) over it. Then is easier to maintain :) Anyway, we are slowly working on it (See Igor & Ronnie work around OSWindow). But is probably too slow :)
Esteban
On 11 Mar 2014, at 11:37, J.F. Rick <self@je77.com> wrote:
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)?
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
J.F. Rick wrote:
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)? The place to ask is [vm-dev] http://lists.squeakfoundation.org/mailman/listinfo/vm-dev
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
2014-03-11 10:37 GMT+00:00 J.F. Rick <self@je77.com>:
As we move from mouse-keyboard devices to touch-based ones
I certainly hope that "we" refers to a small group of people, unless we get onscreen keyboards as good as physical ones, feedback included. The same goes for touch versus physical pointer :) Cheers, Sergi
Hi Sergi, Based on my experience developing touch interfaces since 2007, I believe in touch. I believe it will be the future of interfaces and I believe it can be better than mouse-keyboard interfaces. That said, it is not there yet. Now is the time to innovate and get it there so that touch interfaces won't be poor replicas of their desktop forebears. Currently, text entry is a real problem for touch exclusive devices, but there is progress being made on that front. We don't have the answer yet but I think it will be a significantly different landscape in five years. In regards to touch vs physical pointers, this is not as large of a problem as people make it out to be. It is just that our widgets tend to be created for mouse-based interfaces and therefore don't transition well to touch. Interfaces designed with touch in mind can be just as smooth (if not smoother) as mouse-based interfaces. You just have to take advantage of what touch does well (faster, more precise routes; multi-touch gestures; bimanual interaction) and stay clear of things it does poorly (precise touch down, covering the target with the finger). Cheers, Jeff On Tue, Mar 11, 2014 at 12:34 PM, Sergi Reyner <sergi.reyner@gmail.com>wrote:
2014-03-11 10:37 GMT+00:00 J.F. Rick <self@je77.com>:
As we move from mouse-keyboard devices to touch-based ones
I certainly hope that "we" refers to a small group of people, unless we get onscreen keyboards as good as physical ones, feedback included. The same goes for touch versus physical pointer :)
Cheers, Sergi
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
2014-03-11 11:54 GMT+00:00 J.F. Rick <self@je77.com>:
Hi Sergi,
Based on my experience developing touch interfaces since 2007, I believe in touch.
Don´t get me wrong, I like touch and non-traditional interfaces in general. In fact there are five android devices at home, counting phones and tablets. I also experimented with voice and webcam control for my home theatre. I just think that they are optimal for some scenarios and not-so-optimal for others, and that it wouldn´t be an improvement to make all devices touch-only. As you yourself point out:
You just have to take advantage of what touch does well (faster, more precise routes; multi-touch gestures; bimanual interaction) and stay clear of things it does poorly (precise touch down, covering the target with the finger).
Both have their strengths and weaknesses. Mice have multiple buttons (I use the wheel and the back/forward buttons a lot, both for those functions and as configurable controls in certain games) and are easily identifiable as distinct input devices, while touch interfaces allow more direct (not necessarily precise) control over the actions, taking things like pointer acceleration out of the equation. Almost everyone can draw a lowercase 'g' on a touch screen, but doing it with a mouse takes some skill (well, and a decent mouse). As a more concrete use case, trying to craft some pixel art in a touch device can quickly become tedious. I certainly wouldn´t mind at all having one or two tactile screens in my desk. That said, I once had an interesting conversation with a costumer, in which she told me that she couldn´t play World of Warcraft with her tactile screen because random passing-by insects kept pressing her skills/spells buttons at the worst times :) Cheers, Sergi
I have a pretty decent lineup of devices with touch. The interactions aren't the same on the iPhone as they are on a Galaxy SII phone, not are the same on iPads or on a Nexus7. Magic mouses are also special things. Also my Samsung Windows 8 laptop has a 10 points touch screen. With Windows8.1, I find myself using swipes and touches for a lot of things. And it improves my productivity. Once equipped with a thing like Classic Shell and boot into the desktop right away, Metro apps are really nice little things running in the background and can come fast with a little swipe (cool ones: Skype, Music, PDF reader of a documentation, like a chapter of a Pharo book: swipe: PDF, swipe: Pharo, swiper: PDF... and mouse pointer not moving around). When using my MacBook Pro, I am reaching for the screen and being disappointed to realize it is not touch (Apple, give us touch screens on laptops). Form factors affect the usage patterns as well. But touch is not going displace touchpads or mouses for fine grained control. On the contrary, I tested a Leap motion controller and it felt like crap. Phil On Tue, Mar 11, 2014 at 12:54 PM, J.F. Rick <self@je77.com> wrote:
Hi Sergi,
Based on my experience developing touch interfaces since 2007, I believe in touch. I believe it will be the future of interfaces and I believe it can be better than mouse-keyboard interfaces. That said, it is not there yet. Now is the time to innovate and get it there so that touch interfaces won't be poor replicas of their desktop forebears. Currently, text entry is a real problem for touch exclusive devices, but there is progress being made on that front. We don't have the answer yet but I think it will be a significantly different landscape in five years. In regards to touch vs physical pointers, this is not as large of a problem as people make it out to be. It is just that our widgets tend to be created for mouse-based interfaces and therefore don't transition well to touch. Interfaces designed with touch in mind can be just as smooth (if not smoother) as mouse-based interfaces. You just have to take advantage of what touch does well (faster, more precise routes; multi-touch gestures; bimanual interaction) and stay clear of things it does poorly (precise touch down, covering the target with the finger).
Cheers,
Jeff
On Tue, Mar 11, 2014 at 12:34 PM, Sergi Reyner <sergi.reyner@gmail.com>wrote:
2014-03-11 10:37 GMT+00:00 J.F. Rick <self@je77.com>:
As we move from mouse-keyboard devices to touch-based ones
I certainly hope that "we" refers to a small group of people, unless we get onscreen keyboards as good as physical ones, feedback included. The same goes for touch versus physical pointer :)
Cheers, Sergi
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
Thanks Jeff for this analysis. This is instructing. Cheers, Alexandre
On 11 Mar 2014, at 08:54, "J.F. Rick" <self@je77.com> wrote:
Hi Sergi,
Based on my experience developing touch interfaces since 2007, I believe in touch. I believe it will be the future of interfaces and I believe it can be better than mouse-keyboard interfaces. That said, it is not there yet. Now is the time to innovate and get it there so that touch interfaces won't be poor replicas of their desktop forebears. Currently, text entry is a real problem for touch exclusive devices, but there is progress being made on that front. We don't have the answer yet but I think it will be a significantly different landscape in five years. In regards to touch vs physical pointers, this is not as large of a problem as people make it out to be. It is just that our widgets tend to be created for mouse-based interfaces and therefore don't transition well to touch. Interfaces designed with touch in mind can be just as smooth (if not smoother) as mouse-based interfaces. You just have to take advantage of what touch does well (faster, more precise routes; multi-touch gestures; bimanual interaction) and stay clear of things it does poorly (precise touch down, covering the target with the finger).
Cheers,
Jeff
On Tue, Mar 11, 2014 at 12:34 PM, Sergi Reyner <sergi.reyner@gmail.com> wrote: 2014-03-11 10:37 GMT+00:00 J.F. Rick <self@je77.com>:
As we move from mouse-keyboard devices to touch-based ones
I certainly hope that "we" refers to a small group of people, unless we get onscreen keyboards as good as physical ones, feedback included. The same goes for touch versus physical pointer :)
Cheers, Sergi
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
Hello, the right place is vm-dev. I put it in CC. Regards, 2014-03-11 11:37 GMT+01:00 J.F. Rick <self@je77.com>:
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)?
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
Jeff, since you running linux you can look at what we have done with Ronie not long ago. Basically we changed VM and removed all window/event management code and instead do everything at image side. I think it would be good if you can try and add multi-touch events support to the code. If you wanna try, you need to build new VM first, and follow this: https://pharo.fogbugz.com/f/cases/12743/OSWindow-integration Then you can look at OSWindow package (its X11 part) and see what is needed to support multi-touch events. You should build VM from sources taken from here: https://github.com/ronsaldo/pharo-vm On 11 March 2014 11:37, J.F. Rick <self@je77.com> wrote:
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)?
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
-- Best regards, Igor Stasenko.
Does that mean that normal mouse movement won't work with that VM? I have a couple other things that take precedent but that does sound like it would be worth playing with. Cheers, Jeff On Tue, Mar 11, 2014 at 1:29 PM, Igor Stasenko <siguctua@gmail.com> wrote:
Jeff, since you running linux you can look at what we have done with Ronie not long ago. Basically we changed VM and removed all window/event management code and instead do everything at image side. I think it would be good if you can try and add multi-touch events support to the code.
If you wanna try, you need to build new VM first, and follow this: https://pharo.fogbugz.com/f/cases/12743/OSWindow-integration
Then you can look at OSWindow package (its X11 part) and see what is needed to support multi-touch events.
You should build VM from sources taken from here: https://github.com/ronsaldo/pharo-vm
On 11 March 2014 11:37, J.F. Rick <self@je77.com> wrote:
As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)?
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
-- Best regards, Igor Stasenko.
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
On 12 Mar 2014, at 10:27, J.F. Rick <self@je77.com> wrote:
Does that mean that normal mouse movement won't work with that VM? I have a couple other things that take precedent but that does sound like it would be worth playing with.
What igor and ronie did is to remove the events support from the VM but use NativeBoost to get the OS events.
Cheers,
Jeff
On Tue, Mar 11, 2014 at 1:29 PM, Igor Stasenko <siguctua@gmail.com> wrote: Jeff, since you running linux you can look at what we have done with Ronie not long ago. Basically we changed VM and removed all window/event management code and instead do everything at image side. I think it would be good if you can try and add multi-touch events support to the code.
If you wanna try, you need to build new VM first, and follow this: https://pharo.fogbugz.com/f/cases/12743/OSWindow-integration
Then you can look at OSWindow package (its X11 part) and see what is needed to support multi-touch events.
You should build VM from sources taken from here: https://github.com/ronsaldo/pharo-vm
On 11 March 2014 11:37, J.F. Rick <self@je77.com> wrote: As we move from mouse-keyboard devices to touch-based ones, it would be nice if we could unify touch development across the different Pharo platforms (iOS, Android, Windows 8, Linux, etc.), so that we can start to build touch-aware widgets and frameworks. Currently, there's a lot of interest but no unified movement. For instance, I've got a very nice multi-touch 27" computer that I run in Linux. I develop the applications in Pharo but I currently use a hack to get the touch events: I use OSProcess to start the mtdev2tuio application that takes touch input and sends it via TUIO. I then use a UDP socket to read the events on Pharo. If I am to share my work, I need to get beyond the hack. I (or rather Igor as I watched) investigated the possibility of using NativeBoost calls to do the same, but it would not work since mtdev2tuio has to run as superuser, meaning Pharo would have to run as superuser. We came to the conclusion that this must be a VM-level change.
I'm not a VM hacker but would like to reach the people who maintain the linux and windows VM to see if this might not be a reasonable addition to the standard VM. There does seem to be standard support for touch events in XFree86, so it shouldn't be too large a change on Linux. I assume that Windows 8 similarly makes touch events available. Is this the right place to ask or is there another mailing list (e.g., the cog-vm)?
Cheers,
Jeff
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
-- Best regards, Igor Stasenko.
-- Jochen "Jeff" Rick, Ph.D. http://www.je77.com/ Skype ID: jochenrick
participants (6)
-
Alexandre Bergel -
btc＠openinworld.com -
Clément Bera -
Esteban Lorenzano -
Igor Stasenko -
J.F. Rick -
Pharo4Stef -
phil＠highoctane.be -
Sergi Reyner