Iceberg with SSH on Linux
Hi, I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system. Thank you Esteban and others! Cheers, -- Pavel
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
Yes, the same priority. See: The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority. An exit may be appropriate in some cases, but for most users the above warning is sufficient. https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f321583c69ca27e61... Cheers, -- Pavel 2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
On 30 March 2017 at 11:58, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority. An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f321583c69ca27e61...
Cheers, -- Pavel
From looking at the code, my understanding is that the VM attempts to elevate the priority of the heartbeat thread, if the elevation fails, the old behaviour was to exit with a message. The new behaviour is to just warn the user (with a message to stderr) and continue.
Assuming I've read the code correctly, there's a bug that needs to be fixed: the cat statement won't work as sudo doesn't apply to file redirection; and I've suggested some more informative text: -- This VM uses a separate heartbeat thread to update the internal clock. For best operation, this thread should run at real-time priority, however the VM was unable to change the priority. The effect is that the clock may experience some jitter. To avoid clock jitter, please create the appropriate configuration file in /etc/security/limits.d/ as shown below: cat <<EOF | sudo tee /etc/security/limits.d/VNAME.conf * hard rtprio 2 VNAME * soft rtprio 2 VNAME EOF You will need to logout and back in for the changes to take affect. -- If there's no disagreement or better suggestion, I'll submit a pull request (probably in a couple of days). Cheers, Alistair
2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
2017-03-30 14:03 GMT+02:00 Alistair Grant <akgrant0710@gmail.com>:
From looking at the code, my understanding is that the VM attempts to elevate the priority of the heartbeat thread, if the elevation fails, the old behaviour was to exit with a message. The new behaviour is to just warn the user (with a message to stderr) and continue.
Great!
Hi Alistair,
On Mar 30, 2017, at 5:03 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
On 30 March 2017 at 11:58, Pavel Krivanek <pavel.krivanek@gmail.com> wrote: Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority. An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f321583c69ca27e61...
Cheers, -- Pavel
From looking at the code, my understanding is that the VM attempts to elevate the priority of the heartbeat thread, if the elevation fails, the old behaviour was to exit with a message. The new behaviour is to just warn the user (with a message to stderr) and continue.
Assuming I've read the code correctly, there's a bug that needs to be fixed: the cat statement won't work as sudo doesn't apply to file redirection; and I've suggested some more informative text:
Are you sure? Many people, myself included, have used the sudo version successfully in the past. I think I see what you mean; if sudo is a.command then the shell that invokes it is not running as superuser and it sets up the pipe. But perhaps there's behind-the-scenes magic in the shell (I've used bash) that makes it work? Also isn't a nicer alternative is just to say su cat ... ?
-- This VM uses a separate heartbeat thread to update the internal clock. For best operation, this thread should run at real-time priority, however the VM was unable to change the priority. The effect is that the clock may experience some jitter. To avoid clock jitter, please create the appropriate configuration file in /etc/security/limits.d/ as shown below:
cat <<EOF | sudo tee /etc/security/limits.d/VNAME.conf * hard rtprio 2 VNAME * soft rtprio 2 VNAME EOF
You will need to logout and back in for the changes to take affect. --
If there's no disagreement or better suggestion, I'll submit a pull request (probably in a couple of days).
Cheers, Alistair
2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
Hi Eliot, On 30 March 2017 at 18:37, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Alistair,
On Mar 30, 2017, at 5:03 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
On 30 March 2017 at 11:58, Pavel Krivanek <pavel.krivanek@gmail.com> wrote: Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority. An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f321583c69ca27e61...
Cheers, -- Pavel
From looking at the code, my understanding is that the VM attempts to elevate the priority of the heartbeat thread, if the elevation fails, the old behaviour was to exit with a message. The new behaviour is to just warn the user (with a message to stderr) and continue.
Assuming I've read the code correctly, there's a bug that needs to be fixed: the cat statement won't work as sudo doesn't apply to file redirection; and I've suggested some more informative text:
Are you sure? Many people, myself included, have used the sudo version successfully in the past. I think I see what you mean; if sudo is a.command then the shell that invokes it is not running as superuser and it sets up the pipe. But perhaps there's behind-the-scenes magic in the shell (I've used bash) that makes it work?
Interesting, which OS are you running? It fails on Ubuntu 16.04 (I just tried it again, using bash) and ARCH linux for me (in the past).
Also isn't a nicer alternative is just to say su cat ... ?
This assumes that you can log in as the root account. It's disabled in lots of distributions (and considered poor form to enable it). Of course, my version assumes that sudo is installed and configured, but, at least within the Linux community, I think it is a reasonable assumption. Cheers, Alistair
On Thu, Mar 30, 2017 at 8:03 PM, Alistair Grant <akgrant0710@gmail.com> wrote:
On 30 March 2017 at 11:58, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority. An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/ 32f321583c69ca27e61ffaff6decc2a3e4b6ca5e
Cheers, -- Pavel
From looking at the code, my understanding is that the VM attempts to elevate the priority of the heartbeat thread, if the elevation fails, the old behaviour was to exit with a message. The new behaviour is to just warn the user (with a message to stderr) and continue.
Assuming I've read the code correctly, there's a bug that needs to be fixed: the cat statement won't work as sudo doesn't apply to file redirection; and I've suggested some more informative text:
-- This VM uses a separate heartbeat thread to update
the internal clock.
its internal clock and handle events.
For best operation, this thread should run
at real-time priority,
at a higher priority. (Real-time priority is something else.)
however the VM was unable to change the priority. The effect is that
the clock may experience some jitter.
To avoid clock jitter, please
heavily loaded systems may experience some latency issues. If this occurs, please
create the appropriate configuration file in /etc/security/limits.d/ as shown below:
cat <<EOF | sudo tee /etc/security/limits.d/VNAME.conf * hard rtprio 2 VNAME * soft rtprio 2 VNAME EOF
and report to the mail list whether this improves behaviour.
You will need to logout and back in for the changes to take affect.
--
If there's no disagreement or better suggestion, I'll submit a pull request (probably in a couple of days).
cheers -ben
Cheers, Alistair
2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on
Linux
VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
On Thu, Mar 30, 2017 at 6:34 PM, Ben Coman <btc@openinworld.com> wrote:
On Thu, Mar 30, 2017 at 8:03 PM, Alistair Grant <akgrant0710@gmail.com> wrote:
On 30 March 2017 at 11:58, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority. An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f 321583c69ca27e61ffaff6decc2a3e4b6ca5e
Cheers, -- Pavel
From looking at the code, my understanding is that the VM attempts to elevate the priority of the heartbeat thread, if the elevation fails, the old behaviour was to exit with a message. The new behaviour is to just warn the user (with a message to stderr) and continue.
Assuming I've read the code correctly, there's a bug that needs to be fixed: the cat statement won't work as sudo doesn't apply to file redirection; and I've suggested some more informative text:
-- This VM uses a separate heartbeat thread to update
the internal clock.
its internal clock and handle events.
For best operation, this thread should run
at real-time priority,
at a higher priority.
(Real-time priority is something else.)
however the VM was unable to change the priority. The effect is that
the clock may experience some jitter.
To avoid clock jitter, please
heavily loaded systems may experience some latency issues. If this occurs, please
create the appropriate configuration file in /etc/security/limits.d/ as shown below:
cat <<EOF | sudo tee /etc/security/limits.d/VNAME.conf * hard rtprio 2 VNAME * soft rtprio 2 VNAME EOF
and report to the mail list whether this improves behaviour.
What Ben says. Exactly right on all points.
You will need to logout and back in for the changes to take affect.
--
If there's no disagreement or better suggestion, I'll submit a pull request (probably in a couple of days).
cheers -ben
Cheers, Alistair
2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on
Linux
VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
-- _,,,^..^,,,_ best, Eliot
_,,,^..^,,,_ (phone)
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all. The resolution I thought we had reached means that if correctly installed the VM can work correctly, and will continue to work correctly if and when linux removes the annoyance of the conf files. The resolution you describe Pavel (I hope inaccurately) means the threaded heartbeat VM will never work correctly. One other thing. I know threaded heartbeat is verbose, but I would really appreciate it if we could reserve "threaded vm" to mean a vm that multiplexes Smalltalk processes above native threads, even if not concurrently. This is the threaded ffi plan which we have a good chance of realizing this year.
An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f321583c69ca27e61...
Cheers, -- Pavel
2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM (get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
On 30 Mar 2017, at 18:15, Eliot Miranda <eliot.miranda@gmail.com> wrote:
_,,,^..^,,,_ (phone)
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
The resolution I thought we had reached means that if correctly installed the VM can work correctly, and will continue to work correctly if and when linux removes the annoyance of the conf files. The resolution you describe Pavel (I hope inaccurately) means the threaded heartbeat VM will never work correctly.
this is like this :)
One other thing. I know threaded heartbeat is verbose, but I would really appreciate it if we could reserve "threaded vm" to mean a vm that multiplexes Smalltalk processes above native threads, even if not concurrently. This is the threaded ffi plan which we have a good chance of realizing this year.
+ 1000 Esteban
An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f321583c69ca27e61... <https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/32f321583c69ca27e61...>
Cheers, -- Pavel
2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com <mailto:dionisiydk@gmail.com>>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>>: Hi,
I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM (get.pharo.org/vmTLatest60 <http://get.pharo.org/vmTLatest60>). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
Hi Eliot, On 30 March 2017 at 18:15, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
As Esteban wrote, the vm is trying to raise the priority. You can see the change at: https://github.com/pharo-project/pharo-vm/commit/32f321583c69ca27e61ffaff6de... I'd like to just make sure I understand your comment correctly: Are you saying that the VM will lock up if it gets in to, e.g., an endless loop, so that Ctrl-. fails to interrupt. Or is it more serious than that? Can the VM become locked up for just a long running process (I guess it will be locked while the process is running, but will eventually go back to normal once the process completes)? Also, a change was made to the travis test setup in: https://github.com/pharo-project/pharo-vm/commit/5418a415e9297f601f6d57ee732... The comment is: "No need to raise rtprio limit anymore"
From your comments above, this doesn't seem accurate.
I'm in the middle of adding some details to the text that is printed out by the threaded heartbeat vm, and from what you're saying, maybe we should add some more warnings.
The resolution I thought we had reached means that if correctly installed the VM can work correctly, and will continue to work correctly if and when linux removes the annoyance of the conf files. The resolution you describe Pavel (I hope inaccurately) means the threaded heartbeat VM will never work correctly.
Thanks, Alistair
Hi Alistair, On Thu, Mar 30, 2017 at 9:32 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Eliot,
On 30 March 2017 at 18:15, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute
bound. I
thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
As Esteban wrote, the vm is trying to raise the priority. You can see the change at:
https://github.com/pharo-project/pharo-vm/commit/ 32f321583c69ca27e61ffaff6decc2a3e4b6ca5e
I'd like to just make sure I understand your comment correctly:
Are you saying that the VM will lock up if it gets in to, e.g., an endless loop, so that Ctrl-. fails to interrupt. Or is it more serious than that? Can the VM become locked up for just a long running process (I guess it will be locked while the process is running, but will eventually go back to normal once the process completes)?
The issue is that delays are recognized by periodically checking the time against the next delay expiry. It is the heartbeat thread that causes JITed Smalltalk code to break out of its execution to check for events, including the next delay expiry. So if the heartbeat is at the same priority as the JITed Smalltalk code, and the Smalltalk code is in tight enough a loop (one that does not provoke a GC, does not do any process switches) the heartbeat thread will not run and the delay expiry will never be tested for.
Also, a change was made to the travis test setup in:
https://github.com/pharo-project/pharo-vm/commit/ 5418a415e9297f601f6d57ee732fd7fd942da08c
The comment is: "No need to raise rtprio limit anymore"
From your comments above, this doesn't seem accurate.
Right. What seems to occur is that certain linux distress don't require a conf file to allow raising the priority of a thread. Remember that rtprio is not the issue. We only require that the heartbeat thread has higher priority than the VM thread.
I'm in the middle of adding some details to the text that is printed out by the threaded heartbeat vm, and from what you're saying, maybe we should add some more warnings.
The VM should (& I believe already does) print an error if the attempt to raise the heartbeat thread priority fails. My underatanding is that the positive change is simply not to exit if the attempt fails.
The resolution I thought we had reached means that if correctly installed the VM can work correctly, and will continue to work correctly if and when linux removes the annoyance of the conf files. The resolution you describe Pavel (I hope inaccurately) means the threaded heartbeat VM will never work correctly.
Thanks, Alistair
_,,,^..^,,,_ best, Eliot
On Thu, Mar 30, 2017 at 8:20 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Alistair,
On Thu, Mar 30, 2017 at 9:32 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Eliot,
On 30 March 2017 at 18:15, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at
a
higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
As Esteban wrote, the vm is trying to raise the priority. You can see the change at:
https://github.com/pharo-project/pharo-vm/commit/32f321583c6 9ca27e61ffaff6decc2a3e4b6ca5e
I'd like to just make sure I understand your comment correctly:
Are you saying that the VM will lock up if it gets in to, e.g., an endless loop, so that Ctrl-. fails to interrupt. Or is it more serious than that? Can the VM become locked up for just a long running process (I guess it will be locked while the process is running, but will eventually go back to normal once the process completes)?
The issue is that delays are recognized by periodically checking the time against the next delay expiry. It is the heartbeat thread that causes JITed Smalltalk code to break out of its execution to check for events, including the next delay expiry. So if the heartbeat is at the same priority as the JITed Smalltalk code, and the Smalltalk code is in tight enough a loop (one that does not provoke a GC, does not do any process switches) the heartbeat thread will not run and the delay expiry will never be tested for.
Something like the following should run forever if the heartbeat has the same priority as the VM thread (and should be tried in a headless configuration too): [| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun 1001 Also, a change was made to the travis test setup in:
https://github.com/pharo-project/pharo-vm/commit/5418a415e92 97f601f6d57ee732fd7fd942da08c
The comment is: "No need to raise rtprio limit anymore"
From your comments above, this doesn't seem accurate.
Right. What seems to occur is that certain linux distress don't require a conf file to allow raising the priority of a thread. Remember that rtprio is not the issue. We only require that the heartbeat thread has higher priority than the VM thread.
I'm in the middle of adding some details to the text that is printed out by the threaded heartbeat vm, and from what you're saying, maybe we should add some more warnings.
The VM should (& I believe already does) print an error if the attempt to raise the heartbeat thread priority fails. My underatanding is that the positive change is simply not to exit if the attempt fails.
The resolution I thought we had reached means that if correctly installed the VM can work correctly, and will continue to work correctly if and when linux removes the annoyance of the conf files. The resolution you describe Pavel (I hope inaccurately) means the threaded heartbeat VM will never work correctly.
Thanks, Alistair
_,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
Hi Eliot, On 31 March 2017 at 05:31, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Mar 30, 2017 at 8:20 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Alistair,
On Thu, Mar 30, 2017 at 9:32 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Eliot,
On 30 March 2017 at 18:15, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
As Esteban wrote, the vm is trying to raise the priority. You can see the change at:
https://github.com/pharo-project/pharo-vm/commit/32f321583c69ca27e61ffaff6de...
I'd like to just make sure I understand your comment correctly:
Are you saying that the VM will lock up if it gets in to, e.g., an endless loop, so that Ctrl-. fails to interrupt. Or is it more serious than that? Can the VM become locked up for just a long running process (I guess it will be locked while the process is running, but will eventually go back to normal once the process completes)?
The issue is that delays are recognized by periodically checking the time against the next delay expiry. It is the heartbeat thread that causes JITed Smalltalk code to break out of its execution to check for events, including the next delay expiry. So if the heartbeat is at the same priority as the JITed Smalltalk code, and the Smalltalk code is in tight enough a loop (one that does not provoke a GC, does not do any process switches) the heartbeat thread will not run and the delay expiry will never be tested for.
Something like the following should run forever if the heartbeat has the same priority as the VM thread (and should be tried in a headless configuration too):
[| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun 1001
I get the same result regardless of thread priority, UI or headless. I've just shown the output from the headless VM with the heartbeat thread at normal priority (I haven't incorporated Ben's comments in to the text yet): $ pharo Pharo.image --quit ~/pharo6/pharo/heartbeatTest.st pthread_setschedparam failed: Operation not permitted This VM uses a separate heartbeat thread to update the internal clock. For best operation, this thread should run at real-time priority, however the VM was unable to change the priority. The effect is that the clock may experience some jitter. To avoid clock jitter, please create the appropriate configuration file in /etc/security/limits.d by executing the following: cat <<END | sudo tee /etc/security/limits.d/pharo.conf * hard rtprio 2 * soft rtprio 2 END You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux Time to run: 0:00:00:01 heartbeatTest.st: | time | time := [| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun. FileStream stdout nextPutAll: 'Time to run: '; nextPutAll: time printString; cr. time
Also, a change was made to the travis test setup in:
https://github.com/pharo-project/pharo-vm/commit/5418a415e9297f601f6d57ee732...
The comment is: "No need to raise rtprio limit anymore"
From your comments above, this doesn't seem accurate.
Right. What seems to occur is that certain linux distress don't require a conf file to allow raising the priority of a thread. Remember that rtprio is not the issue. We only require that the heartbeat thread has higher priority than the VM thread.
I'm in the middle of adding some details to the text that is printed out by the threaded heartbeat vm, and from what you're saying, maybe we should add some more warnings.
The VM should (& I believe already does) print an error if the attempt to raise the heartbeat thread priority fails. My underatanding is that the positive change is simply not to exit if the attempt fails.
Right, it does print a message, I'm just trying to make it more accurate and informative. Thanks! Alistair
On 31 March 2017 at 10:14, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Eliot,
On 31 March 2017 at 05:31, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Mar 30, 2017 at 8:20 PM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Alistair,
On Thu, Mar 30, 2017 at 9:32 AM, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Eliot,
On 30 March 2017 at 18:15, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
As Esteban wrote, the vm is trying to raise the priority. You can see the change at:
https://github.com/pharo-project/pharo-vm/commit/32f321583c69ca27e61ffaff6de...
I'd like to just make sure I understand your comment correctly:
Are you saying that the VM will lock up if it gets in to, e.g., an endless loop, so that Ctrl-. fails to interrupt. Or is it more serious than that? Can the VM become locked up for just a long running process (I guess it will be locked while the process is running, but will eventually go back to normal once the process completes)?
The issue is that delays are recognized by periodically checking the time against the next delay expiry. It is the heartbeat thread that causes JITed Smalltalk code to break out of its execution to check for events, including the next delay expiry. So if the heartbeat is at the same priority as the JITed Smalltalk code, and the Smalltalk code is in tight enough a loop (one that does not provoke a GC, does not do any process switches) the heartbeat thread will not run and the delay expiry will never be tested for.
Something like the following should run forever if the heartbeat has the same priority as the VM thread (and should be tried in a headless configuration too):
[| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun 1001
I get the same result regardless of thread priority, UI or headless. I've just shown the output from the headless VM with the heartbeat thread at normal priority (I haven't incorporated Ben's comments in to the text yet):
$ pharo Pharo.image --quit ~/pharo6/pharo/heartbeatTest.st pthread_setschedparam failed: Operation not permitted This VM uses a separate heartbeat thread to update the internal clock. For best operation, this thread should run at real-time priority, however the VM was unable to change the priority. The effect is that the clock may experience some jitter. To avoid clock jitter, please create the appropriate configuration file in /etc/security/limits.d by executing the following:
cat <<END | sudo tee /etc/security/limits.d/pharo.conf * hard rtprio 2 * soft rtprio 2 END
You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux Time to run: 0:00:00:01
heartbeatTest.st:
| time |
time := [| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun. FileStream stdout nextPutAll: 'Time to run: '; nextPutAll: time printString; cr. time
P.S It would be great is someone could reproduce this. Eliot and Ben's descriptions make sense, so I can't help feeling I've done something stupid here. Cheers, Alistair
On Fri, Mar 31, 2017 at 4:54 PM, Alistair Grant <akgrant0710@gmail.com> wrote:
On 31 March 2017 at 10:14, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Eliot,
On 31 March 2017 at 05:31, Eliot Miranda <eliot.miranda@gmail.com> wrote:
On Thu, Mar 30, 2017 at 8:20 PM, Eliot Miranda <eliot.miranda@gmail.com
wrote:
Hi Alistair,
On Thu, Mar 30, 2017 at 9:32 AM, Alistair Grant <akgrant0710@gmail.com
wrote:
Hi Eliot,
On 30 March 2017 at 18:15, Eliot Miranda <eliot.miranda@gmail.com>
wrote:
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <
pavel.krivanek@gmail.com>
wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat
thread
not running at elevated priority.
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
As Esteban wrote, the vm is trying to raise the priority. You can see the change at:
https://github.com/pharo-project/pharo-vm/commit/ 32f321583c69ca27e61ffaff6decc2a3e4b6ca5e
I'd like to just make sure I understand your comment correctly:
Are you saying that the VM will lock up if it gets in to, e.g., an endless loop, so that Ctrl-. fails to interrupt. Or is it more serious than that? Can the VM become locked up for just a long running process (I guess it will be locked while the process is running, but will eventually go back to normal once the process completes)?
The issue is that delays are recognized by periodically checking the time against the next delay expiry. It is the heartbeat thread that causes JITed Smalltalk code to break out of its execution to check for events, including the next delay expiry. So if the heartbeat is at the same priority as the JITed Smalltalk code, and the Smalltalk code is in tight enough a loop (one that does not provoke a GC, does not do any process switches) the heartbeat thread will not run and the delay expiry will never be tested for.
Something like the following should run forever if the heartbeat has the same priority as the VM thread (and should be tried in a headless configuration too):
[| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun 1001
I get the same result regardless of thread priority, UI or headless. I've just shown the output from the headless VM with the heartbeat thread at normal priority (I haven't incorporated Ben's comments in to the text yet):
$ pharo Pharo.image --quit ~/pharo6/pharo/heartbeatTest.st pthread_setschedparam failed: Operation not permitted This VM uses a separate heartbeat thread to update the internal clock. For best operation, this thread should run at real-time priority, however the VM was unable to change the priority. The effect is that the clock may experience some jitter. To avoid clock jitter, please create the appropriate configuration file in /etc/security/limits.d by executing the following:
cat <<END | sudo tee /etc/security/limits.d/pharo.conf * hard rtprio 2 * soft rtprio 2 END
You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk- vm/releases/tag/r3732#linux Time to run: 0:00:00:01
heartbeatTest.st:
| time |
time := [| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun. FileStream stdout nextPutAll: 'Time to run: '; nextPutAll: time printString; cr. time
P.S It would be great is someone could reproduce this. Eliot and Ben's descriptions make sense, so I can't help feeling I've done something stupid here.
Eliot, thanks for the test case. Using VM pharo-linux-i386threaded-201703281011 on an idle PC with pharo UI I get timeToRun of "0:00:00:01.001".
From the command line... $ vi heatbeatTestIdle.st | time | time := [ (Delay forSeconds: 1) wait ] timeToRun. FileStream stdout nextPutAll: 'Time to run: '; nextPutAll: time printString; cr.
$ pharo-linux-i386threaded-201703281011/lib/pharo/5.0-201703281011/pharo \ 60453.image `pwd`/heatbeatTestIdle.st Time to run: 0:00:00:01.002 $ pharo-linux-i386threaded-201703281011/lib/pharo/5.0-201703281011/pharo \ 60453.image `pwd`/heatbeatTest.st Time to run: 0:00:00:01.001 Stressing the system per my test.c from [1] compiled to `fib`, on my four CPUs starting fifteen processes doing fibonnachi of 10E10 spread across three process priorities 0, 1 & 19, and then running heatbeatTest.st. $ N=$((10**10)) ; for NPROC in 1 2 3 4 5 ; do (./fib 19 $N &) && (./fib 1 $N &) && (./fib 0 $N &) ; done && \ pharo-linux-i386threaded-201703281011/lib/pharo/5.0-201703281011/pharo \ 60453.image `pwd`/heatbeatTest.st Time to run: 0:00:00:01.002 14758 @ 0 ==> execution time 23:341844995 14734 @ 0 ==> execution time 23:957766684 14746 @ 0 ==> execution time 24:425043841 14740 @ 0 ==> execution time 24:458788284 14752 @ 0 ==> execution time 24:899062965 14750 @ 1 ==> execution time 26:118315887 14732 @ 1 ==> execution time 26:302822374 14756 @ 1 ==> execution time 26:398550194 14738 @ 1 ==> execution time 26:605458137 14744 @ 1 ==> execution time 26:655449934 14736 @ 19 ==> execution time 36:799776693 14754 @ 19 ==> execution time 37:564087310 14742 @ 19 ==> execution time 39:511492537 14748 @ 19 ==> execution time 39:714399199 14730 @ 19 ==> execution time 39:748212349 Output format is... PID @ nicePriority ==> execution time After output completes, `cat /proc/loadavg` reports a 1 minute load average (i.e. average processes waiting) of 6.36. So to me it looks like the un-raised *static* priority heatbeat-thread is working okay :) cheers -ben [1] http://forum.world.st/Unix-heartbeat-thread-vs-itimer-tp4928943p4938456.html
After re-reading Ben's replies and following the link to the discussion in the Squeak VM list I think I've finally understood this correctly. I've updated the text based on Ben's input to: -- This VM uses a separate heartbeat thread to update its internal clock and handle events. For best operation, this thread should run at a higher priority, however the VM was unable to change the priority. The effect is that heavily loaded systems may experience some latency issues. If this occurs, please create the configuration file in /etc/security/limits.d/ as shown below: cat <<END | sudo tee /etc/security/limits.d/VMNAME.conf * hard rtprio 2 * soft rtprio 2 END and report to the appropriate mailing list for your smalltalk dialect whether this improves behaviour. You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux -- I'm seeing the message printed out three times, which is a little strange, but haven't looked in to it. There are two issues in fogbugz: For the VM: https://pharo.fogbugz.com/f/cases/19890/Linux-Heartbeat-VM-text-and-instruct... For the build scripts: https://pharo.fogbugz.com/f/cases/19891/Suggested-improvement-for-linux-CONF... Given how close the 6.0 release date is and even though there is a bug in the instructions for enabling elevated thread privilege, I've marked this for Pharo 7.0. Of course, if Eliot or someone else decides to put it in 6.0, that's fine by me. As before, if there aren't any conflicting comments I'll submit the pull requests in a couple of days. Thanks again for all your assistance, Alistair
On Sun, Apr 2, 2017 at 2:13 PM, Alistair Grant <akgrant0710@gmail.com> wrote:
After re-reading Ben's replies and following the link to the discussion in the Squeak VM list I think I've finally understood this correctly.
I've updated the text based on Ben's input to:
-- This VM uses a separate heartbeat thread to update its internal clock and handle events. For best operation, this thread should run at a higher priority, however the VM was unable to change the priority. The effect is that heavily loaded systems may experience some latency issues. If this occurs, please create the configuration file in /etc/security/limits.d/ as shown below:
cat <<END | sudo tee /etc/security/limits.d/VMNAME.conf * hard rtprio 2 * soft rtprio 2 END
and report to the appropriate mailing list for your smalltalk dialect whether this improves behaviour.
Looks good. cheers -ben
I'm seeing the message printed out three times, which is a little strange, but haven't looked in to it.
There are two issues in fogbugz:
For the VM: https://pharo.fogbugz.com/f/cases/19890/Linux-Heartbeat- VM-text-and-instructions For the build scripts: https://pharo.fogbugz.com/f/cases/19891/Suggested- improvement-for-linux-CONFIG-file
Given how close the 6.0 release date is and even though there is a bug in the instructions for enabling elevated thread privilege, I've marked this for Pharo 7.0. Of course, if Eliot or someone else decides to put it in 6.0, that's fine by me. As before, if there aren't any conflicting comments I'll submit the pull requests in a couple of days.
Thanks again for all your assistance, Alistair
On Fri, Mar 31, 2017 at 12:15 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
_,,,^..^,,,_ (phone)
On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Yes, the same priority. See:
The VM may have issues with clock jitter due to the heartbeat thread not running at elevated priority.
"may" is fairly broad term and presumptive at this stage.
While the *static* priority of the heartbeat and interpreter threads is the same, the heartbeat-thread mostly sleeps, so its *dynamic* priority will be higher than the interpreter thread. This is even more so when the interpreter-thread is compute intensive. This behaviour is (maybe) Linux specific, but its only Linux that has a problem raising the heartbeat-thread priority. So the threaded-heartbeat "should not" have more jitter than before, but I'm still meaning to confirm this by experiment with a minimal C-lang thread code example. However I speculate that in a heavily loaded system, the itimer-heartbeat may have *slightly* better jitter than even a high-priority-heartbeat-thread, since IIUC signal handling preempts even higher priority threads. But most modern systems operate <20% CPU. Specific experiments would be needed to comment further, but I'd hazard to guess the itimer-heartbeat might only be an advantage for audio on heavily loaded systems. Not your common Pharo system. In balance against itimer conflicts with OSProcess and third-party C libraries, the threaded-heartbeat seems preferable (if all the above remains true in practice). Although I'm reading some stuff that may also resolve the timer-heartbeat issues. cheers -ben
It better /not/ be. This is wrong. The heartbeat thread /must/ run at a higher priority for the VM not to lock up when it becomes compute bound. I thought the resolution was that the VM /will/ try and raise the priority if the heartbeat thread but will /not/ exit if it fails. This is /very/ different from not trying to raise the priority at all.
The resolution I thought we had reached means that if correctly installed the VM can work correctly, and will continue to work correctly if and when linux removes the annoyance of the conf files. The resolution you describe Pavel (I hope inaccurately) means the threaded heartbeat VM will never work correctly.
One other thing. I know threaded heartbeat is verbose, but I would really appreciate it if we could reserve "threaded vm" to mean a vm that multiplexes Smalltalk processes above native threads, even if not concurrently. This is the threaded ffi plan which we have a good chance of realizing this year.
Good point. We can make use of our good friend hyphen... threaded-heatbeat & heartbeat-thread. cheers -ben
An exit may be appropriate in some cases, but for most users the above warning is sufficient.
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/ 32f321583c69ca27e61ffaff6decc2a3e4b6ca5e
Cheers, -- Pavel
2017-03-30 11:51 GMT+02:00 Denis Kudriashov <dionisiydk@gmail.com>:
2017-03-29 20:29 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:
Hi,
I want to let you know that the SSH (git@...) support in Iceberg on Linux VM now works with the latest threaded VM ( get.pharo.org/vmTLatest60). The threaded VM now runs without root priviledges nor special configuration of the system.
And what the logic now? Heartbeat and VM are in same priory?
participants (6)
-
Alistair Grant -
Ben Coman -
Denis Kudriashov -
Eliot Miranda -
Esteban Lorenzano -
Pavel Krivanek