[Pharo-project] What is correct way to save image automatically in background?
In my app I need update pharo image with some resources (new package, icons, configs...) remotelly. After updating I need save image session. For that app show user some special screen, save image and return to ussual app state. My app has process that do system changes when needed and "commit" it as I wrote. I do something like [self blockUI. self doUpdate. SmalltalkImage current snapshot: true andQuit: false. self unblockUI. ] fork And sometimes after image restart I have image crash due to some world cicle activity and freetype plugin failure. I try little [SmalltalkImage current snapshot: true andQuit: false] fork code from workspace. And situation same - sometimes image cant restart. I never see that when I save image manually. What is correct way to save image for my purpose?
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process? 2010/7/23 Lukas Renggli <renggli@gmail.com>
What is correct way to save image for my purpose?
Don't fork.
Lukas
-- Lukas Renggli www.lukas-renggli.ch
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process?
I don't understand what you are saying. The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway. If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive). Lukas -- Lukas Renggli www.lukas-renggli.ch
Sorry for not correct thread title. I dont need save image in background while all system continue work. I just want save image from my smalltalk process. And when I do this I sometimes cant restart image (it crash due to some world cycle activity and freetype plugin failure). But when I save image manually I never see that situation. Maybe somebody know why this happen? And how I must save image from some smalltalk process correctly? 2010/7/23 Lukas Renggli <renggli@gmail.com>
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process?
I don't understand what you are saying.
The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway.
If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive).
Lukas
-- Lukas Renggli www.lukas-renggli.ch
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process?
I don't understand what you are saying.
The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway.
If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive).
"UnixProcess saveImageInBackgroundNicely" Explained in the method comment: UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag "When Squeak is used as a server it is sometimes desirable to periodically save image snapshots. This method forks a headless Squeak to perform a snapshot without impacting the server Squeak. Very little additional memory is required to do this because Unix copy-on-write memory management allows the two Squeak images to share object memory while the save is performed. The saved image is given a time stamped name, and the image name of the main server Squeak remains unchanged. If niceFlag is true, the background OS process runs at lowered scheduling priority." Dave
David, is what is said about memory management really true? I thought that saving the image triggered a garbage collect what forces to copy the whole memory anyway. Lukas On 23 July 2010 14:00, David T. Lewis <lewis@mail.msen.com> wrote:
On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process?
I don't understand what you are saying.
The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway.
If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive).
"UnixProcess saveImageInBackgroundNicely"
Explained in the method comment:
UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag     "When Squeak is used as a server it is sometimes desirable to periodically     save image snapshots. This method forks a headless Squeak to perform a     snapshot without impacting the server Squeak. Very little additional memory     is required to do this because Unix copy-on-write memory management allows     the two Squeak images to share object memory while the save is performed.     The saved image is given a time stamped name, and the image name of     the main server Squeak remains unchanged. If niceFlag is true, the     background OS process runs at lowered scheduling priority."
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
Yes, it really is true. That said, I would expect the practical impact to vary depending on how busy the main image is. If you are running a Squeak/Pharo image that is not busy, such as your own personal desktop image or a Seaside image that is not currently receiving requests, then the additional system memory used by the forked background image is extremely small (it's pretty amazing actually, try watching it on a system monitor such as xosview). However, I would expect that if you were to do this with a busy Seaside image, then that image would be doing a lot of updates to memory and possibly garbage collection, so that will force the OS to use more memory. Unfortunately this means that the busier your image is, the less benefit you get from a memory usage point of view. But even at that, you still get the advantage of doing the image save in a background OS process running at lower OS scheduling priority. I do not actually run any Seaside server images, so someone would need to try this on a real system to find out how much benefit it delivers in practice. I would be interested to know the results. Dave On Fri, Jul 23, 2010 at 02:12:14PM +0200, Lukas Renggli wrote:
David, is what is said about memory management really true? I thought that saving the image triggered a garbage collect what forces to copy the whole memory anyway.
Lukas
On 23 July 2010 14:00, David T. Lewis <lewis@mail.msen.com> wrote:
On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process?
I don't understand what you are saying.
The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway.
If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive).
"UnixProcess saveImageInBackgroundNicely"
Explained in the method comment:
UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag ? ? ? ?"When Squeak is used as a server it is sometimes desirable to periodically ? ? ? ?save image snapshots. This method forks a headless Squeak to perform a ? ? ? ?snapshot without impacting the server Squeak. Very little additional memory ? ? ? ?is required to do this because Unix copy-on-write memory management allows ? ? ? ?the two Squeak images to share object memory while the save is performed. ? ? ? ?The saved image is given a time stamped name, and the image name of ? ? ? ?the main server Squeak remains unchanged. If niceFlag is true, the ? ? ? ?background OS process runs at lowered scheduling priority."
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Cool, thanks for the update. On Friday, July 23, 2010, David T. Lewis <lewis@mail.msen.com> wrote:
Yes, it really is true. That said, I would expect the practical impact to vary depending on how busy the main image is.
If you are running a Squeak/Pharo image that is not busy, such as your own personal desktop image or a Seaside image that is not currently receiving requests, then the additional system memory used by the forked background image is extremely small (it's pretty amazing actually, try watching it on a system monitor such as xosview). However, I would expect that if you were to do this with a busy Seaside image, then that image would be doing a lot of updates to memory and possibly garbage collection, so that will force the OS to use more memory.
Unfortunately this means that the busier your image is, the less benefit you get from a memory usage point of view. But even at that, you still get the advantage of doing the image save in a background OS process running at lower OS scheduling priority.
I do not actually run any Seaside server images, so someone would need to try this on a real system to find out how much benefit it delivers in practice. I would be interested to know the results.
Dave
On Fri, Jul 23, 2010 at 02:12:14PM +0200, Lukas Renggli wrote:
David, is what is said about memory management really true? I thought that saving the image triggered a garbage collect what forces to copy the whole memory anyway.
Lukas
On 23 July 2010 14:00, David T. Lewis <lewis@mail.msen.com> wrote:
On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process?
I don't understand what you are saying.
The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway.
If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive).
"UnixProcess saveImageInBackgroundNicely"
Explained in the method comment:
UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag ? ? ? ?"When Squeak is used as a server it is sometimes desirable to periodically ? ? ? ?save image snapshots. This method forks a headless Squeak to perform a ? ? ? ?snapshot without impacting the server Squeak. Very little additional memory ? ? ? ?is required to do this because Unix copy-on-write memory management allows ? ? ? ?the two Squeak images to share object memory while the save is performed. ? ? ? ?The saved image is given a time stamped name, and the image name of ? ? ? ?the main server Squeak remains unchanged. If niceFlag is true, the ? ? ? ?background OS process runs at lowered scheduling priority."
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
On 23.07.2010, at 17:11, David T. Lewis wrote:
Yes, it really is true. That said, I would expect the practical impact to vary depending on how busy the main image is.
If you are running a Squeak/Pharo image that is not busy, such as your own personal desktop image or a Seaside image that is not currently receiving requests, then the additional system memory used by the forked background image is extremely small (it's pretty amazing actually, try watching it on a system monitor such as xosview). However, I would expect that if you were to do this with a busy Seaside image, then that image would be doing a lot of updates to memory and possibly garbage collection, so that will force the OS to use more memory.
What you are saying depends on the environment underneath. In theory it is pretty easy. The fork doubles the process memory when forking. Systems like linux have something called "memory overcommit". They don't allocate the reserved memory unless something is written. I think that is what you meant by saying "... is extremely small...". Why I am saying this? Because especially for seaside images the situation could be different. It is very common nowadays to use OS virtualization to install things like seaside images. Virtualizers disable "memory overcommitment" because otherwise one virtual instance could have side effects on other instances. So you are back with doubling process memory. You need to be careful if you have a small virtual machine or if you try to fork from several images at the same time. hope this adds anything, Norbert
Unfortunately this means that the busier your image is, the less benefit you get from a memory usage point of view. But even at that, you still get the advantage of doing the image save in a background OS process running at lower OS scheduling priority.
I do not actually run any Seaside server images, so someone would need to try this on a real system to find out how much benefit it delivers in practice. I would be interested to know the results.
Dave
On Fri, Jul 23, 2010 at 02:12:14PM +0200, Lukas Renggli wrote:
David, is what is said about memory management really true? I thought that saving the image triggered a garbage collect what forces to copy the whole memory anyway.
Lukas
On 23 July 2010 14:00, David T. Lewis <lewis@mail.msen.com> wrote:
On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process?
I don't understand what you are saying.
The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway.
If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive).
"UnixProcess saveImageInBackgroundNicely"
Explained in the method comment:
UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag ? ? ? ?"When Squeak is used as a server it is sometimes desirable to periodically ? ? ? ?save image snapshots. This method forks a headless Squeak to perform a ? ? ? ?snapshot without impacting the server Squeak. Very little additional memory ? ? ? ?is required to do this because Unix copy-on-write memory management allows ? ? ? ?the two Squeak images to share object memory while the save is performed. ? ? ? ?The saved image is given a time stamped name, and the image name of ? ? ? ?the main server Squeak remains unchanged. If niceFlag is true, the ? ? ? ?background OS process runs at lowered scheduling priority."
Dave
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- Lukas Renggli www.lukas-renggli.ch
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Here's my understanding of how it works. Garbage collection is completed before the Unix fork system call is made. At the moment of the fork, the original server VM process and the new image saving VM process share the exact same pages in the Unix OS - due to "Unix copy-on-write memory management". Immediately after the fork, the server VM process continues processing requests. Meanwhile, the image saving VM process proceeds to save the image. As each process continues execution, they will "dirty" their memory pages. Each dirty page means that that page is no longer shared between the two processes - leading to increased OS memory resources being allocated to the two processes. If the server VM process is very active, then a lot of dirty pages will be created, before the image saving VM process can complete. So, running the image saving process at a lower priority may actually be detrimental. One thing that occurs to me now, is whether or not the garbage collection is completed before the fork. If not, and it's done first thing by the image saving process, then it will be as you describe - the whole memory gets copied anyway. -- Yanni Lukas Renggli wrote:
David, is what is said about memory management really true? I thought that saving the image triggered a garbage collect what forces to copy the whole memory anyway.
Lukas
On 23 July 2010 14:00, David T. Lewis <lewis@mail.msen.com> wrote:
On Fri, Jul 23, 2010 at 09:49:46AM +0200, Lukas Renggli wrote:
But How I can do that without fork? I track changes in another process and when I do update I save image in this process. How I can initiate image saving from ui process? I don't understand what you are saying.
The thing is that using #fork to save the image has no effect (other than wrecking the image), because #snapshot:andQuit: is eventually calling a primitive that locks the complete VM anyway.
If you want to fork while saving the image you need to use an OS-level fork, not a Smalltalk-level fork. You can do this with OSProcess for example (check the Seaside mailing-list archive). "UnixProcess saveImageInBackgroundNicely"
Explained in the method comment:
UnixProcess class>>saveImageInBackground: savedImageName nice: niceFlag "When Squeak is used as a server it is sometimes desirable to periodically save image snapshots. This method forks a headless Squeak to perform a snapshot without impacting the server Squeak. Very little additional memory is required to do this because Unix copy-on-write memory management allows the two Squeak images to share object memory while the save is performed. The saved image is given a time stamped name, and the image name of the main server Squeak remains unchanged. If niceFlag is true, the background OS process runs at lowered scheduling priority."
Dave
On 2010-07-23, at 9:03 AM, Yanni Chiu wrote:
Here's my understanding of how it works. Garbage collection is completed before the Unix fork system call is made. At the moment of the fork, the original server VM process and the new image saving VM process share the exact same pages in the Unix OS - due to "Unix copy-on-write memory management".
Immediately after the fork, the server VM process continues processing requests. Meanwhile, the image saving VM process proceeds to save the image. As each process continues execution, they will "dirty" their memory pages. Each dirty page means that that page is no longer shared between the two processes - leading to increased OS memory resources being allocated to the two processes.
If the server VM process is very active, then a lot of dirty pages will be created, before the image saving VM process can complete. So, running the image saving process at a lower priority may actually be detrimental.
One thing that occurs to me now, is whether or not the garbage collection is completed before the fork. If not, and it's done first thing by the image saving process, then it will be as you describe - the whole memory gets copied anyway.
-- Yanni
Let me clarify a few things. First the Copy On Write only makes a copy of the original page if the page is going to be written to. Now *usually* 99% of the write activity is in young/new space which starts with zero bytes of memory after the Full GC is triggered. So as the saving task is writing out say the 0-100MB of memory, the running VM is busy allocating and writing to objects at 100+MB so there will be no effect on pages 0-100MB. Of course there are some updates into the 0-100MB area but that is rare as compared to write activity in new space. So if the process doing the write runs at a lower priority it shouldn't have an impact on the process running the VM. The danger is that a full GC should occur which would cause a compaction, but a server image should not be doing full GC more than a few times per hour.... Maybe never... -- =========================================================================== John M. McIntosh <johnmci@smalltalkconsulting.com> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================
participants (6)
-
David T. Lewis -
Denis Kudriashov -
John M McIntosh -
Lukas Renggli -
Norbert Hartl -
Yanni Chiu