[Pharo-project] Zinc Progress
we just discovered a strange side-effect of enabling progress report on ZnClient when loading mc packages: - with progress notification: ~1s - without notification: ~30ms I didn't dig more into the details bug this is quite a diff :D, and most probably the main cause that pharo loads configurations and mc stuff slowly. The latest pharo2.0 release has the progress report disabled. best cami
On 14 September 2012 18:09, Camillo Bruni <camillobruni@gmail.com> wrote:
we just discovered a strange side-effect of enabling progress report on ZnClient when loading mc packages: - with progress notification: ~1s - without notification: ~30ms
I didn't dig more into the details bug this is quite a diff :D, and most probably the main cause that pharo loads configurations and mc stuff slowly. The latest pharo2.0 release has the progress report disabled.
best cami
Does Zn use Notifications? Perhaps all that time's spent crawling up and down the stack. frank
Not good ;-) The actual signalling happens in ZnUtils class>>#streamFrom:to:[size:] every 16Kb using an HTTPProgress notification/exception. Maybe that is too much or too fast; that would mean there are 32 notifications for downloading 512Kb. Any suggestions on what would make more sense ? -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill On 14 Sep 2012, at 19:09, Camillo Bruni <camillobruni@gmail.com> wrote:
we just discovered a strange side-effect of enabling progress report on ZnClient when loading mc packages: - with progress notification: ~1s - without notification: ~30ms
I didn't dig more into the details bug this is quite a diff :D, and most probably the main cause that pharo loads configurations and mc stuff slowly. The latest pharo2.0 release has the progress report disabled.
best cami
Is it possible to notify every time interval? Say, every second or so. That way the cost would also not vary with download/upload speed. frank On 14 September 2012 19:15, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Not good ;-)
The actual signalling happens in ZnUtils class>>#streamFrom:to:[size:] every 16Kb using an HTTPProgress notification/exception. Maybe that is too much or too fast; that would mean there are 32 notifications for downloading 512Kb. Any suggestions on what would make more sense ?
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 14 Sep 2012, at 19:09, Camillo Bruni <camillobruni@gmail.com> wrote:
we just discovered a strange side-effect of enabling progress report on ZnClient when loading mc packages: - with progress notification: ~1s - without notification: ~30ms
I didn't dig more into the details bug this is quite a diff :D, and most probably the main cause that pharo loads configurations and mc stuff slowly. The latest pharo2.0 release has the progress report disabled.
best cami
On 14 Sep 2012, at 20:57, Frank Shearar <frank.shearar@gmail.com> wrote:
Is it possible to notify every time interval? Say, every second or so. That way the cost would also not vary with download/upload speed.
Good idea, it will take some tweaking to get this to work well, I guess. I remember that some changes like this were done for the unit tests progress as well, I'll try find that code. -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 2012-09-14, at 23:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 14 Sep 2012, at 20:57, Frank Shearar <frank.shearar@gmail.com> wrote:
Is it possible to notify every time interval? Say, every second or so. That way the cost would also not vary with download/upload speed.
Good idea, it will take some tweaking to get this to work well, I guess. I remember that some changes like this were done for the unit tests progress as well, I'll try find that code.
I still think it's not the amount of notifications that causes the pauses there must be more to it... given that the download took around 1000ms longer we would have spend more than 30ms per notification! no-way :) running the download in the profiler showed the 1s lost is spent in the idle process ;) IMO that's rather funny :)
On 15 Sep 2012, at 01:02, Camillo Bruni <camillobruni@gmail.com> wrote:
On 2012-09-14, at 23:11, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 14 Sep 2012, at 20:57, Frank Shearar <frank.shearar@gmail.com> wrote:
Is it possible to notify every time interval? Say, every second or so. That way the cost would also not vary with download/upload speed.
Good idea, it will take some tweaking to get this to work well, I guess. I remember that some changes like this were done for the unit tests progress as well, I'll try find that code.
I still think it's not the amount of notifications that causes the pauses there must be more to it... given that the download took around 1000ms longer we would have spend more than 30ms per notification! no-way :)
running the download in the profiler showed the 1s lost is spent in the idle process ;) IMO that's rather funny :)
I wrote some code to try to have something executable to work with, but I can't find anything. (The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 ) I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo: '/tmp/10Mb.bin' asFileReference ensureDeleted. [ ZnClient new systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] timeToRun. [ UIManager default informUserDuring: [ :bar | bar label: 'Downloading...'. [ ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] on: HTTPProgress do: [ :progress | bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. self crLog: progress printString. progress resume ] ] ] timeToRun. As fas as I can tell, both run at the same speed (more or less). Is is possible to show the code where you noticed the slowdown ? Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
I wrote some code to try to have something executable to work with, but I can't find anything.
(The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 )
I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo:
'/tmp/10Mb.bin' asFileReference ensureDeleted.
[ ZnClient new systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] timeToRun.
[ UIManager default informUserDuring: [ :bar | bar label: 'Downloading...'. [ ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] on: HTTPProgress do: [ :progress | bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. self crLog: progress printString. progress resume ] ] ] timeToRun.
As fas as I can tell, both run at the same speed (more or less).
Is is possible to show the code where you noticed the slowdown ?
I assume that the overhead only occurs for small files... ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; get. with progress 2.5secs without, 57ms.. :/ well might be that the server responds with an incomplete header or so...
On 2012-09-16, at 21:50, Camillo Bruni <camillobruni@gmail.com> wrote:
I wrote some code to try to have something executable to work with, but I can't find anything.
(The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 )
I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo:
'/tmp/10Mb.bin' asFileReference ensureDeleted.
[ ZnClient new systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] timeToRun.
[ UIManager default informUserDuring: [ :bar | bar label: 'Downloading...'. [ ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] on: HTTPProgress do: [ :progress | bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. self crLog: progress printString. progress resume ] ] ] timeToRun.
As fas as I can tell, both run at the same speed (more or less).
Is is possible to show the code where you noticed the slowdown ?
I assume that the overhead only occurs for small files...
ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; get.
with progress 2.5secs without, 57ms.. :/ well might be that the server responds with an incomplete header or so...
another remark with signalProgress the time to run highly varies, 250ms - 2500ms. whereas without progress notification it stays <140ms
I assume that the overhead only occurs for small files...
ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; get.
with progress 2.5secs without, 57ms.. :/ well might be that the server responds with an incomplete header or so...
another remark with signalProgress the time to run highly varies, 250ms - 2500ms. whereas without progress notification it stays <140ms
the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a on: Exception do:[ ... retry ]. so that means on an HTTPProgress you initiate a retry? (and wait for retryDelay), AUCH! :D
On 16 Sep 2012, at 21:59, Camillo Bruni <camillobruni@gmail.com> wrote:
I assume that the overhead only occurs for small files...
ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; get.
with progress 2.5secs without, 57ms.. :/ well might be that the server responds with an incomplete header or so...
another remark with signalProgress the time to run highly varies, 250ms - 2500ms. whereas without progress notification it stays <140ms
the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a on: Exception do:[ ... retry ]. so that means on an HTTPProgress you initiate a retry? (and wait for retryDelay), AUCH! :D
Yeah, that is probably it, and that is horrible, terrible. .. ;-) In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly. Tomorrow, I'll have another look, with a fresh head. Thx again! -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a on: Exception do:[ ... retry ]. so that means on an HTTPProgress you initiate a retry? (and wait for retryDelay), AUCH! :D
Yeah, that is probably it, and that is horrible, terrible. .. ;-) In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly. Tomorrow, I'll have another look, with a fresh head.
cool ;) I assume a simple Exception => Error will do :P
On 16 Sep 2012, at 23:14, Camillo Bruni <camillobruni@gmail.com> wrote:
the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a on: Exception do:[ ... retry ]. so that means on an HTTPProgress you initiate a retry? (and wait for retryDelay), AUCH! :D
Yeah, that is probably it, and that is horrible, terrible. .. ;-) In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly. Tomorrow, I'll have another look, with a fresh head.
cool ;) I assume a simple Exception => Error will do :P
Well, no ;-) I think it would be better to only do a retry on NetworkErrors and ZnParseErrors. On the other hand, the ifFail functionality should trigger on any Error. We'll see. Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-) Sven Sven Van Caekenberghe uploaded a new version of Zinc-HTTP to project Zinc HTTP Components: http://www.squeaksource.com/ZincHTTPComponents/Zinc-HTTP-SvenVanCaekenberghe... ==================== Summary ==================== Name: Zinc-HTTP-SvenVanCaekenberghe.301 Author: SvenVanCaekenberghe Time: 17 September 2012, 10:10:49 am UUID: 85632c09-a6c4-40e9-b29b-1c5e86d07ead Ancestors: Zinc-HTTP-SvenVanCaekenberghe.300 Fixed a bug where HTTPProgress notifications would trigger a retry. Thanks Camillo Bruni for finding this problem and suggesting a solution. Now, retries are only triggered by (NetworkError, ZnParseError), while the #ifFailBlock will be trigger on any Error. Furthermore, #noteRetrying: and noteIgnoringExceptionOnReusedConnection: will report on the actual exception. The default #ifFailBlock is now [ :exception | exception pass ] for some cleaner code. -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 2012-09-17, at 11:05, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 17 Sep 2012, at 10:18, Sven Van Caekenberghe <sven@stfx.eu> wrote:
Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-)
After fixing this in my working image, all MC operations just fly. I am happy and sad at the same time. Sorry guys.
haha :D if you have a nice design a performance issue per year is allowed :P
I think that I integrated some of the changes of camillo yesterday. Stef On Sep 16, 2012, at 9:50 PM, Camillo Bruni wrote:
I wrote some code to try to have something executable to work with, but I can't find anything.
(The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 )
I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo:
'/tmp/10Mb.bin' asFileReference ensureDeleted.
[ ZnClient new systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] timeToRun.
[ UIManager default informUserDuring: [ :bar | bar label: 'Downloading...'. [ ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] on: HTTPProgress do: [ :progress | bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. self crLog: progress printString. progress resume ] ] ] timeToRun.
As fas as I can tell, both run at the same speed (more or less).
Is is possible to show the code where you noticed the slowdown ?
I assume that the overhead only occurs for small files...
ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; get.
with progress 2.5secs without, 57ms.. :/ well might be that the server responds with an incomplete header or so...
participants (4)
-
Camillo Bruni -
Frank Shearar -
Stéphane Ducasse -
Sven Van Caekenberghe