For PharoLauncher, I am trying to get a progress bar going for the downloading the Template lists, but the total transfer size is not being filled in for the HTTPProgress announcement. I've pulled out some code to Workspace script into [2] with the resulting Transcript result shown in [3]. I'm guessing this is due to the _"Transfer-Encoding: chunked"_ header clashing with the "Content-Length" header [1]. By putting a 'self halt' in ZnStatusLine>>readFrom: and executing several "Transcript crShow: (ZnLineReader on: stream) nextLine" the real headers were found as shown in [4], from which it seems the Content-Length is faked-up after the download finishes, and no use for a progress bar. So I only today discovered 'chunked' format, so I am curious... is there a particular advantage to use chunked for directory listing at "url=http://files.pharo.org/image/30/" ? Could it be changed to instead set Content-Length instead? btw, This is only for the directory listing (which however takes 10 seconds to get all). The zip files seem to set "Content-Length" when downloaded. cheers -ben [1] http://stackoverflow.com/questions/2419281/content-length-header-versus-chun... "[2]start------------------Workspace-script----------" workBlock := [ ZnClient new signalProgress: true ; url: 'http://files.pharo.org/image/30/' ; enforceHttpSuccess: true ; beOneShot ; accept: 'text/html;charset=utf-8' ; enforceAcceptContentType: true ; logToTranscript ; get ]. UIManager default informUserDuring: [ :bar| [ workBlock value] on: HTTPProgress do: [ : progress | Transcript crShow: '# ' , progress printString , ' total=' , progress total asString , ' amount=' , progress amount asString. bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. progress resume. ]. ]. "[2]end------------------Workspace-script----------" "[3]start--------------Transcript------" # HTTPProgress: Connecting to files.pharo.org total=nil amount=nil 2013-10-02 00:43:36 428852 I Wrote a ZnRequest(GET /image/30/) 2013-10-02 00:43:36 428852 D Sent headers Accept: text/html;charset=utf-8 User-Agent: Zinc HTTP Components 1.0 Connection: close Host: files.pharo.org # HTTPProgress: Writing request total=nil amount=nil # HTTPProgress: Reading response total=nil amount=nil ZnUtils>>signalProgress: 16384 total: nil # HTTPProgress: Tranferred 16.38k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 32768 total: nil # HTTPProgress: Tranferred 32.77k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 49152 total: nil # HTTPProgress: Tranferred 49.15k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 65536 total: nil # HTTPProgress: Tranferred 65.54k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 81920 total: nil # HTTPProgress: Tranferred 81.92k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 85434 total: nil # HTTPProgress: Tranferred 85.43k bytes ... total=nil amount=nil 2013-10-02 00:43:38 428852 I Read a ZnResponse(200 OK text/html;h5ai=0.22.1;charset=UTF-8 85444B) 2013-10-02 00:43:38 428852 D Received headers Date: Tue, 01 Oct 2013 16:43:34 GMT Transfer-Encoding: chunked Content-Length: 85444 Vary: Accept-Encoding Server: Apache Connection: close Content-Type: text/html;h5ai=0.22.1;charset=UTF-8 2013-10-02 00:43:38 428852 T GET /image/30/ 200 85444B 1874ms "[3]end--------------Transcript------" "[4]start----------ZnStatusLine>>readReal-Headers--------" HTTP/1.1 200 OK Date: Tue, 01 Oct 2013 17:23:55 GMT Server: Apache Vary: Accept-Encoding Connection: close Transfer-Encoding: chunked Content-Type: text/html;h5ai=0.22.1;charset=UTF-8 "[4]end-----------Real-Headers--------"
Hi Ben, On 01 Oct 2013, at 19:48, Ben Coman <btc@openInWorld.com> wrote:
For PharoLauncher, I am trying to get a progress bar going for the downloading the Template lists, but the total transfer size is not being filled in for the HTTPProgress announcement. I've pulled out some code to Workspace script into [2] with the resulting Transcript result shown in [3]. I'm guessing this is due to the _"Transfer-Encoding: chunked"_ header clashing with the "Content-Length" header [1]. By putting a 'self halt' in ZnStatusLine>>readFrom: and executing several "Transcript crShow: (ZnLineReader on: stream) nextLine" the real headers were found as shown in [4], from which it seems the Content-Length is faked-up after the download finishes, and no use for a progress bar.
Completely correct! I am happy you made it this far and were able to understand the specs and code.
So I only today discovered 'chunked' format, so I am curious... is there a particular advantage to use chunked for directory listing at "url=http://files.pharo.org/image/30/" ? Could it be changed to instead set Content-Length instead? btw, This is only for the directory listing (which however takes 10 seconds to get all). The zip files seem to set "Content-Length" when downloaded.
The whole idea of chunked transfer encoding is to make it easier for the server: it does not have to buffer and compute a total size upfront. BTW, there are other cases in HTTP where Content-Length is not set as well. It is just how it is. And yes that means silly progress indications. Sven
cheers -ben
[1] http://stackoverflow.com/questions/2419281/content-length-header-versus-chun...
"[2]start------------------Workspace-script----------" workBlock := [ ZnClient new signalProgress: true ; url: 'http://files.pharo.org/image/30/' ; enforceHttpSuccess: true ; beOneShot ; accept: 'text/html;charset=utf-8' ; enforceAcceptContentType: true ; logToTranscript ; get ].
UIManager default informUserDuring: [ :bar| [ workBlock value] on: HTTPProgress do: [ : progress | Transcript crShow: '# ' , progress printString , ' total=' , progress total asString , ' amount=' , progress amount asString. bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. progress resume. ]. ]. "[2]end------------------Workspace-script----------"
"[3]start--------------Transcript------" # HTTPProgress: Connecting to files.pharo.org total=nil amount=nil 2013-10-02 00:43:36 428852 I Wrote a ZnRequest(GET /image/30/) 2013-10-02 00:43:36 428852 D Sent headers Accept: text/html;charset=utf-8 User-Agent: Zinc HTTP Components 1.0 Connection: close Host: files.pharo.org # HTTPProgress: Writing request total=nil amount=nil # HTTPProgress: Reading response total=nil amount=nil ZnUtils>>signalProgress: 16384 total: nil # HTTPProgress: Tranferred 16.38k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 32768 total: nil # HTTPProgress: Tranferred 32.77k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 49152 total: nil # HTTPProgress: Tranferred 49.15k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 65536 total: nil # HTTPProgress: Tranferred 65.54k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 81920 total: nil # HTTPProgress: Tranferred 81.92k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 85434 total: nil # HTTPProgress: Tranferred 85.43k bytes ... total=nil amount=nil 2013-10-02 00:43:38 428852 I Read a ZnResponse(200 OK text/html;h5ai=0.22.1;charset=UTF-8 85444B) 2013-10-02 00:43:38 428852 D Received headers Date: Tue, 01 Oct 2013 16:43:34 GMT Transfer-Encoding: chunked Content-Length: 85444 Vary: Accept-Encoding Server: Apache Connection: close Content-Type: text/html;h5ai=0.22.1;charset=UTF-8 2013-10-02 00:43:38 428852 T GET /image/30/ 200 85444B 1874ms "[3]end--------------Transcript------"
"[4]start----------ZnStatusLine>>readReal-Headers--------" HTTP/1.1 200 OK Date: Tue, 01 Oct 2013 17:23:55 GMT Server: Apache Vary: Accept-Encoding Connection: close Transfer-Encoding: chunked Content-Type: text/html;h5ai=0.22.1;charset=UTF-8 "[4]end-----------Real-Headers--------"
On 02 Oct 2013, at 05:29, btc@openinworld.com wrote:
Sven Van Caekenberghe wrote:
Hi Ben,
On 01 Oct 2013, at 19:48, Ben Coman <btc@openInWorld.com> wrote:
For PharoLauncher, I am trying to get a progress bar going for the downloading the Template lists, but the total transfer size is not being filled in for the HTTPProgress announcement. I've pulled out some code to Workspace script into [2] with the resulting Transcript result shown in [3]. I'm guessing this is due to the _"Transfer-Encoding: chunked"_ header clashing with the "Content-Length" header [1]. By putting a 'self halt' in ZnStatusLine>>readFrom: and executing several "Transcript crShow: (ZnLineReader on: stream) nextLine" the real headers were found as shown in [4], from which it seems the Content-Length is faked-up after the download finishes, and no use for a progress bar.
Completely correct! I am happy you made it this far and were able to understand the specs and code.
So I only today discovered 'chunked' format, so I am curious... is there a particular advantage to use chunked for directory listing at "url=http://files.pharo.org/image/30/ " ? Could it be changed to instead set Content-Length instead? btw, This is only for the directory listing (which however takes 10 seconds to get all). The zip files seem to set "Content-Length" when downloaded.
The whole idea of chunked transfer encoding is to make it easier for the server: it does not have to buffer and compute a total size upfront. BTW, there are other cases in HTTP where Content-Length is not set as well. It is just how it is. And yes that means silly progress indications.
Sven
. Thanks for confirming that. Referring to [5]&[6], do you think the client forcing a HTTP/1.0 connection could help get proper progress bars? ...and how would that look adapting my Workspace script [2] ?
This is an unusual request, I've never needed it, I guess you could do something like | client | client := ZnClient new. client url: 'http://zn.stfx.eu/zn/small.html'. client request requestLine version: 'HTTP/1.0'. client get. client
Maybe a downside of forcing HTTP/1.0 would requiring the server to not be a VirtualHost (??). Does anyone know how the configuration of the servers used by Pharo Launcher relate to this?
But yes, you will break lots of stuff, and for what ? I mean, is it so bad that one of the downloads has no real progress bar ? How long can an 80Kb download take ? There should still be a dialog that says downloading with a moving byte count, the graphic should be an animated in definitive progress bar.
-ben
[5] http://webmasters.stackexchange.com/questions/38738/how-to-avoid-chunked-tra... [6] http://www.jargonsoft.com/support/knowledgebase.php?article=12
cheers -ben
[1] http://stackoverflow.com/questions/2419281/content-length-header-versus-chun...
"[2]start------------------Workspace-script----------" workBlock := [ ZnClient new signalProgress: true ; url: ' http://files.pharo.org/image/30/ ' ; enforceHttpSuccess: true ; beOneShot ; accept: 'text/html;charset=utf-8' ; enforceAcceptContentType: true ; logToTranscript ; get ].
UIManager default informUserDuring: [ :bar| [ workBlock value] on: HTTPProgress do: [ : progress | Transcript crShow: '# ' , progress printString , ' total=' , progress total asString , ' amount=' , progress amount asString. bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. progress resume. ]. ]. "[2]end------------------Workspace-script----------"
"[3]start--------------Transcript------" # HTTPProgress: Connecting to files.pharo.org total=nil amount=nil 2013-10-02 00:43:36 428852 I Wrote a ZnRequest(GET /image/30/) 2013-10-02 00:43:36 428852 D Sent headers Accept: text/html;charset=utf-8 User-Agent: Zinc HTTP Components 1.0 Connection: close Host: files.pharo.org # HTTPProgress: Writing request total=nil amount=nil # HTTPProgress: Reading response total=nil amount=nil ZnUtils>>signalProgress: 16384 total: nil # HTTPProgress: Tranferred 16.38k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 32768 total: nil # HTTPProgress: Tranferred 32.77k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 49152 total: nil # HTTPProgress: Tranferred 49.15k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 65536 total: nil # HTTPProgress: Tranferred 65.54k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 81920 total: nil # HTTPProgress: Tranferred 81.92k bytes ... total=nil amount=nil ZnUtils>>signalProgress: 85434 total: nil # HTTPProgress: Tranferred 85.43k bytes ... total=nil amount=nil 2013-10-02 00:43:38 428852 I Read a ZnResponse(200 OK text/html;h5ai=0.22.1;charset=UTF-8 85444B) 2013-10-02 00:43:38 428852 D Received headers Date: Tue, 01 Oct 2013 16:43:34 GMT Transfer-Encoding: chunked Content-Length: 85444 Vary: Accept-Encoding Server: Apache Connection: close Content-Type: text/html;h5ai=0.22.1;charset=UTF-8 2013-10-02 00:43:38 428852 T GET /image/30/ 200 85444B 1874ms "[3]end--------------Transcript------"
"[4]start----------ZnStatusLine>>readReal-Headers--------" HTTP/1.1 200 OK Date: Tue, 01 Oct 2013 17:23:55 GMT Server: Apache Vary: Accept-Encoding Connection: close Transfer-Encoding: chunked Content-Type: text/html;h5ai=0.22.1;charset=UTF-8 "[4]end-----------Real-Headers--------"
participants (3)
-
Ben Coman -
btc＠openinworld.com
-
Sven Van Caekenberghe