Sven our objectives is that any package published in the metacello repositories (once metacello 1.0 is released) should get treated by the pharo hudson server. Eventually we want to have also the VM automatically generated. I should learn how to set more projects :) Stef

...

As usual, test results, code critics, and ready made images are available here:

http://hudson.lukas-renggli.ch/job/Zinc/

I must say that I am quite addicted to your Hudson server, almost every time I check in I go look at the page to see how it rebuilds.

It is really useful to have an integration server like that, and I can image it must be real fun for the (core) Pharo team as integration is one of their main issues. And it is instructive to so see all the Smalltalk scripting going on.

Sven

Philippe, First of all, thanks a lot for taking the time doing these benchmarks. I know you are very well placed to do these test, so I am honored you did. Second, although this is a static page bypassing encoding, the results are very impressive indeed. It is still Seaside and a 16KB page. Furthermore, all request completed successfully with a good distribution. This makes me quite happy! I do however suspect some serious hardware being used. Could you elaborate on the hardware, OS, VM and other details ? Regards, Sven On 17 Dec 2010, at 20:52, Philippe Marschall wrote:

On 15.12.2010 17:04, Sven Van Caekenberghe wrote:

...

I am pleased to report another milestone was reached in the Zn project.

The default implementation for an HTTP Server is now a multi-threaded (forking) keep-alive implementation called ZnMultiThreadedServer. It passes my initial concurrent load tests. ZnServer is now a facade. This means that all features (test server, static file server, monticello server, seaside adaptor, as well as all unit tests) are now using the new code.

In addition, several useful features were added to the servers, including logging and lastRequest/lastResponse debugging hooks. Extra handlers were added to ZnDefaultServerDelegate (favicon, benchmark, random) which double as examples.

Next up will be some benchmarking and optimalization. As always, users/testers are most welcome.

This is with the AJPFastRequestHandler from the AJP-Benchmark package from http://www.squeaksource.com/ajp (there are no dependencies to AJP).

ab -k -c 10 -n 10000 http://127.0.0.1:8080/fast This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests

Server Software: Zinc Server Hostname: 127.0.0.1 Server Port: 8080

Document Path: /fast Document Length: 16294 bytes

Concurrency Level: 10 Time taken for tests: 5.624 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 10000 Total transferred: 164560000 bytes HTML transferred: 162940000 bytes Requests per second: 1778.08 [#/sec] (mean) Time per request: 5.624 [ms] (mean) Time per request: 0.562 [ms] (mean, across all concurrent requests) Transfer rate: 28574.28 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 0 6 17.1 1 155 Waiting: 0 6 17.1 0 155 Total: 0 6 17.1 1 155

Percentage of the requests served within a certain time (ms) 50% 1 66% 1 75% 1 80% 1 90% 15 95% 44 98% 70 99% 89 100% 155 (longest request)

Not too shabby.

Cheers Philippe

On 18 Dec 2010, at 09:25, Philippe Marschall wrote:

Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz Linux 2.6.36 (64bit) Cog r2316 Pharo 1.1.1 (no memory tweaks)

Basically an almost four year old Linux box.

Philippe, Is that a desktop machine with a normal interactive load, or a server machine ? How much RAM ? Do you run the image headless ? On my development machine (Mac Book Pro, Intel Core 2 Duo 2.4 Ghz, 4 GB RAM, Mac OS X 10.6.5, Squeak 5.8b12, normal image, normal desktop load with lots of apps), I cannot get even close to your numbers (/bytes/16384 is a binary unencoded response of 16Kb direct from Zn): [sven@voyager:~]$ ab -k -n 10000 -c 10 http://127.0.0.1:1701/bytes/16384 This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests Server Software: Zinc Server Hostname: 127.0.0.1 Server Port: 1701 Document Path: /bytes/16384 Document Length: 16384 bytes Concurrency Level: 10 Time taken for tests: 14.829 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 10000 Total transferred: 165610000 bytes HTML transferred: 163840000 bytes Requests per second: 674.37 [#/sec] (mean) Time per request: 14.829 [ms] (mean) Time per request: 1.483 [ms] (mean, across all concurrent requests) Transfer rate: 10906.48 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 1 15 3.1 14 67 Waiting: 0 15 2.8 14 67 Total: 1 15 3.1 14 67 Percentage of the requests served within a certain time (ms) 50% 14 66% 15 75% 16 80% 16 90% 17 95% 18 98% 21 99% 25 100% 67 (longest request) This is no more than 1/3 of your results. I will be trying to find time to get a recent Cog VM on a Linux server machine using a headless deploy image and run benchmarks there. Thanks for the feedback and for pushing this. Sven

Hey Philippe, On 18 Dec 2010, at 17:20, Philippe Marschall wrote:

Is the response static or do you allocate a byte array for every response? Despite everything this benchmark is CPU limited.

You were right: I added caching (reuse on repeated requests) on repeated requests to my /bytes handler and I can now match your results on my machine (development image, interactive machine load): [sven@voyager:~]$ ab -k -n 10000 -c 10 http://127.0.0.1:1701/bytes/16384 This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests Server Software: Zinc Server Hostname: 127.0.0.1 Server Port: 1701 Document Path: /bytes/16384 Document Length: 16384 bytes Concurrency Level: 10 Time taken for tests: 4.319 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 10000 Total transferred: 165610000 bytes HTML transferred: 163840000 bytes Requests per second: 2315.37 [#/sec] (mean) Time per request: 4.319 [ms] (mean) Time per request: 0.432 [ms] (mean, across all concurrent requests) Transfer rate: 37446.17 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 0 4 1.6 4 30 Waiting: 0 4 1.5 4 27 Total: 0 4 1.6 4 30 Percentage of the requests served within a certain time (ms) 50% 4 66% 5 75% 5 80% 5 90% 6 95% 7 98% 8 99% 9 100% 30 (longest request) Benchmarking is indeed fun (and very dangereous and a time sink). Sven

On 18.12.2010 09:17, Stéphane Ducasse wrote:

Hi philippe

thanks for helping sven. Now for the blind like me. What do the tool says? Because not too shabby is difficult to fully interpret :)

Benchmarks always are ;-) Management summary it does: ~30 Mbytes/sec (bytes not bits) and 1778 requests per second. Now comes a long section why this is unrealistic and you can't expect this performance in the real world. You can look at the benchmark as an upper bound of what the server is capable of. It makes 10000 requests using 10 concurrent, recycled connections to the local machine. On the Smalltalk side there's a simple Seaside request handler that takes a 16k byte array (the www.seaside.st homepage) and writes it to the response. Note that a smaller page would obviously result in more requests per second and less throughput while you would expect to see the opposite from a bigger page. There are no sessions, continuations, rendering or encoding involved because the idea is to stress the sever, not Seaside. On the other hand conversion to and from Seaside requests and responses happens as well as the whole context and handler chain are set up. In a real world scenario you would spend much more time in Seaside and the back end of your application¹. The connections would be slower, each connection would not make that may requests and you'd have network latency. To get an idea what impact latency can have see [1]. Other points to note: - The server does not lose any requests. This is a problem I've run into using the same benchmark on other Smalltalk servers. - Most requests come back fairly quickly, 80% in 1 ms. The longest takes 155ms (maybe GC interference). One important point this benchmark does not cover is how well does it clean up hanging or stale connections. This is a much bigger issue in production than a few Mbytes/sec. To reproduce load the package AJP-Benchmark from [2], register AJPFastRequestHandler (see class side), hit the page once with you browser so that the cache gets initialized and run: ab -k -c 10 -n 10000 http://127.0.0.1:8080/fast (ab comes with every Mac) There are other request handlers in there that do encoding and rendering which obviously results in worse performance but more realistic numbers. They are designed to stress Seaside rather than the server. [1] http://blogs.webtide.com/gregw/entry/lies_damned_lies_and_benchmarks [2] http://www.squeaksource.com/ajp ¹ you could use a caching filter though that does a lookup in a dictionary Cheers Philippe

On 18.12.2010 11:43, Stéphane Ducasse wrote:

thanks philippe

now the executive summary: does it mean that zinc is well positioned compared to others?

That's a much more difficult question. First what do you mean by "others"? Second raw performance is often not the most important criteria to chose a web server. Others like reliability and feature set are often more important. And let's be honest, if raw performance was your most important criteria, Pharo probably wouldn't be your first choice anyway. Cheers Philippe

On 18 Dec 2010, at 11:30, Philippe Marschall wrote:

Benchmarks always are ;-)

[...] Excellent writeup, once again. I think / I'm pretty confident that the cleanup goes well, as long as Socket[Stream] respects its timeout (currently set at 10s for all streams, maybe 5s would be better for the server) and throws ConnectionTimedOut and ConnectionClosed exceptions reliably. Doing Smalltalk garbageCollect. Socket allInstances. before and after the benchmarks shows no leaking. But then again, the HTTP request parsing could be made more bullet proof. Adding a load balancer in front of Zn that also sanitizes requests would be a good idea for production setups. Sven