On Sun, Mar 13, 2011 at 11:30 PM, Stefan Marr
<pharo@stefan-marr.de> wrote:
Hi:
For the others, I am currently adapting the benchmarking infrastructure used for Pinocchio to be a bit more general, and enable me to integrate our RoarVM benchmarking tools.
The goal is to have a framework that allows all kind of benchmarking, written like unit tests.
One other idea Henrik was interested in is to be able to easily compare the benchmark results of different versions of method-implementations, to see whether optimizations were successful.
Hi Stef. I would love this. Not only to see how successful an optimization was, but also, how much overhead your solution has ;)
�
Camillo, even so I don't have a working version yet, I was trying to commit my refactoring (perhaps for review).
However, the PinocchioVM project seems to be 'global readonly'.
Would it be better to make it a stand-alone project?
Yes, please. Did you consider the repository PharoBenchmarks (even if the package can be called SBench)?� because it would be good to collect them somewhere, and that seems a good enough place.
BTW, how do you compare the benchmarks from PBenchmark to the ones that are in PharoBenchmarks ?
�
Finally, I would also love to have such benchmarks run in each hudson build to see wether each commit to pharo make it slower/faster.
I know Cyrille was working on that. I cc'ed him, since it would be nice to join forces.
When we go with that step, as already mentioned, I would like to rename it.
PBenchmark is a name that might prevent adoption. Not that I would like to go into politics here, but perhaps we can consider a new name.
Since the general idea is to write benchmarks like unit tests, how about 'SMark'/'SBench' instead of 'SUnit'?
SBench sounds good for me.
�
Best regards
Stefan
PS:
One thing, we might want to use as a source of inspiration in the future is:
http://code.google.com/p/caliper/
Thats a microbenchmark framework for Java, also using the unit-testing metaphor.
There @Param is also neat (http://code.google.com/p/caliper/source/browse/trunk/examples/src/main/java/examples/ArraySortBenchmark.java) and allows to encode input sizes for which the benchmarks should be executed.
That is something I am not to interested in, but in case the framework finds adoption that might be something to keep in mind.
--
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax: � +32 2 629 3525