On 2011-04-05, at 15:02, Nicolas Cellier wrote:
Better than average, take the median
NO! thats why you provide the standard deviation... you should not optimize the results prematurely but provide numbers that let you decide on the quality of the benchmarks. what you can do is optimize for systematic errors, like warm up and stuff, but nevertheless you should provide the raw results... read the paper...
Nicolas
2011/4/5 Camillo Bruni <camillo.bruni@inria.fr>:
This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion.
run it 100 times and take the average and provide the standard deviation.
I am not a big fan of relying on incomplete benchmarking results:
Please read: http://portal.acm.org/citation.cfm?id=1297033
http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it.
r := PBFloat run: 100. r asString
which will give decent results back :). This way it is much easier to make sense out of the numbers.
So here again to remember:
- number of samples - average run times - standard deviation
If one of these results is missing the benchmark results are incomplete.
best regards, camillo
On 2011-04-05, at 13:56, Igor Stasenko wrote:
VariableNode initialize. Compiler recompileAll.
[ TestCase allSubclasses do: [ :cls| cls isAbstract ifFalse: [cls suite run]]. ] timeToRun
178938 183963
(ParseNode classVarNamed: 'StdSelectors') removeKey: #class ifAbsent: []. Compiler recompileAll.
[ TestCase allSubclasses do: [ :cls| cls isAbstract ifFalse: [cls suite run]]. ] timeToRun
187168 184992
the deviation is too big to see if its really so big overhead.
if you compare worst , you got 187/178 ~ 5% and if you compare the best you got 184/183 ~ 0.5%
-- Best regards, Igor Stasenko AKA sig.