Better than average, take the median Nicolas 2011/4/5 Camillo Bruni <camillo.bruni@inria.fr>:
This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion.
run it 100 times and take the average and provide the standard deviation.
I am not a big fan of relying on incomplete benchmarking results:
Please read: http://portal.acm.org/citation.cfm?id=1297033
http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it.
r := PBFloat run: 100. r asString
which will give decent results back :). This way it is much easier to make sense out of the numbers.
So here again to remember:
- number of samples - average run times - standard deviation
If one of these results is missing the benchmark results are incomplete.
best regards, camillo
On 2011-04-05, at 13:56, Igor Stasenko wrote:
VariableNode initialize. Compiler recompileAll.
[ TestCase allSubclasses do: [ :cls|     cls isAbstract         ifFalse: [cls suite run]]. ] timeToRun
178938 183963
(ParseNode classVarNamed: Â 'StdSelectors') removeKey: #class ifAbsent: []. Compiler recompileAll.
[ TestCase allSubclasses do: [ :cls|     cls isAbstract         ifFalse: [cls suite run]]. ] timeToRun
187168 184992
the deviation is too big to see if its really so big overhead.
if you compare worst , you got  187/178 ~ 5% and if you compare the best you got 184/183  ~ 0.5%
-- Best regards, Igor Stasenko AKA sig.