This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion. run it 100 times and take the average and provide the standard deviation. I am not a big fan of relying on incomplete benchmarking results: Please read: http://portal.acm.org/citation.cfm?id=1297033 http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it. r := PBFloat run: 100. r asString which will give decent results back :). This way it is much easier to make sense out of the numbers. So here again to remember: - number of samples - average run times - standard deviation If one of these results is missing the benchmark results are incomplete. best regards, camillo On 2011-04-05, at 13:56, Igor Stasenko wrote:

VariableNode initialize. Compiler recompileAll.

[ TestCase allSubclasses do: [ :cls| cls isAbstract ifFalse: [cls suite run]]. ] timeToRun

178938 183963

(ParseNode classVarNamed: 'StdSelectors') removeKey: #class ifAbsent: []. Compiler recompileAll.

[ TestCase allSubclasses do: [ :cls| cls isAbstract ifFalse: [cls suite run]]. ] timeToRun

187168 184992

the deviation is too big to see if its really so big overhead.

if you compare worst , you got 187/178 ~ 5% and if you compare the best you got 184/183 ~ 0.5%

-- Best regards, Igor Stasenko AKA sig.

On 2011-04-05, at 15:02, Nicolas Cellier wrote:

Better than average, take the median

NO! thats why you provide the standard deviation... you should not optimize the results prematurely but provide numbers that let you decide on the quality of the benchmarks. what you can do is optimize for systematic errors, like warm up and stuff, but nevertheless you should provide the raw results... read the paper...

Nicolas

2011/4/5 Camillo Bruni <camillo.bruni@inria.fr>:

...

This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion.

run it 100 times and take the average and provide the standard deviation.

I am not a big fan of relying on incomplete benchmarking results:

Please read: http://portal.acm.org/citation.cfm?id=1297033

http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it.

r := PBFloat run: 100. r asString

which will give decent results back :). This way it is much easier to make sense out of the numbers.

So here again to remember:

- number of samples - average run times - standard deviation

If one of these results is missing the benchmark results are incomplete.

best regards, camillo

On 2011-04-05, at 13:56, Igor Stasenko wrote:

...

VariableNode initialize. Compiler recompileAll.

[ TestCase allSubclasses do: [ :cls| cls isAbstract ifFalse: [cls suite run]]. ] timeToRun

178938 183963

(ParseNode classVarNamed: 'StdSelectors') removeKey: #class ifAbsent: []. Compiler recompileAll.

[ TestCase allSubclasses do: [ :cls| cls isAbstract ifFalse: [cls suite run]]. ] timeToRun

187168 184992

the deviation is too big to see if its really so big overhead.

if you compare worst , you got 187/178 ~ 5% and if you compare the best you got 184/183 ~ 0.5%

-- Best regards, Igor Stasenko AKA sig.

Hi: On 05 Apr 2011, at 21:27, Nicolas Cellier wrote:

But you probably have more ellaborated and well thought method :) I think we are talking here mostly about quick and dirty benchmarks to confirm intuitions.

The provable best and most cost effective way to do it is: flip a coin (and that is serious, has the same or even better reliability and quality of result) If you think that is not good enough, well then the absolute minimum is something like this: - run your benchmark 5 times - 5 times with your optimization - 5 times without - look at the collected raw values i.e. measure runtime - and use your gut feeling to decide whether you have meaningful values and whether your time to do the optimization was well spent (don't eliminate outliers) If that is still to much to do, well, again, just flip a coin, and you have a high chance that the result of that are actually more reliable. (the approach above does not account for systematic errors...) However, once you got your optimization into the VM your favorite build server should do its thing and that should include performance regression testing which is using a "slightly" more sophisticated approach than the described one. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525

I don't care :) What i found that measurements confirms my previous expectation: the 'slowdown' of removing #class from set of optimized selectors lies in the range of macro-benchmark deviation. And running all tests for 100 times is not fun (it could take half of the day), and i'm not ready to leave my computer for half of the day and simply wait till it finish, because if i will do something else,then it will affect the performance, and measurements will be even more inaccurate. On 5 April 2011 14:53, Camillo Bruni <camillo.bruni@inria.fr> wrote:

This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion.

run it 100 times and take the average and provide the standard deviation.

I am not a big fan of relying on incomplete benchmarking results:

Please read: http://portal.acm.org/citation.cfm?id=1297033

http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it.

r := PBFloat run: 100. r asString

which will give decent results back :). This way it is much easier to make sense out of the numbers.

So here again to remember:

- number of samples - average run times - standard deviation

If one of these results is missing the benchmark results are incomplete.

best regards, camillo

On 2011-04-05, at 13:56, Igor Stasenko wrote:

...

VariableNode initialize. Compiler recompileAll.

[ TestCase allSubclasses do: [ :cls|        cls isAbstract                ifFalse: [cls suite run]]. ] timeToRun

178938 183963

(ParseNode classVarNamed:  'StdSelectors') removeKey: #class ifAbsent: []. Compiler recompileAll.

[ TestCase allSubclasses do: [ :cls|        cls isAbstract                ifFalse: [cls suite run]]. ] timeToRun

187168 184992

the deviation is too big to see if its really so big overhead.

if you compare worst , you got  187/178 ~ 5% and if you compare the best you got 184/183  ~ 0.5%

-- Best regards, Igor Stasenko AKA sig.

-- Best regards, Igor Stasenko AKA sig.

On 05 Apr 2011, at 15:11, Igor Stasenko wrote:

I don't care :) *sigh* then just flip a coin and you have the same amount of insight with less effort.

It is always astonishing how few people know about the basics when it comes to empirical experiments. I disagree with Camillo's demand for 100x repetition, but it should be definitely run at least a few times. And then you have to look at the results and see how they vary. Otherwise you don't know anything and are just wasting your own time. I will try at the next sprint to make some progress on the framework front. Perhaps, someone could trow together something like the unit-test runner to make at least the basics fool-proof. And at the same time the message to measure the execution time of a block should become deprecated or raise a warning that you are about to fool yourself... It does not have to be statistically rigorous for small experiments, but at the very least you have to convince yourself that your measurements are actually of any value. The computer in front of you is just to complex to make an educated guess. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525

Depends at which signal/noise ratio you are ready to drop the signal... Nicolas 2011/4/5 Igor Stasenko <siguctua@gmail.com>:

On 5 April 2011 16:31, Stefan Marr <pharo@stefan-marr.de> wrote:

...

On 05 Apr 2011, at 15:11, Igor Stasenko wrote:

...

I don't care :) *sigh* then just flip a coin and you have the same amount of insight with less effort.

It is always astonishing how few people know about the basics when it comes to empirical experiments.

I disagree with Camillo's demand for 100x repetition, but it should be definitely run at least a few times. And then you have to look at the results and see how they vary. Otherwise you don't know anything and are just wasting your own time.

that's what i did. i run two times and found that slowdown is at scale of deviation. This is why i don't care about exact slowdown anymore, because it is at the level of noise on macro benchmarks.

...

I will try at the next sprint to make some progress on the framework front. Perhaps, someone could trow together something like the unit-test runner to make at least the basics fool-proof.

And at the same time the message to measure the execution time of a block should become deprecated or raise a warning that you are about to fool yourself...

It does not have to be statistically rigorous for small experiments, but at the very least you have to convince yourself that your measurements are actually of any value. The computer in front of you is just to complex to make an educated guess.

Best regards Stefan

-- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax:   +32 2 629 3525

-- Best regards, Igor Stasenko AKA sig.

On 06 Apr 2011, at 00:01, Sven Van Caekenberghe wrote:

On 05 Apr 2011, at 14:53, Camillo Bruni wrote:

...

Please read: http://portal.acm.org/citation.cfm?id=1297033

I would love to read this, but why should I have to pay for this ? Research, especially when it is paid for with public money, should be free for all.

http://lmgtfy.com/?q=Statistically+rigorous+java+performance+evaluation ...

Sven

In case google is only my friend: http://itkovian.net/base/files/papers/oopsla2007-georges-preprint.pdf -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525