I am maybe an ignorant here writing like that. I like to see a nice set of benchmarks popping up in Pharo.

But did you have a look at my benchmark implementation we use in Pinocchio?
Its in the PBenchmark package of the Pinocchio project:

MCHttpRepository
� �location: 'http://www.squeaksource.com/p'
� �user: ''
� �password: ''

Its a fairly straight forward implementation based on UnitTests...

I see several issues in your benchmark implementation which I think are solved much cleaner with my approach:

- currently there is one single class with tons of benchmarks in it
- no statistically valid output ( just the average doesn't mean anything! see http://dx.doi.org/10.1145/1297105.1297033 for the basic scientific backgrounds)
- you interleave model (benchmarks and results) and view (transcript output) which is really evil, there is no way you can ever use this on the command line!

so what I suggest, is that you have a look at my implementation and see how we can improve the current situation.

m(^_-)m
camillo

On 2011-03-11, at 15:05, Alexandre Bergel wrote:

> I am not sure what you mean with vm benchmarks, but in almost all the tools I am contributing come with some benchmarks (spy, Mondrian, I wrote some benchmarks for Glamour as well). Naturally, those are macro benchmarks, which is probably what matter the most.
>
> Cheers,
> Alexandre
>
>
>
> Le 11 mars 2011 � 10:55, Mariano Martinez Peck <marianopeck@gmail.com> a �crit :
>
>> does anyone have news regarding this topic? �we VERY welcome people helping with benchmarks.
>> please feel free to improve http://www.squeaksource.com/PharoBenchmarks
>>
>>
>>
>> On Tue, Jan 4, 2011 at 11:51 PM, Stefan Marr <pharo@stefan-marr.de> wrote:
>> Hi:
>>
>> On 04 Jan 2011, at 23:40, Mariano Martinez Peck wrote:
>>
>>> On Tue, Jan 4, 2011 at 11:35 PM, Stefan Marr <squeak@stefan-marr.de> wrote:
>>> Hi Igor:
>>>
>>> On 04 Jan 2011, at 22:40, Igor Stasenko wrote:
>>>
>>>> Okay, how about creating a separate
>>>> VMBenchmarks repository
>>>> and putting VMBenchmarks package there?
>>>
>>> Sure, sounds good. There are also the Systems benchmarks at http://www.squeaksource.com/PharoBenchmarks.
>>>
>>>
>>> So, at least, I was able to run all benchmarks :)
>>>
>>> However, it still needs cleaning, improvement, testing, blah. �But it is a good start point I think.
>>
>> Well, I am not to sure about the general value of those benchmarks.
>> There are many microbenchmarks which do not tell you a lot. All those test* things.
>> And well, their value for testing is also questionable. They only can help you to identify where it goes *boom* and crashes the VM, but they do not actually assert for anything.
>>
>> Also not sure what the value of Slopstone and Smopstone (names from the top of my head might be slightly different) is nowadays.
>>
>> The compiler benchmark is a reasonable application benchmark.
>> Would be good to have a few others in that collection, too.
>>
>> Best regards
>> Stefan
>>
>>
>>
>>
>> --
>> Stefan Marr
>> Software Languages Lab
>> Vrije Universiteit Brussel
>> Pleinlaan 2 / B-1050 Brussels / Belgium
>> http://soft.vub.ac.be/~smarr
>> Phone: +32 2 629 2974
>> Fax: � +32 2 629 3525
>>
>>
>>