are there benchmarks like deltablue and richards for Pharo?
Hi, I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P) Esteban
A couple of them are in Smark http://smalltalkhub.com/#!/~StefanMarr/SMark On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
I know I am a major pain in the ass, but where do you plan to run these tests? We need a standalone server machine to do them reproducibly :)... OR use https://ci.inria.fr/pharo-contribution/job/Rizel-StackVM/
On May 25, 2013, at 1:05 PM, Camillo Bruni <camillobruni@gmail.com> wrote:
A couple of them are in Smark
http://smalltalkhub.com/#!/~StefanMarr/SMark
On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
I know I am a major pain in the ass, but where do you plan to run these tests?
yes, you are ;) nah, I do not have any idea where to run them... I even doesn't know if I will, I was just thinking in loud. but yes, I think it would be good if we can have recurrent (and reliable) benchmarks to show :)
We need a standalone server machine to do them reproducibly :)... OR use https://ci.inria.fr/pharo-contribution/job/Rizel-StackVM/
On 2013-05-25, at 13:14, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On May 25, 2013, at 1:05 PM, Camillo Bruni <camillobruni@gmail.com> wrote:
A couple of them are in Smark
http://smalltalkhub.com/#!/~StefanMarr/SMark
On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
I know I am a major pain in the ass, but where do you plan to run these tests?
yes, you are ;) nah, I do not have any idea where to run them... I even doesn't know if I will, I was just thinking in loud. but yes, I think it would be good if we can have recurrent (and reliable) benchmarks to show :)
We ran a couple of benchmarks with the FulltextSearch project using the rizel command line handler: https://ci.inria.fr/rmod/job/FullTextSearch/plot/? ./pharo $JOB_NAME.image rizel --packagesMatching='FullTextSearch*' --reporter=RizelXMLReporter >> rizel.xml so basically everything is there :)
On 25 May 2013 13:20, Camillo Bruni <camillobruni@gmail.com> wrote:
On 2013-05-25, at 13:14, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On May 25, 2013, at 1:05 PM, Camillo Bruni <camillobruni@gmail.com> wrote:
A couple of them are in Smark
http://smalltalkhub.com/#!/~StefanMarr/SMark
On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
I know I am a major pain in the ass, but where do you plan to run these tests?
yes, you are ;) nah, I do not have any idea where to run them... I even doesn't know if I will, I was just thinking in loud. but yes, I think it would be good if we can have recurrent (and reliable) benchmarks to show :)
We ran a couple of benchmarks with the FulltextSearch project using the rizel command line handler: https://ci.inria.fr/rmod/job/FullTextSearch/plot/?
./pharo $JOB_NAME.image rizel --packagesMatching='FullTextSearch*' --reporter=RizelXMLReporter >> rizel.xml
so basically everything is there :)
yeah.. that what we actually wanted to do since a while.. when we first started using CI.. Unfortunately for that we need a fully controllable environment: no virtualization, sorry.. If we could have a fully dedicated machine for that, setting things up wouldn't be that hard. -- Best regards, Igor Stasenko.
On 2013-05-25, at 13:46, Igor Stasenko <siguctua@gmail.com> wrote:
On 25 May 2013 13:20, Camillo Bruni <camillobruni@gmail.com> wrote:
On 2013-05-25, at 13:14, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On May 25, 2013, at 1:05 PM, Camillo Bruni <camillobruni@gmail.com> wrote:
A couple of them are in Smark
http://smalltalkhub.com/#!/~StefanMarr/SMark
On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
I know I am a major pain in the ass, but where do you plan to run these tests?
yes, you are ;) nah, I do not have any idea where to run them... I even doesn't know if I will, I was just thinking in loud. but yes, I think it would be good if we can have recurrent (and reliable) benchmarks to show :)
We ran a couple of benchmarks with the FulltextSearch project using the rizel command line handler: https://ci.inria.fr/rmod/job/FullTextSearch/plot/?
./pharo $JOB_NAME.image rizel --packagesMatching='FullTextSearch*' --reporter=RizelXMLReporter >> rizel.xml
so basically everything is there :)
yeah.. that what we actually wanted to do since a while.. when we first started using CI.. Unfortunately for that we need a fully controllable environment: no virtualization, sorry..
well you can perfectly use Rizel in a virtualized environment
On 25 May 2013 13:47, Camillo Bruni <camillobruni@gmail.com> wrote:
On 2013-05-25, at 13:46, Igor Stasenko <siguctua@gmail.com> wrote:
On 25 May 2013 13:20, Camillo Bruni <camillobruni@gmail.com> wrote:
On 2013-05-25, at 13:14, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On May 25, 2013, at 1:05 PM, Camillo Bruni <camillobruni@gmail.com> wrote:
A couple of them are in Smark
http://smalltalkhub.com/#!/~StefanMarr/SMark
On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
I know I am a major pain in the ass, but where do you plan to run these tests?
yes, you are ;) nah, I do not have any idea where to run them... I even doesn't know if I will, I was just thinking in loud. but yes, I think it would be good if we can have recurrent (and reliable) benchmarks to show :)
We ran a couple of benchmarks with the FulltextSearch project using the rizel command line handler: https://ci.inria.fr/rmod/job/FullTextSearch/plot/?
./pharo $JOB_NAME.image rizel --packagesMatching='FullTextSearch*' --reporter=RizelXMLReporter >> rizel.xml
so basically everything is there :)
yeah.. that what we actually wanted to do since a while.. when we first started using CI.. Unfortunately for that we need a fully controllable environment: no virtualization, sorry..
well you can perfectly use Rizel in a virtualized environment
You mean to count stats in number of "message sends/bytecodes"? Yeah, could do.. but i was more about benchmarking VM itself... -- Best regards, Igor Stasenko.
yes, but rizel will give us a good comparison between different versions of our own systems, but now agains others (because rizel=interpreter=old and slow) :) not that it is bad, but I would prefer a dedicated machine. said so... I think if we have the benchmarks, I think we can annoy Stef enough to get a dedicated benchmark machine... not a terrible new one, but good enough :) but we need to have the benchmarks first, otherwise is loosing time, he. Esteban On May 25, 2013, at 1:47 PM, Camillo Bruni <camillobruni@gmail.com> wrote:
On 2013-05-25, at 13:46, Igor Stasenko <siguctua@gmail.com> wrote:
On 25 May 2013 13:20, Camillo Bruni <camillobruni@gmail.com> wrote:
On 2013-05-25, at 13:14, Esteban Lorenzano <estebanlm@gmail.com> wrote:
On May 25, 2013, at 1:05 PM, Camillo Bruni <camillobruni@gmail.com> wrote:
A couple of them are in Smark
http://smalltalkhub.com/#!/~StefanMarr/SMark
On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :) For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
I know I am a major pain in the ass, but where do you plan to run these tests?
yes, you are ;) nah, I do not have any idea where to run them... I even doesn't know if I will, I was just thinking in loud. but yes, I think it would be good if we can have recurrent (and reliable) benchmarks to show :)
We ran a couple of benchmarks with the FulltextSearch project using the rizel command line handler: https://ci.inria.fr/rmod/job/FullTextSearch/plot/?
./pharo $JOB_NAME.image rizel --packagesMatching='FullTextSearch*' --reporter=RizelXMLReporter >> rizel.xml
so basically everything is there :)
yeah.. that what we actually wanted to do since a while.. when we first started using CI.. Unfortunately for that we need a fully controllable environment: no virtualization, sorry..
well you can perfectly use Rizel in a virtualized environment
yeah.. that what we actually wanted to do since a while.. when we first started using CI.. Unfortunately for that we need a fully controllable environment: no virtualization, sorry.. If we could have a fully dedicated machine for that, setting things up wouldn't be that hard.
We can have a machine for this job wihtout problem.
Hi: On 25 May 2013, at 13:05, Camillo Bruni <camillobruni@gmail.com> wrote:
A couple of them are in Smark
SMark includes a pretty complete port of the Language Benchmark Game benchmarks. (http://benchmarksgame.alioth.debian.org/) Other people have been writing the benchmarks, and they are still in the CVS of the benchmark game, I think. The SMark version are slightly adapted but results should remain comparable. That means, taking the benchmark collection, you should be able to compare to all the other languages included in the benchmark game.
On 2013-05-25, at 12:55, Esteban Lorenzano <estebanlm@gmail.com> wrote:
I wonder if there are benchmarks around. Also I wonder if there would be useful :)
DeltaBlue can be found here: ftp://ftp.cs.washington.edu/pub/constraints/code/DeltaBlue/Smalltalk Not sure how much effort it is to get it running in Pharo, but I would assume it is pretty stand-alone. Not sure which Smalltalk it was written for thought. GNU Smalltalk has a version of Richard's benchmarks: https://github.com/gnu-smalltalk/benchmark/blob/master/Richards.st Also looks pretty straight forward to get them running in Pharo. Here is another version: http://www.cs.middlebury.edu/~briggs/Courses/CS313-F10/smalltalk/stx/goodies...
For "marketing" purposes I think yes: if we can put in the web page that we are running those benchmarks, it is cool for "seriousness" (that, of course, assuming the results will not be terribly bad :P)
The Benchmark Game benchmarks would get you quite a bit of the 'marketing'. DeltaBlue and Richards won't hurt, and are also widely ported to other systems. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
participants (5)
-
Camillo Bruni -
Esteban Lorenzano -
Igor Stasenko -
Stefan Marr -
stephane ducasse