Hello, I've just released a utility I've been using for the past few years to help optimize Pharo code. It's a library that makes it easy to check whether an optimization has improved performance by wrapping the optimization and the old code in an if block and executing the optimized and unoptimized paths repeatedly under the same runtime conditions to determine which is faster. Basic usage is to wrap changes to the code in if blocks like so: ABBench a: [ ...existing code...] b: [...optimized code...] And then run the A/B test by printing: ABBench bench: [...some main method...] I have found it to make performance testing simple and quick enough that I actually do it. Perhaps someone else will find it useful. Install with: Metacello new githubUser: 'emdonahue' project: 'ABBench' commitish: 'master' path: ''; baseline: 'ABBench'; load. More details can be found here: https://github.com/emdonahue/ABBench Cheers, Evan

Cheers That looks really interesting - I saw that you use an underlying statistical test for the nul hypothesis. That is a cool way to approach it. Best, Kasper > On 12 Aug 2021, at 19.44, Evan Donahue <emdonahu@gmail.com> wrote: > > Hello, > > I've just released a utility I've been using for the past few years to help optimize Pharo code. It's a library that makes it easy to check whether an optimization has improved performance by wrapping the optimization and the old code in an if block and executing the optimized and unoptimized paths repeatedly under the same runtime conditions to determine which is faster. > > Basic usage is to wrap changes to the code in if blocks like so: > > ABBench a: [ ...existing code...] b: [...optimized code...] > > And then run the A/B test by printing: > > ABBench bench: [...some main method...] > > I have found it to make performance testing simple and quick enough that I actually do it. Perhaps someone else will find it useful. > > Install with: > > Metacello new > githubUser: 'emdonahue' project: 'ABBench' commitish: 'master' path: ''; > baseline: 'ABBench'; > load. > > More details can be found here: > > https://github.com/emdonahue/ABBench <https://github.com/emdonahue/ABBench> > > Cheers, > Evan

Thanks. Yeah, that is to determine a stopping criterion: if one path is clearly faster it needs fewer samples to determine that with confidence. Evan