On Wed, Oct 1, 2008 at 4:56 PM, Alexandre Bergel <Alexandre.Bergel@inria.fr> wrote:

In PharoInbox Gwenael and I put an extension of SUnit that records results when running a set of test. Feel free to suggest improvement....

Open a TestRunner to enjoy its 'History' button.

Please have a look at the SUnit package at http://www.squeaksource.com/Testing/ -- Damien Cassou Peter von der Ahé: «I'm beginning to see why Gilad wished us good luck». (http://blogs.sun.com/ahe/entry/override_snafu)

On Wed, Oct 1, 2008 at 5:13 PM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:

can you tell us more?

http://wiki.squeak.org/squeak/SUnit%20for%203.10 -- Damien Cassou Peter von der Ahé: «I'm beginning to see why Gilad wished us good luck». (http://blogs.sun.com/ahe/entry/override_snafu)

Stéphane Ducasse wrote:

Thanks I will have a look. I do not really understand what is written, this is too cryptic. Does the framework changes remove the fact that tests should start with test?

Stef Yes, although the default behaviour is preserved, TestCase subclasses can publish their own test suites with whatever behaviour they wish. BTW: the framework has changed somewhat since that documentation was written.

For example in SSpec "specs" start with spec* the goal is to enable TestRunner to be used as a runner for SSpec as well as SUnit, if I ever get around to it. Keith

On 2 Oct 2008, at 23:24, Keith Hodges wrote:

Stéphane Ducasse wrote:

...

Thanks I will have a look. I do not really understand what is written, this is too cryptic. Does the framework changes remove the fact that tests should start with test?

Stef Yes, although the default behaviour is preserved, TestCase subclasses can publish their own test suites with whatever behaviour they wish. BTW: the framework has changed somewhat since that documentation was written.

For example in SSpec "specs" start with spec*

the goal is to enable TestRunner to be used as a runner for SSpec as well as SUnit, if I ever get around to it.

In the latest version of the SSpec runner that we use (which is also the latest in Squeaksource iirc, as John here made the most recent changes) specs can be whatever format you want, but the spec method(s) must be in the method category "specs" in order to run. Generally specs (ie BDD) are of the format "shouldDoBlahBlah" so that they read in a BDD manner when they report problems. In practice this means It parses the camel case in the output message, so for example: if the spec class is AddTwoNumbersSpec then you'd get something like "Add two numbers spec should get negative number for negative arguments". This sentence is English, and is relatively easy to parse and understand what the desired behaviour is. So: using "should" as a prefix makes sense to me, and I like the use of the method categorisation to define what runs as a spec: it makes it easy to turn some specs on and off within a spec class without having to do method renames, which is generally more hassle (changing a method category is a drag and drop operation). My €0.02 as usual. S ****************************************************************************************************************************************** This email is from Pinesoft Limited. Its contents are confidential to the intended recipient(s) at the email address(es) to which it has been addressed. It may not be disclosed to or used by anyone other than the addressee(s), nor may it be copied in anyway. If received in error, please contact the sender, then delete it from your system. Although this email and attachments are believed to be free of virus, or any other defect which might affect any computer or IT system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Pinesoft for any loss or damage arising in any way from receipt or use thereof. ******************************************************************************************************************************************* Pinesoft Limited are registered in England, Registered number: 2914825. Registered office: 266-268 High Street, Waltham Cross, Herts, EN8 7EA

Stef, what do you mean by "the Testing package" ? S On 3 Oct 2008, at 09:46, Stéphane Ducasse wrote:

Simon what would be good is a feedback on whether the Testing package is good, useful, ....

Stef

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

****************************************************************************************************************************************** This email is from Pinesoft Limited. Its contents are confidential to the intended recipient(s) at the email address(es) to which it has been addressed. It may not be disclosed to or used by anyone other than the addressee(s), nor may it be copied in anyway. If received in error, please contact the sender, then delete it from your system. Although this email and attachments are believed to be free of virus, or any other defect which might affect any computer or IT system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Pinesoft for any loss or damage arising in any way from receipt or use thereof. ******************************************************************************************************************************************* Pinesoft Limited are registered in England, Registered number: 2914825. Registered office: 266-268 High Street, Waltham Cross, Herts, EN8 7EA

The latest SUnit in ss/Testing allows suites and filters (e.g. #myCustomSuite , #todoFilter ) to be assembled adding and subtracting TestSuite's (collections or tests) obtained using the following methods. #suiteWithFlagged: "looks for flag: #literal or Pragma: <suite: #literal>" #suiteWithMethodCategoryMatching: pattern "for sspec it would be specs*" #suiteWithSelectorsMatching: pattern "the default is test*" e.g. MyTestCaseClass-myCustomSuite ^ (self suiteWithSelectorsMatching: 'test*') , ( self suiteWithFlagged: #test ) The custom suite is published on the class side of your TestCase class in #publishedSuites ^ #( #allStandardTests #myCustomSuite ) , likewise for #publishedFilters. The TestRunner allows you to chose a suite, only the classes included in the suite are displayed. This finally allows you to define a project to work within, and have the UI ignore the other tests. The TestRunner also allows you to either reject or select items in the suite that match the filter. So you could run all of the todoItems from mySuite, or all of the non-todo items from mySuite. TestCaseVersioned makes extensive use of this. Tests in subclasses can flag specifically which versions of squeak the test is expected to work in, and on which platform. This allow you to have some means of flagging pharo specific tests, i.e. tests that you would only expect to work in pharo, and tests that you would only expect to work in squeak. Thus we can still keep an "all tests green" policy, even though we may have different tests for different image-kernels. As we move to images that may have modules removed, we can add a mechanism for specifying "tests that only apply if a particular package is loaded." hope this helps to explain things a bit Keith

Stéphane Ducasse wrote:
> Thanks keith
> After I really wonder if we need all that.
> I would be really interested in understanding what are the 20% nost
> useful features
> and focus on them.
>
> Stef
1. The old suite building code was a mess and had lots of duplication
for very little functionality. Here we get much more for less code.

For the rest, I suggest looking at the feature set backwards, start with
the basic uber-use case* and see what features are needed to support it.

* I implemented much of this for my own project in st/x 10 years ago,
and have missed it ever since.

2. Use Case: Be able to run...

"All tests or specifications for my project which are expected to run on
this platform, this image version, and this vm... with the ability to
mark tests as 'expected failures', 'to do' , 'slow', 'long', 'uses
network' etc, and then be able to run a suite with or without the marked
tests.

btw: The next refactoring is to remove the whole notion of
expectedFailures from TestResult and further simplify things. I asked on
squeak-dev over a year ago to ascertain whether there are any users of
this confusing feature, and I don't think there are any.

Keith

Hi Alex, I tried your extension and have the following questions/problems: - when I first run all the tests in the system, I get 4 failures (as expected) but the bar on top stays green where it should be yellow (it worked before) - I don't understand the "Progress" feature. What I did is run the tests, then "Store results", then I fixed a failing test, then "Run Failures", then "Show progress". But nothing happens. Maybe I did something wrong? Thanks, Adrian On Oct 1, 2008, at 16:56 , Alexandre Bergel wrote:

Dear list,

In PharoInbox Gwenael and I put an extension of SUnit that records results when running a set of test. Feel free to suggest improvement....

Open a TestRunner to enjoy its 'History' button.

Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

the idea adrian is that we store the tests with a packgae then when we load the package we can then compare if we made progress or not. For moose this is really nice beacuse I got far too muc h red tests at the beginning. On Oct 10, 2008, at 10:10 AM, Adrian Lienhard wrote:

Hi Alex,

I tried your extension and have the following questions/problems: - when I first run all the tests in the system, I get 4 failures (as expected) but the bar on top stays green where it should be yellow (it worked before) - I don't understand the "Progress" feature. What I did is run the tests, then "Store results", then I fixed a failing test, then "Run Failures", then "Show progress". But nothing happens. Maybe I did something wrong?

Thanks, Adrian

On Oct 1, 2008, at 16:56 , Alexandre Bergel wrote:

...

Dear list,

In PharoInbox Gwenael and I put an extension of SUnit that records results when running a set of test. Feel free to suggest improvement....

Open a TestRunner to enjoy its 'History' button.

Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

There is the history button. You can save a result to compare them later. Alexandre On 10 Oct 2008, at 16:44, Damien Pollet wrote:

On Fri, Oct 10, 2008 at 1:25 PM, Alexandre Bergel <alexandre@bergel.eu> wrote:

...

...

- I don't understand the "Progress" feature. What I did is run the tests, then "Store results", then I fixed a failing test, then "Run Failures", then "Show progress". But nothing happens. Maybe I did something wrong?

Result is displayed in the Transcript. Sorry to not have said it before. We though first about opening a workspace with the result inside, but I did not find an easy way to do it. So, for now, it prints in the Transcript.

What about summarizing progress with the passes/failures/errors statistics after each run, and if one wants more details, then clicking on the statistics opens a workspace with the detailed list of fixed/broken tests ?

-- Damien Pollet type less, do more [ | ] http://people.untyped.org/damien.pollet

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

good feedback. This means that soon we will also be able to run test server and systematically and easily know if we made progress or not. Now we may face the problem that running the tests make the packages dirty. Stef

Hi Alex,

On Oct 10, 2008, at 13:25 , Alexandre Bergel wrote:

...

...

I tried your extension and have the following questions/problems: - when I first run all the tests in the system, I get 4 failures (as expected) but the bar on top stays green where it should be yellow (it worked before)

that's strange. I run all the test, and it turns yellow saying: "10 October 2008 1:07:37 pm 2286 run, 2279 passes, 4 expected failures, 3 failures, 0 errors, 0 unexpected passes"

ok, I downloaded a fresh image and updated to 10091 and now it also works for me.

...

...

- I don't understand the "Progress" feature. What I did is run the tests, then "Store results", then I fixed a failing test, then "Run Failures", then "Show progress". But nothing happens. Maybe I did something wrong?

Result is displayed in the Transcript. Sorry to not have said it before.

oh, I see. Yes, this seems to be a really useful feature! What is the difference between storing the result via the "History" button vs. "Store result into classes". Then also the results of "show diff" and "Show Progress" are almost the same. I guess this is to make it possible to store the results with the code. It seems to me that this could be simplified...

...

We though first about opening a workspace with the result inside, but I did not find an easy way to do it. So, for now, it prints in the Transcript.

If you want to open a workspace you can do it like this: Workspace new contents: 'foo'; openLabel: 'SUnit Progress'

I think the feature should be more intuitive so that a user can figure out how it works just by trying it out. Wouldn't the following 2 actions be sufficient for most cases: 'Remember Result', 'Compare Result'? You would always only remember one result and no diff selection and history deleting would be necessary. The usability would increase because you would go down from 8 possible user actions to 2.

Cheers, Adrian

...

Cheers, Alexandre

...

On Oct 1, 2008, at 16:56 , Alexandre Bergel wrote:

...

Dear list,

In PharoInbox Gwenael and I put an extension of SUnit that records results when running a set of test. Feel free to suggest improvement....

Open a TestRunner to enjoy its 'History' button.

Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Alex, On Oct 12, 2008, at 11:59 , Alexandre Bergel wrote:

...

I think the feature should be more intuitive so that a user can figure out how it works just by trying it out. Wouldn't the following 2 actions be sufficient for most cases: 'Remember Result', 'Compare Result'? You would always only remember one result and no diff selection and history deleting would be necessary. The usability would increase because you would go down from 8 possible user actions to 2.

It makes sense. I do not have problem to remove the 'history' button, and to keep (and rename) 'Remember Result', 'Compare Result'. I was just wondering whether having a local in time history and a persistent one would make sense. We could only keep the latter.

I think that would be a good tradeoff between functionality and complexity of the user interface. Could you also please check that it works with all "Run XXX" buttons? Cheers, Adrian