[Pharo-project] akuhn/SUnit, chrisma edition :)
Cleanup of SUnit is almost done. There are 179 tests with 75% coverage. Missing are tests for the new resource setup and tests for the new history storage. Changes - uses announcements - new class factory DSL - does not print 'unexpected/expected' when empty - factored all assertions into TAssertable - copied Niall's 3-valued setup logic for test resources - new expectation matcher DSL - replaced history dictionary with proper class - keeps history as TestResult in memory - stores history as array in compiled methods - unexpected/expected linked with stored history - migrates #expectedDefects to stored history on the fly - bug fix: 'run failures/errors' updates history - bug fix: coverage is now able to cover itself - added flags for compatibility and deprecation - removed *a lot of* code duplication I tried to run *all* tests in the latest image with akuhn/SUnit and it seems to run fine. There are 5 failures and 2 errors (both with akuhn/SUnit and pharo/SUnit). The new test suite starts faster. You click and tests start running immediately! Progress: numClasses: 13 -> 19 numKLOC: 965 -> 1209 numMethods: 224 -> 291 numStatements: 3086 -> 3903 numTestCases: 51 -> 177 numTestKLOC: 591 -> 1200 numTestStatements: 2175 -> 4663 testCoverage: 61% -> 75% eventually, a SUnit with more test than production code! Give it a try and lemme know whether it runs with your tests. All your tests should still run. Gofer it disablePackageCache; squeaksource: 'akuhn'; package: 'SUnit'; package: 'SUnitGUI'; load cheers, AA
Hi Adrian,
Gofer it disablePackageCache; squeaksource: 'akuhn'; package: 'SUnit'; package: 'SUnitGUI'; load
Where is the update of Gofer that understand your expression? I tried with Gofer-AdrianLienhard.79 and Gofer-DamienCassou.88 from squeaksource.com/Pharo, but disablePackageCache: and package: are not understood. Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Where is the update of Gofer that understand your expression?
[ Gofer gofer load ] on: Error do: [ :err | err retry ]. Gofer gofer recompile. Lukas -- Lukas Renggli http://www.lukas-renggli.ch
Alexandre Bergel <alexandre@...> writes:
Gofer it disablePackageCache; squeaksource: 'akuhn'; package: 'SUnit'; package: 'SUnitGUI'; load
Where is the update of Gofer that understand your expression?
See Lukas's mail. Alternatively use Gofer new "beware, might load wrong version from local cache with latest gofer!" squeaksource: 'akuhn'; addPackage: 'SUnit'; addPackage: 'SUnitGUI'; load the #disablePackageCache is only used to work around a glitch in the latest Gofer. (It attempts to load packages from the local cache that do *not* originate from akuhn. As it seems gofer always loads the package with the highest number independent of its origin. And keith's SUnit does have a higher version number than mine. You see the mess.) --AA
On Sun, Dec 27, 2009 at 4:03 AM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Cleanup of SUnit is almost done. There are 179 tests with 75% coverage. Missing are tests for the new resource setup and tests for the new history storage.
Changes
- uses announcements - new class factory DSL - does not print 'unexpected/expected' when empty - factored all assertions into TAssertable - copied Niall's 3-valued setup logic for test resources - new expectation matcher DSL - replaced history dictionary with proper class - keeps history as TestResult in memory - stores history as array in compiled methods - unexpected/expected linked with stored history - migrates #expectedDefects to stored history on the fly - bug fix: 'run failures/errors' updates history - bug fix: coverage is now able to cover itself - added flags for compatibility and deprecation - removed *a lot of* code duplication
I tried to run *all* tests in the latest image with akuhn/SUnit and it seems to run fine. There are 5 failures and 2 errors (both with akuhn/SUnit and pharo/SUnit). The new test suite starts faster. You click and tests start running immediately!
Progress:
numClasses: 13 -> 19 numKLOC: 965 -> 1209 numMethods: 224 -> 291 numStatements: 3086 -> 3903 numTestCases: 51 -> 177 numTestKLOC: 591 -> 1200 numTestStatements: 2175 -> 4663 testCoverage: 61% -> 75%
eventually, a SUnit with more test than production code!
Give it a try and lemme know whether it runs with your tests. All your tests should still run.
This is great :) I run all my SqueakDBX and GlorpDBX tests and I have one only problem: the expectedFailures are not working any more :( I put a halt are they are not even being called. All the rest seem to work (I have even test resources in both projects) Cheers Mariano
Gofer it disablePackageCache; squeaksource: 'akuhn'; package: 'SUnit'; package: 'SUnitGUI'; load
cheers, AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Adrian: Can I still bother you with new requirements ? jajaja First of all, thank you very much for putting a context menu when you press "right button" over a failing or error test. I really hate when it was run again instead of opening a context menu. Now, I noticed that you open a menu with the options "History" "Store..." and "Show progress". I really love one item more called "browse it" and that opens a System Browser with that class, with that test method selected. What do you think? Cheers Mariano On Mon, Dec 28, 2009 at 12:25 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Sun, Dec 27, 2009 at 4:03 AM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Cleanup of SUnit is almost done. There are 179 tests with 75% coverage. Missing are tests for the new resource setup and tests for the new history storage.
Changes
- uses announcements - new class factory DSL - does not print 'unexpected/expected' when empty - factored all assertions into TAssertable - copied Niall's 3-valued setup logic for test resources - new expectation matcher DSL - replaced history dictionary with proper class - keeps history as TestResult in memory - stores history as array in compiled methods - unexpected/expected linked with stored history - migrates #expectedDefects to stored history on the fly - bug fix: 'run failures/errors' updates history - bug fix: coverage is now able to cover itself - added flags for compatibility and deprecation - removed *a lot of* code duplication
I tried to run *all* tests in the latest image with akuhn/SUnit and it seems to run fine. There are 5 failures and 2 errors (both with akuhn/SUnit and pharo/SUnit). The new test suite starts faster. You click and tests start running immediately!
Progress:
numClasses: 13 -> 19 numKLOC: 965 -> 1209 numMethods: 224 -> 291 numStatements: 3086 -> 3903 numTestCases: 51 -> 177 numTestKLOC: 591 -> 1200 numTestStatements: 2175 -> 4663 testCoverage: 61% -> 75%
eventually, a SUnit with more test than production code!
Give it a try and lemme know whether it runs with your tests. All your tests should still run.
This is great :)
I run all my SqueakDBX and GlorpDBX tests and I have one only problem: the expectedFailures are not working any more :(
I put a halt are they are not even being called.
All the rest seem to work (I have even test resources in both projects)
Cheers
Mariano
Gofer it disablePackageCache; squeaksource: 'akuhn'; package: 'SUnit'; package: 'SUnitGUI'; load
cheers, AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Mariano Martinez Peck <marianopeck@...> writes:
Adrian: Can I still bother you with new requirements ? jajaja
Your welcome :)
First of all, thank you very much for putting a context menu when you press "right button" over a failing or error test. I really hate when it was run again instead of opening a context menu. Now, I noticed that you open a menu with the options "History" "Store..." and "Show progress".
The history menu should *only* show up when you right-click in the colored status bar, but not on failure and error list. Can you conform that?
I really love one item more called "browse it" and that opens a System Browser with that class, with that test method selected. What do you think?
Awesome idea, I'll add that. PS, I was also thinking about a 'step through test' option that open a debugger at the start of the test, rather than at the failing assertions. Will add that to the same menu... --AA
Name: SUnitGUI-AdrianKuhn.58 Author: AdrianKuhn Time: 28 December 2009, 4:21:29 am UUID: 88a3fdcc-e794-4666-8f50-80be6c8afcf4 Ancestors: SUnitGUI-AdrianKuhn.57 Context menu for failures and errors. When you *right* click on a failure or error then a menu should show up. The menu offers to browse or debug the defect (as requested by Mariano Martinez Peck on the mailing list). As before, when you *left* click on a failure or error then it open ins the debugger.
Adrian I would love to see a videos of the new features you did :) Stef (back writing and closing mail) On Dec 28, 2009, at 1:57 AM, Adrian Kuhn wrote:
Mariano Martinez Peck <marianopeck@...> writes:
Adrian: Can I still bother you with new requirements ? jajaja
Your welcome :)
First of all, thank you very much for putting a context menu when you press "right button" over a failing or error test. I really hate when it was run again instead of opening a context menu. Now, I noticed that you open a menu with the options "History" "Store..." and "Show progress".
The history menu should *only* show up when you right-click in the colored status bar, but not on failure and error list. Can you conform that?
I really love one item more called "browse it" and that opens a System Browser with that class, with that test method selected. What do you think?
Awesome idea, I'll add that.
PS, I was also thinking about a 'step through test' option that open a debugger at the start of the test, rather than at the failing assertions. Will add that to the same menu...
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Stéphane Ducasse <stephane.ducasse@...> writes:
Adrian I would love to see a videos of the new features you did :)
A blog post is in work... --AA -- 2nd Workshop on Source Code Search etc. Submit papers by January 19, 2010. http://bit.ly/suite2010
On Mon, Dec 28, 2009 at 1:57 AM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Mariano Martinez Peck <marianopeck@...> writes:
Adrian: Can I still bother you with new requirements ? jajaja
Your welcome :)
First of all, thank you very much for putting a context menu when you press "right button" over a failing or error test. I really hate when it was run again instead of opening a context menu. Now, I noticed that you open a menu with the options "History" "Store..." and "Show progress".
The history menu should *only* show up when you right-click in the colored status bar, but not on failure and error list. Can you conform that?
I think that yes :) I attach an screenshot. Tell me is that's fine. In that screenshot you can see also the error tests that actually are expectedFailures and with the previous version of Sunit, the were green.
I really love one item more called "browse it" and that opens a System Browser with that class, with that test method selected. What do you think?
Awesome idea, I'll add that.
Excellent :)
PS, I was also thinking about a 'step through test' option that open a debugger at the start of the test, rather than at the failing assertions. Will add that to the same menu...
Yes, that's cool also. Many times I have to go to the assertion and then a restart in the debugger. Thank you very much. Mariano --AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Mon, Dec 28, 2009 at 12:33 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Mon, Dec 28, 2009 at 1:57 AM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Mariano Martinez Peck <marianopeck@...> writes:
Adrian: Can I still bother you with new requirements ? jajaja
Your welcome :)
First of all, thank you very much for putting a context menu when you press "right button" over a failing or error test. I really hate when it was run again instead of opening a context menu. Now, I noticed that you open a menu with the options "History" "Store..." and "Show progress".
The history menu should *only* show up when you right-click in the colored status bar, but not on failure and error list. Can you conform that?
I think that yes :) I attach an screenshot. Tell me is that's fine. In that screenshot you can see also the error tests that actually are expectedFailures and with the previous version of Sunit, the were green.
I now tested with your latests commits and it seems to work ok. Only the items "Browse" and "Debug" are shown over a failure/error test. And the other three ("History" "Store..." and "Show progress".) are only in the status bar :)
I really love one item more called "browse it" and that opens a System Browser with that class, with that test method selected. What do you think?
Awesome idea, I'll add that.
Excellent :)
PS, I was also thinking about a 'step through test' option that open a debugger at the start of the test, rather than at the failing assertions. Will add that to the same menu...
Yes, that's cool also. Many times I have to go to the assertion and then a restart in the debugger.
Thank you very much.
Mariano
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
MMariano Martinez Peck <marianopeck@...> writes:
This is great :) I run all my SqueakDBX and GlorpDBX tests
Thanks for using it!
and I have one only problem: the expectedFailures are not working any more :( I put a halt are they are not even being called. All the rest seem to work (I have even test resources in both projects)
Cannot reproduce that. Can you check which version of SUnit is loaded in your image? --AA
On Mon, Dec 28, 2009 at 1:00 AM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
MMariano Martinez Peck <marianopeck@...> writes:
This is great :) I run all my SqueakDBX and GlorpDBX tests
Thanks for using it!
thanks you for the develop :)
and I have one only problem: the expectedFailures are not working any more :( I put a halt are they are not even being called. All the rest seem to work (I have even test resources in both projects)
Cannot reproduce that. Can you check which version of SUnit is loaded in your image?
SUnit-AdrianKuhn.114 SUnitGUI-AdrianKuhn.57 As an example, I implement a instance side method of a test that doesn't extend directly from TestCase but from 2 o 3 in the middle that finally extends from TestCase: expectedFailures ^ (SqueakDatabaseAccessor DefaultDriver = SqueakDBXDriver) ifTrue: [ #(testBlob testBlobWithSubstring testLargeBlob)] ifFalse: [ #()]. HTH Mariano
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Mariano Martinez Peck <marianopeck@...> writes:
As an example, I implement a instance side method of a test that doesn't extend directly from TestCase but from 2 o 3 in the middle that finally extends from TestCase:
expectedFailures ^ (SqueakDatabaseAccessor DefaultDriver = SqueakDBXDriver) ifTrue: [ #(testBlob testBlobWithSubstring testLargeBlob)] ifFalse: [ #()].
Oh, dynamically generated expected failures! Awesome, I did not think of that use. Currently they are copied once into the stored history and then ignored upon further runs. I'll fix that. --AA
Name: SUnit-AdrianKuhn.115 Author: AdrianKuhn Time: 28 December 2009, 2:22:16 am UUID: 21af8078-0c75-4323-89aa-5f0142fa09d4 Ancestors: SUnit-AdrianKuhn.114 Takes #expectedFailure into account. As required by Mariano Martinez Peck on the mailing list. He uses the #expectedFailure method to dynamically create expected failures, rather than a fixed set of failures as is stored in stored history.
Hi! Apparently, expectedFailure has a different behavior from the original SUnit. I have a test method declared in a expectedFailures method. However, this test method does not fail, but it triggers a MNU. Running the unit test tells me there is 1 error, the method that I declared as expectedFailure. I guess this behavior is the right one. How can I fix this? There is no sender or implementer of expectedErrors. By the way, why TestCase>>expectedFailures is in the protocol 'history management'? There is also two empty method categories in TestCase. Cheers, Alexandre On 28 Dec 2009, at 02:27, Adrian Kuhn wrote:
Name: SUnit-AdrianKuhn.115 Author: AdrianKuhn Time: 28 December 2009, 2:22:16 am UUID: 21af8078-0c75-4323-89aa-5f0142fa09d4 Ancestors: SUnit-AdrianKuhn.114
Takes #expectedFailure into account.
As required by Mariano Martinez Peck on the mailing list. He uses the #expectedFailure method to dynamically create expected failures, rather than a fixed set of failures as is stored in stored history.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Alexandre Bergel <alexandre@...> writes:
Apparently, expectedFailure has a different behavior from the original SUnit.
Yes, now as you mention errors. In the original it has the (undocumented) behavior of declaring expected failures *and* errors. I'll see how I can fix that.
I have a test method declared in a expectedFailures method. However, this test method does not fail, but it triggers a MNU. Running the unit test tells me there is 1 error, the method that I declared as expectedFailure. I guess this behavior is the right one.
How can I fix this? There is no sender or implementer of expectedErrors.
The preferred way is to store your results into the stored history. You can store the history with the context menu in the test runner. To do so, right click in the colored status field and choose `store history`. The stored results are treated as the expected results then.
By the way, why TestCase>>expectedFailures is in the protocol 'history management'? There is also two empty method categories in TestCase.
The empty protocols are left overs of the old implementations. I dont know how to tell gofer/monticello to delete them. Do you know how? --AA
Apparently, expectedFailure has a different behavior from the original SUnit.
Yes, now as you mention errors. In the original it has the (undocumented) behavior of declaring expected failures *and* errors. I'll see how I can fix that.
Maybe introducing a method expectedErrors would do the thing.
I have a test method declared in a expectedFailures method. However, this test method does not fail, but it triggers a MNU. Running the unit test tells me there is 1 error, the method that I declared as expectedFailure. I guess this behavior is the right one.
How can I fix this? There is no sender or implementer of expectedErrors.
The preferred way is to store your results into the stored history. You can store the history with the context menu in the test runner. To do so, right click in the colored status field and choose `store history`. The stored results are treated as the expected results then.
Yes, but... how is this related to the expectedError?
By the way, why TestCase>>expectedFailures is in the protocol 'history management'? There is also two empty method categories in TestCase.
The empty protocols are left overs of the old implementations. I dont know how to tell gofer/monticello to delete them. Do you know how?
No idea. But this looks like to be a bug in Monticello no? Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Alexandre Bergel <alexandre@...> writes:
The preferred way is to store your results into the stored history. You can store the history with the context menu in the test runner. To do so, right click in the colored status field and choose `store history`. The stored results are treated as the expected results then.
Yes, but... how is this related to the expectedError?
The stored history is compiled into a method on the class side. So you get the exact benefit of #expectedFailures/#expectedError. It is persistent across source control and will mark your failures/errors as expected. So you dont need #expectedError at all. The history system has two levels - "history" is put in a class variable on the class side. This information is used for the browser icons by OB and O2, such that you can see the result of the latest test run. This information is local to your image. - "stored history" is put into a compiled method on the class side. This information is used by the test runner to mark unexpected passes and expected defects (failures, errors *and* any custom defects). This information is global and persists together with your source code. Programatically they are accessible with `testClass history` and `testClass storedHistory`, and `testClass storeHistoryIntoCompiledMethod` will put the current history into stored history. Marking of unexpected results can be done between *any* two test results with `testResult markUnexpectedResults: anotherTestResult`, so other tools builders are free to compare any two results. Also you can define your very own test results (eg `aTestFinished outcome: #ignored`) and they will be processed by result marking. --AA
The preferred way is to store your results into the stored history. You can store the history with the context menu in the test runner. To do so, right click in the colored status field and choose `store history`. The stored results are treated as the expected results then.
Yes, but... how is this related to the expectedError?
The stored history is compiled into a method on the class side. So you get the exact benefit of #expectedFailures/#expectedError. It is persistent across source control and will mark your failures/errors as expected. So you dont need #expectedError at all.
Yep, I know this, I pair programmed the history mechanism with Simon :-) But I still do not understand how do you distinguish a method that always fails from an expectedFailure (i.e., a method that I know will fail, but I do not want it to make my unit test red)? If we need expectedFailure, we will probably need expectedError no? Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
This is really cool that you work on SUnit. This is really needed. Cheers, Alexandre On 28 Dec 2009, at 18:47, Alexandre Bergel wrote:
The preferred way is to store your results into the stored history. You can store the history with the context menu in the test runner. To do so, right click in the colored status field and choose `store history`. The stored results are treated as the expected results then.
Yes, but... how is this related to the expectedError?
The stored history is compiled into a method on the class side. So you get the exact benefit of #expectedFailures/#expectedError. It is persistent across source control and will mark your failures/errors as expected. So you dont need #expectedError at all.
Yep, I know this, I pair programmed the history mechanism with Simon :-) But I still do not understand how do you distinguish a method that always fails from an expectedFailure (i.e., a method that I know will fail, but I do not want it to make my unit test red)? If we need expectedFailure, we will probably need expectedError no?
Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
-- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
Alexandre Bergel <alexandre@...> writes:
The stored history is compiled into a method on the class side. So you get the exact benefit of #expectedFailures/#expectedError. It is persistent across source control and will mark your failures/errors as expected. So you dont need #expectedError at all.
Yep, I know this, I pair programmed the history mechanism with Simon But I still do not understand how do you distinguish a method that always fails from an expectedFailure (i.e., a method that I know will fail, but I do not want it to make my unit test red)? If we need expectedFailure, we will probably need expectedError no?
This is an aspect that I changed. An outcome is unexpected if and only if it does not match what is in the stored history. (For compatibility only #expectedFailures are added to the stored history.) So the preferred way to mark an outcome as unexpected is to store the history. Try it - remove #expectedFailures - run all your tests (you'll get failures and errors, -> not green) - store the history - run all your tests again (all your defects are expected, -> green) And just for demonstration - introduce a new defect - run all your tests again (one unexpected defect, -> not green) Does this match your workflow? --AA
On Mon, Dec 28, 2009 at 7:01 PM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Alexandre Bergel <alexandre@...> writes:
The stored history is compiled into a method on the class side. So you get the exact benefit of #expectedFailures/#expectedError. It is persistent across source control and will mark your failures/errors as expected. So you dont need #expectedError at all.
Yep, I know this, I pair programmed the history mechanism with Simon But I still do not understand how do you distinguish a method that always fails from an expectedFailure (i.e., a method that I know will fail, but I do not want it to make my unit test red)? If we need expectedFailure, we will probably need expectedError no?
This is an aspect that I changed. An outcome is unexpected if and only if it does not match what is in the stored history. (For compatibility only #expectedFailures are added to the stored history.) So the preferred way to mark an outcome as unexpected is to store the history.
Try it
- remove #expectedFailures - run all your tests (you'll get failures and errors, -> not green) - store the history - run all your tests again (all your defects are expected, -> green)
And just for demonstration
- introduce a new defect - run all your tests again (one unexpected defect, -> not green)
Does this match your workflow?
The problem is if you have several others failing/error tests but that they are not expected :( Ok...I can run only the expected ones and store them, or don't run the error/failures ones and run them later
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
This is an aspect that I changed. An outcome is unexpected if and only if it does not match what is in the stored history. (For compatibility only #expectedFailures are added to the stored history.) So the preferred way to mark an outcome as unexpected is to store the history.
Ok, I understand now. Could be okay. But how can I distinguish an expected failure from a real failure ? At the first run, I get some failed tests. Are they real or expected failures? Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
On Mon, Dec 28, 2009 at 2:27 AM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Name: SUnit-AdrianKuhn.115 Author: AdrianKuhn Time: 28 December 2009, 2:22:16 am UUID: 21af8078-0c75-4323-89aa-5f0142fa09d4 Ancestors: SUnit-AdrianKuhn.114
Takes #expectedFailure into account.
As required by Mariano Martinez Peck on the mailing list. He uses the #expectedFailure method to dynamically create expected failures, rather than a fixed set of failures as is stored in stored history.
Hi Adrian! Thanks for trying to fix this. However, I can still reproduce the problem. I put a halt in my expectedFailures method and it is being called, here: importExpectedFailures: aTestClass aTestClass new expectedFailures do: [ :each | self testFinished: (TestFinished new testClass: aTestClass; selector: each; outcome: TestFinished failures; yourself) ] The iteration throught the do: seems to be ok, but I don't know why, the tests are still considered as error (red) instead of expectedFailures in green. Is correct to put "outcome: TestFinished failures;" ??? I guess it would be something like "outcome: TestFInished expectedFailures" but I look in the class side methods of TestFInished and I only see passed, failure and error. Let me know if I can test something more. The versions I am using are: SUnit-AdrianKuhn.116 SUnitGUI-AdrianKuhn.58 Cheers Mariano
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Mariano Martinez Peck <marianopeck@...> writes:
Hi Adrian! Thanks for trying to fix this. However, I can still reproduce the problem. I put a halt in my expectedFailures method and it is being called [..] The iteration throught the do: seems to be ok, but I don't know why, the tests are still considered as error (red) instead of expectedFailures in green. Is correct to put "outcome: TestFinished failures;" ??? I guess it would be something like "outcome: TestFinished expectedFailures" but I look in the class side methods of TestFinished and I only see passed, failure and error. Let me know if I can test something more.
I see, you got the same case as Alex. You declared an expected *error* in #expectedFailures. That used to work with the old SUnit but the new SUnit takes you literally and expects it to be a failure and a failure only. I will work on a solution for that. --AA
On Mon, Dec 28, 2009 at 6:42 PM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Mariano Martinez Peck <marianopeck@...> writes:
Hi Adrian! Thanks for trying to fix this. However, I can still reproduce the problem. I put a halt in my expectedFailures method and it is being called [..] The iteration throught the do: seems to be ok, but I don't know why, the tests are still considered as error (red) instead of expectedFailures in green. Is correct to put "outcome: TestFinished failures;" ??? I guess it would be something like "outcome: TestFinished expectedFailures" but I look in the class side methods of TestFinished and I only see passed, failure and error. Let me know if I can test something more.
I see, you got the same case as Alex.
You declared an expected *error* in #expectedFailures. That used to work with the old SUnit but the new SUnit takes you literally and expects it to be a failure and a failure only. I will work on a solution for that.
Ahhh yes, sorry. I just discovered that. Wait...I will suggest this: It is true what you said about expectedFailure is expected failure and not expected error. I just did it like that because I thought it was like that. But now that I see it, that is probably wrong. So, what I would do is to "fix" it but (I don't know) put is as deprecated so that all the tests of the people are still green and give them time to change it to expectedErrors or similar. After a second release you can remove this fix :) What do you think ? Cheers Mariano
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Mariano Martinez Peck <marianopeck@...> writes:
So, what I would do is to "fix" it but (I don't know) put is as deprecated so that all the tests of the people are still green and give them time to change it to expectedErrors or similar. After a second release you can remove this fix :) What do you think?
Yes, that is how I planned it ... and then you came with your dynamically generated expectations. Which the new system does not support. It's an awesome idea, so I am thinking about a way to support dynamic expectations in the new system. I have one solution in mind: given your #expectedFailures, would the following work for you? testBlob self assume: SqueakDatabaseAccessor DefaultDriver ~~ SqueakDBXDriver. ... which runs the test if and only if the default driver is not SqueakDBXDriver. This would have the same effect as your #expectedFailures implementation, where you expected an error for that case. Like this your expectation is put closer to the code. --AA
On Mon, Dec 28, 2009 at 7:29 PM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Mariano Martinez Peck <marianopeck@...> writes:
So, what I would do is to "fix" it but (I don't know) put is as deprecated so that all the tests of the people are still green and give them time to change it to expectedErrors or similar. After a second release you can remove this fix :) What do you think?
Yes, that is how I planned it ... and then you came with your dynamically generated expectations. Which the new system does not support.
It's an awesome idea, so I am thinking about a way to support dynamic expectations in the new system. I have one solution in mind: given your #expectedFailures, would the following work for you?
testBlob self assume: SqueakDatabaseAccessor DefaultDriver ~~ SqueakDBXDriver. ...
which runs the test if and only if the default driver is not SqueakDBXDriver. This would have the same effect as your #expectedFailures implementation, where you expected an error for that case. Like this your expectation is put closer to the code.
Yes, this will work and will give me a benefit and a drawback :) Benefit: I can declare a expected failure only for a single test and not for all the tests of the class. Right now, I have situations like this and I a doing exactly that (more ugly actually), an if to see something and then a ^ self is it shouldn't be run. Drawback: Most of the times (in my case) the condition (SqueakDatabaseAccessor DefaultDriver ~~ SqueakDBXDrive) is always the same, so that's why it was easier as I did it. Here I should declare it in every test. However, this is VERYYYYYY little detail that I really don't care (I am not eve sure if this solution is better). I am bothering too much hahaha. Of course, I don't know what other people think about this but in my point of view, this is ok :) Cheers Mariano
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Mariano Martinez Peck <marianopeck@...> writes: Assumptions are in the box! If you expect your test case to fail unless a given condition is met, you can now use assumptions to declare that. Gofer the latest akuhn/SUnit to get assumptions. For example, when your test case depends on the availability of a certain database you can write testBlob self assumeThat: Database default = DatabaseWithBlobs. ... if the assumption is true, the test case is run as a normal test case. If however the assumption is false, the test case is *not* run but still marked as passed. This is supposed to replace conditional #expectedFailures in the new SUnit. Typically, assumptions are put at the begin of a test case. However, you can put assumptions anywhere. The running tests case is aborted as soon as the first failing assumption is reached. PS regarding the naming: the message #assumeThat: has been preferred over a simple #assume: to make confusions with a simple #assert: less likely. --AA
On Mon, Dec 28, 2009 at 11:19 PM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Mariano Martinez Peck <marianopeck@...> writes:
Assumptions are in the box!
If you expect your test case to fail unless a given condition is met, you can now use assumptions to declare that. Gofer the latest akuhn/SUnit to get assumptions.
For example, when your test case depends on the availability of a certain database you can write
testBlob self assumeThat: Database default = DatabaseWithBlobs. ...
if the assumption is true, the test case is run as a normal test case. If however the assumption is false, the test case is *not* run but still marked as passed. This is supposed to replace conditional #expectedFailures in the new SUnit.
Typically, assumptions are put at the begin of a test case. However, you can put assumptions anywhere. The running tests case is aborted as soon as the first failing assumption is reached.
PS regarding the naming: the message #assumeThat: has been preferred over a simple #assume: to make confusions with a simple #assert: less likely.
Thanks Adrian. All your improvments are really cool. The only thing that I am concerned is that if I change all my tests to this style, Squeak people won't be able to run the tests :( because that method won't be understood. Anyway, I will si what can I do. Thank you very much for doing every day something better from SUnit!!! --AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Tue, Dec 29, 2009 at 12:17 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Mon, Dec 28, 2009 at 11:19 PM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
Mariano Martinez Peck <marianopeck@...> writes:
Assumptions are in the box!
If you expect your test case to fail unless a given condition is met, you can now use assumptions to declare that. Gofer the latest akuhn/SUnit to get assumptions.
For example, when your test case depends on the availability of a certain database you can write
testBlob self assumeThat: Database default = DatabaseWithBlobs. ...
if the assumption is true, the test case is run as a normal test case. If however the assumption is false, the test case is *not* run but still marked as passed. This is supposed to replace conditional #expectedFailures in the new SUnit.
Typically, assumptions are put at the begin of a test case. However, you can put assumptions anywhere. The running tests case is aborted as soon as the first failing assumption is reached.
PS regarding the naming: the message #assumeThat: has been preferred over a simple #assume: to make confusions with a simple #assert: less likely.
Thanks Adrian. All your improvments are really cool. The only thing that I am concerned is that if I change all my tests to this style, Squeak people won't be able to run the tests :( because that method won't be understood. Anyway, I will si what can I do.
Sorry I forgot to say that I just realized about that (because you asked me if it was ok and I said that yes).
Thank you very much for doing every day something better from SUnit!!!
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Mariano Martinez Peck <marianopeck@...> writes:
Thanks Adrian. All your improvments are really cool. The only thing that I am concerned is that if I change all my tests to this style, Squeak people won't be able to run the tests :( because that method won't be understood.
True enough, that applies to any new feature. We cannot have evolution and full compatibility at the same time. I will try to ensure that old tests keep running. But regarding new features there is no guarantees (unless someone backports my changes). --AA
That's irght. You can't break eggs without baking a cake! Dale ----- "Adrian Kuhn" <akuhn@iam.unibe.ch> wrote: | Mariano Martinez Peck <marianopeck@...> writes: | | > Thanks Adrian. All your improvments are really cool. The only thing | that I am | > concerned is that if I change all my tests to this style, Squeak | people won't | > be able to run the tests :( because that method won't be | understood. | | True enough, that applies to any new feature. We cannot have evolution | and | full compatibility at the same time. I will try to ensure that old | tests keep | running. But regarding new features there is no guarantees (unless | someone | backports my changes). | | --AA | | | | _______________________________________________ | Pharo-project mailing list | Pharo-project@lists.gforge.inria.fr | http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Mon, Dec 28, 2009 at 2:19 PM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
For example, when your test case depends on the availability of a certain  database you can write
  testBlob     self assumeThat: Database default = DatabaseWithBlobs.     ...
Can I suggest simply #assume: ? The colon kind of has the same meaning as "that" so it ends up redundant (would be the same with existing methods: #assert(That): #deny(That):). Then, having said that, #assume: sounds a bit like the test should assume the statement is true and then carry on running with that assumption. Maybe #expect: would be more descriptive? Or #precondition: ? Not sure... Julian
yes expect: seems better. Stef On Dec 29, 2009, at 7:21 AM, Julian Fitzell wrote:
On Mon, Dec 28, 2009 at 2:19 PM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
For example, when your test case depends on the availability of a certain database you can write
testBlob self assumeThat: Database default = DatabaseWithBlobs. ...
Can I suggest simply #assume: ? The colon kind of has the same meaning as "that" so it ends up redundant (would be the same with existing methods: #assert(That): #deny(That):).
Then, having said that, #assume: sounds a bit like the test should assume the statement is true and then carry on running with that assumption. Maybe #expect: would be more descriptive? Or #precondition: ? Not sure...
Julian
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Dec 29, 2009, at 7:21 AM, Julian Fitzell wrote:
Then, having said that, #assume: sounds a bit like the test should assume the statement is true and then carry on running with that assumption. Maybe #expect: would be more descriptive? Or #precondition: ? Not sure...
Stéphane Ducasse <stephane.ducasse@...> writes:
yes expect: seems better.
Hmm, that sound to me as if the test should fail when the expectations is not met, whereas in fact it is skipped unless the condition is true. What about something like #skipUnless: ? --AA
yes much much better to me. On Dec 29, 2009, at 11:13 AM, Adrian Kuhn wrote:
On Dec 29, 2009, at 7:21 AM, Julian Fitzell wrote:
Then, having said that, #assume: sounds a bit like the test should assume the statement is true and then carry on running with that assumption. Maybe #expect: would be more descriptive? Or #precondition: ? Not sure...
Stéphane Ducasse <stephane.ducasse@...> writes:
yes expect: seems better.
Hmm, that sound to me as if the test should fail when the expectations is not met, whereas in fact it is skipped unless the condition is true.
What about something like #skipUnless: ?
--AA
_______________________________________________ Pharo-project mailing list Pharo-project@lists.gforge.inria.fr http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
On Tue, Dec 29, 2009 at 2:13 AM, Adrian Kuhn <akuhn@iam.unibe.ch> wrote:
On Dec 29, 2009, at 7:21 AM, Julian Fitzell wrote:
Then, having said that, #assume: sounds a bit like the test should assume the statement is true and then carry on running with that assumption. Maybe #expect: would be more descriptive? Or #precondition: ? Not sure...
Stéphane Ducasse <stephane.ducasse@...> writes:
yes expect: seems better.
Hmm, that sound to me as if the test should fail when the expectations is  not met, whereas in fact it is skipped unless the condition is true.
What about something like #skipUnless: ?
Sure... I think #expect: could be interpreted either way; #skipUnless: pretty unambiguous. Julian
Name: SUnit-AdrianKuhn.122 Author: AdrianKuhn Time: 29 December 2009, 5:14:01 pm UUID: cc99ab5b-294e-4142-a5da-baf8af550a0f Ancestors: SUnit-AdrianKuhn.120 Renamed assumptions to #skipUnless:. As suggested on mailing list renamed #assumeThat: to #skipUnless. A skipped test is marked as 'skipped' which counts as passed and is thus a green result. Also, added unconditional #skip message.
participants (7)
-
Adrian Kuhn -
Alexandre Bergel -
Dale Henrichs -
Julian Fitzell -
Lukas Renggli -
Mariano Martinez Peck -
Stéphane Ducasse