[Pharo-users] TestAsserter>>assertCollection:hasSameElements:

jtuchel at objektfabrik.de jtuchel at objektfabrik.de
Tue Oct 27 00:41:33 EDT 2015


I see you clearly understand what I wanted to get into and don't give up 
because of my aggressive tone in the first place.

Am 26.10.15 um 23:53 schrieb Peter Uhnák:
> On Mon, Oct 26, 2015 at 8:27 AM, jtuchel at objektfabrik.de 
> <mailto:jtuchel at objektfabrik.de> <jtuchel at objektfabrik.de 
> <mailto:jtuchel at objektfabrik.de>> wrote:
>     Am 25.10.15 um 16:33 schrieb Peter Uhnák:
>>         > assert:equals:  it's just more typing than #= with no
>>         additional outcome
>>     I also disagree, but that may be also because maybe we write
>>     tests for different purpse.
>>     If you write tests just to test your code, then whatever... I
>>     don't do that so I can't comment on that.
>>     However if you do TDD, then tests are to provide feedback, and
>>     from that perspective you want to see immediately what is the
>>     difference. If I see that the assertion failed I have to start
>>     digging to find out what is the difference, which is extra work
>>     and bad from feedback perspective. If I instead immediately see
>>     what I need to see it allows me to faster deduce the source of
>>     the problem and resolve it.
>     I'm sorry, but what you are saying doesn't make any sense. Even if
>     I "only" want to test code (which is exactly what you do in TDD,
>     btw.), I need good feedback.
> Well, that was a bit of generalization on my part. The point was, that 
> many people write tests as an afterthought... which means they have 
> different means of getting feedback from the system (e.g. logging, 
> running manually the code, etc.). In such situation they often end up 
> testing already mostly working code, and thus the feedback from the 
> tests is not as important, because you end up less digging up problems 
> (since large portion were found and resolved during manual testing).

I don't think there really is a difference between test first and tests 
as an afterthought. You always need good and precise feedback. What you 
say about having learned much more during development of a body of code 
is not necessarily true in teams or when you maintain software that has 
grown for years or decades, Remember, there are Smalltalk systems out 
there whose development started in the late 80s or early 90s. In fact, 
most commercial Smalltalk projects out in the wild are rather old.
If you have to make sure you don't break anything by putting a system 
under test, you need precise feeddback, because errors can happen in 
areas you haven't touched in the last 5 or 10 years or even never before.

There is, btw, another area in which SUnit can be extremely helpful: 
Understanding an existing body of code and proving your theories about 
code to yourself while you dive deeper. Here, feedback is also very 

So let's just assume that feedback of SUnit can never be too specific 
and is one of the very most important aspects of the framework.

>     We could think about subclassing TestFailure and a way to hand
>     information to the TestFailure so that a nice String can be
>     produced. Like a method like cull: that adds arguments'
>     printString representation into the failure description.
> You mean #assert:description: ? Because we already have that.
Well, I actually mean a new generation of #assert:description: that 
concentrates more on the #description: part. Remember: some assertion 
methods have been added as a way to provide "more suitable" feedback, 
like handing in an object or two to be mentioned in the description 
string. So the problem at hand wasn't that the standard assert: method 
was insufficient, just its output.
I clearly understand how this can happen. You want to provide a better 
log output and since you are testing something that even others might 
need, you just add that damn method that writes a nice "expecte $A but 
got $B".
In fact, this solves a problem in the wrong way, because it will result 
in many more or less well-named assertion methods that basically are 
there to provide a better failure String.

So what we have is a way to display a description without parameters. 
But we sometimes want more than that. In case of #assert:equals: we 
wanted a String that says "Expected $A and got $B". Let's not speculate 
whether this String really saves us much time if we are in a TDD 
scenario or on a build server. Let's just assume it is never bad to know 
as much as possible about a problem to do something about it.

Let's also not forget that Kent Beck had this clear idea from the very 
beginning of SUnit that it must be lightweight and easy to understand to 
not shy people away from trying. You must be able to use SUnit within an 
hour and it must be easy and provide instant feedback. If I have to type 
too much, or keep too many things in mind, I won't use it.

>     Please step back for a second and think again: these are two very
>     different things. The job of an Assertion is to make a problem
>     visible. The representation of the problem is something else, even
>     if these are closely related. This is object thinking lesson #2 or so.
> Object thinking lesson #3 tells me that I should not care about what 
> is going on behind the curtains.
Well, that is probably not really true for unit tests, because you may 
want to test something in a very specific way to prove your thesis about 
how to solve a problem is true under several circumstances.

> And while I could explicitly separate the two, if I do it all the time 
> I don't see what's bad about having a convenience method. (And by 
> looking at Object or String protocol, Pharo is a lot about having 
> convenience over engineering rigidness).
Convenience methods are not bad per se, as long as they add value and 
clearly communicate what they do. Pharo and other Smalltalks are stuffed 
with lots of bad examples here. Sometimes Convenience methods are a 
symptom of design problems or the lack of multiple inheritance. 
Sometimes they are just a good idea.

>     But: adding more and more misnamed and misleading assertion
>     methods makes the use of SUnit frustrating and will make it
>     obsolete over time. If I have to hunt for design problems in SUnit
>     because it assumes something to be wrong even though my
>     understanding of waht I tested is different, I lose way more time
>     than I am ready to accept. This doesn't happen to me often. If
>     finding that I misunderstood an assertion method means I lost a
>     few hours, the best thing that may happen is that I never use that
>     method again. In the worst case, I decide I think SUnit is useless
>     for me. That would be really bad, don't you think?
> I am not sure if we are talking about the same SUnit. Sure, there are 
> 32 methods in the "asserting" protocol, however most of them are 
> either opposites of one another "#assert: vs #deny:", or they provide 
> some customization such has "#assert:description:", "#should:raise:" 
> ... so if I count only meaningfully different methods the number 7 
> (not to mention that some of the methods are not even used). But if 
> you have trouble understanding the purpose of seven methods, then the 
> problem is on your end, and don't blame SUnit for it.

I think should:/shouldnt are great examples of convenience methods that 
are not named good enough. I know nobody who can use them without taking 
a second to think about them.
So what you are saying is that SUnit already has roughly 25 convenience 
methods that aren't used or hard to understand. And the ones we 
understand are used because they are a good shortcut to a better 
description string (assert:equals:). So we are not far away from each 
other... ;-)
>>      > assertCollection:hasSameElements:
>>         > So, let's start by asking what the question really means.
>>         Does it mean that one collection is a subset of the other?
>>         What about ordering then? Does it mean both contain the same
>>         elements at the same position or just the fact that if they
>>         both were Sets contained the exact same elements. The
>>         question itself is not exact, so how could an answer possibly be?
>>     There is no question about this method, since this is implemented
>>     in code there is nothing ambiguous. This method effectively
>>     converts both arguments to sets and compares them like that.
>     Sorry to say that, but this is ambiguity by design: you define
>     hasSameElements: as "both result in the same Set". So the name of
>     this assertion method is a great example of bad naming, IMO.
> Yes, the naming is confusing. My point was, that instead of 
> philosophizing about the meaning you can look at the code. Of course 
> if you use the method for the first time (like I did), you will get 
> burned by it (as I did).

Hmm - just by reading these 3 last sentences, you see that naming is 
important. But we don't disagree here. Let's forget about the concrete 
naming of the method from now on. It's an example.
My point was that if the naming of a method opens room for speculation 
and philosophy, it is most likely either named badly or it is a bad idea 
in itself. Maybe it is not a candidate for addition as a core method in 
SUnit. SUnit should make you think about your code, not about itself.

>>         self assert: result asOrderedCollection asSortedCollection
>>         equals: (1 to: 10) asOrderedCollection
>>     This is what I usually do now (although I convert to Array, not
>>     OrderedCollection, because the expected one is usually created by
>>     hand with #() or {} ).
>     I don't really care. If what you try t say is that the testing
>     code can be ugly and long, then I agree. If you need tests like
>     this very often and want something to make this easier, I
>     understand and agree that some additions to SUnit can be helpful.
>     But the way this has been tried so far seems completely wrong to me.
> How would you test it then? Some problem domains deal with certain 
> kind of problems more than others and thus benefit more from 
> appropriate assertions.
This is not about how to test for something. Our discussion is about the 
question whether a certain wayto test something should be added to the 
framework or not. Is the test generic and important enough so that it 
should make its way into SUnit. And if it should be added, what is a 
good name for it, so that others find the method as the right thing to 
use for their purpose or will it disturb them in their proces more than 
it helps.

You can always use helper objects and non-SUnit-classes in your test 
packages for generalizations of your project's needs. If you need to 
walk trees a lot for your purposes, you can add Visitors and stuff, to 
your test package, put them under test to make sure they work and use 
them in your assertions. This is not necessarily a candidate for 
inclusion in SUnit, just because chances are others need to test Trees 
as well.

>>         > Just a few weeks ago, we discussed something similar about
>>         the equality of two Collections. Endless discussions where
>>         people try convince others that their definition of equality
>>         and/or sameness is correct with no outcome.
>>     I don't see a problem with that because collections truly can
>>     have different equalities based on the context and their purpose.
>>     And while you can call this rat poison, it effectively tells what
>>     kind of behavior you expect from the collections, which seems ok.
>     So what, again, was the point of naming a method after a general
>     Collection class and use a question that is very unspecific? A
>     Collection has the same elements as another does not necessarily
>     mean they both result in the same set. Can we agree on that? All
>     the question asks if all Elements in Collection A can also be
>     found in Collection B. The method name states nothing more than that.
> We agree that the method is badly named. (However what equality of two 
> Collections means is context-dependent.)
Phew. Thank you. I am glad we agree on this. So let's carry on with the 
SUnit related stuff.

>     My point here is that a general purpose framework like SUnit
>     should be free of such debatable things. SUnit has to be reliable
>     and understandable.
>     There is nothing wrong with providing some "plugins" for problems
>     like Collections that make life easier.
>     It would be desirable to have more control over SUnit's feedback
>     with little typing.
> Well but we need a way to provide context for the assertion. If I 
> could type less then I would be happier user, however currently I 
> don't see a way how to make it more general (so I don't need to type) 
> and more precise (it still understands context) at the same time.
Now I have you in the corner I want you in ;-)
We should talk about ways to make it easier to provide clearer feedback 

> So going back to your earlier suggestion with the class... you imagine 
> something like this?
> self assert: a equals: b strategy: 
> CollectionHasExactlyTheSameElementsAsAnotherAtTheSamePosition
No. For several reasons (based on how I interpret the names you provide 
as an example):

* You are back to the idea that you need a way to specify a specialized 
testing strategy. That is exactly the wrong thought here. What and how 
to test is highly dependent on your code base and needs. A simple 
#asserts: and your very special way of testing formulated either by 
writing complex code or using your own specialized helpers is the only 
way to keep SUnit lean and mean (BTW, there is nobody keeping us from 
providing SUnit helper packages that make life easier, like the Tree 
example - but not as part of core SUnit)

* what if I need a test to prove/check that two collections are NOT the 
same in a very special case? would you add a #deny:equals:strategy: 
also? It will blow the testing protocol up from 32 to 100 very soon.

I would like to start a discussion on ways to separate the assertion 
stuff from the feedback generation. The first steps I can think of are 
not sufficient in the "little typing" arena, and I must admit I have not 
yet found a good idea.

But to start with something, I'd like to start with the assumption that 
we only have #assert: and all we want is a second parameter, like in 

So I am now just brainstorming, not saying this is a good solution already.

Let's assume we have some way to add parameters into Strings, like 
#bindWith: and friends (or Mustache, or whatever). Let's also assume 
there is something like Block>>#cull: that makes it easy to fill such a 
String with any number of arguments (e.g. 'I accept up to %1 arguments' 
cull: {arg1, arg2}).
Couldn't we just put some machinery into the test runner that uses this 
facility to provide better feedback? like in: #assert:failureText: 
'Expected %1 but got %2'.

At first sight, this is not so easy, because we need a place to store 
the arguments into, and since assert: just accepts a Block or simply the 
result of executing an expression, there is no such thing at hand (if it 
was easy, SUnit would have had such a feature from the beginning).

So the assertion needs to be at least a Block or an object which offers 
its arguments to the "feedback machine" as well.

The api might change from assert: to something like #assertExecuting: 
aBlockOrObject with: listOfArguments resultsIn: expectedResult

Typing this, I see that this is not nice, because you cannot just write 
some expression like self assert: myObject *3 = 15. You'd have to write 
assertExecuting: [:myObject| myObject * 3] with: 5 resultsIn: 15
Adding the feedback generation into makes the API even more complicated, 
because you'd have to ar least pass a String as a fourth parameter.
So we'd end up with:

assertExecuting: aBlock with: listOfArguments resultsIn: expectedResult 
otherwise: failureString.

In this form, I am sure nobody would use it. Or just for very special 
and important tests. So all that I can come up with for now is 
insufficient, unfortunately. Sure, there can be variants that provide 
some standard parameters for most of the arguments, but the base problem 
remains that the API is clumsy.

So what I am just showing is that what has happened with the assert: 
variants was the best thing one could possibly do without making SUnit 
heavy-weight and unpleasant to use.
Does it mean I am wrong? Does it mean it would be a good idea to put 
some more thought into this? Does it mean we should just go on and add 
more convenience methods? Does it mean we shouldn't change SUnit and 
keep it as it is?

I think it means we shouldn't just go on adding new convenience methods, 
but think about ways to improve its internals and make it more flexible 
in its feedback generation facilities.
Maybe we'll end up in something that you can use as a base 
implementation for today's #assert: method...

> 1. #assertCollection:hasSameElements: is a bad name, because it treats 
> arguments as sets, not generic collections
> it could be renamed or removed (no code in Pharo itself uses it (only 
> Spec, but the use there is wrong))
let's put a check mark on this one. It was just an example
> 2. Providing selectors such as #assertCollection:equals: and similar 
> just to get different description may not be the best idea, which 
> leads to point 3
I think I made it clear that this will be a less than perfect solution 
that doesn't jump far enough
> 3. An easy way to extend printing/assertions/plugins; but what would 
> that look like?
I am convinced this is the way to go. But I have no good answer to it. 
Just a spark for now.
> Maybe
> #assert: a = b description: [ Diff collectionDiffBetween: a and: b ]

Possibly the best thing we can come up with for now. By repeating the 
arguments in the feedback generation part, we keep the complexity out of 
the test runner's internals

> or
> #assert: a equals: b strategy: 
> CollectionHasExactlyTheSameElementsAsAnotherAtTheSamePosition
See above. Better than another assert:-variation, but still too heavy

> or
> something else...
> Peter


Objektfabrik Joachim Tuchel          mailto:jtuchel at objektfabrik.de
Fliederweg 1                         http://www.objektfabrik.de
D-71640 Ludwigsburg                  http://joachimtuchel.wordpress.com
Telefon: +49 7141 56 10 86 0         Fax: +49 7141 56 10 86 1

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pharo.org/pipermail/pharo-users_lists.pharo.org/attachments/20151027/28cdddd9/attachment.html>

More information about the Pharo-users mailing list