And now it is in latest Pharo 2017-11-09 12:16 GMT+01:00 Denis Kudriashov <dionisiydk@gmail.com>:

Hi Tim.

Fix is here 20661-Fixing-test-from-debugger-should-mark-test-as- green-when-proceed <https://github.com/pharo-project/pharo/pull/456> . Thank's for reporting. It forces me to fix. I always noticed it but never take it seriously :)

2017-11-09 11:32 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Hi - I really like the build in test runner in the Pharo browsers, and I was preparing a talk to show how great TDD is in Pharo and how we aren’t ashamed of our debugger when testing (it augments the experience in fact - letting you poke around and get your thoughts straight).

However - if I pick rerun in the test runner debugger - and step through a test and then correct the failing code, and hit resume - the browser always shows a red failure, even though the execution is now correct. I have to run the test again.

This doesn’t seem right to me - are we missing a success event or something?

Tim

I saw this earlier and thought it was wrong to make that change. You cannot assign a colour to the test if it does not run all the way through without interruption. You cannot know whether it truly succeeded or not until you finally run it all the way through without error or interruption. Leave it uncoloured until you (can) know for certain. On Thu, Nov 9, 2017 at 9:29 AM, Tim Mackinnon <tim@testit.works> wrote:

Thanks for looking at this - there is an issue however - when you apply that change (at least in a Pharo 6.1 image) - it shows green even when a test fails? So I think its turned one problem into the opposite one.

Unfortunately I haven’t got a chance to look a bit deeper to help - but it might be worth rolling back this change for now. We should fix it though - and the answer must be in the area you have identified.

tim

On 9 Nov 2017, at 12:43, Denis Kudriashov <dionisiydk@gmail.com> wrote:

And now it is in latest Pharo

2017-11-09 12:16 GMT+01:00 Denis Kudriashov <dionisiydk@gmail.com>:

...

Hi Tim.

Fix is here 20661-Fixing-test-from-debugger-should-mark-test-as-gre en-when-proceed <https://github.com/pharo-project/pharo/pull/456> . Thank's for reporting. It forces me to fix. I always noticed it but never take it seriously :)

2017-11-09 11:32 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Hi - I really like the build in test runner in the Pharo browsers, and I was preparing a talk to show how great TDD is in Pharo and how we aren’t ashamed of our debugger when testing (it augments the experience in fact - letting you poke around and get your thoughts straight).

However - if I pick rerun in the test runner debugger - and step through a test and then correct the failing code, and hit resume - the browser always shows a red failure, even though the execution is now correct. I have to run the test again.

This doesn’t seem right to me - are we missing a success event or something?

Tim

Thanks Denis - I'll try my simple TDD test case in Pharo 7 (I haven't tried it yet - probably time to). I think it is correct to show green if you fix your problem in the debugger and then proceed and it passes all assertions. But it should equally fail if you proceed and it asserts false or craps out. Strange it would vary in 6.1 - but I'll check. Tim Sent from my iPhone Sent from my iPhone

On 9 Nov 2017, at 19:54, Denis Kudriashov <dionisiydk@gmail.com> wrote:

In Pharo 7 it works.

2017-11-09 18:29 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Thanks for looking at this - there is an issue however - when you apply that change (at least in a Pharo 6.1 image) - it shows green even when a test fails? So I think its turned one problem into the opposite one.

Unfortunately I haven’t got a chance to look a bit deeper to help - but it might be worth rolling back this change for now. We should fix it though - and the answer must be in the area you have identified.

tim

...

On 9 Nov 2017, at 12:43, Denis Kudriashov <dionisiydk@gmail.com> wrote:

And now it is in latest Pharo

2017-11-09 12:16 GMT+01:00 Denis Kudriashov <dionisiydk@gmail.com>:

...

Hi Tim.

Fix is here 20661-Fixing-test-from-debugger-should-mark-test-as-green-when-proceed . Thank's for reporting. It forces me to fix. I always noticed it but never take it seriously :)

2017-11-09 11:32 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Hi - I really like the build in test runner in the Pharo browsers, and I was preparing a talk to show how great TDD is in Pharo and how we aren’t ashamed of our debugger when testing (it augments the experience in fact - letting you poke around and get your thoughts straight).

However - if I pick rerun in the test runner debugger - and step through a test and then correct the failing code, and hit resume - the browser always shows a red failure, even though the execution is now correct. I have to run the test again.

This doesn’t seem right to me - are we missing a success event or something?

Tim

Hi Richard 2017-11-09 23:51 GMT+01:00 Richard Sargent < richard.sargent@gemtalksystems.com>:

I think it is correct to show green if you fix your problem in the

...

debugger and then proceed and it passes all assertions. But it should equally fail if you proceed and it asserts false or craps out.

It would be a lie. The truth is only that from that point on, it is correct. You have no guarantee that your correction hasn't broken something that would occur in the next run prior to the point at which you applied the correction.

I think it is correct because icon shows the last run state. It is not depends on the way how user manages to pass or fail the test. So user can hack the test from the debugger and make it pass or fail using inspector, scripting or whatever interaction with available objects. When he closes the debugger (by proceed or window close) it is expected to see the result of this hacking in the UI. So test icon should be updated accordingly. Then it is up to user to check how test is working in normal way.

Hence my assertion that you cannot determine the colour until the next run.

On Thu, Nov 9, 2017 at 2:47 PM, Tim Mackinnon <tim@testit.works> wrote:

...

Thanks Denis - I'll try my simple TDD test case in Pharo 7 (I haven't tried it yet - probably time to).

I think it is correct to show green if you fix your problem in the debugger and then proceed and it passes all assertions. But it should equally fail if you proceed and it asserts false or craps out.

Strange it would vary in 6.1 - but I'll check.

Tim

Sent from my iPhone

Sent from my iPhone On 9 Nov 2017, at 19:54, Denis Kudriashov <dionisiydk@gmail.com> wrote:

In Pharo 7 it works.

2017-11-09 18:29 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Thanks for looking at this - there is an issue however - when you apply that change (at least in a Pharo 6.1 image) - it shows green even when a test fails? So I think its turned one problem into the opposite one.

Unfortunately I haven’t got a chance to look a bit deeper to help - but it might be worth rolling back this change for now. We should fix it though - and the answer must be in the area you have identified.

tim

On 9 Nov 2017, at 12:43, Denis Kudriashov <dionisiydk@gmail.com> wrote:

And now it is in latest Pharo

2017-11-09 12:16 GMT+01:00 Denis Kudriashov <dionisiydk@gmail.com>:

...

Hi Tim.

Fix is here 20661-Fixing-test-from-debugger-should-mark-test-as-gre en-when-proceed <https://github.com/pharo-project/pharo/pull/456> . Thank's for reporting. It forces me to fix. I always noticed it but never take it seriously :)

2017-11-09 11:32 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Hi - I really like the build in test runner in the Pharo browsers, and I was preparing a talk to show how great TDD is in Pharo and how we aren’t ashamed of our debugger when testing (it augments the experience in fact - letting you poke around and get your thoughts straight).

However - if I pick rerun in the test runner debugger - and step through a test and then correct the failing code, and hit resume - the browser always shows a red failure, even though the execution is now correct. I have to run the test again.

This doesn’t seem right to me - are we missing a success event or something?

Tim

Richard Sargent wrote

The only reliable conclusion one can make from such an interrupted run is whether it failed again. So, it would be possible to determine that the test should be coloured red, but it is impossible to *reliably* claim that the test should be coloured green.

What if we ran the test again as if from the browser/runner before setting the icon? ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

My specific usecase is from a pragmatic TDD perspective - failing test, in the debugger you fix the test and press proceed - expecting green. Getting red, and then immediately running again to get red takes away from our story of love coding and loving your debugger - and even Cassie me to mistrust the tools. I get the idea that you can jiffy in the debugger and cause a false pass - but I feel you are penalised for doing it right at the moment. Of course these tests will get run again, but I like the idea that if I did it right, it should recognise this, not incur an extra click and moment of doubt. A button to rerun the whole lot, or automatically rerun, or just run it would work for me. Tim Sent from my iPhone

On 10 Nov 2017, at 17:56, Richard Sargent <richard.sargent@gemtalksystems.com> wrote:

That would be fine. The point is that, without running the test in its entirety, from start to finish, without interruption, error, or failure, one cannot claim success.

...

On Fri, Nov 10, 2017 at 9:34 AM, Sean P. DeNigris <sean@clipperadams.com> wrote: Richard Sargent wrote

...

The only reliable conclusion one can make from such an interrupted run is whether it failed again. So, it would be possible to determine that the test should be coloured red, but it is impossible to *reliably* claim that the test should be coloured green.

What if we ran the test again as if from the browser/runner before setting the icon?

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

I hear you and I understand your pain. However, if you corrected the problem, not by a code change, but by playing in the object's inspector or otherwise causing its internal state to change, and then resumed from the debugger, would you still claim the method was successful and should be coloured green? The only thing we can claim is that there were or were not further errors in the test. Colour it red if there were, but you cannot honestly colour it green. The code doesn't actually work. On Fri, Nov 10, 2017 at 11:29 AM, Tim Mackinnon <tim@testit.works> wrote:

My specific usecase is from a pragmatic TDD perspective - failing test, in the debugger you fix the test and press proceed - expecting green. Getting red, and then immediately running again to get red takes away from our story of love coding and loving your debugger - and even Cassie me to mistrust the tools.

I get the idea that you can jiffy in the debugger and cause a false pass - but I feel you are penalised for doing it right at the moment.

Of course these tests will get run again, but I like the idea that if I did it right, it should recognise this, not incur an extra click and moment of doubt.

A button to rerun the whole lot, or automatically rerun, or just run it would work for me.

Tim

Sent from my iPhone

On 10 Nov 2017, at 17:56, Richard Sargent <richard.sargent@ gemtalksystems.com> wrote:

That would be fine. The point is that, without running the test in its entirety, from start to finish, without interruption, error, or failure, one cannot claim success.

On Fri, Nov 10, 2017 at 9:34 AM, Sean P. DeNigris <sean@clipperadams.com> wrote:

...

Richard Sargent wrote

...

The only reliable conclusion one can make from such an interrupted run is whether it failed again. So, it would be possible to determine that the test should be coloured red, but it is impossible to *reliably* claim that the test should be coloured green.

What if we ran the test again as if from the browser/runner before setting the icon?

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Richard - I better understand what you are saying now. If you change the method and save it (hence restarting on the stack) I would expect it to turn green if I press resume. That is the TDD flow I expect, and one which sunit and coding in the debugger was predicated on. However, there is the potential that some memory object that cached a result and is now running a second time could be the cause of a pass and so it is remotely possible that this is a false pass…. (And this I think is the crux of your argument - if any memory object is affected, theoretically you should rerun the whole transaction from a virgin state - which is effectively rerun the test from the beginning). So I guess we are discussing that we don’t have fully transactional memory executions? However I would argue that its way more common that you edit a method and fix a text and have 0 side effects than mucking around in memory or having something that rerunning locally messes up memory as well. So its much more useful to get the confirmation of green and move on. YES you could have a subtle error, and when you re-run it may later go red - but I would favour the 99% case as its a annoying if you are doing TDD. Tim

On 10 Nov 2017, at 19:45, Richard Sargent <richard.sargent@gemtalksystems.com> wrote:

I hear you and I understand your pain.

However, if you corrected the problem, not by a code change, but by playing in the object's inspector or otherwise causing its internal state to change, and then resumed from the debugger, would you still claim the method was successful and should be coloured green?

The only thing we can claim is that there were or were not further errors in the test. Colour it red if there were, but you cannot honestly colour it green. The code doesn't actually work.

On Fri, Nov 10, 2017 at 11:29 AM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote: My specific usecase is from a pragmatic TDD perspective - failing test, in the debugger you fix the test and press proceed - expecting green. Getting red, and then immediately running again to get red takes away from our story of love coding and loving your debugger - and even Cassie me to mistrust the tools.

I get the idea that you can jiffy in the debugger and cause a false pass - but I feel you are penalised for doing it right at the moment.

Of course these tests will get run again, but I like the idea that if I did it right, it should recognise this, not incur an extra click and moment of doubt.

A button to rerun the whole lot, or automatically rerun, or just run it would work for me.

Tim

Sent from my iPhone

On 10 Nov 2017, at 17:56, Richard Sargent <richard.sargent@gemtalksystems.com <mailto:richard.sargent@gemtalksystems.com>> wrote:

...

That would be fine. The point is that, without running the test in its entirety, from start to finish, without interruption, error, or failure, one cannot claim success.

On Fri, Nov 10, 2017 at 9:34 AM, Sean P. DeNigris <sean@clipperadams.com <mailto:sean@clipperadams.com>> wrote: Richard Sargent wrote

...

The only reliable conclusion one can make from such an interrupted run is whether it failed again. So, it would be possible to determine that the test should be coloured red, but it is impossible to *reliably* claim that the test should be coloured green.

What if we ran the test again as if from the browser/runner before setting the icon?

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html <http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html>

On 15 November 2017 at 00:14, Richard Sargent <rsargent@5x5.on.ca> wrote:

...

...

I would expect it to turn green if I press resume.

I disagree with your expectations. You are arguing that we should operate is if "probably correct" is the same as "correct". That's why we have ****ty software.

I have no objection to leaving the method uncoloured when you resume after correcting the error. I have no objection to eliminating the red colouring in these scenarios. I strongly object to pretending that it is known to be correct.

Or it could go to Amber, half-way between green & red to mean probably correct. cheers -ben

In my experience, we end up with better results when we act on what we absolutely know to be factual and stop relying on guessing.

------------------------------ *From:* Pharo-users [mailto:pharo-users-bounces@lists.pharo.org] *On Behalf Of *Tim Mackinnon *Sent:* November 14, 2017 06:19 *To:* Any question about pharo is welcome *Subject:* Re: [Pharo-users] Why does the test runner show red when I correct a test?

Richard - I better understand what you are saying now. If you change the method and save it (hence restarting on the stack) I would expect it to turn green if I press resume. That is the TDD flow I expect, and one which sunit and coding in the debugger was predicated on.

However, there is the potential that some memory object that cached a result and is now running a second time could be the cause of a pass and so it is remotely possible that this is a false pass…. (And this I think is the crux of your argument - if any memory object is affected, theoretically you should rerun the whole transaction from a virgin state - which is effectively rerun the test from the beginning). So I guess we are discussing that we don’t have fully transactional memory executions?

However I would argue that its way more common that you edit a method and fix a text and have 0 side effects than mucking around in memory or having something that rerunning locally messes up memory as well. So its much more useful to get the confirmation of green and move on. YES you could have a subtle error, and when you re-run it may later go red - but I would favour the 99% case as its a annoying if you are doing TDD.

Tim

On 10 Nov 2017, at 19:45, Richard Sargent <richard.sargent@ gemtalksystems.com> wrote:

I hear you and I understand your pain.

However, if you corrected the problem, not by a code change, but by playing in the object's inspector or otherwise causing its internal state to change, and then resumed from the debugger, would you still claim the method was successful and should be coloured green?

The only thing we can claim is that there were or were not further errors in the test. Colour it red if there were, but you cannot honestly colour it green. The code doesn't actually work.

On Fri, Nov 10, 2017 at 11:29 AM, Tim Mackinnon <tim@testit.works> wrote:

...

My specific usecase is from a pragmatic TDD perspective - failing test, in the debugger you fix the test and press proceed - expecting green. Getting red, and then immediately running again to get red takes away from our story of love coding and loving your debugger - and even Cassie me to mistrust the tools.

I get the idea that you can jiffy in the debugger and cause a false pass - but I feel you are penalised for doing it right at the moment.

Of course these tests will get run again, but I like the idea that if I did it right, it should recognise this, not incur an extra click and moment of doubt.

A button to rerun the whole lot, or automatically rerun, or just run it would work for me.

Tim

Sent from my iPhone

On 10 Nov 2017, at 17:56, Richard Sargent <richard.sargent@gemtalksystem s.com> wrote:

That would be fine. The point is that, without running the test in its entirety, from start to finish, without interruption, error, or failure, one cannot claim success.

On Fri, Nov 10, 2017 at 9:34 AM, Sean P. DeNigris <sean@clipperadams.com> wrote:

...

Richard Sargent wrote

...

The only reliable conclusion one can make from such an interrupted run is whether it failed again. So, it would be possible to determine that the test should be coloured red, but it is impossible to *reliably* claim that the test should be coloured green.

What if we ran the test again as if from the browser/runner before setting the icon?

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run

Absolutely! (Although, I was kind of looking forward to the third colour discussions. My vote would have been for paisley!) On Tue, Nov 14, 2017 at 4:49 PM, Sean P. DeNigris <sean@clipperadams.com> wrote:

Ben Coman wrote

...

Or it could go to Amber, half-way between green & red to mean probably correct.

Ha ha.

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run addresses all points on both sides, no?

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

2017-11-15 1:49 GMT+01:00 Sean P. DeNigris <sean@clipperadams.com>:

Ben Coman wrote

...

Or it could go to Amber, half-way between green & red to mean probably correct.

Ha ha.

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run addresses all points on both sides, no?

Except that sometimes we are debugging slow test and running it second time automatically after "proceed" can be not appropriate. We are talking about single test run. If user have any doubts about result It is his responsibility to rerun the test. User knows what he is doing when he debug and fix the test. No intelligence is required here. And anyway current fix just provides consistent behaviour to debugging from explicit breakpoint/halt. In that case the result was always in sync with debug session.

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

2017-11-15 11:00 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com>:

And just putting it back to gray? As "not run"?

We can implement any logic. Personally I need current behaviour.

On Wed, Nov 15, 2017 at 10:44 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

...

2017-11-15 1:49 GMT+01:00 Sean P. DeNigris <sean@clipperadams.com>:

...

Ben Coman wrote

...

Or it could go to Amber, half-way between green & red to mean probably correct.

Ha ha.

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run addresses all points on both sides, no?

Except that sometimes we are debugging slow test and running it second time automatically after "proceed" can be not appropriate. We are talking about single test run. If user have any doubts about result It is his responsibility to rerun the test. User knows what he is doing when he debug and fix the test. No intelligence is required here.

And anyway current fix just provides consistent behaviour to debugging from explicit breakpoint/halt. In that case the result was always in sync with debug session.

...

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

2017-11-15 11:08 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com>:

On Wed, Nov 15, 2017 at 11:06 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

...

2017-11-15 11:00 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com>:

...

And just putting it back to gray? As "not run"?

We can implement any logic. Personally I need current behaviour.

But it is not about you personally. It is about implementing the most common and the less strange for newcomers.

To know what is the most common case people should tell personal opinion. And in this thread only Richard was against current logic.

...

...

On Wed, Nov 15, 2017 at 10:44 AM, Denis Kudriashov <dionisiydk@gmail.com

...

wrote:

...

2017-11-15 1:49 GMT+01:00 Sean P. DeNigris <sean@clipperadams.com>:

...

Ben Coman wrote

...

Or it could go to Amber, half-way between green & red to mean probably correct.

Ha ha.

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run addresses all points on both sides, no?

Except that sometimes we are debugging slow test and running it second time automatically after "proceed" can be not appropriate. We are talking about single test run. If user have any doubts about result It is his responsibility to rerun the test. User knows what he is doing when he debug and fix the test. No intelligence is required here.

And anyway current fix just provides consistent behaviour to debugging from explicit breakpoint/halt. In that case the result was always in sync with debug session.

...

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

I think that setting the button back to gray is a good behaviour. - it is the same thing that happens once you modify a method (which is what is happening during debugging) - it explicitly says "please rerun the test because you may have introduced side effects"

I would go a little further. Any method modified by the developer during the course of running a test voids the ability to claim the test succeeded. Likewise, for any object editted in an inspector. In other words, if the test run was interrupted, it cannot be considered successful. Keep things simple. (as simple as possible) On Wed, Nov 15, 2017 at 2:49 AM, Guillermo Polito <guillermopolito@gmail.com

wrote:

On Wed, Nov 15, 2017 at 11:41 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

...

2017-11-15 11:08 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com>:

...

On Wed, Nov 15, 2017 at 11:06 AM, Denis Kudriashov <dionisiydk@gmail.com

...

wrote:

...

2017-11-15 11:00 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com> :

...

And just putting it back to gray? As "not run"?

We can implement any logic. Personally I need current behaviour.

But it is not about you personally. It is about implementing the most common and the less strange for newcomers.

To know what is the most common case people should tell personal opinion. And in this thread only Richard was against current logic.

But you're assuming here that: - people that is not reading this email do not care and don't have a say - so pleople that is not subscribed to the mailing list don't care - and that includes newbies

Our role of experienced guys it not only look after "our" best defaults. But also after the defaults of people without experience.

I think that setting the button back to gray is a good behaviour. - it is the same thing that happens once you modify a method (which is what is happening during debugging) - it explicitly says "please rerun the test because you may have introduced side effects"

Unless you make the debugger more intelligent, you cannot be sure that the result you obtained at the end of the test is really reproducible. And moreover, to be able to make such assumption you should be an expert that understands how the underlying framework behaves.

...

...

...

...

On Wed, Nov 15, 2017 at 10:44 AM, Denis Kudriashov < dionisiydk@gmail.com> wrote:

...

2017-11-15 1:49 GMT+01:00 Sean P. DeNigris <sean@clipperadams.com>:

...

Ben Coman wrote

...

Or it could go to Amber, half-way between green & red to mean probably correct.

Ha ha.

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run addresses all points on both sides, no?

Except that sometimes we are debugging slow test and running it second time automatically after "proceed" can be not appropriate. We are talking about single test run. If user have any doubts about result It is his responsibility to rerun the test. User knows what he is doing when he debug and fix the test. No intelligence is required here.

And anyway current fix just provides consistent behaviour to debugging from explicit breakpoint/halt. In that case the result was always in sync with debug session.

...

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

I would agree that grey is better than red - but I personally think we’re being too pedantic on this - particularly when doing TDD and coding in the debugger. If I’m writing straight forward tests and like to see a red failure (either by deliberately returning false, or a subclass responsibility, or -1) and then correct that failure in the debugger to return the correct result - then its tedious to have to run the test again (in fact it feels odd). For the rare time that I got it wrong - it will show up when I run all tests on the next phase. I accept that others may see this the other way around - but I’m a more optimistic guy. This said - maybe we make it an option (or an easy code switch) - I’d default it to the optimistic TDD mode personally. My CI server will give me the full lowdown. Tim

On 15 Nov 2017, at 21:50, Sean P. DeNigris <sean@clipperadams.com> wrote:

Richard Sargent wrote

...

I would go a little further. Any method modified by the developer during the course of running a test voids the ability to claim the test succeeded. Likewise, for any object editted in an inspector.

That makes sense to me.

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

A test that asserts nothing has only asserted that the code ran through without throwing an error. :-) I like your proposal that such a test is an inherently failing test. (Of course, that will result in the student adding a single assertion at the end, of the form "self assert: true"!) On Tue, Nov 21, 2017 at 4:07 PM, Prof. Andrew P. Black <black@cs.pdx.edu> wrote:

While we are discussing colors, what should we do about a test that does not make any assertions at all?

A couple of years ago, a smart student who was working on a testing dialect for me decided that such tests should be in a new category all of their own. Later I simplified things and just made these tests fail. I am pretty convinced now that this is the right behaviour. It certainly suits my purposes (teaching TDD); if the student omits to make any assertions, they have a failing test. The message is: "Failure: test made no assertions".

Currently, such a test is green in Pharo. In 2013, I filed a bug report on a bunch of tests of #printOn: on collections that made no assertions. (The test writer didn’t understand how streams worked, and made assertions for each element of the empty string.) I was reminded of this just this week, because those tests have yet to be fixed. I suspect that if they had been yellow, rather than green, then they would have been fixed before now.

I plan to fix those tests on Friday, but I also wonder about changing the behaviour of the testing framework. What do you think?

Andrew

BTW I like grey for a method that was edited in the debugger. Because red is not good. The proof is that when I rerun my test is green. While saying I do not know you have to rerun is a good solution. I like it. I got always frustrated with the red. On Fri, Nov 24, 2017 at 12:05 AM, Stephane Ducasse <stepharo.self@gmail.com> wrote:

Hi andrew

I like your idea. It is fun and at least we can spot test. Now sometimes we will have to have self assert: true because running a code can be also considered as a test. Now what I do not like is empty test method because there are green.

stef

On Tue, Nov 21, 2017 at 10:07 PM, Prof. Andrew P. Black <black@cs.pdx.edu> wrote:

...

While we are discussing colors, what should we do about a test that does not make any assertions at all?

A couple of years ago, a smart student who was working on a testing dialect for me decided that such tests should be in a new category all of their own. Later I simplified things and just made these tests fail. I am pretty convinced now that this is the right behaviour. It certainly suits my purposes (teaching TDD); if the student omits to make any assertions, they have a failing test. The message is: "Failure: test made no assertions".

Currently, such a test is green in Pharo. In 2013, I filed a bug report on a bunch of tests of #printOn: on collections that made no assertions. (The test writer didn’t understand how streams worked, and made assertions for each element of the empty string.) I was reminded of this just this week, because those tests have yet to be fixed. I suspect that if they had been yellow, rather than green, then they would have been fixed before now.

I plan to fix those tests on Friday, but I also wonder about changing the behaviour of the testing framework. What do you think?

Andrew

Ben, I think you understand the dilemma. My philosophy is to avoid claims that are known to be not provably true. i.e. don't colour it green, because the test runner cannot claim that knowledge. I am agnostic to whether it should be left uncoloured or coloured red to document the fact that it failed (as in was unable to run to completion without a problem). I truly don't care. Our profession has a tendency to make claims which aren't provably true. It is one of the many reasons we have poor software. So I am adamant about avoiding that particular mistake. And, I am adamant about dissuading others from the same kinds of mistakes. Write software which conveys exactly what it can claim without Deus ex machina intervention. The test failed. It was unable to run to completion without an error. It doesn't matter whether that particular error was corrected or not. The software running the test is incapable of determining whether the change that allowed the test to run to completion was code, data editing, or something else and it is incapable of knowing whether whatever allowed the test to finish is a correct fix for the error. Be adamant about what your software can know to be true. Equally important, be adamant about your own approach to problems and solutions. Godel had some good advice. On Nov 23, 2017 19:31, "Ben Coman" <btc@openinworld.com> wrote:

On 16 November 2017 at 00:20, Richard Sargent <richard.sargent@ gemtalksystems.com> wrote:

...

I think that setting the button back to gray is a good behaviour.

...

- it is the same thing that happens once you modify a method (which is what is happening during debugging) - it explicitly says "please rerun the test because you may have introduced side effects"

I would go a little further. Any method modified by the developer during the course of running a test voids the ability to claim the test succeeded. Likewise, for any object editted in an inspector.ere

So to summarise the viewpoints as I understand them, consider an interrupted test that later runs to completion and is then...

A. incorrectly marked red (or grey) [false negative] ==> frequent during TDD ==> need to manually rerun test *every* time ==> large extra effort for developer

B. incorrectly marked green [false positive] ==> infrequent (presumed) ==> error picked up anytime test is run again, or when test group is run ==> small extra effort for developer ==> developer may be aware when they make suspect changes which undermine test result, to judge to run it again.

C. test is automatically rerun a second time ==> infrequently some tests run too long for this to be practical

A and B are like the philosophical difference between engineers and scientists. i.e. engineers deal with approximations** that make the design process more efficient. For C, I still don't fully understand the concrete problem. How much time do such tests take?

Anyway, I prefer early efficiency with late bound correctness, so my vote would be to avoid A.

In other words, if the test run was interrupted, it cannot be considered

...

successful. Keep things simple. (as simple as possible)

There are costs associated with perfect correctness. i.e. costs associated with both false-negatives and false-positives. The alternate approach is to consider the consequence of temporary false-positive. I'd vote for making things as simple and *efficient* as possible.

cheers -ben

**A mathematician, a scientist, and an engineer are given the task of finding how high a particular red rubber ball will bounce when dropped from a given height onto a given surface. The mathematician derives the elasticity of the ball from its chemical makeup, derives the equations to determine how high it will bounce and calculates it. The physicist takes the ball into the lab, measures its elasticity, and plugs the variables into a formula. The engineer looks it up in his red rubber ball book.

...

On Wed, Nov 15, 2017 at 2:49 AM, Guillermo Polito < guillermopolito@gmail.com> wrote:

...

On Wed, Nov 15, 2017 at 11:41 AM, Denis Kudriashov <dionisiydk@gmail.com

...

wrote:

...

2017-11-15 11:08 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com> :

...

On Wed, Nov 15, 2017 at 11:06 AM, Denis Kudriashov < dionisiydk@gmail.com> wrote:

...

2017-11-15 11:00 GMT+01:00 Guillermo Polito < guillermopolito@gmail.com>:

...

And just putting it back to gray? As "not run"?

We can implement any logic. Personally I need current behaviour.

But it is not about you personally. It is about implementing the most common and the less strange for newcomers.

To know what is the most common case people should tell personal opinion. And in this thread only Richard was against current logic.

But you're assuming here that: - people that is not reading this email do not care and don't have a say - so pleople that is not subscribed to the mailing list don't care - and that includes newbies

Our role of experienced guys it not only look after "our" best defaults. But also after the defaults of people without experience.

I think that setting the button back to gray is a good behaviour. - it is the same thing that happens once you modify a method (which is what is happening during debugging) - it explicitly says "please rerun the test because you may have introduced side effects"

Unless you make the debugger more intelligent, you cannot be sure that the result you obtained at the end of the test is really reproducible. And moreover, to be able to make such assumption you should be an expert that understands how the underlying framework behaves.

...

...

...

...

On Wed, Nov 15, 2017 at 10:44 AM, Denis Kudriashov < dionisiydk@gmail.com> wrote:

...

2017-11-15 1:49 GMT+01:00 Sean P. DeNigris <sean@clipperadams.com>:

...

Ben Coman wrote

...

Or it could go to Amber, half-way between green & red to mean probably correct.

Ha ha.

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run addresses all points on both sides, no?

Except that sometimes we are debugging slow test and running it second time automatically after "proceed" can be not appropriate. We are talking about single test run. If user have any doubts about result It is his responsibility to rerun the test. User knows what he is doing when he debug and fix the test. No intelligence is required here.

And anyway current fix just provides consistent behaviour to debugging from explicit breakpoint/halt. In that case the result was always in sync with debug session.

...

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Sm alltalk-Users-f1310670.html

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

2017-11-15 17:16 GMT+01:00 Richard Sargent < richard.sargent@gemtalksystems.com>:

And in this thread only Richard was against current logic.

...

I am against colouring it green without having run the test without interruption or error. I don't think that is the current logic.

Fix is already in Pharo 7. So it is current logic now. And it was same for manual halt (instead of failure).

On Wed, Nov 15, 2017 at 2:41 AM, Denis Kudriashov <dionisiydk@gmail.com> wrote:

...

2017-11-15 11:08 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com>:

...

On Wed, Nov 15, 2017 at 11:06 AM, Denis Kudriashov <dionisiydk@gmail.com

...

wrote:

...

2017-11-15 11:00 GMT+01:00 Guillermo Polito <guillermopolito@gmail.com> :

...

And just putting it back to gray? As "not run"?

We can implement any logic. Personally I need current behaviour.

But it is not about you personally. It is about implementing the most common and the less strange for newcomers.

To know what is the most common case people should tell personal opinion. And in this thread only Richard was against current logic.

...

...

...

On Wed, Nov 15, 2017 at 10:44 AM, Denis Kudriashov < dionisiydk@gmail.com> wrote:

...

2017-11-15 1:49 GMT+01:00 Sean P. DeNigris <sean@clipperadams.com>:

...

Ben Coman wrote

...

Or it could go to Amber, half-way between green & red to mean probably correct.

Ha ha.

Again, it seems that just automatically rerunning the test immediately after a human-manipulated run and setting the color based on that second run addresses all points on both sides, no?

Except that sometimes we are debugging slow test and running it second time automatically after "proceed" can be not appropriate. We are talking about single test run. If user have any doubts about result It is his responsibility to rerun the test. User knows what he is doing when he debug and fix the test. No intelligence is required here.

And anyway current fix just provides consistent behaviour to debugging from explicit breakpoint/halt. In that case the result was always in sync with debug session.

...

----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

--

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>*

*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>

So after a restart we would expect green

Yes. Add having the test tool able to automate the tear-down and restart of the test would be a good automation. On Fri, Nov 10, 2017 at 11:41 AM, stephan <stephan@stack.nl> wrote:

On 10-11-17 18:56, Richard Sargent wrote:

...

That would be fine. The point is that, without running the test in its entirety, from start to finish, without interruption, error, or failure, one cannot claim success.

So after a restart we whould expect green

Stephan

Love this: “Many people in our industry, our profession have a disturbing tendency to be vague and practice self-deception. Take the concept of a "bug" as an example. They don't "fly into" our code; we *put* them in it. *We* put them in. That's why they should be called errors or defects.” Or, sometimes, “lazy stupidity” (particularly in my case). And in many others “don’t really know what I’m doing, really, so I copied/pasted a bunch of code off random websites until it “sorta/kinda” worked.” The worst is that I’ve had a number of team leads and architects on projects for whom the last is the most accurate. Andrew

And give me concrete steps to see that it is not working (if you will find it) 2017-11-09 23:47 GMT+01:00 Tim Mackinnon <tim@testit.works>:

Thanks Denis - I'll try my simple TDD test case in Pharo 7 (I haven't tried it yet - probably time to).

I think it is correct to show green if you fix your problem in the debugger and then proceed and it passes all assertions. But it should equally fail if you proceed and it asserts false or craps out.

Strange it would vary in 6.1 - but I'll check.

Tim

Sent from my iPhone

Sent from my iPhone On 9 Nov 2017, at 19:54, Denis Kudriashov <dionisiydk@gmail.com> wrote:

In Pharo 7 it works.

2017-11-09 18:29 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Thanks for looking at this - there is an issue however - when you apply that change (at least in a Pharo 6.1 image) - it shows green even when a test fails? So I think its turned one problem into the opposite one.

Unfortunately I haven’t got a chance to look a bit deeper to help - but it might be worth rolling back this change for now. We should fix it though - and the answer must be in the area you have identified.

tim

On 9 Nov 2017, at 12:43, Denis Kudriashov <dionisiydk@gmail.com> wrote:

And now it is in latest Pharo

2017-11-09 12:16 GMT+01:00 Denis Kudriashov <dionisiydk@gmail.com>:

...

Hi Tim.

Fix is here 20661-Fixing-test-from-debugger-should-mark-test-as-gre en-when-proceed <https://github.com/pharo-project/pharo/pull/456> . Thank's for reporting. It forces me to fix. I always noticed it but never take it seriously :)

2017-11-09 11:32 GMT+01:00 Tim Mackinnon <tim@testit.works>:

...

Hi - I really like the build in test runner in the Pharo browsers, and I was preparing a talk to show how great TDD is in Pharo and how we aren’t ashamed of our debugger when testing (it augments the experience in fact - letting you poke around and get your thoughts straight).

However - if I pick rerun in the test runner debugger - and step through a test and then correct the failing code, and hit resume - the browser always shows a red failure, even though the execution is now correct. I have to run the test again.

This doesn’t seem right to me - are we missing a success event or something?

Tim