The 6.1 Space Monkey is unhappy
Hi Esteban & Marcus, I'm getting repeated validation failures for: https://pharo.manuscript.com/f/cases/21431 It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted. Do you know if this is happening on other tests? Thanks very much, Alistair
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests?
I saw that Saturday but decided to wait till Monday (weekends are important..). So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method. Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasons⦠And I have no idea why. (And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not. The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.) Marcus
Hi Marcus, On 5 March 2018 at 09:23, Marcus Denker <marcus.denker@inria.fr> wrote:
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests?
I saw that Saturday but decided to wait till Monday (weekends are important..).
So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method.
Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasonsâ¦
And I have no idea why.
(And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not.
The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.)
Thanks for the update. I took a look at the failures and it appears that BehaviorTest>>testBehaviorRespectsPolymorphismWithTraitBehavior ClassDescriptionTest>>testClassDescriptionRespectsPolymorphismWithTraitDescription ClassTest>>testClassRespectsPolymorphismWithTrait are all failing due to changes in Fuel - methods were changed from traits to local methods. I'll try and track down when the change was introduced and by whom, but hopefully it is just a matter of updating the tests. Cheers, Alistair
On 5 Mar 2018, at 10:27, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Marcus,
On 5 March 2018 at 09:23, Marcus Denker <marcus.denker@inria.fr> wrote:
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests?
I saw that Saturday but decided to wait till Monday (weekends are important..).
So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method.
Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasonsâ¦
And I have no idea why.
(And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not.
The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.)
Thanks for the update.
Oh, and it was completely unrelated. Your change is for Pharo6...
I took a look at the failures and it appears that
BehaviorTest>>testBehaviorRespectsPolymorphismWithTraitBehavior ClassDescriptionTest>>testClassDescriptionRespectsPolymorphismWithTraitDescription ClassTest>>testClassRespectsPolymorphismWithTrait
are all failing due to changes in Fuel - methods were changed from traits to local methods.
Yes, the problem is that the monkey (the contribution checker) fails as soon as there are errors even in the main image. The last Pharo6 has these tests failing, so now all contribution checks for Pharo6 fail. What needs to be done? -> your change can be accepted as we know it does not fail more fixes -> then we need fix the tests in Pharo6 -> in a perfect world we would update the slice checker to only fail for now test failing⦠(it used to be lille thatâ¦). As I said: this is a full time job⦠Marcus
2018-03-05 10:54 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 5 Mar 2018, at 10:27, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Marcus,
On 5 March 2018 at 09:23, Marcus Denker <marcus.denker@inria.fr> wrote:
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests?
I saw that Saturday but decided to wait till Monday (weekends are important..).
So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method.
Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasonsâ¦
And I have no idea why.
(And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not.
The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.)
Thanks for the update.
Oh, and it was completely unrelated. Your change is for Pharo6...
I took a look at the failures and it appears that
BehaviorTest>>testBehaviorRespectsPolymorphismWithTraitBehavior ClassDescriptionTest>>testClassDescriptionRespectsPolymorphismWithTraitDescription ClassTest>>testClassRespectsPolymorphismWithTrait
are all failing due to changes in Fuel - methods were changed from traits to local methods.
Yes, the problem is that the monkey (the contribution checker) fails as soon as there are errors even in the main image.
The last Pharo6 has these tests failing, so now all contribution checks for Pharo6 fail.
What needs to be done?
-> your change can be accepted as we know it does not fail more fixes -> then we need fix the tests in Pharo6 -> in a perfect world we would update the slice checker to only fail for now test failing⦠(it used to be lille thatâ¦).
I must say that It made the validation two times slower, fragile and led to the hiding of problems instead of solving them... -- Pavel
As I said: this is a full time jobâ¦
Marcus
What needs to be done?
-> your change can be accepted as we know it does not fail more fixes -> then we need fix the tests in Pharo6 -> in a perfect world we would update the slice checker to only fail for now test failing⦠(it used to be lille thatâ¦).
I must say that It made the validation two times slower, fragile and led to the hiding of problems instead of solving themâ¦
Yes, true. Marcus
Hi pavel when I'm back can you explain to me because I did not get it :). Stef On Mon, Mar 5, 2018 at 11:07 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
2018-03-05 10:54 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 5 Mar 2018, at 10:27, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Marcus,
On 5 March 2018 at 09:23, Marcus Denker <marcus.denker@inria.fr> wrote:
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests?
I saw that Saturday but decided to wait till Monday (weekends are important..).
So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method.
Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasonsâ¦
And I have no idea why.
(And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not.
The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.)
Thanks for the update.
Oh, and it was completely unrelated. Your change is for Pharo6...
I took a look at the failures and it appears that
BehaviorTest>>testBehaviorRespectsPolymorphismWithTraitBehavior ClassDescriptionTest>>testClassDescriptionRespectsPolymorphismWithTraitDescription ClassTest>>testClassRespectsPolymorphismWithTrait
are all failing due to changes in Fuel - methods were changed from traits to local methods.
Yes, the problem is that the monkey (the contribution checker) fails as soon as there are errors even in the main image.
The last Pharo6 has these tests failing, so now all contribution checks for Pharo6 fail.
What needs to be done?
-> your change can be accepted as we know it does not fail more fixes -> then we need fix the tests in Pharo6 -> in a perfect world we would update the slice checker to only fail for now test failing⦠(it used to be lille thatâ¦).
I must say that It made the validation two times slower, fragile and led to the hiding of problems instead of solving them...
-- Pavel
As I said: this is a full time jobâ¦
Marcus
2018-03-05 14:14 GMT+01:00 Stephane Ducasse <stepharo.self@gmail.com>:
Hi pavel
when I'm back can you explain to me because I did not get it :).
:) it is simple. If you want to ignore issues that the slice is not adding, you need to know which of them to ignore. That's why the original monkey run all the validations twice - first time to collect a list of failing tests in the fresh unchanged image and then with the slice or configuration loaded. So it doubled the issue validation time. There are several alternative strategies like to cache the failing test results for every build and use them for validations but the best strategy is simply to keep the amount of failing tests in the clean image on the zero level and force people to keep the system clean. The original monkey hasn't exposed the list of ignored tests. So it was possible that some test was ignored because of a temporal network issue but for the second time it failed for o good reason and you even didn't know. Cheers, -- Pavel
Stef
On Mon, Mar 5, 2018 at 11:07 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
2018-03-05 10:54 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 5 Mar 2018, at 10:27, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Marcus,
On 5 March 2018 at 09:23, Marcus Denker <marcus.denker@inria.fr> wrote:
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests?
I saw that Saturday but decided to wait till Monday (weekends are important..).
So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method.
Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasonsâ¦
And I have no idea why.
(And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not.
The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.)
Thanks for the update.
Oh, and it was completely unrelated. Your change is for Pharo6...
I took a look at the failures and it appears that
BehaviorTest>>testBehaviorRespectsPolymorphismWithTraitBehavior ClassDescriptionTest>>testClassDescriptionRespectsPolymorphismWithTraitDescription ClassTest>>testClassRespectsPolymorphismWithTrait
are all failing due to changes in Fuel - methods were changed from traits to local methods.
Yes, the problem is that the monkey (the contribution checker) fails as soon as there are errors even in the main image.
The last Pharo6 has these tests failing, so now all contribution checks for Pharo6 fail.
What needs to be done?
-> your change can be accepted as we know it does not fail more fixes -> then we need fix the tests in Pharo6 -> in a perfect world we would update the slice checker to only fail for now test failing⦠(it used to be lille thatâ¦).
I must say that It made the validation two times slower, fragile and led to the hiding of problems instead of solving them...
-- Pavel
As I said: this is a full time jobâ¦
Marcus
this sounds a bit like smart-test (that tries to identify the tests to re-run after a method is changed) Smarttest can work - on dynamic data: first run the tests to know what test cause what methods to be executed; then if such method is modified, re-run the associated tests - on static data: analyse senders of a method recursively to find the tests that could possibly call it 1- we are doing now experiment to find out what method (static or dynamic) is best. Surprisingly, the static method gives very good results (often better than dynamic) even if we all know that senders-of is not always very precise in smalltalk 2- may be something similar could be used to find out in the monkey what failing tests are caused by a new slice ? (going back from the changes in the slice to the tests, or forward from the failing test toward the changes in the slice ...) nicolas On 05/03/2018 14:29, Pavel Krivanek wrote:
2018-03-05 14:14 GMT+01:00 Stephane Ducasse <stepharo.self@gmail.com>:
Hi pavel
when I'm back can you explain to me because I did not get it :). :) it is simple. If you want to ignore issues that the slice is not adding, you need to know which of them to ignore. That's why the original monkey run all the validations twice - first time to collect a list of failing tests in the fresh unchanged image and then with the slice or configuration loaded. So it doubled the issue validation time. There are several alternative strategies like to cache the failing test results for every build and use them for validations but the best strategy is simply to keep the amount of failing tests in the clean image on the zero level and force people to keep the system clean. The original monkey hasn't exposed the list of ignored tests. So it was possible that some test was ignored because of a temporal network issue but for the second time it failed for o good reason and you even didn't know.
Cheers, -- Pavel
Stef
On Mon, Mar 5, 2018 at 11:07 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
2018-03-05 10:54 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 5 Mar 2018, at 10:27, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Marcus,
On 5 March 2018 at 09:23, Marcus Denker <marcus.denker@inria.fr> wrote:
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests? I saw that Saturday but decided to wait till Monday (weekends are important..).
So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method.
Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasonsâ¦
And I have no idea why.
(And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not.
The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.)
Thanks for the update.
Oh, and it was completely unrelated. Your change is for Pharo6...
I took a look at the failures and it appears that
BehaviorTest>>testBehaviorRespectsPolymorphismWithTraitBehavior ClassDescriptionTest>>testClassDescriptionRespectsPolymorphismWithTraitDescription ClassTest>>testClassRespectsPolymorphismWithTrait
are all failing due to changes in Fuel - methods were changed from traits to local methods.
Yes, the problem is that the monkey (the contribution checker) fails as soon as there are errors even in the main image.
The last Pharo6 has these tests failing, so now all contribution checks for Pharo6 fail.
What needs to be done?
-> your change can be accepted as we know it does not fail more fixes -> then we need fix the tests in Pharo6 -> in a perfect world we would update the slice checker to only fail for now test failing⦠(it used to be lille thatâ¦).
I must say that It made the validation two times slower, fragile and led to the hiding of problems instead of solving them...
-- Pavel
As I said: this is a full time jobâ¦
Marcus
-- Nicolas Anquetil RMod team -- Inria Lille
On Mon, Mar 5, 2018 at 2:29 PM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
2018-03-05 14:14 GMT+01:00 Stephane Ducasse <stepharo.self@gmail.com>:
Hi pavel
when I'm back can you explain to me because I did not get it :).
:) it is simple. If you want to ignore issues that the slice is not adding, you need to know which of them to ignore. That's why the original monkey run all the validations twice - first time to collect a list of failing tests in the fresh unchanged image and then with the slice or configuration loaded. So it doubled the issue validation time.
It sounds good :)
There are several alternative strategies like to cache the failing test results for every build and use them for validations but the best strategy is simply to keep the amount of failing tests in the clean image on the zero level and force people to keep the system clean.
Yes!
The original monkey hasn't exposed the list of ignored tests. So it was possible that some test was ignored because of a temporal network issue but for the second time it failed for o good reason and you even didn't know.
Cheers, -- Pavel
Stef
On Mon, Mar 5, 2018 at 11:07 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
2018-03-05 10:54 GMT+01:00 Marcus Denker <marcus.denker@inria.fr>:
On 5 Mar 2018, at 10:27, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Marcus,
On 5 March 2018 at 09:23, Marcus Denker <marcus.denker@inria.fr> wrote:
On 5 Mar 2018, at 09:16, Alistair Grant <akgrant0710@gmail.com> wrote:
Hi Esteban & Marcus,
I'm getting repeated validation failures for:
https://pharo.manuscript.com/f/cases/21431
It's the same set of tests that fail each time, and as far as I can tell they have nothing to do with the patch I submitted.
Do you know if this is happening on other tests?
I saw that Saturday but decided to wait till Monday (weekends are important..).
So: no, I have *no* idea what happened. From one CI run to the next, suddenly around 160 tests related to Calypso started failing due to a missing method.
Now starting from sometime today, this problem stoped. The last failing PR checks fail due to different reasonsâ¦
And I have no idea why.
(And yes, we al know that 1) the PR checks need more compute power, too slow 2) we *need* to track down the reason why still *a lot* of times the PR fails even though it should not.
The problem is that just keeping a build alive of this kind is a full time job.. that we have nobody doing, so many many people do as much as they can and we hope it will get betterâ¦.)
Thanks for the update.
Oh, and it was completely unrelated. Your change is for Pharo6...
I took a look at the failures and it appears that
BehaviorTest>>testBehaviorRespectsPolymorphismWithTraitBehavior ClassDescriptionTest>>testClassDescriptionRespectsPolymorphismWithTraitDescription ClassTest>>testClassRespectsPolymorphismWithTrait
are all failing due to changes in Fuel - methods were changed from traits to local methods.
Yes, the problem is that the monkey (the contribution checker) fails as soon as there are errors even in the main image.
The last Pharo6 has these tests failing, so now all contribution checks for Pharo6 fail.
What needs to be done?
-> your change can be accepted as we know it does not fail more fixes -> then we need fix the tests in Pharo6 -> in a perfect world we would update the slice checker to only fail for now test failing⦠(it used to be lille thatâ¦).
I must say that It made the validation two times slower, fragile and led to the hiding of problems instead of solving them...
-- Pavel
As I said: this is a full time jobâ¦
Marcus
participants (5)
-
Alistair Grant -
Marcus Denker -
Nicolas Anquetil -
Pavel Krivanek -
Stephane Ducasse