They seem like the right priorities. We have a work around to report it manually for now. Just wanted to get the idea on the list while I thought of it. cheers -ben On Sun, Aug 23, 2015 at 6:15 PM, stepharo <stepharo@free.fr> wrote:
HI ben
you are right. We are all running like mad. We are focusing on the bootstrap because guille will leave to brussels end of the month. Esteban is fighting with FFI to get migrating to Spur. But yes this issue is a real one. Stef
Le 23/8/15 04:08, Ben Coman a écrit :
Is there somewhere that statistics of failures from the CI validation report are gathered?
We've all experienced transient CI-monkey validation failures unrelated to our submitted fix, where just resubmitting to the monkey without change works fine. The problem is:
* this may have been introduced several builds ago, but the transient failure passed the monkey at the time
* it can be hard to identify which build introduced a transient error
* doubly hard if the failure is due to some CI infrastructure change, or one particular build server
* its hard to get an overview of how many issues get the same failure - is this a "once in six months" failure or is it happening 50% of the time in every issue.
* it takes longer for an individual to observes a false-failure is common enough to worry about, than if reports were merged. Since different slices should produce different errors, failures common across all slices should stand out in statistics.
If such statistics are not currently available, in cases where a simple resubmission passed the monkey, should we:
* have an issue for each such false-failure, to get some early indication from manually track occurrences
* where such an issue hasn't yet been created, first report comes to the list
------ So, I just got the following false-failures in separate runs for: https://pharo.fogbugz.com/f/cases/16265 * ProcessTerminateBug>>#testTerminationDuringUnwind * a CISmallIntRule: Timeout (after 0:00:40:00) occured while loading...
Anyone else?
cheers -ben
P.S. I understand some errors are filtered. Can this list be output as part of the validation report (or is it already) ?