[Pharo-project] VM unstable
we still get very undeterministic results from our build server. I relaunched the Pharo-2.0-Tests several times with the exact same VM & Image configuration. BUT, the outcome of the test results varies each time a little bit |( https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/176/testReport/ (windows crashed) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/177/testReport/ (windows crashed, KernelTests.Chronology.DateTest.testDateAndTimeNow failing...) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/178/testReport/ (KernelTests.Chronology.DateTest.testDateAndTimeNow on all) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/179/testReport/ (!BOOM!) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/180/testReport/ ... ditto ... https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/181/testReport/ (!BOOM! plus windows crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/182/testReport/ (!BOOM! plus windows AND linux crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/183/testReport/ (lots of failures all systems running, no chronology tests failing) ... so we still have to investigate this instability...
Camillo Bruni-3 wrote
we still get very undeterministic results from our build server.
What are the most stable Stack/Jit VMs we have right now? -- View this message in context: http://forum.world.st/VM-unstable-tp4672363p4672366.html Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
On Feb 27, 2013, at 12:57 AM, Camillo Bruni <camillobruni@gmail.com> wrote:
we still get very undeterministic results from our build server. I relaunched the Pharo-2.0-Tests several times with the exact same VM & Image configuration.
BUT, the outcome of the test results varies each time a little bit |(
https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/176/testReport/ (windows crashed) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/177/testReport/ (windows crashed, KernelTests.Chronology.DateTest.testDateAndTimeNow failing...) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/178/testReport/ (KernelTests.Chronology.DateTest.testDateAndTimeNow on all) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/179/testReport/ (!BOOM!) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/180/testReport/ ... ditto ... https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/181/testReport/ (!BOOM! plus windows crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/182/testReport/ (!BOOM! plus windows AND linux crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/183/testReport/ (lots of failures all systems running, no chronology tests failing) ...
so we still have to investigate this instabilityâ¦
Yes what would be good is to find a reproducible case. Stef
On Feb 27, 2013, at 12:57 AM, Camillo Bruni <camillobruni@gmail.com> wrote:
we still get very undeterministic results from our build server. I relaunched the Pharo-2.0-Tests several times with the exact same VM & Image configuration.
BUT, the outcome of the test results varies each time a little bit |(
https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/176/testReport/ (windows crashed) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/177/testReport/ (windows crashed, KernelTests.Chronology.DateTest.testDateAndTimeNow failing...) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/178/testReport/ (KernelTests.Chronology.DateTest.testDateAndTimeNow on all) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/179/testReport/ (!BOOM!) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/180/testReport/ ... ditto ... https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/181/testReport/ (!BOOM! plus windows crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/182/testReport/ (!BOOM! plus windows AND linux crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/183/testReport/ (lots of failures all systems running, no chronology tests failing) ...
so we still have to investigate this instability...
It is strange. Form the image side: First problem: Build #177 (Feb 26, 2013 11:25:31 PM) Good: Build #176 (Feb 25, 2013 5:40:31 PM) -> No Image change in between. Latest was good This means that the image is not to blame? But the VM: 91 is from 24-Feb-2013 21:25 92 (last VM build): 25-Feb-2013 17:50 So if the clocks are in sync. we ran 91 for the last good and the last fail. Hmm⦠strange. The first thing is that we need to make issues district. 1) We now there is some NB related Crash. Let's not run that test. (and set up a CI on all architectures for NBâ¦) Marcus
On Feb 27, 2013, at 8:53 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Feb 27, 2013, at 12:57 AM, Camillo Bruni <camillobruni@gmail.com> wrote:
we still get very undeterministic results from our build server. I relaunched the Pharo-2.0-Tests several times with the exact same VM & Image configuration.
BUT, the outcome of the test results varies each time a little bit |(
https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/176/testReport/ (windows crashed) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/177/testReport/ (windows crashed, KernelTests.Chronology.DateTest.testDateAndTimeNow failing...) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/178/testReport/ (KernelTests.Chronology.DateTest.testDateAndTimeNow on all) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/179/testReport/ (!BOOM!) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/180/testReport/ ... ditto ... https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/181/testReport/ (!BOOM! plus windows crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/182/testReport/ (!BOOM! plus windows AND linux crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/183/testReport/ (lots of failures all systems running, no chronology tests failing) ...
so we still have to investigate this instability...
It is strange. Form the image side:
Even stranger https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/ My theory: strange clock related issues due to virtualization? But then why did it fail on the mac? Marcus
On 27 Feb 2013, at 09:31, Marcus Denker <marcus.denker@inria.fr> wrote:
On Feb 27, 2013, at 8:53 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Feb 27, 2013, at 12:57 AM, Camillo Bruni <camillobruni@gmail.com> wrote:
we still get very undeterministic results from our build server. I relaunched the Pharo-2.0-Tests several times with the exact same VM & Image configuration.
BUT, the outcome of the test results varies each time a little bit |(
https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/176/testReport/ (windows crashed) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/177/testReport/ (windows crashed, KernelTests.Chronology.DateTest.testDateAndTimeNow failing...) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/178/testReport/ (KernelTests.Chronology.DateTest.testDateAndTimeNow on all) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/179/testReport/ (!BOOM!) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/180/testReport/ ... ditto ... https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/181/testReport/ (!BOOM! plus windows crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/182/testReport/ (!BOOM! plus windows AND linux crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/183/testReport/ (lots of failures all systems running, no chronology tests failing) ...
so we still have to investigate this instability...
It is strange. Form the image side:
Even stranger
https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/
My theory: strange clock related issues due to virtualisation?
To rule out clock problems, you should check the SNTP client that I announced - this was exactly why I wrote it. The ZTimestamp unit tests will only pass if the clock difference is not greater than 2 seconds. If the clock is wrong on the machine, on Linux you can do sudo ntpdate pool.ntp.org to manually force a date/time sync.
But then why did it fail on the mac?
Marcus
On 2013-02-27, at 09:31, Marcus Denker <marcus.denker@inria.fr> wrote:
On Feb 27, 2013, at 8:53 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Feb 27, 2013, at 12:57 AM, Camillo Bruni <camillobruni@gmail.com> wrote:
we still get very undeterministic results from our build server. I relaunched the Pharo-2.0-Tests several times with the exact same VM & Image configuration.
BUT, the outcome of the test results varies each time a little bit |(
https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/176/testReport/ (windows crashed) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/177/testReport/ (windows crashed, KernelTests.Chronology.DateTest.testDateAndTimeNow failing...) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/178/testReport/ (KernelTests.Chronology.DateTest.testDateAndTimeNow on all) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/179/testReport/ (!BOOM!) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/180/testReport/ ... ditto ... https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/181/testReport/ (!BOOM! plus windows crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/182/testReport/ (!BOOM! plus windows AND linux crash) https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/183/testReport/ (lots of failures all systems running, no chronology tests failing) ...
so we still have to investigate this instability...
It is strange. Form the image side:
Even stranger
https://ci.inria.fr/pharo/job/Pharo-2.0-Tests/
My theory: strange clock related issues due to virtualization? But then why did it fail on the mac?
I thought the same... but there is a time related problem see https://code.google.com/p/pharo/issues/detail?id=7600
Marcus we were looking at delay and dateAndTime with igor and I wonder if there is not a relationship between all the date and time delays and the fact that you put the cleanForRelease systematically for each build. We may have found a bug in/related startTimerEventLoop when you look at the code you should get really concentrated and fresh ⦠:( Stef
On Feb 27, 2013, at 10:38 PM, stephane ducasse <stephane.ducasse@free.fr> wrote:
Marcus we were looking at delay and dateAndTime with igor and I wonder if there is not a relationship between all the date and time delays and the fact that you put the cleanForRelease systematically for each build.
yes, that could be. We can turn it off again.. and move that to 3.0 But we really need to have a process of building the image that does not require *anything* to be done before the relase. Because anything, be it even the most trivial thing, will break if you do it only once every year.
We may have found a bug in/related startTimerEventLoop
when you look at the code you should get really concentrated and fresh ⦠:(
Stef
On Feb 28, 2013, at 8:08 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Feb 27, 2013, at 10:38 PM, stephane ducasse <stephane.ducasse@free.fr> wrote:
Marcus we were looking at delay and dateAndTime with igor and I wonder if there is not a relationship between all the date and time delays and the fact that you put the cleanForRelease systematically for each build.
yes, that could be. We can turn it off again.. and move that to 3.0
But we really need to have a process of building the image that does not require *anything* to be done before the relase. Because anything, be it even the most trivial thing, will break if you do it only once every year.
I think that it may just solve the dateAndTime delay problems Now it shows that there is something wrong in the logic of the cleanup and probably on the delay In adition there is probabbly a bug that crash the vm
We may have found a bug in/related startTimerEventLoop
when you look at the code you should get really concentrated and fresh ⦠:(
Stef
On Feb 28, 2013, at 9:29 AM, stephane ducasse <stephane.ducasse@free.fr> wrote:
On Feb 28, 2013, at 8:08 AM, Marcus Denker <marcus.denker@inria.fr> wrote:
On Feb 27, 2013, at 10:38 PM, stephane ducasse <stephane.ducasse@free.fr> wrote:
Marcus we were looking at delay and dateAndTime with igor and I wonder if there is not a relationship between all the date and time delays and the fact that you put the cleanForRelease systematically for each build.
yes, that could be. We can turn it off again.. and move that to 3.0
But we really need to have a process of building the image that does not require *anything* to be done before the relase. Because anything, be it even the most trivial thing, will break if you do it only once every year.
I think that it may just solve the dateAndTime delay problems Now it shows that there is something wrong in the logic of the cleanup and probably on the delay
Ok, I added a removal for the cleanup.
In adition there is probabbly a bug that crash the vm
participants (5)
-
Camillo Bruni -
Marcus Denker -
Sean P. DeNigris -
stephane ducasse -
Sven Van Caekenberghe