[Pharo-dev] CI Build order and strange "latest" Pharo 7 effect

Guillermo Polito guillermopolito at gmail.com
Fri Oct 13 05:37:12 EDT 2017


Well, apparently there is an option to disable concurrent builds, and we
should be able to make it work only for the development branch.

https://jenkins.io/doc/book/pipeline/syntax/#options
https://stackoverflow.com/questions/41492688/how-to-limit-jenkins-concurrent-multibranch-pipeline-builds
https://issues.jenkins-ci.org/browse/JENKINS-46593

I've opened the following issue:

https://pharo.fogbugz.com/f/cases/20541/Diable-concurrent-builds-for-the-development-branch

and the following PR:

https://github.com/pharo-project/pharo/pull/363

Could you review/check? I'm in the middle of a meeting right now :)


On Fri, Oct 13, 2017 at 9:40 AM, Guillermo Polito <guillermopolito at gmail.com
> wrote:

>
>
> On Thu, Oct 12, 2017 at 9:23 PM, Torsten Bergmann <astares at gmx.de> wrote:
>
>> Hi Guille,
>>
>> I understand that we want run the pull requests for contributions in
>> parallel. But do we want to run also  the main build for
>> the daily official relase in parallel to have such side effects? I doubt
>> that.
>>
>> As I wrote it is not only an issue of the file server. The zero conf does
>> not give you the latest
>> although available (which is bad if you want to do the next contribution)
>> and we inform about
>> later build earlier than previous builds, etc.
>>
>
> Yes, I agree that this is a real issue. The reason is the following: every
> build, all artifacts are uploaded to the file server + a copy named
> latest.zip, which is used by zeroconf scripts. latest.zip is overwritten on
> every build. This means that the last build to finish is the last that
> writes to latest.zip.
>
>
>>
>> Sure it would be "nice" if our future Pharo 12 would already be available
>> before we release Pharo 7 - but it
>> still is not correct and confuses people ;)
>>
>> So is there anything that can be done to improve/solve this?
>>
>
> I'll check the options and come back to everybody here.
>
>
>>
>> Thanks
>> T.
>>
>>
>> Am 12.10.2017 um 16:12 schrieb Guillermo Polito:
>>
>>>
>>>
>>> On Thu, Oct 12, 2017 at 4:01 PM, Norbert Hartl <norbert at hartl.name
>>> <mailto:norbert at hartl.name>> wrote:
>>>
>>>
>>>     > Am 12.10.2017 um 14:23 schrieb Torsten Bergmann <astares at gmx.de
>>>     <mailto:astares at gmx.de>>:
>>>
>>>     >
>>>     > Hi,
>>>     >
>>>     > today there was an interesting effect visible for the Pharo
>>>     build on CI for Pharo 7:
>>>     >
>>>     > Esteban merged three contributions for Pharo 7 into #development
>>>     (see [1]). All three merges on git triggered the CI.
>>>     > For each a separate build for Pharo 7 started on CI independently.
>>>     >
>>>     > While usually one would expect the builds in regular build order
>>>     183, 184, 185 the CI job for building 185 job was faster
>>>     > (as you can see on the attached screenshot).
>>>     >
>>>     > So the 185 series of images was copied first to the file server
>>>     [3] because the job was already finished.
>>>     > Afterwards 183 build completed on CI and these images were
>>>     copied to the file server with a newer timestamp.
>>>     >
>>>     > So you have the order 185, 183 with 183 having the files with
>>>     the newest timestamp (but not being the most recent state of Pharo).
>>>     >
>>>     > This leads to several unfortunate effects:
>>>     >  1. When you sort on file server [3] descending by date the
>>>     build 183 images are newer than the build 185 images
>>>     >  2. When you use PharoLauncher to download "latest-32" you get
>>>     an 183 image instead of the "real latest" 185 image
>>>     >  3. When you ask Zeroconf to download the latest using
>>>     http://get.pharo.org/70+vm you also get 183 instead of the already
>>>     >     available 185
>>>     >
>>>     > So as the CI build of an older "update" (here 183) finished
>>>     later than the most recent version (here 185)
>>>     > you get 183 as "latest" instead of 185.
>>>     >
>>>     > Any ideas how we could avoid such a side effect? Maybe a chain
>>>     on the builds so 183 has to finished before 184 and 185 could be
>>>     > built?
>>>     >
>>>     I don't understand why it is working like this. Usually only build
>>>     for a job is active, others will be queued. So the problem here is
>>>     that
>>>     the jenkins spawns more builds in parallel for the same job.
>>>
>>>
>>> Yes, the thing is that the same job is being used to build the
>>> development branch and the pull requests.
>>> If you think about it, this is the same setup as in travis.
>>>
>>> The thing is that this does not bother me too much, considering the
>>> tradeoffs.
>>>
>>> - I want pull request to run in parallel.
>>> - I want to have less things to maintain :)
>>>
>>> Now,
>>>  - we could split the job in two (integration vs pr validation). But
>>> this requires setting up a separate jenkins because otherwise they share
>>> the same slave configuration :/. And there is no way in the actual state of
>>> the plugins we use to limit the number of parallel builds...
>>>  - we could try to make a build wait until a build with lower build
>>> number is building...
>>>  - we could also forbid Esteban to push on the merge button while a
>>> build is running
>>> But doing any of those is a lot of work...
>>>
>>> - Moreover, I think the problem is that we are using the file server as
>>> something it is not. The file server is... a file server.
>>> Ideally, I'd like to attach meta-data to a release/build artifact, and
>>> be able to browse/search/sort using that metadata. Something alike
>>> artifactory or bintray.
>>>
>>> - Also, I don't know how the pharo launcher is getting the latest image,
>>> but if instead of using the file timestamp it uses the build number, then
>>> this may be fixed, isn't it?
>>>
>>>
>>>     Norbert
>>>
>>>     > Thx
>>>     > T.
>>>     >
>>>     >
>>>     >
>>>     > [1] https://github.com/pharo-project/pharo/commits/development
>>>     <https://github.com/pharo-project/pharo/commits/development>
>>>     > [2]
>>>     https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20p
>>> ull%20request%20and%20branch%20Pipeline/job/development
>>>     <https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20
>>> pull%20request%20and%20branch%20Pipeline/job/development>
>>>     > [3] http://files.pharo.org/image/70/
>>>     <http://files.pharo.org/image/70/>
>>>     > <buildOrder.png>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>> Guille Polito
>>>
>>> Research Engineer
>>>
>>> Centre de Recherche en Informatique, Signal et Automatique de Lille
>>>
>>> CRIStAL - UMR 9189
>>>
>>> French National Center for Scientific Research - _http://www.cnrs.fr_
>>>
>>>
>>> *Web:* _http://guillep.github.io_
>>>
>>> *Phone: *+33 06 52 70 66 13
>>>
>>>
>>
>
>
> --
>
>
>
> Guille Polito
>
> Research Engineer
>
> Centre de Recherche en Informatique, Signal et Automatique de Lille
>
> CRIStAL - UMR 9189
>
> French National Center for Scientific Research - *http://www.cnrs.fr
> <http://www.cnrs.fr>*
>
>
> *Web:* *http://guillep.github.io* <http://guillep.github.io>
>
> *Phone: *+33 06 52 70 66 13 <+33%206%2052%2070%2066%2013>
>



-- 



Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - *http://www.cnrs.fr
<http://www.cnrs.fr>*


*Web:* *http://guillep.github.io* <http://guillep.github.io>

*Phone: *+33 06 52 70 66 13
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20171013/2a602643/attachment-0002.html>


More information about the Pharo-dev mailing list