June 2018 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [CI] Cause of the random failures in the CI

by Alistair Grant

Hi Cyril & Guille, > On Tue, Jun 19, 2018 at 5:06 PM Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com> wrote: >> >> > Slightly off-topic, but I'm also wondering why the test scripts >> > download the VM again? Why not just use the one that is already in >> > vmtarget? >> > >> >> Because it is a vm linux and tests are executed on OSX and Windows too. On Tue, 19 Jun 2018 at 18:49, Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > Thanks Cyril, thanks for pushing. I've only given you permissions to re-run the build, so keep yourself and Vincent all the credit! > > > Slightly off-topic, but I'm also wondering why the test scripts > > download the VM again? Why not just use the one that is already in > > vmtarget? > > Yes, on one side there is what Cyril says, we need to download a new VM on each new slave, because > - we build the image on a single slave (unix) > - we then transfer it to different slaves (mac, windows) to run the tests. > > This is like that because originally running the bootstrap was about three times as expensive. > Right now we are on 7minutes to create the minimal image and a couple more to load the rest on top of it. Thanks for your explanations, I hadn't considered these requirements. Apart from avoiding duplication and resource waste by re-running the bootstrap, testing a single image does a sanity check that we haven't broken cross-platform compatibility in some way. Thanks again, Alistair > There is another thing also: we always assumed that slaves do not share disk, so either we re-downloaded a new unix vm or we shared it. > Apparently our assumptions weren't right?

June 19, 2018

Re: [Pharo-dev] [CI] Cause of the random failures in the CI

by Guillermo Polito

Thanks Cyril, thanks for pushing. I've only given you permissions to re-run the build, so keep yourself and Vincent all the credit! > Slightly off-topic, but I'm also wondering why the test scripts > download the VM again? Why not just use the one that is already in > vmtarget? Yes, on one side there is what Cyril says, we need to download a new VM on each new slave, because - we build the image on a single slave (unix) - we then transfer it to different slaves (mac, windows) to run the tests. This is like that because originally running the bootstrap was about three times as expensive. Right now we are on 7minutes to create the minimal image and a couple more to load the rest on top of it. There is another thing also: we always assumed that slaves do not share disk, so either we re-downloaded a new unix vm or we shared it. Apparently our assumptions weren't right? On Tue, Jun 19, 2018 at 5:06 PM Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com> wrote: > On 19/06/2018 17:02, Alistair Grant wrote: > > Hi Cyril, > > > Great work! > > > > Thank you. Also thanks to Guille, Vincent, Marcus that helped to find > the problem. > > > It would be nice if all the temporary files were in a single folder. > > Cleaning the environment then becomes a matter of deleting that one > > folder. (At the moment, there is a vm installed in to the root > > directory, a vmtarget directory (I'm guilty here) and > > bootstrap-cache). > > > > Maybe yes :) > > > Slightly off-topic, but I'm also wondering why the test scripts > > download the VM again? Why not just use the one that is already in > > vmtarget? > > > > Because it is a vm linux and tests are executed on OSX and Windows too. > > > > > > > This definitely helps :-) > > > > > > Thanks, > > Alistair > > > > > -- > Cyril Ferlicot > https://ferlicot.fr > > -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

June 19, 2018

Re: [Pharo-dev] [CI] Cause of the random failures in the CI

by Cyril Ferlicot D.

On 19/06/2018 17:02, Alistair Grant wrote: > Hi Cyril, > Great work! > Thank you. Also thanks to Guille, Vincent, Marcus that helped to find the problem. > It would be nice if all the temporary files were in a single folder. > Cleaning the environment then becomes a matter of deleting that one > folder. (At the moment, there is a vm installed in to the root > directory, a vmtarget directory (I'm guilty here) and > bootstrap-cache). > Maybe yes :) > Slightly off-topic, but I'm also wondering why the test scripts > download the VM again? Why not just use the one that is already in > vmtarget? > Because it is a vm linux and tests are executed on OSX and Windows too. > > > This definitely helps :-) > > > Thanks, > Alistair > -- Cyril Ferlicot https://ferlicot.fr

June 19, 2018

Re: [Pharo-dev] [CI] Cause of the random failures in the CI

by Alistair Grant

Hi Cyril, On Tue, 19 Jun 2018 at 16:55, Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com> wrote: > > Hi, > > Since months now there are a lot of random failure on the CI making it > hard to work. > > There is different kind of failures: > - Network problems > - Failing tests > - Incomprehensible problems > > Now I don't see much failure due to Network. I suppose the Inria > infrastructure improved. > > Failing tests were corrected those past months and we see less and less > of them. > > Now the big problem are the incomprehensible crashes such as "The > workspace was not found" or "FileDoesNotExistException" or "pharo-vm/ is > already present". > > We just found the problem :) > > During the validation of the Bootstrap multiple tests are launched on > OSX/Windows/linux in parallel. Each task is on a different slave of the > Jenkins. But, apparently we discovered that two slaves could have the > same disk. Usually it does not cause any trouble since a job is only run > by one slave. But in this particular case, two slaves can be used by the > same job and mess with the resources of each other. > > We highlighted the problem by adding logs to the CI. Now when we launch > tests we create a file with the name of the task. > > Today we got a crash and in the log we see that the same workspace has > two of those files, proving that they are executed on the same disk, in > the same folder : > > [â¦] > -rw-rw-r-- 1 ci ci 0 Jun 19 16:01 Kernel-tests-unix-32 > [â¦] > -rw-rw-r-- 1 ci ci 0 Jun 19 16:01 Tests-unix-32 > > As a solution we will execute the tests inside a subfolder with the name > of the task and it should reduce a lot the number of problems. Great work! It would be nice if all the temporary files were in a single folder. Cleaning the environment then becomes a matter of deleting that one folder. (At the moment, there is a vm installed in to the root directory, a vmtarget directory (I'm guilty here) and bootstrap-cache). Slightly off-topic, but I'm also wondering why the test scripts download the VM again? Why not just use the one that is already in vmtarget? > Have a nice day :) This definitely helps :-) > -- > Cyril Ferlicot > https://ferlicot.fr Thanks, Alistair

June 19, 2018

Re: [Pharo-dev] [CI] Cause of the random failures in the CI

by Sven Van Caekenberghe

Well done. Great work. > On 19 Jun 2018, at 16:55, Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com> wrote: > > Hi, > > Since months now there are a lot of random failure on the CI making it > hard to work. > > There is different kind of failures: > - Network problems > - Failing tests > - Incomprehensible problems > > Now I don't see much failure due to Network. I suppose the Inria > infrastructure improved. > > Failing tests were corrected those past months and we see less and less > of them. > > Now the big problem are the incomprehensible crashes such as "The > workspace was not found" or "FileDoesNotExistException" or "pharo-vm/ is > already present". > > We just found the problem :) > > During the validation of the Bootstrap multiple tests are launched on > OSX/Windows/linux in parallel. Each task is on a different slave of the > Jenkins. But, apparently we discovered that two slaves could have the > same disk. Usually it does not cause any trouble since a job is only run > by one slave. But in this particular case, two slaves can be used by the > same job and mess with the resources of each other. > > We highlighted the problem by adding logs to the CI. Now when we launch > tests we create a file with the name of the task. > > Today we got a crash and in the log we see that the same workspace has > two of those files, proving that they are executed on the same disk, in > the same folder : > > [â¦] > -rw-rw-r-- 1 ci ci 0 Jun 19 16:01 Kernel-tests-unix-32 > [â¦] > -rw-rw-r-- 1 ci ci 0 Jun 19 16:01 Tests-unix-32 > > As a solution we will execute the tests inside a subfolder with the name > of the task and it should reduce a lot the number of problems. > > Have a nice day :) > > -- > Cyril Ferlicot > https://ferlicot.fr >

June 19, 2018

[CI] Cause of the random failures in the CI

by Cyril Ferlicot D.

Hi, Since months now there are a lot of random failure on the CI making it hard to work. There is different kind of failures: - Network problems - Failing tests - Incomprehensible problems Now I don't see much failure due to Network. I suppose the Inria infrastructure improved. Failing tests were corrected those past months and we see less and less of them. Now the big problem are the incomprehensible crashes such as "The workspace was not found" or "FileDoesNotExistException" or "pharo-vm/ is already present". We just found the problem :) During the validation of the Bootstrap multiple tests are launched on OSX/Windows/linux in parallel. Each task is on a different slave of the Jenkins. But, apparently we discovered that two slaves could have the same disk. Usually it does not cause any trouble since a job is only run by one slave. But in this particular case, two slaves can be used by the same job and mess with the resources of each other. We highlighted the problem by adding logs to the CI. Now when we launch tests we create a file with the name of the task. Today we got a crash and in the log we see that the same workspace has two of those files, proving that they are executed on the same disk, in the same folder : [â¦] -rw-rw-r-- 1 ci ci 0 Jun 19 16:01 Kernel-tests-unix-32 [â¦] -rw-rw-r-- 1 ci ci 0 Jun 19 16:01 Tests-unix-32 As a solution we will execute the tests inside a subfolder with the name of the task and it should reduce a lot the number of problems. Have a nice day :) -- Cyril Ferlicot https://ferlicot.fr

June 19, 2018

Re: [Pharo-dev] [Pharo-users] [Ann] Iceberg v1.1.1

by Esteban A. Maringolo

On 19/06/2018 09:06, Norbert Hartl wrote: >> We can agree that there is no hard rule on versionning, do we? But I >> try to follow the following guidelines (delta my own interpretation >> that adds some subjectivity :P) >> - Major Version will change when we break backwards compatibility >> - Minor Version will change when new features are added >> - Otherwise, patch version will change. >> > There is only one hard rule for me and that is knowing about the risk to > take. So if we take the patch version it should only include important > bug fixes and nothing else. I would argue that only #864, #862, #858 and > #854 qualify for such a patch if at all. Not sure about #860 because the > title is not specific enough. > The point for me is that I want my project to rely on something like > 1.1.x because I donât want anything to change that breaks my software. > And I can tell you that most developers underestimate the side-effects > of changes.Â +1 to this. Only by introducing a new feature you MUST increment the minor version. >> Now, this is the kind of subjective topic that starts a flamewar, but >> Iâd prefer to use my time on somewhat else ^^. > > You may not like to talk about these things but I do. And you should > listen. I have no use for an environment where people only care about > coding new cool stuff. > If it would be like this I would quit all my > pharo businesses, move to mainstream and would use pharo only for > hobbyist projects because that would be the level that can be met. I've used commercial Smalltalks before Pharo and VW for the last year, and although they're _ages_ behind some of the "modern" stuff in Pharo, they never compromise version semantics regarding these things. And if I give it a second thought I can point to other open-source non-smalltak big projects/libraries which are also cool but also follow strict versioning, and being bigger the cascade effect of changing artifacts semantics is way bigger. Please take this comment with a grain of salt, it's not a critic of all the work done in Iceberg or other projects, but it seems that these kind of "meta" (aka "PM", "strategy", or anything non-coding) discussions are avoided or muted, and they shouldn't. Regards, -- Esteban A. Maringolo

June 19, 2018

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Launcher v1.2 release

by Sven Van Caekenberghe

> On 19 Jun 2018, at 16:01, Tudor Girba <tudor(a)tudorgirba.com> wrote: > > Great work! Indeed. Thank you. > Doru > > >> On Jun 19, 2018, at 3:55 PM, Christophe Demarey <Christophe.Demarey(a)inria.fr> wrote: >> >> Hi all, >> >> I just released PharoLauncher 1.2. It includes a new windows installer that you can use without administrator privileges as well as binary signing for OS X and Windows. Also, Pharo Launcher is not anymore identified as âPharoâ application and comes with its own icon. >> >> Here is the changelog (details on https://github.com/pharo-project/pharo-launcher/issues) >> New features: >> #21 Bless the DMG >> #46 sign pharo launcher app for windows >> #103 No way to rename a local template >> #107 Unable to add a description for the image using the Launcher UI >> #121 You can't see/sort images by last modified date >> Improvements: >> #69 Import command should also import pharo-local directory >> #70 Import command should delete origin folder if empty >> #73 Managers of Download of VMs and images should be in their own packages >> #76 Use https instead of http to requests the pharo file server >> #82 Official Distributions loads 32bit versions on 64bit System (i.e. provide better information on templates architecture) >> #86 Sort Existing Images Case-Insensitive >> #98 Copy and subfolders problem (contents no copied) >> #101 Templates from a local image are listed in "downloaded". "local" would be a better name >> #102 Template Cleared at Startup setting is enabled, making it weird when trying to use the template feature >> #106 Import could work if we select the parent folder of an image >> #109 Use latest pre-Spur VM to determine the image version >> #122 The Run without settings icon looks like a funny grey/which blob (missing alpha correction) >> Bux fixes: >> #41 #selectedMorphList was sent to nil >> #67 bash is not a command usable under windows >> #68 Does not launch images on Windows >> #85 Double click on an existing image open a file selector >> #88 Pharo Launcher on Windows > Failing >> #104 GUI bug makes Launcher unusable >> #110 Image launch not reliable on Windows >> #119 MessageNotUnderstood exception on launch >> #123 The status bar of the Launcher is broken, so can't easily show image descriptions >> >> >> Big thanks to all contributors: code, issues report, comments, advices. >> >> You can get platform bundles from pharo download page or files.pharo.org: http://files.pharo.org/pharo-launcher/1.2/ >> >> Regards, >> Christophe. > > -- > www.tudorgirba.com > www.feenk.com > > "The coherence of a trip is given by the clearness of the goal." > > > > > >

June 19, 2018

[ANN] TechTalk Dates second half 2018

by Marcus Denker

Here are the next dates for the Pharo Techtalks. 12 Jul 2018: Pharo TechTalk: Machine Learning with TensorFlow and Pharo https://association.pharo.org/event-2973748 For the others we have no topic yet, but dates are pre-fixed to: 23 Aug 2018: Pharo TechTalk https://association.pharo.org/event-2973753 11 Oct 2018: Pharo TechTalk https://association.pharo.org/event-2973754 15 Nov 2018: Pharo TechTalk https://association.pharo.org/event-2973755 13 Dec 2018: Pharo TechTalk https://association.pharo.org/event-2973756 We are looking for topics! â¢ If you have ideas for topics, please tell us! â¢ With no topic, we will use this slot for a general free discussion. â¢ If you want to do a TechTalk about a topic but no date is good, we can change the date! For recording of the past ones, see: http://pharo.org/TechTalk

June 19, 2018

Re: [Pharo-dev] [Pharo-users] [Ann] Iceberg v1.1.1

by Norbert Hartl

> Am 19.06.2018 um 15:22 schrieb Guillermo Polito <guillermopolito(a)gmail.com>: > > > On Tue, Jun 19, 2018 at 2:07 PM Norbert Hartl <norbert(a)hartl.name <mailto:norbert@hartl.name>> wrote: > Hi, > > let me wear the project manager hat for a moment. > > let me too, because the fact that I'm younger does not mean I don't know, right? :) > It has more to do with experience then with age. Well, the one thing enables sometimes the other but there is no strict causality, right. > >> Am 19.06.2018 um 10:59 schrieb Guillermo Polito <guillermopolito(a)gmail.com <mailto:guillermopolito@gmail.com>>: >> >> Hi, >> >> About why 1.1.1 and not 1.2.0. Itâs not about cheap or not, but about semantics :) > > for me âcaring about semanticsâ is just one of the top ten justifications developers use for the changes they did. > > Maybe, and putting the hat of project manager is usually a justification for somebody that is not good at technical stuff. > But I know that's not like it, so please let's not enter into this, I've felt a little insulted by this comment... > That is pretty clear by what you wrote. It wasnât meant to be personal so please donât take it like that. So do I. > >> We can agree that there is no hard rule on versionning, do we? But I try to follow the following guidelines (delta my own interpretation that adds some subjectivity :P) >> - Major Version will change when we break backwards compatibility >> - Minor Version will change when new features are added >> - Otherwise, patch version will change. >> > There is only one hard rule for me and that is knowing about the risk to take. > > That's a matter of conventions. We agree that version 1.1.x is compatible with 1.1.y. > > So if we take the patch version it should only include important bug fixes and nothing else. I would argue that only #864, #862, #858 and #854 qualify for such a patch if at all. > > So they are to my view. They should not introduce any compatibility issue. > And if they do, that's an error, but we are too few helping here, doing our best... > My mail was meant to be a plea for exactly this. If you donât think the hot-fix (I like that much more than patch) solves a show stopper for a lot of users of this version you donât put it in a patch version. If it is in the slightest sense an improvement donât put into a patch version but a minor on. Because I donât want to have my stuff broken but I choose to update in order to get the improvement hence a deliberate action. If all of these commits are hot-fixes then take my big excuse for bringing this up. > Not sure about #860 because the title is not specific enough. > > Please, I'll let you judge it for yourself > > https://github.com/pharo-vcs/iceberg/pull/860/files <https://github.com/pharo-vcs/iceberg/pull/860/files> > > But to me that change applies to patch. It actually fixes a compatibility issue that was introduced in 1.1.0. > > The point for me is that I want my project to rely on something like 1.1.x because I donât want anything to change that breaks my software. And I can tell you that most developers underestimate the side-effects of changes. > > I'm well aware of this. But do you have a concrete issue? > For what? > >> So I donât assign a new version number regarding the number of changes but about what they mean... > > To mean they mean it is a risk to use that version and you define how big that is. > > We are trying to do weekly releases, we could do better but again. I can count with my hand fingers people contributing with actual commits and issues in the issue tracker. > This time it is not about having more but less in a version. > >> Now, I considered myself this release as a patch because mostly little bugs here and there were fixed. >> Moreover, one of the changes done in the credentials manager was to *recover* some backwards compatibility for people setting up credentials in settings files. >> Of course, to this we add to this that my own interpretation saying that the changes do not break compatibility nor add features :) >> > You see you said âmostly bugsâ and that is the error already. > > I'm sorry for not being perfect... > > I mean we come from an amateurish behaviour that we change released artefacts. > > I assure you I do my best on it, and I'm one of the first that cries aloud when there are versionning and dependencies problems. > What I do not understand if this mail was meant as a lesson for the community or should I take it personally... > I know you do your best. And I know that you are a really good developer that does great stuff all the time. It is neither meant as a lesson nor something you should take personally. I do not often raise my wishes because I always need to assurance it is something really important and not something I want out of my current mood. But this is something I have no doubt about so I bring it up. And I do all of this for quite some time and that does count, too. > That is not discussable just a no-go. > > I know and I'm against it. So we agree, right? > Yes. > The reason was it would have wasted a huge amount of time to do a new version. So ok it was a loose-loose situation. Now we can do it better and I want something far less amateurish. So you can discuss your semantics about what major and minor versions in pharo mean but patch needs to be the definition of the combination: least risk - highest value. > > Would you help us measuring the risk? > Sure. > >> Now, this is the kind of subjective topic that starts a flamewar, but Iâd prefer to use my time on somewhat else ^^. > > You may not like to talk about these things but I do. And you should listen. > > I mostly do [enjoy such discussions] but what I learn from this discussion is: > - you think I'm stupid, or young, or both, so I don't know > - we are all amateurs > No, it is more like - you took it as a personal insult. That was not my intention and I apologize - well, we includes me. There are things that I have a pretty solid opinion about and that is what I want to raise. And I want to bring that into the community. It is also important work > I have no use for an environment where people only care about coding new cool stuff. > > I'm fu*** trying to make Iceberg as stable as possible, If I was just wanting to do new cool stuff I would not be doing this. > I know. Norbert

June 19, 2018