September 2017 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [ci-announces] [CI] July 28, 2017 Incident Report

by David T. Lewis

Kudos for providing a good incident report and explanation of the cause of the problem. I know little about this actual incident, but that it not important. It is really good to see constructive responses to problems like this with a focus on inprovement and problem prevention in the future. Dave On Thu, Sep 21, 2017 at 03:35:19PM +0200, Christophe Demarey wrote: > July 28, 2017 Incident Report > > This summer, we experienced a hard incident on the Continuous Integration service infrastructure.Today we???re providing an incident report that details the nature of the incident and our response. > We understand this service issue has impacted all Inria developers using the CI service and their valued time, and we apologize to everyone who was affected. > > Issue summary > Due to a manipulation error, a lot of Virtual Machines (VM) hosted on CloudStack were destroyed and not recoverable! > Jenkins servers are not concerned by this issue, so all Jenkins jobs and history are safe. > > Root cause > After the successful migration of Jenkins servers and security fixes (July 18-19, 2017), a few projects were not able to reach their slaves hosted on the CI build farm. This problem was due to a synchronization problem between the CI database and CloudStack (powering the CI build farm) having its own way to manage projects and users (through domains). > An attempt to reproduce and debug this problem on the qualification infrastructure failed. So, we added some logging on the production infrastructure. To avoid troubles to the production infrastructure, we limited the synchronization to one project. It was the mistake! > The synchronization of one project led to the deletion of all other CloudStack domains (i.e. projects). Indeed, the synchronization code expected to get the full environment (all CI projects) and if it finds a domain not bound to a CI project, it deletes it... > The synchronization process was aborted before its termination but it was too late. Some user Virtual Machines were still alive during some hours but were finally purged by CloudStack. > It means we lost most VM and templates hosted on the CI build farm. > > Resolution and recovery > It was impossible to recover destroyed virtual machines. CloudStack is configured to keep VM data 24 hours before actually destroying it but it does not work when the domain hosting the VM is destroyed. > Primary storage hosting running VM is a high-performance and very expensive storage. That???s why the CI team chose (at the CI service setup) to do not backup VM but rather to rely on both the expunge delay and the snapshot / template mechanism to save VM state. This mechanism was useless in relation to this incident. > Templates and snapshots are hosted on the secondary storage that is a redundant storage in two different buildings to ensure data reliability and recovery. The incident led CloudStack to perform a ?? clean ?? deletion of all the domain data including templates. That???s why they also became unavailable. > > We were able to rebuild all domains from the CI database but CI service users had to create new VM to replace the destroyed ones. > > Corrective and preventative measures > All members of the CI team (DSI, SED) worked and are still working all together to find the best solution to mitigate the incident and prevent same situations in the future. > The synchronization code responsible of the deletion of CloudStack domain has been deactivated. CloudStack domain deletion is a critical action and will no longer be automated. Deletions will be reviewed and approved by the CI team before being completed. > This incident showed us that backup mechanism in place are not strong enough and we are now evaluating the cost to backup, with history: > all VM, snapshots and templates or > all snapshots and templates. > We are also working on providing a way to download templates created on CloudStack so that you can easily get a copy of them. We encourage you to create templates for virtual machines that are time consuming to set up from scratch. > > > Sincerely, > The CI Team

Sept. 22, 2017

How to get started writing a little booklet on Pharo?

by Stephane Ducasse

- Create a repository xxx on github - git clone xxx - cd xxx wget https://raw.githubusercontent.com/pillar-markup/pillar/master/download.sh chmod +x download.sh ./download.sh ./pillar archetype book make spiral You can also run the texlive distribution in docker. https://github.com/cdlm/docker-texlive Stef

Sept. 21, 2017

Call to read and give feedback on Booklet-Mocking

by Stephane Ducasse

HI guys we should increase our culture around documentation. https://github.com/SquareBracketAssociates/Booklet-Mocking Please read the booklet. Attila told me that he was planning to rewrite the chapter on baby mock. Now each time you commit on github the pdf is generated. Stef

Sept. 21, 2017

Re: [Pharo-dev] Iceberg Default Subdirectory Setting?

by stephan

On 21-09-17 18:54, Esteban Lorenzano wrote: > I do not understand what is the problem. > You have different approaches: > > 1) you work always on same branch (even if you âjumpâ from branchs): like today I work on âissue-1â and tomorrow on âissue-2â. You just switch branchs on your iceberg tool and thatâs all. > 2) you work on different branchs at same time and you do not want to switch branches all the time. Then you just have two clones and points to this different clones. > > in general, Iâm working on different branches all the time (pharo branches, so big projects) and I prefer to use pattern (1), but you can also use pattern (2)â¦ as I saidâ¦ I do not understand what is the problem :) You cannot use pattern 2 with shared repos. Stephan

Sept. 21, 2017

Non-deterministic ombu errors

by Gabriel Cotelli

Hi, I'm having some non-deterministic errors in the Travis builds, the error is always something like: *OmFileStoreReadingError: Reading entry from File @ pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: Invalid utf8 input detected* If I retried the build it works without any change. I think it happens during the code loading phase. Is anyone experiencing this issue? Here's an extract of the travis log: *OmFileStoreReadingError: Reading entry from File @ pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: Invalid utf8 input detected* [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] in [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :error | ... BlockClosure>>cull: Context>>evaluateSignal: Context>>handleSignal: UTF8InvalidText(Exception)>>signal UTF8InvalidText(Exception)>>signal: UTF8TextConverter>>errorMalformedInput UTF8TextConverter>>nextFromStream: MultiByteFileStream>>next MultiByteFileStream(PositionableStream)>>match: MultiByteFileStream(PositionableStream)>>upToAll: OmSTONEntryReader>>nextEntryPositionIfFound:ifNone: OmBlockFileStore>>refreshNewBlocksFrom: [ :readStream | self refreshNewBlocksFrom: readStream ] in OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since: in Block: [ :readStream | self refreshNewBlocksFrom: readStr...etc... [ ^ aBlockClosure value: readStream ascii ] in [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ ^ aBlockClosure value: readStream ascii ] BlockClosure>>on:do: [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :readStream | ... [ aBlock value: stream ] in FileReference(AbstractFileReference)>>readStreamDo: in Block: [ aBlock value: stream ] BlockClosure>>ensure: FileReference(AbstractFileReference)>>readStreamDo: OmBlockFileStore(OmFileStore)>>readEntriesWith: OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since: [ | initialPosition initialLocalName | self entryBuffer isEmpty ifTrue: [ ^ self ]. fileReference writeStreamDo: [ :fileStream | fileStream setToEnd. initialPosition := fileStream position. initialLocalName := self entryBuffer first value. ZnBufferedWriteStream on: fileStream do: [ :aWriteStream | | anEntryWriter | anEntryWriter := self newEntryWriter. [ self entryBuffer isEmpty ] whileFalse: [ | next entry | next := self entryBuffer removeFirst. entry := next key. "Write entry to file" anEntryWriter on: aWriteStream nextEntryPut: entry ] ]. "In Linux it was necessary to explicitely flush the file stream" fileStream flush. lastStreamPosition := fileStream size ]. self refreshEntryPositionsByLocalNameStartingAt: initialPosition since: initialLocalName ] in OmBlockFileStore(OmFileStore)>>flushEntryBuffer in Block: [ | initialPosition initialLocalName |... [ caught := true. self wait. blockValue := mutuallyExcludedBlock value ] in Semaphore>>critical: in Block: [ caught := true.... BlockClosure>>ensure: Semaphore>>critical: OmBlockFileStore(OmFileStore)>>critical: OmBlockFileStore(OmFileStore)>>flushEntryBuffer OmDeferrer>>sendMessage [ self sendMessage ] in OmDeferrer>>flush in Block: [ self sendMessage ]

Sept. 21, 2017

Re: [Pharo-dev] Iceberg Default Subdirectory Setting?

by Esteban Lorenzano

I do not understand what is the problem. You have different approaches: 1) you work always on same branch (even if you âjumpâ from branchs): like today I work on âissue-1â and tomorrow on âissue-2â. You just switch branchs on your iceberg tool and thatâs all. 2) you work on different branchs at same time and you do not want to switch branches all the time. Then you just have two clones and points to this different clones. in general, Iâm working on different branches all the time (pharo branches, so big projects) and I prefer to use pattern (1), but you can also use pattern (2)â¦ as I saidâ¦ I do not understand what is the problem :) Esteban > On 21 Sep 2017, at 18:33, Sean P. DeNigris <sean(a)clipperadams.com> wrote: > > Stephan Eggermont-3 wrote >> The problem with that is that nearly none of the power users have such a >> workflow. > > I assume that's why it's a setting ;) > > > > ----- > Cheers, > Sean > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >

Sept. 21, 2017

Re: [Pharo-dev] Iceberg Default Subdirectory Setting?

by Sean P. DeNigris

Stephan Eggermont-3 wrote > The problem with that is that nearly none of the power users have such a > workflow. I assume that's why it's a setting ;) ----- Cheers, Sean -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Sept. 21, 2017

Re: [Pharo-dev] Iceberg Default Subdirectory Setting?

by stephan

On 21-09-17 16:07, Sean P. DeNigris wrote: > Stephan Eggermont-3 wrote >> I'm not quite sure how that share repos works. > > My understanding is that it only works for certain straightforward > workflows. Luckily, my guess is that many if not most users have such a > workflow (e.g. single contributor/user projects where everything is > committed to master) The problem with that is that nearly none of the power users have such a workflow. That is what killed versionner. And also for single contributor you need to compare with old versions regularly Stephan

Sept. 21, 2017

Re: [Pharo-dev] slow Pharo downloading and Travis

by Ben Coman

btw, I got 17MB in good time (~25s) this link... https://ci.inria.fr/pharo/view/Launcher/job/Launcher/149/PHARO=61,VERSION=b… So here is my entirely unsubstantiated exotic random troubleshooting tip... check for a mismatch in auto-negotiate setting between server ethernet port and etherswitch I've seen more than once strange slow downs due to one side being configured for 100Mb, and the other side configured to auto-negotiate and it craps out. Both sides need to be set autonegotiate, or both sides need to be set 100Mb, not mixed. cheers -ben On Thu, Sep 21, 2017 at 11:13 PM, Ben Coman <btc(a)openinworld.com> wrote: > Still seems a bit slow... > http://files.pharo.org/get-files/70/pharo64.zip > 16MB ==> 10 minutes (and actually it stalled after 3.4MB downloaded) > Local speed test: 40Mb/s down, 20Mb/s up > > but then I'm a long way across the internet, the cause could be anywhere... > $ traceroute files.pharo.org > traceroute to files.pharo.org (128.93.162.72), 30 hops max, 60 byte > packets > 1 192.168.43.1 (192.168.43.1) 2.040 ms 2.403 ms 4.222 ms > 2 * * * > 3 * * * > 4 Bundle-Ether14.wel-edge901.perth.telstra.net (139.130.16.49) 67.741 > ms 67.739 ms 67.730 ms > 5 bundle-ether6.wel-core3.perth.telstra.net (203.50.6.214) 67.696 ms > 67.697 ms 67.697 ms > 6 bundle-ether7.fli-core1.adelaide.telstra.net (203.50.11.18) 95.723 > ms 91.753 ms 100.574 ms > 7 bundle-ether9.win-core10.melbourne.telstra.net (203.50.11.91) > 106.550 ms 108.849 ms 108.851 ms > 8 bundle-ether12.ken-core10.sydney.telstra.net (203.50.11.122) 117.770 > ms 117.767 ms 117.751 ms > 9 bundle-ether1.pad-gw11.sydney.telstra.net (203.50.6.61) 117.720 ms > 103.603 ms 102.171 ms > 10 bundle-ether1.sydp-core04.sydney.reach.com (203.50.13.90) 93.276 ms > 95.822 ms 102.577 ms > 11 i-52.tlot-core02.bx.telstraglobal.net (202.84.137.101) 330.128 ms > 321.231 ms 322.522 ms > 12 i-0-4-0-10.tlot-core01.bi.telstraglobal.net (202.84.253.21) 321.187 > ms i-0-2-0-10.tlot-core01.bi.telstraglobal.net (202.40.149.181) 321.181 > ms i-0-2-0-9.tlot-core01.bi.telstraglobal.net (202.40.149.177) 308.786 ms > 13 gtt-peer.tlot02.pr.telstraglobal.net (134.159.63.182) 273.784 ms > 283.530 ms 281.079 ms > 14 xe-1-2-0.cr0-par7.ip4.gtt.net (89.149.185.53) 466.648 ms 479.910 ms > 481.254 ms > 15 renater-gw-ix1.gtt.net (77.67.123.206) 479.900 ms 479.882 ms > 478.641 ms > 16 * * * > 17 inria-rocquencourt-te1-4-inria-rtr-021.noc.renater.fr > (193.51.184.177) 436.776 ms 479.819 ms 478.907 ms > 18 * * * > 19 * * * > > $ httping -h files.pharo.org > PING files.pharo.org:80 (http://files.pharo.org:80/) > connected to 128.93.162.72:80 (169 bytes), seq=0 time=970.27 ms > connected to 128.93.162.72:80 (169 bytes), seq=1 time=928.26 ms > connected to 128.93.162.72:80 (169 bytes), seq=2 time=860.06 ms > connected to 128.93.162.72:80 (169 bytes), seq=3 time=976.04 ms > connected to 128.93.162.72:80 (169 bytes), seq=4 time=1238.72 ms > connected to 128.93.162.72:80 (169 bytes), seq=5 time=1239.65 ms > connected to 128.93.162.72:80 (169 bytes), seq=6 time=918.06 ms > > > cheers -ben > > On Thu, Sep 21, 2017 at 5:43 PM, Marcus Denker <marcus.denker(a)inria.fr> > wrote: > >> >> > On 18 Sep 2017, at 08:37, Marcus Denker <marcus.denker(a)inria.fr> wrote: >> > >> > >> >> On 16 Sep 2017, at 09:40, Stephane Ducasse <stepharo.self(a)gmail.com> >> wrote: >> >> >> >> Hi marcus >> >> >> >> I do not see why they could not find it. >> >> Can you tell me how you ask because we should reask? >> >> >> >> >> > >> > I spend a ticket and they wanted to close it last week. I have told >> them to not close it >> > >> The problems seems to be fixed now. >> >> If someone sees any slowness -> send me a mail. >> >> Marcus >> >> >> >

Sept. 21, 2017

Re: [Pharo-dev] slow Pharo downloading and Travis

by Ben Coman

Still seems a bit slow... http://files.pharo.org/get-files/70/pharo64.zip 16MB ==> 10 minutes (and actually it stalled after 3.4MB downloaded) Local speed test: 40Mb/s down, 20Mb/s up but then I'm a long way across the internet, the cause could be anywhere... $ traceroute files.pharo.org traceroute to files.pharo.org (128.93.162.72), 30 hops max, 60 byte packets 1 192.168.43.1 (192.168.43.1) 2.040 ms 2.403 ms 4.222 ms 2 * * * 3 * * * 4 Bundle-Ether14.wel-edge901.perth.telstra.net (139.130.16.49) 67.741 ms 67.739 ms 67.730 ms 5 bundle-ether6.wel-core3.perth.telstra.net (203.50.6.214) 67.696 ms 67.697 ms 67.697 ms 6 bundle-ether7.fli-core1.adelaide.telstra.net (203.50.11.18) 95.723 ms 91.753 ms 100.574 ms 7 bundle-ether9.win-core10.melbourne.telstra.net (203.50.11.91) 106.550 ms 108.849 ms 108.851 ms 8 bundle-ether12.ken-core10.sydney.telstra.net (203.50.11.122) 117.770 ms 117.767 ms 117.751 ms 9 bundle-ether1.pad-gw11.sydney.telstra.net (203.50.6.61) 117.720 ms 103.603 ms 102.171 ms 10 bundle-ether1.sydp-core04.sydney.reach.com (203.50.13.90) 93.276 ms 95.822 ms 102.577 ms 11 i-52.tlot-core02.bx.telstraglobal.net (202.84.137.101) 330.128 ms 321.231 ms 322.522 ms 12 i-0-4-0-10.tlot-core01.bi.telstraglobal.net (202.84.253.21) 321.187 ms i-0-2-0-10.tlot-core01.bi.telstraglobal.net (202.40.149.181) 321.181 ms i-0-2-0-9.tlot-core01.bi.telstraglobal.net (202.40.149.177) 308.786 ms 13 gtt-peer.tlot02.pr.telstraglobal.net (134.159.63.182) 273.784 ms 283.530 ms 281.079 ms 14 xe-1-2-0.cr0-par7.ip4.gtt.net (89.149.185.53) 466.648 ms 479.910 ms 481.254 ms 15 renater-gw-ix1.gtt.net (77.67.123.206) 479.900 ms 479.882 ms 478.641 ms 16 * * * 17 inria-rocquencourt-te1-4-inria-rtr-021.noc.renater.fr (193.51.184.177) 436.776 ms 479.819 ms 478.907 ms 18 * * * 19 * * * $ httping -h files.pharo.org PING files.pharo.org:80 (http://files.pharo.org:80/) connected to 128.93.162.72:80 (169 bytes), seq=0 time=970.27 ms connected to 128.93.162.72:80 (169 bytes), seq=1 time=928.26 ms connected to 128.93.162.72:80 (169 bytes), seq=2 time=860.06 ms connected to 128.93.162.72:80 (169 bytes), seq=3 time=976.04 ms connected to 128.93.162.72:80 (169 bytes), seq=4 time=1238.72 ms connected to 128.93.162.72:80 (169 bytes), seq=5 time=1239.65 ms connected to 128.93.162.72:80 (169 bytes), seq=6 time=918.06 ms cheers -ben On Thu, Sep 21, 2017 at 5:43 PM, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > > On 18 Sep 2017, at 08:37, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > > > > >> On 16 Sep 2017, at 09:40, Stephane Ducasse <stepharo.self(a)gmail.com> > wrote: > >> > >> Hi marcus > >> > >> I do not see why they could not find it. > >> Can you tell me how you ask because we should reask? > >> > >> > > > > I spend a ticket and they wanted to close it last week. I have told them > to not close it > > > The problems seems to be fixed now. > > If someone sees any slowness -> send me a mail. > > Marcus > > >

Sept. 21, 2017