October 2018 - Pharo-dev - lists.pharo.org

[Pharo 7.0-dev] Build #1288: Use /tmp (or analog in other OSs) instead of the working "directory"

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1288 was: SUCCESS. The Pull Request #1849 was integrated: "Use /tmp (or analog in other OSs) instead of the working "directory"" Pull request url: https://github.com/pharo-project/pharo/pull/1849 Issue Url: https://pharo.fogbugz.com/f/cases/21167 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Oct. 2, 2018

Re: [Pharo-dev] Reach of Iceberg/git/tonel ?

by Sven Van Caekenberghe

> On 30 Sep 2018, at 11:00, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > Hi, > > > > >> On 29 Sep 2018, at 14:39, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >> Hi, >> >> So the future for Pharo is Iceberg/git/tonel and that is fine. >> >> My question however is: if I convert my external libraries to this new format, what is the reach of this new technology ? Can people still load code in older versions of Pharo ? I think 6.x is no problem, but what about earlier versions ? What about other Smalltalk implementations ? What is the official answer here ? > > In absence of Iceberg, Tonel can be used as any other MCRepository. You can save and read your packages as it would be with a âmetadataless filetreeâ. > That means, in Pharo without iceberg you need to save/read from your cloned repository. > Saving to your cloned repository will write files to disk. Which means you need to go there and do a regular git commit/push cycle. > > While this is annoying compared with the usage of Iceberg, is perfectly doable: it is the same as while using metadataless filetree. > > Tonel is tested to work up to Pharo 3. Older Pharos may work (there is no reason why not) but maybe some minor adjustments are needed. > > I designed Tonel to be easily portable. I didnât used a lot of Pharo elements that would have done my life a lot easier (like using RBParser instead parsing manual). This way other dialects can take it and make it work. Now, making it work, is a problem of other dialects users (Iâm willing to help, but that work will fall in other persons). > I know it is already ported to Gemstone and it was being ported to Squeak. > >> A second question: suppose a repo is converted, is then still possible to copy a version over to an old MC repo ? Just to keep it in sync. > > You can save your packages as you want. > You can save a package with Iceberg and then read it with Monticello., > And viceversa. > Now, it you want to "transferâ the history from git to an mcz, you will need to do some migration tool. > > Esteban Thank you Esteban, for the explanation and for the implementation, that all sounds really good. Iceberg has been very stable for me this past week, great work. >> Thx, >> >> Sven >> >> > >

Oct. 2, 2018

Re: [Pharo-dev] [ANN] Migrated Artefact to GitHub

by Sven Van Caekenberghe

> On 2 Oct 2018, at 10:39, Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > Also, everybody seems to assume that it is "really easy to do" in emails. > But nobody is taking the time to do it, so why should I? > > Now apparently you also missed a point in my email: what I did is not incompatible with migrating the history. > History could have been migrated separately, and then my changes merged in them. > > Now we can focus on the mistakes, or try to build something together. > But again, we are discussing again this instead of actually migrating the history :). So true: I very much appreciate that you did something that you were not required to do. And to be clear: you have done a tremendous amount of work lately, Guille, so thank you for all your contributions. It is better to err on the action side than to do nothing and talk. Sven

Oct. 2, 2018

Re: [Pharo-dev] [ANN] Migrated Artefact to GitHub

by Guillermo Polito

Hi, Yes, Eliot. I see the value in authorship, I never said the opposite. I also see that this could be taken bad. Now, every time I read **again** a mail like this I just confirm that I had only two ways out: - doing the actual migration of the history (which I will not, because if I do it wrong, I will get complaints again, and we loop! So to me there is no point in doing it now) - retracting my repository Also, everybody seems to assume that it is "really easy to do" in emails. But nobody is taking the time to do it, so why should I? Now apparently you also missed a point in my email: what I did is not incompatible with migrating the history. History could have been migrated separately, and then my changes merged in them. Now we can focus on the mistakes, or try to build something together. But again, we are discussing again this instead of actually migrating the history :). On Tue, Oct 2, 2018 at 1:00 AM Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > Hi Guille, > > this issue is /really/ important to me. People are helping me migrate > VMMaker to GitHub and it is /treally/ important to the project that > authorship history is maintained, because finding out who to ask when code > is affected is essential. VMMaker is large, very complex and has had many > contributors. Wiping authorship is unacceptable to me. I'm glad that > Peter's tool is being used in the migration. Hence my responses below.. > > > On Sat, Sep 29, 2018 at 4:34 AM Guillermo Polito < > guillermopolito(a)gmail.com> wrote: > >> >> On Sat, Sep 29, 2018 at 12:41 PM Peter Uhnak <i.uhnak(a)gmail.com> wrote: >> >>> Hi Stef, >>> >>> I understand that everyone is short on time, but I consider not >>> preserving the history problematic for two reasons >>> >>> * it is appropriating someone else's work as one's own -- this seems >>> borderline illegal, or at the very least in violation of MIT license >>> >> >> Not really. If the contributors are specified in the copyright, is it >> stealing? >> >> Now, my point is manners. I got in 4/5 emails: >> - you're stupid because you did not use Peter's tool >> - you're inefficient because you took one hour to do it >> - you're stealing >> >> The first two points show first one big problem that I've seem many times >> with people and software: missing context. >> People think most of the time that "I would have done it better". But >> usually they don't take into account >> - the time constraints (I had one hour, and one hour I had) >> - the knowledge (where is the Artefact Repo?, why is it failing for >> Milton?, Do I know enough about metacello/streams/artefact to do it well?) >> - the working environment (does the person that did it have all the >> tools to work properly? does he have a healthy working environment? For >> example, I've worked on several big companies where you can find really bad >> environments...) >> - technical and not technical problems that are sometimes independent of >> the problem itself (take into account that for example, fighting against a >> metacello baseline is completely orthogonal to your tool or even iceberg, >> good internet connection) >> >> All these details are important also at the end, and they should be put >> into the balance too when we make a judgement. >> >> >>> * it is sending bad signals to potential contributors that we can scrub >>> them anytime we want >>> >>> as you yourself have said: >>> > @People try to avoid to piss on good will of others. >>> Yet this is what it feels like to some when traces of their >>> contributions are voided. >>> >> >> And this is my third point. This "stealing" idea is mainly a matter of >> manners. >> I really hope nobody here really thinks I wanted to take credit for >> Olivier, Guillaume or any of the other contributors. >> Still I preserved pointers to the original authors and their original >> website in google sites. >> > > That's simply not good enough. The minimum acceptable solution is that > within Pharo, within a browser, one can find out who authored what method, > class comment, and preferably class declaration (we don't have this yet). > Going outside to find out who authored is an unacceptable regression. > > But in the case somebody did think that, I removed the repository to >> remove any doubt. >> So again, I apologize if somebody felt offended, but I also prefer to not >> be called a thief. >> >> Now, when I do stuff I'm not thinking about "oh yes, I'm getting famous", >> that would be pretty sad for me :/. >> I do stuff because I just think it's useful. >> I DON'T CARE personally about artefact, and I don't want to take credit >> for it. >> I don't even care about the fu***ng 2 commits I did to port it to Pharo >> 7, I can tell anybody what I did so she/he can re-doit. >> Because I don't use Artefact. Now, Somebody wants the "credit"? I could >> have even amended a commit and put anybody else as author. >> > > But dot you see that voiding authorship a) gives the impression of > stealing, and clearly opens you up to the accusation of stealing, no matter > what your actual intent is? And do you see that vitally important > information is being lost? If this application had, as VMMaker does, > hundreds of contributors then tracking down who last modified what, which > is really important information, is made much harder. > > My problem here is people assuming stealing by default, instead of >> assuming, for example, mistake. >> Imagine an alternative scenario: >> - X: "Hey Guille, could you add in the copyright X, and Y and Z? They >> also contributed to the project, you should take them into account..." >> - Guille: "Ah sure, sorry, this was not my intention, I'm so stupid, I >> forgot about Z. Commit push, done". >> >> If instead of bashing on people, we wanted to discuss on how to actually >> FIX the thing, here are my 2 cents: >> - From a copyright perspective it should have been enough to check the >> licence file and name the contributors there >> - The history could have been retrieved in a separate branch and then >> merged (and look, we had the best of the two worlds!) >> - both of the things could have been then integrated through a pull >> request (luckily in less than one hour :)) >> > > Well, for me, it its not about copyright, it is about authorship, > attribution and communication. I /have/ to be able to determine original > authors to have a chance at getting help with issues that arise. I hope > the use case helps motivate you to preserve authorship. > > >> >> And at the end, with the apport of everybody we could have got a >> repository with history, baseline and working on pharo 7. >> >> Now, from a human perspective, please let's try everybody to assume the >> best of the other by default. >> That will just make all interactions much more healthy. >> >> And please, this is not particularly directed to anybody, I've seen such >> remarks many times. Let's just think positive. >> Problems can be fixed if we talk about them, but more specifically if we >> are looking for solutions. >> > > _,,,^..^,,,_ > best, Eliot > -- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13

Oct. 2, 2018

[ANN] 22477 DelayScheduler cleanup and refactoring [was: Where do we go now ?]

by Ben Coman

On Fri, 13 Apr 2018 at 13:56, Benoit St-Jean via Pharo-users < pharo-users(a)lists.pharo.org> wrote: > Do we really need 8 delay schedulers (DelayMicrosecondScheduler, > DelayMillisecondScheduler, DelayNullScheduler, > DelayExperimentalSpinScheduler, DelaySpinScheduler, DelayTicklessScheduler, > DelayExperimentalCourageousScheduler, DelayExperimentalSemaphoreScheduler) > ? > I've cleaned the delay scheduling subsystem [1] to condense these alternatives and separate orthogonal functionality. After a couple of reviews this was integrated and made active in last week's Build 1273. Could anyone with test scenarios from previous issues with delays have a bash at stressing the latest build. The old(existing) DelaySpinScheduler remains in the system so it can be activated as a point of comparison (System > Settings > System > Delay Scheduler). Pending any adverse reports the final step will be to remove the old hierarchy next week. There remain separate mutex and semaphore based schedulers since: * they differ by only a couple of overridden methods * their slightly different implementations help highlight the core algorithm * they provide a simple in-Image example of different synchronisation mechanisms. * in isolating edge cases its useful to be able to compare results between implementations I've retained both microsecond and millisecond operation, but extracted into "ticker" classes orthogonal to the "scheduling" classes since: * it makes the core scheduling algorithm independent of time base * millisecond (or other custom) timebase might(?) be more efficient on smaller 32-bit embedded systems * tests can now simulate ticker time to avoid tests interfering with system's-active-scheduler VM interaction (which may be the source of some random CI failures) * delay scheduler tests are now independent of real-time (which may be the source of some random CI failures where delays are affected by varying CI server loads) * when multi-threaded FFI callbacks become available, may facilitate experimenting with: * wake-up from native timers * wake-up of embedded Pharo from the encompassing system Other refactors: * during system snapshot, save/restore of resumption times was happening at user priority which risks a race condition reported at [2]. *All* modification of resumption times now occurs at timing/highest priority. [1] https://pharo.manuscript.com/f/cases/22477/DelayScheduler-cleanup-and-refac… [2] https://pharo.manuscript.com/f/cases/18359/

Oct. 2, 2018

Re: [Pharo-dev] [ANN] Migrated Artefact to GitHub

by Eliot Miranda

Hi Guille, this issue is /really/ important to me. People are helping me migrate VMMaker to GitHub and it is /treally/ important to the project that authorship history is maintained, because finding out who to ask when code is affected is essential. VMMaker is large, very complex and has had many contributors. Wiping authorship is unacceptable to me. I'm glad that Peter's tool is being used in the migration. Hence my responses below.. On Sat, Sep 29, 2018 at 4:34 AM Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > On Sat, Sep 29, 2018 at 12:41 PM Peter Uhnak <i.uhnak(a)gmail.com> wrote: > >> Hi Stef, >> >> I understand that everyone is short on time, but I consider not >> preserving the history problematic for two reasons >> >> * it is appropriating someone else's work as one's own -- this seems >> borderline illegal, or at the very least in violation of MIT license >> > > Not really. If the contributors are specified in the copyright, is it > stealing? > > Now, my point is manners. I got in 4/5 emails: > - you're stupid because you did not use Peter's tool > - you're inefficient because you took one hour to do it > - you're stealing > > The first two points show first one big problem that I've seem many times > with people and software: missing context. > People think most of the time that "I would have done it better". But > usually they don't take into account > - the time constraints (I had one hour, and one hour I had) > - the knowledge (where is the Artefact Repo?, why is it failing for > Milton?, Do I know enough about metacello/streams/artefact to do it well?) > - the working environment (does the person that did it have all the tools > to work properly? does he have a healthy working environment? For example, > I've worked on several big companies where you can find really bad > environments...) > - technical and not technical problems that are sometimes independent of > the problem itself (take into account that for example, fighting against a > metacello baseline is completely orthogonal to your tool or even iceberg, > good internet connection) > > All these details are important also at the end, and they should be put > into the balance too when we make a judgement. > > >> * it is sending bad signals to potential contributors that we can scrub >> them anytime we want >> >> as you yourself have said: >> > @People try to avoid to piss on good will of others. >> Yet this is what it feels like to some when traces of their contributions >> are voided. >> > > And this is my third point. This "stealing" idea is mainly a matter of > manners. > I really hope nobody here really thinks I wanted to take credit for > Olivier, Guillaume or any of the other contributors. > Still I preserved pointers to the original authors and their original > website in google sites. > That's simply not good enough. The minimum acceptable solution is that within Pharo, within a browser, one can find out who authored what method, class comment, and preferably class declaration (we don't have this yet). Going outside to find out who authored is an unacceptable regression. But in the case somebody did think that, I removed the repository to remove > any doubt. > So again, I apologize if somebody felt offended, but I also prefer to not > be called a thief. > > Now, when I do stuff I'm not thinking about "oh yes, I'm getting famous", > that would be pretty sad for me :/. > I do stuff because I just think it's useful. > I DON'T CARE personally about artefact, and I don't want to take credit > for it. > I don't even care about the fu***ng 2 commits I did to port it to Pharo 7, > I can tell anybody what I did so she/he can re-doit. > Because I don't use Artefact. Now, Somebody wants the "credit"? I could > have even amended a commit and put anybody else as author. > But dot you see that voiding authorship a) gives the impression of stealing, and clearly opens you up to the accusation of stealing, no matter what your actual intent is? And do you see that vitally important information is being lost? If this application had, as VMMaker does, hundreds of contributors then tracking down who last modified what, which is really important information, is made much harder. My problem here is people assuming stealing by default, instead of > assuming, for example, mistake. > Imagine an alternative scenario: > - X: "Hey Guille, could you add in the copyright X, and Y and Z? They > also contributed to the project, you should take them into account..." > - Guille: "Ah sure, sorry, this was not my intention, I'm so stupid, I > forgot about Z. Commit push, done". > > If instead of bashing on people, we wanted to discuss on how to actually > FIX the thing, here are my 2 cents: > - From a copyright perspective it should have been enough to check the > licence file and name the contributors there > - The history could have been retrieved in a separate branch and then > merged (and look, we had the best of the two worlds!) > - both of the things could have been then integrated through a pull > request (luckily in less than one hour :)) > Well, for me, it its not about copyright, it is about authorship, attribution and communication. I /have/ to be able to determine original authors to have a chance at getting help with issues that arise. I hope the use case helps motivate you to preserve authorship. > > And at the end, with the apport of everybody we could have got a > repository with history, baseline and working on pharo 7. > > Now, from a human perspective, please let's try everybody to assume the > best of the other by default. > That will just make all interactions much more healthy. > > And please, this is not particularly directed to anybody, I've seen such > remarks many times. Let's just think positive. > Problems can be fixed if we talk about them, but more specifically if we > are looking for solutions. > _,,,^..^,,,_ best, Eliot

Oct. 1, 2018

Re: [Pharo-dev] [Pharo-users] [ANN] Success story Mobility Map

by Nicolas Cellier

Hi Norbert, Le lun. 1 oct. 2018 Ã 19:22, Norbert Hartl <norbert(a)hartl.name> a Ã©crit : > > > Am 26.09.2018 um 19:22 schrieb Nicolas Cellier < > nicolas.cellier.aka.nice(a)gmail.com>: > > Very nice proof that we can leverage efficient and up to date technologies > ! > > > That was part of the plan ;) > > Thank you so much for sharing, that's the right way to make Pharo (and > Smalltalk) alive and kicking. How did the Pharo IDE help in such context? > (did you use debugging facility extensively?). > What tool is missing? > > > At first pharo is always a bliss to work with. In such a project I try to > make the whole application managable on different levels. The most > important one is to have the whole application in one image. We have tests > for the broker and for each micro service. But we also have tests that > operate on the whole stack. Here the Q is being shortcut and handling is > more synchronous than in the Q case. With this it is easier to get a stack > in the debugger of the whole roundtrip. > > Using docker we can have the whole application started on a local laptop. > This way all components are much closer to investigate. We can switch the > frontend server with a local development entity. I started to do that for > the pharo images but did not finish yet. The idea is to start the same > stack as in the swarm but replace one image with an actual development > image where you can use the debugger. > > On the swarm itself we configure each microservice to upload a fuel > context dump to a central server on exception. From my development image I > have a simple client to look for exceptions for a project and version > number. Clicking on one it downloads the dump and opens a debugger locally. > I can fix the bug and commit with iceberg. This goes well with our > continuous deployment. When a commit is done, jenkins builds the whole > product and deploys that automatically on the alpha swarm. This way from > seeing an error the only thing to do is clicking on it solve the problem in > the debugger and commit. Well, in most of the cases ;) > > What Iâm working on and gave a quick preview on ESUg is to have a client > for docker swarm. It is another way to close the cycle to use pharo to > manage things in a swarm that is build from pharo images. I did a first > prototype how to connect on a particular image in the swarm and start > TelePharo on it so you can set breakpoints for certain things and have a > debugger in your image from the live swarm. > > The last things we did is to add proper monitoring of metrics withing the > service so you can spot problems that belong to any kind of resource > shortage. In this case it would be especially useful to connect to such an > image to do live investigations. Yes and object centric debugging/logging > will help here Steven/guys ;) > > Or two say it in less words :) The two problems we had to solve is to > remove complexity where possible and to have all the mentioned approaches > to enable our team to tackle an occurring problem from different angles. If > no one is blocked in work the project does not stagnate. It is not a > guarantee to succeed but a requirement > > Hope this is the information you were asking for. > > Norbert > Yes, thanks for these details, that's exactly the kind of testimony i'm after. I wish i were able to demonstrate the advantages by myself to some teams not working with Pharo, but i belong to dinosaures era of desktop apps, i can preach, but not easily practice. The team has chosen to work in heterogeneous languages/environments so the idea of having services integrated in single image while very interesting, won't directly translate. I don't know if Pharo will shine in this context for developing a single service as a POC, and i guess that remote debugging will indeed be a must in such more adverse ( less integrated) environment. I'll take time to reread your detailed answer and also transmit the pointers to the team. Thanks again. > > > Le mar. 25 sept. 2018 Ã 17:45, Sven Van Caekenberghe <sven(a)stfx.eu> a > Ã©crit : > >> >> >> > On 25 Sep 2018, at 14:39, Norbert Hartl <norbert(a)hartl.name> wrote: >> > >> > >> > >> >> Am 25.09.2018 um 12:52 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: >> >> >> >> Wow. Very nice, well done. >> >> >> >> Any chance on some more technical details, as in what 'connected by a >> message queue for the communication' exactly means ? How did you approach >> micro services exactly ? >> >> >> > Sure :) >> >> Thanks, this is very interesting ! >> >> > The installation spawns multiple physical machines. All the machines >> are joined to a docker swarm. The installation is reified as either task or >> service from the view on the docker swarm. Meaning you instantiate an >> arbitrary amount of services and docker swarm distributes them among the >> physical machines. Usually you donât take control which is running where >> but you can. At this point you have spread dozens of pharo images among >> multiple machines and each of them has an IP address. Furthermore in docker >> swarm you have a reification of a network meaning that every instance in a >> network can see all other instances on this network. Each service can be >> reached by its service name in that network. Docker swarm does all the >> iptables/firewall and DNS setup for you. >> >> Are you happy with docker swarm's availability/fail-over behaviour ? In >> other words: does it work when one image/instance goes bad, does it detect >> and restore the missing functionality ? >> >> > In order to have communication between those runtimes we use rabbitmq >> because you were so nice writing a driver for it ;) The rabbitmq does have >> a support for cluster setup, meaning each of the physical machines has a >> rabbitmq installation and they know each other. So it does not matter to >> which instance you send messages to and on which you register for receiving >> messages. So every pharo image connects to the service rabbitmq and opens a >> queue for interaction. >> >> Same question: does RabbitMQ's clustering work well under stress/problems >> ? Syncing all queues between all machines sounds quite heavy (I never tried >> it, but maybe it just works). >> >> > Each service like the car sharing opens a queue e.g. /queue/carSharing >> and listens on it. The broker images are stateful so they open queues like >> /queue/mobility-map-afdeg32 where afdeg32 is the container id of the >> instance (hostname in docker). In each request the queue name to reply is >> sent as a header. So we can make sure that the right image gets the message >> back. This way we can have sticky sessions keeping volatile data in memory >> for the lifecycle of a session. There is one worker image which opens a >> queue /queue/mobility-map where session independent requests can be >> processed. >> >> I think I understand ;-) >> >> > In order to ease development we are sharing code between the broker and >> the micro service. Each micro service has a -Common package where the >> classes are in that build the interface. The classes in here are a kind of >> data entity facades. They use NeoJSON to map to and from a stream. The >> class name is send with the message as a header so the remote side knows >> what to materialize. The handling is unified for the four cases >> > >> > - Request as inquiry to another micro service >> > - Response returns values to a Request >> > - Error is transferred like a Response but is then signalled on the >> receiving side >> > - Notification connects the announcers on the broker and the micro >> service side. >> >> Yes, makes total sense. >> >> > Asynchronous calls we solved using Promises and Futures. Each async >> call to the Q becomes a promise (that blocks on #value) and is combined to >> a future value containing all promises with support to generate a delta of >> all resolved promises. This we need because you issue a search that takes >> longer and you want to display results as soon as they are resolved not >> after all haven been resolved. >> >> Which Promise/Future framework/library are you using in Pharo ? >> >> You did not go for single threaded worker images ? >> >> > And a lot more. This is a coarse grained overview over the >> architecture. Iâm happy to answer further questions about this. >> > >> > Norbert >> > >> >>> On 25 Sep 2018, at 12:20, Norbert Hartl <norbert(a)hartl.name> wrote: >> >>> >> >>> As presented on ESUG here is the brief description of one of our >> current projects. >> >>> >> >>> Mobility Map >> >>> ââââââ >> >>> >> >>> Mobility Map is a broker for mobility services. It offers multi-modal >> routing search enabling users to find the best travel options between >> locations. Travel options include car sharing, bikes, trains, busses etc. >> Rented cars can be offered for ride sharing on booking time letting other >> people find it to participate in the ride. Single travel options are >> combined in travel plans that can be booked and managed in a very easy way. >> >>> >> >>> For this project main requirements were scalability to serve a large >> user base and flexibility to add more additional providers to the broker. >> The application has been realized using web technologies for the frontend >> and pharo for the backend. Using a microservice architecture combined with >> a broker it is easy to extend the platform with additional brokers. >> Deployment is done using docker swarm for distributing dozens of pharo >> images among multiple server machines connected by a message queue for the >> communication. Pharo supported that scenario very well enabling us the meet >> the requirements with less effort. >> >>> >> >>> Pharo turned out to be a perfect fit to develop the application in a >> agile way. Small development cycles with continuous integration and >> continuous delivery enables fast turnarounds for the customers to validate >> progress. >> >>> >> >>> This is a screenshot of the search page for multi-modal results: >> >>> >> >>> >> >>> <Screen Shot 2018-09-21 at 16.54.30.png> >> >> >> >> >> > >> > >> >> >> >

Oct. 1, 2018

Re: [Pharo-dev] [Pharo-users] [ANN] Success story Mobility Map

by Norbert Hartl

> Am 30.09.2018 um 13:01 schrieb Pierce Ng <pierce(a)samadhiweb.com>: > > On Wed, Sep 26, 2018 at 07:49:10PM +0200, Norbert Hartl wrote: >>>> And a lot more. This is a coarse grained overview over the >>>> architecture. Iâm happy to answer further questions about this. >>>> [very nice writeup] > > Hi Norbert, > > Very nice write-up, thanks. > thanks > What persistence mechanism are you using - Gemstone/S, Glorp, Voyage, â¦? We use voyage with a mongo replica set. Norbert

Oct. 1, 2018

Re: [Pharo-dev] [Pharo-users] [ANN] Success story Mobility Map

by Norbert Hartl

> Am 26.09.2018 um 19:22 schrieb Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: > > Very nice proof that we can leverage efficient and up to date technologies ! That was part of the plan ;) > Thank you so much for sharing, that's the right way to make Pharo (and Smalltalk) alive and kicking. How did the Pharo IDE help in such context? (did you use debugging facility extensively?). > What tool is missing? At first pharo is always a bliss to work with. In such a project I try to make the whole application managable on different levels. The most important one is to have the whole application in one image. We have tests for the broker and for each micro service. But we also have tests that operate on the whole stack. Here the Q is being shortcut and handling is more synchronous than in the Q case. With this it is easier to get a stack in the debugger of the whole roundtrip. Using docker we can have the whole application started on a local laptop. This way all components are much closer to investigate. We can switch the frontend server with a local development entity. I started to do that for the pharo images but did not finish yet. The idea is to start the same stack as in the swarm but replace one image with an actual development image where you can use the debugger. On the swarm itself we configure each microservice to upload a fuel context dump to a central server on exception. From my development image I have a simple client to look for exceptions for a project and version number. Clicking on one it downloads the dump and opens a debugger locally. I can fix the bug and commit with iceberg. This goes well with our continuous deployment. When a commit is done, jenkins builds the whole product and deploys that automatically on the alpha swarm. This way from seeing an error the only thing to do is clicking on it solve the problem in the debugger and commit. Well, in most of the cases ;) What Iâm working on and gave a quick preview on ESUg is to have a client for docker swarm. It is another way to close the cycle to use pharo to manage things in a swarm that is build from pharo images. I did a first prototype how to connect on a particular image in the swarm and start TelePharo on it so you can set breakpoints for certain things and have a debugger in your image from the live swarm. The last things we did is to add proper monitoring of metrics withing the service so you can spot problems that belong to any kind of resource shortage. In this case it would be especially useful to connect to such an image to do live investigations. Yes and object centric debugging/logging will help here Steven/guys ;) Or two say it in less words :) The two problems we had to solve is to remove complexity where possible and to have all the mentioned approaches to enable our team to tackle an occurring problem from different angles. If no one is blocked in work the project does not stagnate. It is not a guarantee to succeed but a requirement Hope this is the information you were asking for. Norbert > > Le mar. 25 sept. 2018 Ã 17:45, Sven Van Caekenberghe <sven(a)stfx.eu <mailto:sven@stfx.eu>> a Ã©crit : > > > > On 25 Sep 2018, at 14:39, Norbert Hartl <norbert(a)hartl.name <mailto:norbert@hartl.name>> wrote: > > > > > > > >> Am 25.09.2018 um 12:52 schrieb Sven Van Caekenberghe <sven(a)stfx.eu <mailto:sven@stfx.eu>>: > >> > >> Wow. Very nice, well done. > >> > >> Any chance on some more technical details, as in what 'connected by a message queue for the communication' exactly means ? How did you approach micro services exactly ? > >> > > Sure :) > > Thanks, this is very interesting ! > > > The installation spawns multiple physical machines. All the machines are joined to a docker swarm. The installation is reified as either task or service from the view on the docker swarm. Meaning you instantiate an arbitrary amount of services and docker swarm distributes them among the physical machines. Usually you donât take control which is running where but you can. At this point you have spread dozens of pharo images among multiple machines and each of them has an IP address. Furthermore in docker swarm you have a reification of a network meaning that every instance in a network can see all other instances on this network. Each service can be reached by its service name in that network. Docker swarm does all the iptables/firewall and DNS setup for you. > > Are you happy with docker swarm's availability/fail-over behaviour ? In other words: does it work when one image/instance goes bad, does it detect and restore the missing functionality ? > > > In order to have communication between those runtimes we use rabbitmq because you were so nice writing a driver for it ;) The rabbitmq does have a support for cluster setup, meaning each of the physical machines has a rabbitmq installation and they know each other. So it does not matter to which instance you send messages to and on which you register for receiving messages. So every pharo image connects to the service rabbitmq and opens a queue for interaction. > > Same question: does RabbitMQ's clustering work well under stress/problems ? Syncing all queues between all machines sounds quite heavy (I never tried it, but maybe it just works). > > > Each service like the car sharing opens a queue e.g. /queue/carSharing and listens on it. The broker images are stateful so they open queues like /queue/mobility-map-afdeg32 where afdeg32 is the container id of the instance (hostname in docker). In each request the queue name to reply is sent as a header. So we can make sure that the right image gets the message back. This way we can have sticky sessions keeping volatile data in memory for the lifecycle of a session. There is one worker image which opens a queue /queue/mobility-map where session independent requests can be processed. > > I think I understand ;-) > > > In order to ease development we are sharing code between the broker and the micro service. Each micro service has a -Common package where the classes are in that build the interface. The classes in here are a kind of data entity facades. They use NeoJSON to map to and from a stream. The class name is send with the message as a header so the remote side knows what to materialize. The handling is unified for the four cases > > > > - Request as inquiry to another micro service > > - Response returns values to a Request > > - Error is transferred like a Response but is then signalled on the receiving side > > - Notification connects the announcers on the broker and the micro service side. > > Yes, makes total sense. > > > Asynchronous calls we solved using Promises and Futures. Each async call to the Q becomes a promise (that blocks on #value) and is combined to a future value containing all promises with support to generate a delta of all resolved promises. This we need because you issue a search that takes longer and you want to display results as soon as they are resolved not after all haven been resolved. > > Which Promise/Future framework/library are you using in Pharo ? > > You did not go for single threaded worker images ? > > > And a lot more. This is a coarse grained overview over the architecture. Iâm happy to answer further questions about this. > > > > Norbert > > > >>> On 25 Sep 2018, at 12:20, Norbert Hartl <norbert(a)hartl.name <mailto:norbert@hartl.name>> wrote: > >>> > >>> As presented on ESUG here is the brief description of one of our current projects. > >>> > >>> Mobility Map > >>> ââââââ > >>> > >>> Mobility Map is a broker for mobility services. It offers multi-modal routing search enabling users to find the best travel options between locations. Travel options include car sharing, bikes, trains, busses etc. Rented cars can be offered for ride sharing on booking time letting other people find it to participate in the ride. Single travel options are combined in travel plans that can be booked and managed in a very easy way. > >>> > >>> For this project main requirements were scalability to serve a large user base and flexibility to add more additional providers to the broker. The application has been realized using web technologies for the frontend and pharo for the backend. Using a microservice architecture combined with a broker it is easy to extend the platform with additional brokers. Deployment is done using docker swarm for distributing dozens of pharo images among multiple server machines connected by a message queue for the communication. Pharo supported that scenario very well enabling us the meet the requirements with less effort. > >>> > >>> Pharo turned out to be a perfect fit to develop the application in a agile way. Small development cycles with continuous integration and continuous delivery enables fast turnarounds for the customers to validate progress. > >>> > >>> This is a screenshot of the search page for multi-modal results: > >>> > >>> > >>> <Screen Shot 2018-09-21 at 16.54.30.png> > >> > >> > > > > > >

Oct. 1, 2018

[Pharo 7.0-dev] Build #1287: 22271-Fix-IndexedSlots-in-combination-with-StatefulTraits-

by ci-pharo-ci-jenkins2＠inria.fr

There is a new Pharo build available! The status of the build #1287 was: SUCCESS. The Pull Request #1730 was integrated: "22271-Fix-IndexedSlots-in-combination-with-StatefulTraits-" Pull request url: https://github.com/pharo-project/pharo/pull/1730 Issue Url: https://pharo.fogbugz.com/f/cases/22271 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%2…

Oct. 1, 2018