December 2017 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Flexibility of ZnUrl (For Metacello evolution)

by Dale Henrichs

On 12/29/17 3:55 AM, Sven Van Caekenberghe wrote: > >> On 29 Dec 2017, at 10:35, Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com> wrote: >> >> Hi, >> >> In order to improve the usage of Pharo and the git integration there is >> some discussions around Metacello in order to be able to: >> >> - Load a private git project from Metacello API >> - Load a project hosted on a private server from Metacello API >> - Load a project hosted on a private server with non default ssh port >> from Metacello API >> - Depend on a private git project in a baseline >> - Depend on a project hosted on a private server in a baseline >> - Depend on a project hosted on a private server with non default ssh >> port in a baseline >> >> To achieve this, Metacello needs to be able to manage urls like: >> >> - 'git@gitlab.ferlicot.fr:Projet/Bazard.git' >> - 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' >> >> and many others. >> >> Dale wants to use Zinc to manage urls's in Pharo. >> >> It works great for this url: >> >> 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' asUrl >> >> It gives an instance of ZnUrl of the ssh scheme, gitlab.ferlicot.fr as >> host, 3456 as port, git as username and #('Project' 'Bazard.git') as >> segments. >> >> Now the problem is that it does not work well for this url: >> >> 'git@gitlab.ferlicot.fr:Projet/Bazard.git' asUrl > ZnUrl fromString: 'git(a)gitlab.ferlicot.fr/Projet/Bazard.git' defaultScheme: #http. > > ZnUrl fromString: 'git(a)gitlab.ferlicot.fr/Projet/Bazard.git' defaultScheme: #ssh. > > URI/URL are put in classes or types based on their scheme. If there is no explicit scheme in the URI/URL itself, you have to supply a default one. > > That being said, the general syntax supported by ZnUrl is approximately > > scheme://username:password@host:port/segments?query#fragment > > The colon right after the host is reserved for a port, a path has to start with a / (unless you are using relative URL's, but then you need a parent context). This is why I changed your example, to make it work. > > There literally exist hundreds of schemes, some with very weird syntax. That is indeed the prerogative of a scheme, to define its own special syntax. However, the straightforward approach of the ZnUrl parser cannot recognise those. > > The question then becomes: how to provide an extension mechanism for exotic schemes ? > > Although the question has come up a couple of times, it has never been very urgent. > > I know that some people have programmed around this limitation, maybe they could tell us what they did. > Thanks Sven. As Cyril points out we are considering adding a new batch of urls for repository descriptions and the first order of business is to solve the parsing issue --- I would prefer NOT to bring back the old Url hiearchy --- which is what I would do if forced to provide a Metacello-specific solution ... The overall goal is to be able have Metacello execute an expression that looks like the following: Â repositoryDescriptionString asUrl asMetacelloRepository where the repositoryDescriptionString could be one of the following: Â 'http://seaside.gemtalksystems.com/ss/metacello' Â 'github://dalehenrich/filetree:pharo1.1/repository' Â 'filetree://$GS_HOME/shared/repos/metacello/repository' Note that the last two repository descriptions are accepted by ZnUrl, but require additional parsing tease out the required fields. Cyril's examples are what is used by git as a repository description: Â 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' Â 'git@gitlab.ferlicot.fr:Projet/Bazard.git' Note that in these two examples from Cyril, we are missing some critical information: commitish and repository path need for a complete Metacello repository description, so I don't consider these repository descriptions complete. Thierry Goubier addressed these particular issues with his GitFileTree repository description[1]: Â 'gitfiletree://gitlab.ferlicot.fr:3456:Projet/Bazard:dev/src' My inclination is to base the custom urls on Thierry's gitfiletree model, but we are still in the process of gathering requirements and parsing technology will be an issue once we've settled on the standard set of repository descriptions. In the past Thierry has mentioned that "I would have reused ZnUrl to do that [iceberg using regex], but maybe they wanted to avoid ZnUrl errors about unknown schemes. I had to copy/paste some of the ZnUrl code into GitFileTree for that purpose."[2] Looking at Thierry's implementation[1] it does look like he is able to pack all of the critical information into the repository description, but as with the github:// and filetree:// schemes additional parsing is required to tease out all of the fields ... I have not seen examples of the repository descriptions used by iceberg (yet) ... So, Sven, it would be interesting to hear from you about any ideas that you might have to allow for the handling of custom url parsing within the ZnUrl framework... ideally we'd not have to do any scheme-specific post-parsing of the url and the #asMetacelloRepository method could be implemented to just send messages to extract the various values of the custom fields ... as with the old Url hierarchy, allowing subclasses to implement custom parsers based on the scheme could be done, but it isn't the only approach. I do want to point out that the #asMetacelloRepository is intended to return a repository instance from which packages can be loaded by Metacello... and the intent is to allow developers to be able to choose which platform tool will be used to manage the repository. The idea is that platform tools will register to handle a particular scheme, so the a developer can choose whether to use gitfiletree or iceberg or <the next generation repository management tool> to manage the repository for the github:// scheme, etc. This would eliminate the current implementation where we have a fixed map of schemes to tools and developers to invent a custom url (and patch Metacello) for each new tool. Dale [1] https://github.com/dalehenrich/filetree/blob/734eed46ea57ebf5e24e5d935768bd… [2] https://github.com/Metacello/metacello/issues/474#issuecomment-350871020

Dec. 29, 2017

Re: [Pharo-dev] Flexibility of ZnUrl (For Metacello evolution)

by Norbert Hartl

> Am 29.12.2017 um 10:35 schrieb Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com>: > > Hi, > > In order to improve the usage of Pharo and the git integration there is > some discussions around Metacello in order to be able to: > > - Load a private git project from Metacello API > - Load a project hosted on a private server from Metacello API > - Load a project hosted on a private server with non default ssh port > from Metacello API > - Depend on a private git project in a baseline > - Depend on a project hosted on a private server in a baseline > - Depend on a project hosted on a private server with non default ssh > port in a baseline > > To achieve this, Metacello needs to be able to manage urls like: > > - 'git@gitlab.ferlicot.fr:Projet/Bazard.git' > - 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' > > and many others. > > Dale wants to use Zinc to manage urls's in Pharo. > > It works great for this url: > > 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' asUrl > > It gives an instance of ZnUrl of the ssh scheme, gitlab.ferlicot.fr as > host, 3456 as port, git as username and #('Project' 'Bazard.git') as > segments. > > Now the problem is that it does not work well for this url: > > 'git@gitlab.ferlicot.fr:Projet/Bazard.git' asUrl > > It gives an instance of ZnUrl with #('git@gitlab.ferlicot.fr:Projet' > 'Bazard.git') as segments. > > What do you think guys about this issue? Could the flexibility of ZnUrls > be improved in order to be able to use it? Should Metacello use another way? > Why ZnURL should be flexible to treat something that is not a URI? An URI is something like <some ascii chars>:<some arbitrary stuff> The important part is the colon. If you split on it then the first part is addressing a name that can be resolved to get an entity that can digest the second part. Thatâs it! A locator is just a special form of URI that cheats by saying that the second part can be used directly to locate the resource (thatâs the difference by identifiable resources and locatable resources meaning the ones that really exist) So I cannot see how this can work. But nobody says you need to fold this into ZnURL. A Client is free to do something like if it starts with git I prepend that with ssh:// or git:// or git+ssh:// So the point is that if your model is the URI (it does not have to) then you should create valid URLs by adding missing stuff. Otherwise just have a model that treats URLs and your special cases. Norbert > To see the full discussions: > - Github issue: https://github.com/Metacello/metacello/issues/474 > - Metacello group discussion: > https://groups.google.com/forum/#!topic/metacello/QbpngL6Jhg8 > > Have a nice day :) > > -- > Cyril Ferlicot > https://ferlicot.fr > > http://www.synectique.eu > 2 rue Jacques PrÃ©vert 01, > 59650 Villeneuve d'ascq France >

Dec. 29, 2017

Re: [Pharo-dev] Flexibility of ZnUrl (For Metacello evolution)

by Sven Van Caekenberghe

> On 29 Dec 2017, at 10:35, Cyril Ferlicot D. <cyril.ferlicot(a)gmail.com> wrote: > > Hi, > > In order to improve the usage of Pharo and the git integration there is > some discussions around Metacello in order to be able to: > > - Load a private git project from Metacello API > - Load a project hosted on a private server from Metacello API > - Load a project hosted on a private server with non default ssh port > from Metacello API > - Depend on a private git project in a baseline > - Depend on a project hosted on a private server in a baseline > - Depend on a project hosted on a private server with non default ssh > port in a baseline > > To achieve this, Metacello needs to be able to manage urls like: > > - 'git@gitlab.ferlicot.fr:Projet/Bazard.git' > - 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' > > and many others. > > Dale wants to use Zinc to manage urls's in Pharo. > > It works great for this url: > > 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' asUrl > > It gives an instance of ZnUrl of the ssh scheme, gitlab.ferlicot.fr as > host, 3456 as port, git as username and #('Project' 'Bazard.git') as > segments. > > Now the problem is that it does not work well for this url: > > 'git@gitlab.ferlicot.fr:Projet/Bazard.git' asUrl ZnUrl fromString: 'git(a)gitlab.ferlicot.fr/Projet/Bazard.git' defaultScheme: #http. ZnUrl fromString: 'git(a)gitlab.ferlicot.fr/Projet/Bazard.git' defaultScheme: #ssh. URI/URL are put in classes or types based on their scheme. If there is no explicit scheme in the URI/URL itself, you have to supply a default one. That being said, the general syntax supported by ZnUrl is approximately scheme://username:password@host:port/segments?query#fragment The colon right after the host is reserved for a port, a path has to start with a / (unless you are using relative URL's, but then you need a parent context). This is why I changed your example, to make it work. There literally exist hundreds of schemes, some with very weird syntax. That is indeed the prerogative of a scheme, to define its own special syntax. However, the straightforward approach of the ZnUrl parser cannot recognise those. The question then becomes: how to provide an extension mechanism for exotic schemes ? Although the question has come up a couple of times, it has never been very urgent. I know that some people have programmed around this limitation, maybe they could tell us what they did. > It gives an instance of ZnUrl with #('git@gitlab.ferlicot.fr:Projet' > 'Bazard.git') as segments. > > What do you think guys about this issue? Could the flexibility of ZnUrls > be improved in order to be able to use it? Should Metacello use another way? > > To see the full discussions: > - Github issue: https://github.com/Metacello/metacello/issues/474 > - Metacello group discussion: > https://groups.google.com/forum/#!topic/metacello/QbpngL6Jhg8 > > Have a nice day :) > > -- > Cyril Ferlicot > https://ferlicot.fr > > http://www.synectique.eu > 2 rue Jacques PrÃ©vert 01, > 59650 Villeneuve d'ascq France >

Dec. 29, 2017

Re: [Pharo-dev] New Year Wishlist (2018) ?

by philippe.back＠highoctane.be

On Dec 29, 2017 02:48, "Pierce Ng" <pierce(a)samadhiweb.com> wrote: On Wed, Dec 27, 2017 at 04:41:56PM +0100, Cyril Ferlicot D. wrote: > I also forgot: a real headless :) And real "embeddability". :-) I'll write GUIs with Pascal/Lazarus if I must. That would actually suit me very well too. Being able to implement domain logic stuff in Pharo embedded into the application will super charge productivity. +1 million Phil Pierce

Dec. 29, 2017

Re: [Pharo-dev] Best wishes for 2018

by Hilaire

Don't worrry, we love you guy! Happy new year Le 29/12/2017 Ã 10:27, Esteban Lorenzano a Ã©critÂ : > Thanks to everybody for allow me to be your âguy to hateâ:) > I will continue doing my best to make Pharo a better place for everybody, one step at a time. -- Dr. Geo http://drgeo.eu

Dec. 29, 2017

Flexibility of ZnUrl (For Metacello evolution)

by Cyril Ferlicot D.

Hi, In order to improve the usage of Pharo and the git integration there is some discussions around Metacello in order to be able to: - Load a private git project from Metacello API - Load a project hosted on a private server from Metacello API - Load a project hosted on a private server with non default ssh port from Metacello API - Depend on a private git project in a baseline - Depend on a project hosted on a private server in a baseline - Depend on a project hosted on a private server with non default ssh port in a baseline To achieve this, Metacello needs to be able to manage urls like: - 'git@gitlab.ferlicot.fr:Projet/Bazard.git' - 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' and many others. Dale wants to use Zinc to manage urls's in Pharo. It works great for this url: 'ssh://git@gitlab.ferlicot.fr:3456/Projet/Bazard.git' asUrl It gives an instance of ZnUrl of the ssh scheme, gitlab.ferlicot.fr as host, 3456 as port, git as username and #('Project' 'Bazard.git') as segments. Now the problem is that it does not work well for this url: 'git@gitlab.ferlicot.fr:Projet/Bazard.git' asUrl It gives an instance of ZnUrl with #('git@gitlab.ferlicot.fr:Projet' 'Bazard.git') as segments. What do you think guys about this issue? Could the flexibility of ZnUrls be improved in order to be able to use it? Should Metacello use another way? To see the full discussions: - Github issue: https://github.com/Metacello/metacello/issues/474 - Metacello group discussion: https://groups.google.com/forum/#!topic/metacello/QbpngL6Jhg8 Have a nice day :) -- Cyril Ferlicot https://ferlicot.fr http://www.synectique.eu 2 rue Jacques PrÃ©vert 01, 59650 Villeneuve d'ascq France

Dec. 29, 2017

Re: [Pharo-dev] Best wishes for 2018

by Esteban Lorenzano

Thanks Torsten, I was about to write a similar mail. Thanks to everybody for allow me to be your âguy to hateâ :) I will continue doing my best to make Pharo a better place for everybody, one step at a time. happy new year! Esteban > On 29 Dec 2017, at 09:39, Torsten Bergmann <astares(a)gmx.de> wrote: > > Only a few days left for 2017. Thanks to all who helped shaping Pharos future. > May the lighthouse be with you in 2018 as well: > > World backgroundImage: > (ZnEasy > getJpeg: 'https://spotlight.it-notes.ru/wp-content/uploads/2017/08/02c48424be88ee36d5…') > layout: #scaled > > Have fun! > > Bye > T. >

Dec. 29, 2017

Re: [Pharo-dev] an error in ZnResourceMetaUtils>>queryKeyValueSafeSet ?

by Esteban Lorenzano

Hi Sven, A POST with this form will fail on twitter api: POST /1.1/statuses/update.json HTTP/1.1 Host: api.twitter.com Content-Type: application/x-www-form-urlencoded Accept: */* Content-Length: 0 User-Agent: Zinc HTTP Components 1.0 (Pharo/7.0) status=testing? It will fail because of the â?â, if I remove ? from save and I encode as this: POST /1.1/statuses/update.json HTTP/1.1 Host: api.twitter.com Content-Type: application/x-www-form-urlencoded Accept: */* Content-Length: 0 User-Agent: Zinc HTTP Components 1.0 (Pharo/7.0) status=testing%3F it will be posted correctly. My doubt if this is a bug in queryKeyValueSafeSet or this is a particular problem on twitter. If the last, I didnât see a good way to change value of safe set for just one case (for now I just override the method, but this is clearly not good ;) ). cheers! Esteban > On 28 Dec 2017, at 13:00, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > Hi Esteban, > >> On 28 Dec 2017, at 11:54, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: >> >> Hi all, Sven >> >> Iâm doing some experiments with twitter and posting and I found that twitter does not accept a post with $? in the status string (it throws an error). Now, if I change ZnResourceMetaUtils>>queryKeyValueSafeSet removing $? from the safe set it works correctly. >> >> the question is: this is a twitter error or a ZnResourceMetaUtils>>queryKeyValueSafeSet error and the $? shouldnât be considered safe (and then it should be percent encoded?) >> >> cheers, >> Esteban > > I know that the last time I did the implementation, we looked very carefully in what characters are safe or not, and had some discussions about it. > > I am not familiar with the API you are trying to use. I am a bit confused by POST and 'status string', while encoding a query part of the URL. It is possible to show exactly the call that you are trying to make ? What does an equivalent curl -v call show ? Could you point to the actual API call documentation ? > > Sven > >

Dec. 29, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Denis Kudriashov

2017-12-29 10:07 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: > > > 2017-12-28 22:29 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > >> >> >> 2017-12-28 20:02 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: >> >>> For string searching in source codes it is 1.5s (external sources) vs >>> 3.7s (in image compressed), for recompilation of the image it was for me >>> 51s vs 56s. >>> >> >> I expected much better performance than external file. Can it be >> optimized? >> I guess that every "aMethod sourceCode" decompresses full section. And it >> can be easily cached. >> To measure maximum speed can you check the version with uncompressed >> inmemory sources? >> > > Raw memory file has time for searching for a string in the image 2.1s BUT > if the file is stored on disk, the time is 16.5s so about 10-times slower > than current version. I suppose that the original fast mechanism used > MultiByteFileStream but the new mechanism uses ZnCharacterReadStream. > For file sources we should use ZnBufferedReadStream over ZnCharacterReadStream. It should lead to similar performance as old MultiByteFileStream. > So, in the same conditions the compressed memory stream is almost 5-times > faster than disk access. We should try to find some optimization but I > think that it is not necessary to do it immediately. > > -- Pavel > > >> >> >>> >>> -- Pavel >>> >>> 2017-12-28 17:10 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: >>> >>>> 2017-12-28 17:07 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: >>>> >>>>> Interesting to compare speed of search operation like "find string in >>>>> sources" >>>>> >>>> >>>> With Calypso you can do it by: >>>> >>>> [(ClyMethodSources withString: 'example' from: ClyNavigationEnvironment >>>> currentImageScope ) execute] timeToRun >>>> >>>> >>>> >>>>> >>>>> 2017-12-28 16:21 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: >>>>> >>>>>> >>>>>> >>>>>> 2017-12-28 16:00 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: >>>>>> >>>>>>> Nice. >>>>>>> >>>>>>> 2017-12-28 15:50 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com> >>>>>>> : >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> when the Pharo 7 image is being built on top of the bootstrapped >>>>>>>> image, all loaded code is placed in the changes file so this process ends >>>>>>>> with an empty sources file, huge changes file and the image. We condense >>>>>>>> the sources file for every build so we distribute the Pharo 7 as >>>>>>>> three files - image, huge sources (over 30 MB) and almost empty changes >>>>>>>> file. >>>>>>>> As consequence every build has own sources file that can be shared >>>>>>>> between images based on the same build - so when we will release the >>>>>>>> Pharo 7, the sources file can be shared as it was for previous >>>>>>>> releases, but as soon as some fixing version of Pharo 7 will be >>>>>>>> released, it will require a new specific sources file. >>>>>>>> >>>>>>>> As one possible solution I tried to create a pull request that >>>>>>>> resurrects from oblivion an old code made for Squeak 3.7 that introduced >>>>>>>> option of compressed sources files. Such files use segments that are stored >>>>>>>> in compressed form so it is not so small as one single zip file but still >>>>>>>> good enough. From original uncompressed file of size 30.8 MB it generates >>>>>>>> 7.1 MB file while single zip has 6.1 MB. >>>>>>>> >>>>>>>> Of course the image file is bigger but: >>>>>>>> - the amount of new objects is low so it should not make much more >>>>>>>> stress on GC >>>>>>>> >>>>>>> >>>>>>> And how many objects it adds? (how many sections are compressed?) >>>>>>> >>>>>> >>>>>> It uses currently 1619 segments of size 20000 bytes but that does not >>>>>> mean that such amount of objects is needed because in the memory filesystem >>>>>> all the (compressed) data are in single ByteArray and the stream class uses >>>>>> a very simple segments table - one big array of 1619 integer items that >>>>>> contain indexes to the ByteArray. >>>>>> >>>>>> >>>>>>> >>>>>>>> - we already used to have big images because the condenser was not >>>>>>>> working and it was not a big issue >>>>>>>> - the amount of distributed files will be the same as in case of >>>>>>>> Pharo 6 (image+changes) so it means less troubles >>>>>>>> - the sources file can be easily restored, deleted from the object >>>>>>>> memory and used the same way as before >>>>>>>> - we are not using the nice concept of the image well enough. All >>>>>>>> data that are not really shared between images or serve as backup >>>>>>>> information should be stored inside image >>>>>>>> >>>>>>> >>>>>>> Can we avoid empty changes file? We can create it on demand (when >>>>>>> user modifies code for example). >>>>>>> >>>>>> >>>>>> I think we can because it contains only do-its. We should be able to >>>>>> simply delete it and only modify the image start-up do do not require it. >>>>>> >>>>>> -- Pavel >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Please review this PR. As a side effect it makes the sources file >>>>>>>> mechanism (but not changes file) independent on the FileStream >>>>>>>> classes that are deprecated. >>>>>>>> >>>>>>>> PR: https://github.com/pharo-project/pharo/pull/631 >>>>>>>> >>>>>>>> The pre-built image: >>>>>>>> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20 >>>>>>>> pull%20request%20and%20branch%20Pipeline/job/PR-631/lastSucc >>>>>>>> essfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip >>>>>>>> >>>>>>>> Cheers, >>>>>>>> -- Pavel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >

Dec. 29, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Pavel Krivanek

2017-12-28 22:29 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > > > 2017-12-28 20:02 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: > >> For string searching in source codes it is 1.5s (external sources) vs >> 3.7s (in image compressed), for recompilation of the image it was for me >> 51s vs 56s. >> > > I expected much better performance than external file. Can it be optimized? > I guess that every "aMethod sourceCode" decompresses full section. And it > can be easily cached. > To measure maximum speed can you check the version with uncompressed > inmemory sources? > Raw memory file has time for searching for a string in the image 2.1s BUT if the file is stored on disk, the time is 16.5s so about 10-times slower than current version. I suppose that the original fast mechanism used MultiByteFileStream but the new mechanism uses ZnCharacterReadStream. So, in the same conditions the compressed memory stream is almost 5-times faster than disk access. We should try to find some optimization but I think that it is not necessary to do it immediately. -- Pavel > > >> >> -- Pavel >> >> 2017-12-28 17:10 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: >> >>> 2017-12-28 17:07 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: >>> >>>> Interesting to compare speed of search operation like "find string in >>>> sources" >>>> >>> >>> With Calypso you can do it by: >>> >>> [(ClyMethodSources withString: 'example' from: ClyNavigationEnvironment >>> currentImageScope ) execute] timeToRun >>> >>> >>> >>>> >>>> 2017-12-28 16:21 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: >>>> >>>>> >>>>> >>>>> 2017-12-28 16:00 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: >>>>> >>>>>> Nice. >>>>>> >>>>>> 2017-12-28 15:50 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> when the Pharo 7 image is being built on top of the bootstrapped >>>>>>> image, all loaded code is placed in the changes file so this process ends >>>>>>> with an empty sources file, huge changes file and the image. We condense >>>>>>> the sources file for every build so we distribute the Pharo 7 as >>>>>>> three files - image, huge sources (over 30 MB) and almost empty changes >>>>>>> file. >>>>>>> As consequence every build has own sources file that can be shared >>>>>>> between images based on the same build - so when we will release the >>>>>>> Pharo 7, the sources file can be shared as it was for previous >>>>>>> releases, but as soon as some fixing version of Pharo 7 will be >>>>>>> released, it will require a new specific sources file. >>>>>>> >>>>>>> As one possible solution I tried to create a pull request that >>>>>>> resurrects from oblivion an old code made for Squeak 3.7 that introduced >>>>>>> option of compressed sources files. Such files use segments that are stored >>>>>>> in compressed form so it is not so small as one single zip file but still >>>>>>> good enough. From original uncompressed file of size 30.8 MB it generates >>>>>>> 7.1 MB file while single zip has 6.1 MB. >>>>>>> >>>>>>> Of course the image file is bigger but: >>>>>>> - the amount of new objects is low so it should not make much more >>>>>>> stress on GC >>>>>>> >>>>>> >>>>>> And how many objects it adds? (how many sections are compressed?) >>>>>> >>>>> >>>>> It uses currently 1619 segments of size 20000 bytes but that does not >>>>> mean that such amount of objects is needed because in the memory filesystem >>>>> all the (compressed) data are in single ByteArray and the stream class uses >>>>> a very simple segments table - one big array of 1619 integer items that >>>>> contain indexes to the ByteArray. >>>>> >>>>> >>>>>> >>>>>>> - we already used to have big images because the condenser was not >>>>>>> working and it was not a big issue >>>>>>> - the amount of distributed files will be the same as in case of >>>>>>> Pharo 6 (image+changes) so it means less troubles >>>>>>> - the sources file can be easily restored, deleted from the object >>>>>>> memory and used the same way as before >>>>>>> - we are not using the nice concept of the image well enough. All >>>>>>> data that are not really shared between images or serve as backup >>>>>>> information should be stored inside image >>>>>>> >>>>>> >>>>>> Can we avoid empty changes file? We can create it on demand (when >>>>>> user modifies code for example). >>>>>> >>>>> >>>>> I think we can because it contains only do-its. We should be able to >>>>> simply delete it and only modify the image start-up do do not require it. >>>>> >>>>> -- Pavel >>>>> >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Please review this PR. As a side effect it makes the sources file >>>>>>> mechanism (but not changes file) independent on the FileStream >>>>>>> classes that are deprecated. >>>>>>> >>>>>>> PR: https://github.com/pharo-project/pharo/pull/631 >>>>>>> >>>>>>> The pre-built image: >>>>>>> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20 >>>>>>> pull%20request%20and%20branch%20Pipeline/job/PR-631/lastSucc >>>>>>> essfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip >>>>>>> >>>>>>> Cheers, >>>>>>> -- Pavel >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >

Dec. 29, 2017