- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Denis Kudriashov

2017-12-28 17:07 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > Interesting to compare speed of search operation like "find string in > sources" > With Calypso you can do it by: [(ClyMethodSources withString: 'example' from: ClyNavigationEnvironment currentImageScope ) execute] timeToRun > > 2017-12-28 16:21 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: > >> >> >> 2017-12-28 16:00 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: >> >>> Nice. >>> >>> 2017-12-28 15:50 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: >>> >>>> Hi, >>>> >>>> when the Pharo 7 image is being built on top of the bootstrapped >>>> image, all loaded code is placed in the changes file so this process ends >>>> with an empty sources file, huge changes file and the image. We condense >>>> the sources file for every build so we distribute the Pharo 7 as three >>>> files - image, huge sources (over 30 MB) and almost empty changes file. >>>> As consequence every build has own sources file that can be shared >>>> between images based on the same build - so when we will release the >>>> Pharo 7, the sources file can be shared as it was for previous >>>> releases, but as soon as some fixing version of Pharo 7 will be >>>> released, it will require a new specific sources file. >>>> >>>> As one possible solution I tried to create a pull request that >>>> resurrects from oblivion an old code made for Squeak 3.7 that introduced >>>> option of compressed sources files. Such files use segments that are stored >>>> in compressed form so it is not so small as one single zip file but still >>>> good enough. From original uncompressed file of size 30.8 MB it generates >>>> 7.1 MB file while single zip has 6.1 MB. >>>> >>>> Of course the image file is bigger but: >>>> - the amount of new objects is low so it should not make much more >>>> stress on GC >>>> >>> >>> And how many objects it adds? (how many sections are compressed?) >>> >> >> It uses currently 1619 segments of size 20000 bytes but that does not >> mean that such amount of objects is needed because in the memory filesystem >> all the (compressed) data are in single ByteArray and the stream class uses >> a very simple segments table - one big array of 1619 integer items that >> contain indexes to the ByteArray. >> >> >>> >>>> - we already used to have big images because the condenser was not >>>> working and it was not a big issue >>>> - the amount of distributed files will be the same as in case of Pharo >>>> 6 (image+changes) so it means less troubles >>>> - the sources file can be easily restored, deleted from the object >>>> memory and used the same way as before >>>> - we are not using the nice concept of the image well enough. All data >>>> that are not really shared between images or serve as backup information >>>> should be stored inside image >>>> >>> >>> Can we avoid empty changes file? We can create it on demand (when user >>> modifies code for example). >>> >> >> I think we can because it contains only do-its. We should be able to >> simply delete it and only modify the image start-up do do not require it. >> >> -- Pavel >> >> >>> >>> >>>> >>>> Please review this PR. As a side effect it makes the sources file >>>> mechanism (but not changes file) independent on the FileStream classes >>>> that are deprecated. >>>> >>>> PR: https://github.com/pharo-project/pharo/pull/631 >>>> >>>> The pre-built image: >>>> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20 >>>> pull%20request%20and%20branch%20Pipeline/job/PR-631/lastSuccessfulBuild >>>> /artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip >>>> >>>> Cheers, >>>> -- Pavel >>>> >>>> >>>> >>> >>> >> >

Dec. 28, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Denis Kudriashov

Interesting to compare speed of search operation like "find string in sources" 2017-12-28 16:21 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: > > > 2017-12-28 16:00 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > >> Nice. >> >> 2017-12-28 15:50 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: >> >>> Hi, >>> >>> when the Pharo 7 image is being built on top of the bootstrapped image, >>> all loaded code is placed in the changes file so this process ends with an >>> empty sources file, huge changes file and the image. We condense the >>> sources file for every build so we distribute the Pharo 7 as three >>> files - image, huge sources (over 30 MB) and almost empty changes file. >>> As consequence every build has own sources file that can be shared >>> between images based on the same build - so when we will release the >>> Pharo 7, the sources file can be shared as it was for previous >>> releases, but as soon as some fixing version of Pharo 7 will be >>> released, it will require a new specific sources file. >>> >>> As one possible solution I tried to create a pull request that >>> resurrects from oblivion an old code made for Squeak 3.7 that introduced >>> option of compressed sources files. Such files use segments that are stored >>> in compressed form so it is not so small as one single zip file but still >>> good enough. From original uncompressed file of size 30.8 MB it generates >>> 7.1 MB file while single zip has 6.1 MB. >>> >>> Of course the image file is bigger but: >>> - the amount of new objects is low so it should not make much more >>> stress on GC >>> >> >> And how many objects it adds? (how many sections are compressed?) >> > > It uses currently 1619 segments of size 20000 bytes but that does not mean > that such amount of objects is needed because in the memory filesystem all > the (compressed) data are in single ByteArray and the stream class uses a > very simple segments table - one big array of 1619 integer items that > contain indexes to the ByteArray. > > >> >>> - we already used to have big images because the condenser was not >>> working and it was not a big issue >>> - the amount of distributed files will be the same as in case of Pharo >>> 6 (image+changes) so it means less troubles >>> - the sources file can be easily restored, deleted from the object >>> memory and used the same way as before >>> - we are not using the nice concept of the image well enough. All data >>> that are not really shared between images or serve as backup information >>> should be stored inside image >>> >> >> Can we avoid empty changes file? We can create it on demand (when user >> modifies code for example). >> > > I think we can because it contains only do-its. We should be able to > simply delete it and only modify the image start-up do do not require it. > > -- Pavel > > >> >> >>> >>> Please review this PR. As a side effect it makes the sources file >>> mechanism (but not changes file) independent on the FileStream classes >>> that are deprecated. >>> >>> PR: https://github.com/pharo-project/pharo/pull/631 >>> >>> The pre-built image: >>> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20 >>> pull%20request%20and%20branch%20Pipeline/job/PR-631/lastSuccessfulBuild >>> /artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip >>> >>> Cheers, >>> -- Pavel >>> >>> >>> >> >> >

Dec. 28, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Denis Kudriashov

2017-12-28 16:21 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: > > > 2017-12-28 16:00 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > >> Nice. >> >> 2017-12-28 15:50 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: >> >>> Hi, >>> >>> when the Pharo 7 image is being built on top of the bootstrapped image, >>> all loaded code is placed in the changes file so this process ends with an >>> empty sources file, huge changes file and the image. We condense the >>> sources file for every build so we distribute the Pharo 7 as three >>> files - image, huge sources (over 30 MB) and almost empty changes file. >>> As consequence every build has own sources file that can be shared >>> between images based on the same build - so when we will release the >>> Pharo 7, the sources file can be shared as it was for previous >>> releases, but as soon as some fixing version of Pharo 7 will be >>> released, it will require a new specific sources file. >>> >>> As one possible solution I tried to create a pull request that >>> resurrects from oblivion an old code made for Squeak 3.7 that introduced >>> option of compressed sources files. Such files use segments that are stored >>> in compressed form so it is not so small as one single zip file but still >>> good enough. From original uncompressed file of size 30.8 MB it generates >>> 7.1 MB file while single zip has 6.1 MB. >>> >>> Of course the image file is bigger but: >>> - the amount of new objects is low so it should not make much more >>> stress on GC >>> >> >> And how many objects it adds? (how many sections are compressed?) >> > > It uses currently 1619 segments of size 20000 bytes but that does not mean > that such amount of objects is needed because in the memory filesystem all > the (compressed) data are in single ByteArray and the stream class uses a > very simple segments table - one big array of 1619 integer items that > contain indexes to the ByteArray. > > >> >>> - we already used to have big images because the condenser was not >>> working and it was not a big issue >>> - the amount of distributed files will be the same as in case of Pharo >>> 6 (image+changes) so it means less troubles >>> - the sources file can be easily restored, deleted from the object >>> memory and used the same way as before >>> - we are not using the nice concept of the image well enough. All data >>> that are not really shared between images or serve as backup information >>> should be stored inside image >>> >> >> Can we avoid empty changes file? We can create it on demand (when user >> modifies code for example). >> > > I think we can because it contains only do-its. We should be able to > simply delete it and only modify the image start-up do do not require it. > It is unbelievable that we can have single image file. I think it is very attractive goal for Pharo 7. And it is so close. > > -- Pavel > > >> >> >>> >>> Please review this PR. As a side effect it makes the sources file >>> mechanism (but not changes file) independent on the FileStream classes >>> that are deprecated. >>> >>> PR: https://github.com/pharo-project/pharo/pull/631 >>> >>> The pre-built image: >>> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20 >>> pull%20request%20and%20branch%20Pipeline/job/PR-631/lastSuccessfulBuild >>> /artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip >>> >>> Cheers, >>> -- Pavel >>> >>> >>> >> >> >

Dec. 28, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Pavel Krivanek

I'm trying this DrGeo image on Linux VM and I have no problems with Athens but with the FreeType. The image does not start and reports following bugs: FT2Error: Freetype2 primitive failed [error 1][cannot open resource] FreeTypeFace(FT2Handle)>>primitiveFailed: FreeTypeFace(FT2Handle)>>primitiveFailed FreeTypeFace(FT2Face)>>primNewFaceFromFile:index: FreeTypeFace(FT2Face)>>newFaceFromFile:index: FreeTypeFace>>create FreeTypeFace>>validate FreeTypeFont>>face To be able to run it I had to remove the FreeType plugin from the VM but then I have no Athens error. Try to build this image without opening of the main DrGeo window. -- Pavel 2017-12-28 16:29 GMT+01:00 Hilaire <hilaire(a)drgeo.eu>: > Did a successful build of drgeo with this image. However Athens Canvas > broken again (see screenshot). Is it because missing image? There was not > complain during the image build though. Still don't know how to get the > trace of the error log. > > Build image is here[1] in case it is useful. > > Hilaire > > [1] https://www.dropbox.com/s/wc18e21p371z28f/DrGeo.app-18.01a.zip?dl=0 > > > > Le 28/12/2017 Ã 15:50, Pavel Krivanek a Ã©crit : > >> >> Please review this PR. As a side effect it makes the sources file >> mechanism (but not changes file) independent on the FileStream classes that >> are deprecated. >> >> > -- > Dr. Geo > http://drgeo.eu > >

Dec. 28, 2017

Re: [Pharo-dev] Simple game framework

by Hilaire

IF there is a Pharo binding for SDL, why not audio from SDSL? Le 25/12/2017 Ã 19:32, ClÃ©ment Bera a Ã©critÂ : > Audio has always been a problem to me. I successfully run audio files > with the open AL binding, but I failed to do so cross-platform. > Already key-bindings coherent cross platform is difficult. > -- Dr. Geo http://drgeo.eu

Dec. 28, 2017

Re: [Pharo-dev] Simple game framework

by Hilaire

What is the reason using Cairo over SDSL? Is not SDL just enought? Hilaire Le 25/12/2017 Ã 19:32, ClÃ©ment Bera a Ã©critÂ : > > https://github.com/clementbera/wizard-battle-arena > > https://github.com/clementbera/SpiderInvasion > > Audio has always been a problem to me. I successfully run audio files > with the open AL binding, but I failed to do so cross-platform. > Already key-bindings coherent cross platform is difficult. > -- Dr. Geo http://drgeo.eu

Dec. 28, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Hilaire

Did a successful build of drgeo with this image. However Athens Canvas broken again (see screenshot). Is it because missing image? There was not complain during the image build though.Â Still don't know how to get the trace of the error log. Build image is here[1] in case it is useful. Hilaire [1] https://www.dropbox.com/s/wc18e21p371z28f/DrGeo.app-18.01a.zip?dl=0 Le 28/12/2017 Ã 15:50, Pavel Krivanek a Ã©critÂ : > > Please review this PR. As a side effect it makes the sources file > mechanism (but not changes file) independent on the FileStream classes > that are deprecated. > -- Dr. Geo http://drgeo.eu

Dec. 28, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Pavel Krivanek

2017-12-28 16:00 GMT+01:00 Denis Kudriashov <dionisiydk(a)gmail.com>: > Nice. > > 2017-12-28 15:50 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: > >> Hi, >> >> when the Pharo 7 image is being built on top of the bootstrapped image, >> all loaded code is placed in the changes file so this process ends with an >> empty sources file, huge changes file and the image. We condense the >> sources file for every build so we distribute the Pharo 7 as three files >> - image, huge sources (over 30 MB) and almost empty changes file. >> As consequence every build has own sources file that can be shared >> between images based on the same build - so when we will release the >> Pharo 7, the sources file can be shared as it was for previous releases, >> but as soon as some fixing version of Pharo 7 will be released, it will >> require a new specific sources file. >> >> As one possible solution I tried to create a pull request that resurrects >> from oblivion an old code made for Squeak 3.7 that introduced option of >> compressed sources files. Such files use segments that are stored in >> compressed form so it is not so small as one single zip file but still good >> enough. From original uncompressed file of size 30.8 MB it generates 7.1 MB >> file while single zip has 6.1 MB. >> >> Of course the image file is bigger but: >> - the amount of new objects is low so it should not make much more stress >> on GC >> > > And how many objects it adds? (how many sections are compressed?) > It uses currently 1619 segments of size 20000 bytes but that does not mean that such amount of objects is needed because in the memory filesystem all the (compressed) data are in single ByteArray and the stream class uses a very simple segments table - one big array of 1619 integer items that contain indexes to the ByteArray. > >> - we already used to have big images because the condenser was not >> working and it was not a big issue >> - the amount of distributed files will be the same as in case of Pharo 6 >> (image+changes) so it means less troubles >> - the sources file can be easily restored, deleted from the object memory >> and used the same way as before >> - we are not using the nice concept of the image well enough. All data >> that are not really shared between images or serve as backup information >> should be stored inside image >> > > Can we avoid empty changes file? We can create it on demand (when user > modifies code for example). > I think we can because it contains only do-its. We should be able to simply delete it and only modify the image start-up do do not require it. -- Pavel > > >> >> Please review this PR. As a side effect it makes the sources file >> mechanism (but not changes file) independent on the FileStream classes >> that are deprecated. >> >> PR: https://github.com/pharo-project/pharo/pull/631 >> >> The pre-built image: >> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20 >> pull%20request%20and%20branch%20Pipeline/job/PR-631/lastSuccessfulBuild >> /artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip >> >> Cheers, >> -- Pavel >> >> >> > >

Dec. 28, 2017

Re: [Pharo-dev] PR: Compressed sources inside the object memory

by Denis Kudriashov

Nice. 2017-12-28 15:50 GMT+01:00 Pavel Krivanek <pavel.krivanek(a)gmail.com>: > Hi, > > when the Pharo 7 image is being built on top of the bootstrapped image, > all loaded code is placed in the changes file so this process ends with an > empty sources file, huge changes file and the image. We condense the > sources file for every build so we distribute the Pharo 7 as three files > - image, huge sources (over 30 MB) and almost empty changes file. > As consequence every build has own sources file that can be shared between > images based on the same build - so when we will release the Pharo 7, the > sources file can be shared as it was for previous releases, but as soon as > some fixing version of Pharo 7 will be released, it will require a new > specific sources file. > > As one possible solution I tried to create a pull request that resurrects > from oblivion an old code made for Squeak 3.7 that introduced option of > compressed sources files. Such files use segments that are stored in > compressed form so it is not so small as one single zip file but still good > enough. From original uncompressed file of size 30.8 MB it generates 7.1 MB > file while single zip has 6.1 MB. > > Of course the image file is bigger but: > - the amount of new objects is low so it should not make much more stress > on GC > And how many objects it adds? (how many sections are compressed?) > - we already used to have big images because the condenser was not working > and it was not a big issue > - the amount of distributed files will be the same as in case of Pharo 6 > (image+changes) so it means less troubles > - the sources file can be easily restored, deleted from the object memory > and used the same way as before > - we are not using the nice concept of the image well enough. All data > that are not really shared between images or serve as backup information > should be stored inside image > Can we avoid empty changes file? We can create it on demand (when user modifies code for example). > > Please review this PR. As a side effect it makes the sources file > mechanism (but not changes file) independent on the FileStream classes > that are deprecated. > > PR: https://github.com/pharo-project/pharo/pull/631 > > The pre-built image: > https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending% > 20pull%20request%20and%20branch%20Pipeline/job/PR-631/lastSuccessfulBuild/ > artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip > > Cheers, > -- Pavel > > >

Dec. 28, 2017

PR: Compressed sources inside the object memory

by Pavel Krivanek

Hi, when the Pharo 7 image is being built on top of the bootstrapped image, all loaded code is placed in the changes file so this process ends with an empty sources file, huge changes file and the image. We condense the sources file for every build so we distribute the Pharo 7 as three files - image, huge sources (over 30 MB) and almost empty changes file. As consequence every build has own sources file that can be shared between images based on the same build - so when we will release the Pharo 7, the sources file can be shared as it was for previous releases, but as soon as some fixing version of Pharo 7 will be released, it will require a new specific sources file. As one possible solution I tried to create a pull request that resurrects from oblivion an old code made for Squeak 3.7 that introduced option of compressed sources files. Such files use segments that are stored in compressed form so it is not so small as one single zip file but still good enough. From original uncompressed file of size 30.8 MB it generates 7.1 MB file while single zip has 6.1 MB. Of course the image file is bigger but: - the amount of new objects is low so it should not make much more stress on GC - we already used to have big images because the condenser was not working and it was not a big issue - the amount of distributed files will be the same as in case of Pharo 6 (image+changes) so it means less troubles - the sources file can be easily restored, deleted from the object memory and used the same way as before - we are not using the nice concept of the image well enough. All data that are not really shared between images or serve as backup information should be stored inside image Please review this PR. As a side effect it makes the sources file mechanism (but not changes file) independent on the FileStream classes that are deprecated. PR: https://github.com/pharo-project/pharo/pull/631 The pre-built image: https://ci.inria.fr/pharo-ci -jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/PR-631/ lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0-32bit-57ddc75.zip Cheers, -- Pavel

Dec. 28, 2017