Pharo 7 and changes/sources files
Hi, because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build. So Pharo 7 development versions will be delivered in an archive with three files named like: Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes. You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuil... Cheers, -- Pavel
On Fri, Jun 9, 2017 at 4:01 PM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
So what we need is for CompiledMethod>>#sourceCode to be based on git-hashes and directly access git, with one git repository shared by multiple Images. Debugging into... (Object>>#printString) sourceCode I see the trailer has "kind == #SourcePointer" and in #getSourceFromFile method #decodeSourcePointer does some interesting magic to calculate its "data" that later is returned by #sourcePointer in "SourceFiles sourceCodeAt: self sourcePointer" where SourceFiles is ...PharoV60.sources and ...PharoV60.changes. So I guess we'd need something like "kind == #ContentHash" and then in CompiledMethod>>sourceCode ahead of getSourceFromFile we add ... [ self getSourceFromUserLocalGit: contentHash ] ifNotNil: [:s | ^s]. [ self getSourceFromUserGitServer: contentHash ] ifNotNil: [:s | ^s]. [ self getSourceFromPharoGitServer: contentHash ] ifNotNil: [:s | ^s]. [ self getSourceFromPrivateEncryptedRepository: contentHash ] ifNotNil: [:s | ^s]. [ self getSourceFromIPFS: contentHash ] ifNotNil: [:s | ^s]. and such like. Perhaps some would need separate "kind"s, and this is oversimplistic since I haven't considered saving sources, but you get the idea. cheers -ben
Why cannot we have the sources in the image? When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image? Uko
On 9 Jun 2017, at 10:01, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuil... <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuil...>
Cheers, -- Pavel
We plan to change the sources management. But not immediately. -- Pavel 2017-06-09 14:51 GMT+02:00 Yuriy Tymchuk <yuriy.tymchuk@me.com>:
Why cannot we have the sources in the image?
When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?
Uko
On 9 Jun 2017, at 10:01, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70- Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0- 117934d.zip
Cheers, -- Pavel
Aha, good to know!
On 9 Jun 2017, at 14:57, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
We plan to change the sources management. But not immediately.
-- Pavel
2017-06-09 14:51 GMT+02:00 Yuriy Tymchuk <yuriy.tymchuk@me.com <mailto:yuriy.tymchuk@me.com>>: Why cannot we have the sources in the image?
When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?
Uko
On 9 Jun 2017, at 10:01, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuil... <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuil...>
Cheers, -- Pavel
On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Why cannot we have the sources in the image?
When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?
We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing. So I'm not against it, but we cannot just make a change like this as we are right now facing lots of key changes at the same time: - migration to git, iceberg - bootstrap based process
Uko
On 9 Jun 2017, at 10:01, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70- Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0- 117934d.zip
Cheers, -- Pavel
-- Guille Polito Research Engineer French National Center for Scientific Research - *http://www.cnrs.fr* <http://www.cnrs.fr> *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13
On Friday 09 June 2017 06:38 PM, Guillermo Polito wrote:
On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?
We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing.
What if the sources is appended to the image file with a special header. New code can relocate this pointer on the fly. The same technique can be applied to changes file too. A small script can split files for older code or combine them for new code. The resulting image file will be backward compatible. New code can preserve these files across saves. Just a thought .. Subbu
On Fri, Jun 9, 2017 at 10:01 PM, K K Subbu <kksubbu.ml@gmail.com> wrote:
On Friday 09 June 2017 06:38 PM, Guillermo Polito wrote:
On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?
We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing.
What if the sources is appended to the image file with a special header. New code can relocate this pointer on the fly. The same technique can be applied to changes file too. A small script can split files for older code or combine them for new code.
The resulting image file will be backward compatible. New code can preserve these files across saves.
Just a thought .. Subbu
Or perhaps MemoryFileSystem could be effective? But no rush. cheers -ben
On Fri, Jun 9, 2017 at 9:08 AM, Guillermo Polito <guillermopolito@gmail.com> wrote:
On Fri, Jun 9, 2017 at 2:51 PM, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Why cannot we have the sources in the image?
We can, but that's something that requires changing lot's of things inside the image and it requires probably lot of testing.
So I'm not against it, but we cannot just make a change like this as we are right now facing lots of key changes at the same time: - migration to git, iceberg - bootstrap based process
How about saving the sources in-memory via an instance of MemoryStore, so it should still look just like a file.
Hi Yiury, On Fri, Jun 9, 2017 at 5:51 AM, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Why cannot we have the sources in the image?
When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?
It's not a "micro saving". A Pharo6 image is about 48Mb and a Pharo sources file is about 32Mb. So you'd increase the size of the image by 75%. Further, that space has to be dealt with by the GC. The compactor may have to move this objects around; so it's certainly overhead. It might make sense for development, especially if you use Igor's include-source-in-the-method-trailer approach. But it doesn't make sense for deployment where the application isn't going to look at the sources anyway. So why put a heavy burden on deploying applications when the current system works, and off loads a substantial amount of memory from the system?
Uko
On 9 Jun 2017, at 10:01, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70- Bootstrap-32bit/lastSuccessfulBuild/artifact/bootstrap-cache/Pharo7.0- 117934d.zip
Cheers, -- Pavel
-- _,,,^..^,,,_ best, Eliot
+ 1 :) It is just a file and this is only for the development of Pharo and for users of Pharo 70 it will be exactly the same. Let us focus on the place where we can get a good impact/ratio/plus for business/plus for innovation. I have a lot of dreams for Pharo: from the coffee machine to the big data servers and this story with source files is not interesting. We download thousands of files in any application. So let us focus our energy on something else. stef On Sat, Jun 10, 2017 at 6:13 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Yiury,
On Fri, Jun 9, 2017 at 5:51 AM, Yuriy Tymchuk <yuriy.tymchuk@me.com> wrote:
Why cannot we have the sources in the image?
When I asked it 2 years ago people were like: "oh, but you can just download 1 sources file and use it by all the images". I donât care about that micro space savings, but fine, I said: have it your way. Now as we have to download sources all the time, why cannot we just have it IN the image?
It's not a "micro saving". A Pharo6 image is about 48Mb and a Pharo sources file is about 32Mb. So you'd increase the size of the image by 75%. Further, that space has to be dealt with by the GC. The compactor may have to move this objects around; so it's certainly overhead. It might make sense for development, especially if you use Igor's include-source-in-the-method-trailer approach. But it doesn't make sense for deployment where the application isn't going to look at the sources anyway. So why put a heavy burden on deploying applications when the current system works, and off loads a substantial amount of memory from the system?
Uko
On 9 Jun 2017, at 10:01, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuil...
Cheers, -- Pavel
-- _,,,^..^,,,_ best, Eliot
This is excellent to get the bootstrap in production.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! On Fri, Jun 9, 2017 at 10:01 AM, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:
Hi,
because during the image bootstrapping all source code is reloaded, it is all placed in the changes file and the sources file is useless. It would be strange to deliver Pharo with empty sources file and big changes so we decided to do sources compacting for every build.
So Pharo 7 development versions will be delivered in an archive with three files named like:
Pharo7.0-117934d.image Pharo7.0-117934d.changes Pharo7.0-117934d.sources
where "117934d" is an abbreviated hash of the Git commit from which the Pharo image was bootstrapped. The sources file will be build-specific and of course you will be able to share it between images based on the same build. In changes file you will have only you own changes.
You can test it here: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/lastSuccessfulBuil...
Cheers, -- Pavel
participants (8)
-
Ben Coman -
Eliot Miranda -
Guillermo Polito -
K K Subbu -
Pavel Krivanek -
Stephane Ducasse -
Yanni Chiu -
Yuriy Tymchuk