On Fri, Jul 14, 2017 at 09:06:44AM +0100, Tim Mackinnon wrote:

Related to this question - in the Linux vm there are several .so files that have numerical extensions as well as the same named .so file. Are these merge conflicts or platform variations of the same files?

libSDL2-2.0.so.0.4.1 & libSDL2-2.0.so & libSDL2.so

They are the same files, typically sym-links. Pierce

Hi - the script I'm using is in https://gitlab.com/macta/PharoLambda - it's nothing fancy, look at the Gitlab .yml file. This is all Linux running in an ubuntu docker image on Gitlab CI. Tim Sent from my iPhone Sent from my iPhone

On 15 Jul 2017, at 06:27, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi Tim,

2017-07-14 4:59 GMT-03:00 Tim Mackinnon <tim@testit.works>:

...

Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

Can you share a script to reproduce the steps to build a minimal image?

Is it reproducible under Windows?

Cheers,

Hernán

Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can. I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right? The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?) I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?) The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps) Tim

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/ <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/>, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone

If you want to stay with Pharo 6 image, you can try the bootstrapped version of the minimal image: https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... -- Pavel 2017-07-15 10:33 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:

Try the Pharo 7 metacello image (=Pharo 7 minimal image that the CI is already converting to 64bit). There should be no problem with STON because whole Pharo is loaded into it using metacello and filetree. Pharo 6 minimal image is done differently (by shrinking) and not so well tested.

For the conversion of 32-bit image to 64-bit image you need a VMMaker image: https://ci.inria.fr/pharo/job/Spur-Git-Tracker/ lastSuccessfulBuild/artifact/vmmaker-image.zip and then evaluate: ./pharo generator.image eval "[Spur32to64BitBootstrap new bootstrapImage: 'conversion.image'] on: AssertionFailure do: [ :fail | fail resumeUnchecked: nil ]"

-- Pavel

2017-07-15 10:19 GMT+02:00 Tim Mackinnon <tim@testit.works>:

...

Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can.

I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right?

The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?)

I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?)

The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps)

Tim

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works>:

...

Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone

Well I’ve been shooting in the dark a bit - but I also left out the sound and display so’s (e.g. -x execlude the following and add back the null so's zip -r --symlinks ../deploy/$LAMBDA_NAME.zip * -x pharo-local/\* \*.sources \*.changes \*.st \*.log */libgit2.* */libSDL2* */B3DAccelerator* */JPEGRead* */vm-sound* */vm-display* tmp/\* */__MACOSX\* - zip -uR ../deploy/$LAMBDA_NAME.zip *-null.so And everything seems to run clean. (Would be useful to get some feedback from those in the know - does just leaving out so’s incurred a penalty if you don’t recompile the VM? Presumably something would get written to std error or pharodebug if it was an issue). In fact my run times on EC2 are pretty impressive: PharoLambdaMin]$ time ./pharo Pharo.image exec "Lambda processJSON: '{}'" {"outputSpeech":{"text":"Good Morning, it's eleven twenty-six","type":"PlainText"},"shouldEndSession":true,"card":{"content":"Operation Successful","title":"Pharo Lambda","type":"Simple"}} real 0m0.039s user 0m0.028s sys 0m0.000s [ec2-user@ip-172-31-44-73 PharoLambdaMin]$ time ./pharo Pharo.image exec "Lambda processJSON: '{}'" {"outputSpeech":{"text":"Good Morning, it's eleven twenty-six","type":"PlainText"},"shouldEndSession":true,"card":{"content":"Operation Successful","title":"Pharo Lambda","type":"Simple"}} real 0m0.039s user 0m0.020s sys 0m0.008s Not bad eh? Tim

On 17 Jul 2017, at 07:00, Tim Mackinnon <tim@testit.works> wrote:

Thanks again Pavel - I'll try the 6.0 step 4 or possibly step 5 with sunit (as many libraries don't separate out their tests).

I've also tried leaving out libgit and libsdl2 .so's on my server build and that seems fine too - making me wonder what others I can safely leave out? Sound is a candidate (but small fry in size but do you need the null variant?).

Libcrypto is big - but I wonder if https routines would use that (and it sounds server processing'y so maybe best left).

I was hoping to find a list explaining them somewhere - but it remains rather mysterious.

However, at this point, I think I may have hit the sweet spot in size where AWS seems to load efficiently below a zip of 10mb?

Tim

Sent from my iPhone

On 15 Jul 2017, at 09:35, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

...

If you want to stay with Pharo 6 image, you can try the bootstrapped version of the minimal image: https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...>

-- Pavel

2017-07-15 10:33 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>>: Try the Pharo 7 metacello image (=Pharo 7 minimal image that the CI is already converting to 64bit). There should be no problem with STON because whole Pharo is loaded into it using metacello and filetree. Pharo 6 minimal image is done differently (by shrinking) and not so well tested.

For the conversion of 32-bit image to 64-bit image you need a VMMaker image: https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/... <https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/...> and then evaluate: ./pharo generator.image eval "[Spur32to64BitBootstrap new bootstrapImage: 'conversion.image'] on: AssertionFailure do: [ :fail | fail resumeUnchecked: nil ]"

-- Pavel

2017-07-15 10:19 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can.

I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right?

The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?)

I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?)

The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps)

Tim

...

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/ <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/>, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone

On 19 Jul 2017, at 14:55, Esteban A. Maringolo <emaringolo@gmail.com> wrote:

I don't know how "mainstream" solutions perform on AWS Lambda or EC2, but this seems really fast to me. 50 ms is great, assuming it bills by every 100ms, you still have room to perform your computation.

Yes, it seems incredibly fast. I'll have to try this myself to check, but I have no time now.

Thank you for pursuing this path, it could open a new territory for using Pharo at big scale.

Esteban A. Maringolo

2017-07-17 8:32 GMT-03:00 Tim Mackinnon <tim@testit.works>:

...

Well I’ve been shooting in the dark a bit - but I also left out the sound and display so’s (e.g. -x execlude the following and add back the null so's

zip -r --symlinks ../deploy/$LAMBDA_NAME.zip * -x pharo-local/\* \*.sources \*.changes \*.st \*.log */libgit2.* */libSDL2* */B3DAccelerator* */JPEGRead* */vm-sound* */vm-display* tmp/\* */__MACOSX\* - zip -uR ../deploy/$LAMBDA_NAME.zip *-null.so

And everything seems to run clean. (Would be useful to get some feedback from those in the know - does just leaving out so’s incurred a penalty if you don’t recompile the VM? Presumably something would get written to std error or pharodebug if it was an issue).

In fact my run times on EC2 are pretty impressive:

PharoLambdaMin]$ time ./pharo Pharo.image exec "Lambda processJSON: '{}'" {"outputSpeech":{"text":"Good Morning, it's eleven twenty-six","type":"PlainText"},"shouldEndSession":true,"card":{"content":"Operation Successful","title":"Pharo Lambda","type":"Simple"}}

real 0m0.039s user 0m0.028s sys 0m0.000s [ec2-user@ip-172-31-44-73 PharoLambdaMin]$ time ./pharo Pharo.image exec "Lambda processJSON: '{}'" {"outputSpeech":{"text":"Good Morning, it's eleven twenty-six","type":"PlainText"},"shouldEndSession":true,"card":{"content":"Operation Successful","title":"Pharo Lambda","type":"Simple"}}

real 0m0.039s user 0m0.020s sys 0m0.008s

Not bad eh?

Tim

On 17 Jul 2017, at 07:00, Tim Mackinnon <tim@testit.works> wrote:

Thanks again Pavel - I'll try the 6.0 step 4 or possibly step 5 with sunit (as many libraries don't separate out their tests).

I've also tried leaving out libgit and libsdl2 .so's on my server build and that seems fine too - making me wonder what others I can safely leave out? Sound is a candidate (but small fry in size but do you need the null variant?).

Libcrypto is big - but I wonder if https routines would use that (and it sounds server processing'y so maybe best left).

I was hoping to find a list explaining them somewhere - but it remains rather mysterious.

However, at this point, I think I may have hit the sweet spot in size where AWS seems to load efficiently below a zip of 10mb?

Tim

Sent from my iPhone

On 15 Jul 2017, at 09:35, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

If you want to stay with Pharo 6 image, you can try the bootstrapped version of the minimal image: https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...

-- Pavel

2017-07-15 10:33 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:

...

Try the Pharo 7 metacello image (=Pharo 7 minimal image that the CI is already converting to 64bit). There should be no problem with STON because whole Pharo is loaded into it using metacello and filetree. Pharo 6 minimal image is done differently (by shrinking) and not so well tested.

For the conversion of 32-bit image to 64-bit image you need a VMMaker image:

https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/... and then evaluate: ./pharo generator.image eval "[Spur32to64BitBootstrap new bootstrapImage: 'conversion.image'] on: AssertionFailure do: [ :fail | fail resumeUnchecked: nil ]"

-- Pavel

2017-07-15 10:19 GMT+02:00 Tim Mackinnon <tim@testit.works>:

...

Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can.

I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right?

The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?)

I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?)

The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps)

Tim

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works>:

...

Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone

I'm really curious to see how these numbers will changes with Sista because pharo will be to start hot from a Jit point of view. So you will be able to run your app save it and ship it hot with the JIT optimisation already there. On Wed, Jul 19, 2017 at 6:20 PM, Tim Mackinnon <tim@testit.works> wrote:

Hi - I neglected to mentioned “the catch” with Lambda, next to my results. So on a tiny EC2 instance you get those kinds of results (this is where I measured the numbers of 50ms) - however on Lambda you aren’t entirely clear what hardware its running on - and there are 2 aspects to consider - a cold start (where you are allocated a new Lambda instance, and so it has to bring in your deployed package) and then there appears to be a cached start - where it seems that one of your old Lambda environments can be reused. On top of both of these states - there is an extra cost of spawning out to Pharo as its not supported natively.

I mention this in the Readme on the gitlab page (it’s possibly a bit subtle) - but I was pointed to the Sparta GoLang project (who aren’t supported natively either) where they have measured that that cost of spawning out to GoLang (and it looks fairly similar for Pharo) is 700ms. Essentially this spawning is the cost of loading up a NodeJS environment (presumably some Docker like image they have already prepared - although they don’t reveal how this is done), “requiring” the ‘child-process’ node module to get an exec method, and then your code to shell out. (In my repo - this is the PharoLambda.js file).

Empirically I am seeing results from 500ms to 1200ms which are in line with Sparta (possibly better? I haven’t loaded up a Go environment to understand what they need to package up to deploy an app that can be exec’d and how that compares to our 10mb'ish footprint).

If I look at a basic NodeJS hello world app - I see .5ms to 290ms responses - (the min billing unit is 100ms). I got the impression for a recent serverless meet-up that sub 500 is what people aim for. Which means we are at least in the running.

I don’t know how sensitive the ‘overhead’ load time is due to the package size you deploy (I saw a big increase when I got my package below 10mb) or whether it truly is the NodeJS tax. I would love to get hold of the AWS team and suggest they provide another fixed solution that efficiently exec’s in C, a named executable with configurable parameters and the “event” parameter serialised in JSON (on the surface it seems overkill to use NodeJS for just that simple operation).

All this said the free tier gives you "1M free requests per month and 400,000 GB-seconds of compute time per month” - so assuming we can do interesting things in under a second (which I’ve shown), then you can process 400,000 of them a month for free (which isn’t bad really).

Tim

...

On 19 Jul 2017, at 13:59, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 19 Jul 2017, at 14:55, Esteban A. Maringolo <emaringolo@gmail.com> wrote:

I don't know how "mainstream" solutions perform on AWS Lambda or EC2, but this seems really fast to me. 50 ms is great, assuming it bills by every 100ms, you still have room to perform your computation.

Yes, it seems incredibly fast. I'll have to try this myself to check, but I have no time now.

...

Thank you for pursuing this path, it could open a new territory for using Pharo at big scale.

Esteban A. Maringolo

2017-07-17 8:32 GMT-03:00 Tim Mackinnon <tim@testit.works>:

...

Well I’ve been shooting in the dark a bit - but I also left out the sound and display so’s (e.g. -x execlude the following and add back the null so's

zip -r --symlinks ../deploy/$LAMBDA_NAME.zip * -x pharo-local/\* \*.sources \*.changes \*.st \*.log */libgit2.* */libSDL2* */B3DAccelerator* */JPEGRead* */vm-sound* */vm-display* tmp/\* */__MACOSX\* - zip -uR ../deploy/$LAMBDA_NAME.zip *-null.so

And everything seems to run clean. (Would be useful to get some feedback from those in the know - does just leaving out so’s incurred a penalty if you don’t recompile the VM? Presumably something would get written to std error or pharodebug if it was an issue).

In fact my run times on EC2 are pretty impressive:

PharoLambdaMin]$ time ./pharo Pharo.image exec "Lambda processJSON: '{}'" {"outputSpeech":{"text":"Good Morning, it's eleven twenty-six","type":"PlainText"},"shouldEndSession":true,"card":{"content":"Operation Successful","title":"Pharo Lambda","type":"Simple"}}

real 0m0.039s user 0m0.028s sys 0m0.000s [ec2-user@ip-172-31-44-73 PharoLambdaMin]$ time ./pharo Pharo.image exec "Lambda processJSON: '{}'" {"outputSpeech":{"text":"Good Morning, it's eleven twenty-six","type":"PlainText"},"shouldEndSession":true,"card":{"content":"Operation Successful","title":"Pharo Lambda","type":"Simple"}}

real 0m0.039s user 0m0.020s sys 0m0.008s

Not bad eh?

Tim

On 17 Jul 2017, at 07:00, Tim Mackinnon <tim@testit.works> wrote:

Thanks again Pavel - I'll try the 6.0 step 4 or possibly step 5 with sunit (as many libraries don't separate out their tests).

I've also tried leaving out libgit and libsdl2 .so's on my server build and that seems fine too - making me wonder what others I can safely leave out? Sound is a candidate (but small fry in size but do you need the null variant?).

Libcrypto is big - but I wonder if https routines would use that (and it sounds server processing'y so maybe best left).

I was hoping to find a list explaining them somewhere - but it remains rather mysterious.

However, at this point, I think I may have hit the sweet spot in size where AWS seems to load efficiently below a zip of 10mb?

Tim

Sent from my iPhone

On 15 Jul 2017, at 09:35, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

If you want to stay with Pharo 6 image, you can try the bootstrapped version of the minimal image: https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...

-- Pavel

2017-07-15 10:33 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com>:

...

Try the Pharo 7 metacello image (=Pharo 7 minimal image that the CI is already converting to 64bit). There should be no problem with STON because whole Pharo is loaded into it using metacello and filetree. Pharo 6 minimal image is done differently (by shrinking) and not so well tested.

For the conversion of 32-bit image to 64-bit image you need a VMMaker image:

https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/... and then evaluate: ./pharo generator.image eval "[Spur32to64BitBootstrap new bootstrapImage: 'conversion.image'] on: AssertionFailure do: [ :fail | fail resumeUnchecked: nil ]"

-- Pavel

2017-07-15 10:19 GMT+02:00 Tim Mackinnon <tim@testit.works>:

...

Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can.

I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right?

The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?)

I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?)

The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps)

Tim

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works>:

...

Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone

Hi Pavel - I’m just revisiting a few of your previous messages on minimal images as I’m trying to get things working again with Pharo 6.1 now that you guys have rejigged all of the build pipelines. I tried a previous suggestion of: For Pharo 6: https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf... <https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf...> For Pharo 7: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc... <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc...> However the Pharo 6 version is no longer being built, and it doesn’t seem to work well with the newer 64bit vm (? I seem to get a load error that I didn’t have before - although its possible that I’m loading a new pre-req with metacello and this is tipping it over the edge). Anyway - a prev suggestion for 6.x from you was:

https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...> However this is a 32bit image. Is there an equivalent 64bit image for 6.1? OR should I use the Pharo 7 one for now (I guess the minimal image will probably be pretty stable for a little while as I’m sure the action is higher up the chain?)

Tim

On 15 Jul 2017, at 09:35, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

If you want to stay with Pharo 6 image, you can try the bootstrapped version of the minimal image: https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...>

-- Pavel

2017-07-15 10:33 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>>: Try the Pharo 7 metacello image (=Pharo 7 minimal image that the CI is already converting to 64bit). There should be no problem with STON because whole Pharo is loaded into it using metacello and filetree. Pharo 6 minimal image is done differently (by shrinking) and not so well tested.

For the conversion of 32-bit image to 64-bit image you need a VMMaker image: https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/... <https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/...> and then evaluate: ./pharo generator.image eval "[Spur32to64BitBootstrap new bootstrapImage: 'conversion.image'] on: AssertionFailure do: [ :fail | fail resumeUnchecked: nil ]"

-- Pavel

2017-07-15 10:19 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can.

I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right?

The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?)

I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?)

The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps)

Tim

...

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/ <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/>, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone

Ok (I am reading there is a lot going on for you guys to sort out at the moment). It sounds like a more stable conversion of Pharo 6.1 for 64 bit would be best to give me a stable platform to run on. In that minimal image - what can I rely on to load code (in trying the 7.0 version it seems like Gofer isn’t there, and I’m now wondering if Metacello isn’t there either) - so I’m wondering how you guys load baselinesOf of configurationsOf? Tim

On 31 Jul 2017, at 14:17, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

H Tim

2017-07-31 15:07 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi Pavel - I’m just revisiting a few of your previous messages on minimal images as I’m trying to get things working again with Pharo 6.1 now that you guys have rejigged all of the build pipelines.

I tried a previous suggestion of:

For Pharo 6: https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf... <https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf...> For Pharo 7: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc... <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc...>

However the Pharo 6 version is no longer being built, and it doesn’t seem to work well with the newer 64bit vm (? I seem to get a load error that I didn’t have before - although its possible that I’m loading a new pre-req with metacello and this is tipping it over the edge).

Anyway - a prev suggestion for 6.x from you was:

...

https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...> However this is a 32bit image. Is there an equivalent 64bit image for 6.1? OR should I use the Pharo 7 one for now (I guess the minimal image will probably be pretty stable for a little while as I’m sure the action is higher up the chain?)

You should use an image that is bootstrapped, so version from SysConf jobs or Pharo 7. For Pharo 7 we are preparing a lot of big changes in the kernel so do not expect it will be stable. We can do a conversion job of of it to 64-bit version as soon as the CI infrastructure will be on knees again.

-- Pavel

Tim

...

On 15 Jul 2017, at 09:35, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

If you want to stay with Pharo 6 image, you can try the bootstrapped version of the minimal image: https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...>

-- Pavel

2017-07-15 10:33 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>>: Try the Pharo 7 metacello image (=Pharo 7 minimal image that the CI is already converting to 64bit). There should be no problem with STON because whole Pharo is loaded into it using metacello and filetree. Pharo 6 minimal image is done differently (by shrinking) and not so well tested.

For the conversion of 32-bit image to 64-bit image you need a VMMaker image: https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/... <https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/...> and then evaluate: ./pharo generator.image eval "[Spur32to64BitBootstrap new bootstrapImage: 'conversion.image'] on: AssertionFailure do: [ :fail | fail resumeUnchecked: nil ]"

-- Pavel

2017-07-15 10:19 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can.

I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right?

The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?)

I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?)

The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps)

Tim

...

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/ <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/>, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone

I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf... <https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf...> one still shows as being last updated 7 days ago. The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...> one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521). The https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc... <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc...> one gives me a walkback when trying to run my install script : 25 UndefinedObject(Object)>>doesNotUnderstand: #addTo: 26 MCRepositoryGroup>>addRepository: 27 createRepository | repo | repo := self project createRepository: self. ^ MCRepositoryGroup default repositories detect: [ :each | each = repo ] ifNone: [ MCRepositoryGroup default addRepository: repo. repo ] in MetacelloRepositorySpec>>createRepository I think this is because the image doesn’t support Metacello? As in: Metacello new repository: 'filetree://../src'; baseline: 'Lambda'; load. How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?) I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this. Tim

On 31 Jul 2017, at 14:42, Pavel Krivanek <pavel.krivanek@gmail.com> wrote:

2017-07-31 15:28 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Ok (I am reading there is a lot going on for you guys to sort out at the moment).

It sounds like a more stable conversion of Pharo 6.1 for 64 bit would be best to give me a stable platform to run on.

In that minimal image - what can I rely on to load code (in trying the 7.0 version it seems like Gofer isn’t there, and I’m now wondering if Metacello isn’t there either) - so I’m wondering how you guys load baselinesOf of configurationsOf?

Try it again, there was a bug in the job configuration

.. Pavel

Tim

...

On 31 Jul 2017, at 14:17, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

H Tim

2017-07-31 15:07 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi Pavel - I’m just revisiting a few of your previous messages on minimal images as I’m trying to get things working again with Pharo 6.1 now that you guys have rejigged all of the build pipelines.

I tried a previous suggestion of:

For Pharo 6: https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf... <https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf...> For Pharo 7: https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc... <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc...>

However the Pharo 6 version is no longer being built, and it doesn’t seem to work well with the newer 64bit vm (? I seem to get a load error that I didn’t have before - although its possible that I’m loading a new pre-req with metacello and this is tipping it over the edge).

Anyway - a prev suggestion for 6.x from you was:

...

https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...> However this is a 32bit image. Is there an equivalent 64bit image for 6.1? OR should I use the Pharo 7 one for now (I guess the minimal image will probably be pretty stable for a little while as I’m sure the action is higher up the chain?)

You should use an image that is bootstrapped, so version from SysConf jobs or Pharo 7. For Pharo 7 we are preparing a lot of big changes in the kernel so do not expect it will be stable. We can do a conversion job of of it to 64-bit version as soon as the CI infrastructure will be on knees again.

-- Pavel

Tim

...

On 15 Jul 2017, at 09:35, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

If you want to stay with Pharo 6 image, you can try the bootstrapped version of the minimal image: https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...>

-- Pavel

2017-07-15 10:33 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>>: Try the Pharo 7 metacello image (=Pharo 7 minimal image that the CI is already converting to 64bit). There should be no problem with STON because whole Pharo is loaded into it using metacello and filetree. Pharo 6 minimal image is done differently (by shrinking) and not so well tested.

For the conversion of 32-bit image to 64-bit image you need a VMMaker image: https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/... <https://ci.inria.fr/pharo/job/Spur-Git-Tracker/lastSuccessfulBuild/artifact/...> and then evaluate: ./pharo generator.image eval "[Spur32to64BitBootstrap new bootstrapImage: 'conversion.image'] on: AssertionFailure do: [ :fail | fail resumeUnchecked: nil ]"

-- Pavel

2017-07-15 10:19 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi Pavel - thanks for getting me to the point where I could even have a minimal image. As I’m on the edge of my Pharo knowledge here, I’ll try and run with this as best I can.

I’d been using the 6.0 image you suggested to me - but maybe I could use a 70 image with Pharo 6 for a while (until the VM diverges) right?

The bit I haven’t quite understood however, is how the 64bit image is created - as your reference is to a 32bit version? Is the 64bit one converted from 32 in a later stage? (For AWS Lambda I need 64bit) - am I right in thinking the pipeline stage after this one is the one you sent me - and the travis.yml file shows me what it does? But I can’t see a trivis.yml in the conversion stage so I’m not sure how it does that. (Question - how do I see what the pipelines do to answer my own questions?)

I was hoping that there was a basic image that got me up to metacello baseline level to load git file tree packages/baselines in my own repo as well baselines on the internet. The one you sent me is fairly close to that (its just missing STON in the image and seems to have an issue with resolving undeclared classes that get loaded in - should do a fogbugz on that?)

The follow-on from a metacello image is how we can get people to create better baselines that give you more minimal loading options (e.g. conditionally leave out the test cases perhaps)

Tim

...

On 15 Jul 2017, at 08:24, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:

Hi Tim,

you can base the your work on the bootstrapped image, see https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/ <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit/>, file Pharo7.0-core-*.zip

This image does not have a lot of basic components like Monticello or network but it has a compiler so the code can be imported as *.st files. Then we have Pharo7.0-monticello-*.zip which will be easier to use and probably can fit your needs. Monticello and network support are included. But you cannot use baselines nor configurations to load your code.

-- Pavel

2017-07-14 9:59 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>: Hi - buoyed by the success of a minimal image (thanks Pavel), I'm wondering if I can get even smaller.

There are lots of .so's in the vm which wouldn't make sense on a server once deployed - sound, maybe libgit ...

Is there a list of the essential ones, or tips on what I can strip out of the Linux deployment? I also recall that i can leave out .sources and .changes as well right?

Tim

Sent from my iPhone