[Pharo-dev] Creating the smallest server runtime footprint

Tim Mackinnon tim at testit.works
Fri Aug 4 05:47:24 EDT 2017


If Traits are supported then I must be doing something else wrong - maybe load order? (To be honest, I don’t fully understand how traits work - but I will keep digging).

It could be that my latest error (having just done Class compile: ‘…. Missing method code … ‘) is something deeper related to these traits methods not applying somehow… 

I’ll try a few things this morning.

It’s amazing how easily the image size jumps up when you add things though. With the original minimal v6 64bit image you sorted out for me I had a footprint of 21.6mb (image was 12mb).
Now with the v7 image - and adding fuel and some S3 handling - it jumps up to 32.4mb (image is 15mb).

I know that the S3 support has a lot of XML crud - but it seems quite astounding that this takes almost 8mb of code/objects.

AWS loading (and hence response time) seems quite sensitive to the size of the package it has to load - although equally I wonder if there is also more #startUp/initalisation cost that is going on as well as when you get a warm lambda instance (so its not just loading your package) I’m still not seeing the 400-500ms times but more 800-1000ms times. 

Its all quite interesting - and shows why the work behind a minimal image is very useful. Of course there is tiny-minimal (which I know you guys are after) and then there is small-server- minimal (which is realistically what I’m after, so I can just focus on creating my app and just carefully load code and dependencies - but still have core server services like Fuel, and Git loading).

Thanks for doing this work - as it brings a lot of useful things together.

Tim


> On 4 Aug 2017, at 06:48, Pavel Krivanek <pavel.krivanek at gmail.com> wrote:
> 
> 
> 
> 2017-08-03 18:17 GMT+02:00 Tim Mackinnon <tim at testit.works <mailto:tim at testit.works>>:
> Hi guys - I think I’m getting close - I’m able to load remote packages, and I’ve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames
> 
> I can see that this is a trait - is trait support missing in the minimal image?
> 
> No, this image supports them 
> 
> (I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?
> 
> Tim
> 
> 
> "errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
>   "errorType": "Error",
>   "stackTrace": [
>     "Writing fuel context to S3...",
>     "Using S3 bucket morethan-technology-projects",
>     "\u001b[31m",
>     "--- RUNNING ERROR HANDLER ---",
>     "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
>     "",
>     "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
>     "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
>     "FLVariablesMapping>>instanceVariableNamesToSerialize",
>     "FLVariablesMapping>>initializeAnalyzing",
>     "FLVariablesMapping class>>newAnalyzing:references:",
>     "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
>     "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
>     "clusterKeyedByObjectClass: t1 class: t2",
>     "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
>     "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
>     "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
>     "at: t1 ifAbsentPut: t2",
>     "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
>     "IdentityDictionary(Dictionary)>>at:ifAbsent:",
>     "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
>     "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
>     "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
>     "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
>     "FLLightGeneralMapper>>visitMethodContext:",
>     "Context>>fuelAccept:",
>     "FLLightGeneralMapper>>mapAndTrace:",
>     "FLLightGlobalMapper>>mapAndTrace:",
>     "FLPluggableSubstitutionMapper>>mapAndTrace:",
>     "FLAnalysis>>mapAndTrace:",
>     "FLAnalysis>>run",
>     "setDefaultAnalysis",
>     "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
>     "FLAnalyzer>>analysisFor:",
>     "FLSerialization>>analysisStep",
>     "FLSerialization>>run",
>     "setDefaultSerialization",
>     "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
>     "serialize: t1 on: t2",
>     "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
>     "on: t1 globalEnvironment: t2 do: t3",
>     "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
>     "BlockClosure>>ensure:",
>     "FLEncoder class>>on:globalEnvironment:do:",
>     "\u001b[0m"
>> On 3 Aug 2017, at 13:16, Mariano Martinez Peck <marianopeck at gmail.com <mailto:marianopeck at gmail.com>> wrote:
>> 
>> 
>> 
>> On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <tim at testit.works <mailto:tim at testit.works>> wrote:
>> I’m so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - I’m just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)… so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?
>> 
>> 
>> The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
>> Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello. 
>> 
>> Sometimes you can print the result of #record:
>> 
>> Metacello new 
>> 	configuration: 'Fuel';
>> 	smalltalkhubUser: 'Pharo' project: 'Fuel';
>> 	version: #stable;
>> 	record: #('Core').
>> 
>> ->
>> 
>> "linear load : 
>> 	linear load : 2.1.10 [ConfigurationOfFuel]
>> 		linear load : baseline [ConfigurationOfFuelPlatform]
>> 			load : FuelPlatform-Core
>> 			load : FuelPlatform-Pharo-Core
>> 			load : FuelPlatform-Pharo-06"
>> 
>> 
>> ------
>> 
>> Anyway.... I guess I would try to load these packages (in this order):
>> 
>> FuelPlatform-Core
>> FuelPlatform-Pharo-Core
>> FuelPlatform-Pharo-06
>> Fuel
>> 
>> And when you are down loading, then execute:
>> 
>> (Smalltalk at: #FLPlatform) current addHacks
>> 
>>  
>> I’m also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)
>> 
>> Tim
>> 
>>> On 3 Aug 2017, at 12:07, Pavel Krivanek <pavel.krivanek at gmail.com <mailto:pavel.krivanek at gmail.com>> wrote:
>>> 
>>> 
>>> 
>>> 2017-08-03 13:02 GMT+02:00 Tim Mackinnon <tim at testit.works <mailto:tim at testit.works>>:
>>> I really appreciate your patience and help on this - it looks very promising and I am giving it a spin now…
>>> 
>>> When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src <https://github.com/pharo-project/pharo/tree/master/src>. ? Just so I know for the future.
>>> 
>>> yes
>>> 
>>> -- Pavel 
>>> 
>>> 
>>> Tim
>>> 
>>>> On 3 Aug 2017, at 09:16, Pavel Krivanek <pavel.krivanek at gmail.com <mailto:pavel.krivanek at gmail.com>> wrote:
>>>> 
>>>> 
>>>> 
>>>> 2017-08-03 10:14 GMT+02:00 Pavel Krivanek <pavel.krivanek at gmail.com <mailto:pavel.krivanek at gmail.com>>:
>>>> The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):
>>>> 
>>>> Metacello-GitBasedRepository.package
>>>> Metacello-GitHub.package
>>>> SUnit-Core.package
>>>> 
>>>> and create baseline to load them. I already tried it so you can test it:
>>>> https://drive.google.com/open?id=0BzSsmZhqtUTeMVJacTZtdW5UQW8 <https://drive.google.com/open?id=0BzSsmZhqtUTeMVJacTZtdW5UQW8>
>>>> 
>>>> Then you will do something like:
>>>> 
>>>> (accidental message send ;-) )
>>>> 
>>>> BASELINE=MetacelloGitBasedRepository
>>>> pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
>>>> 	"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src <>'; load."
>>>> 
>>>> BASELINE=Lambda
>>>> pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
>>>> 	"Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src <>'; load."
>>>> 
>>>> As I tried that. The baseline of Lambda is then able to be loaded.
>>>> 
>>>> -- Pavel
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 2017-08-02 15:18 GMT+02:00 Tim Mackinnon <tim at testit.works <mailto:tim at testit.works>>:
>>>> Ah, I think I’m starting to get closer to what I need…
>>>> 
>>>> So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? I’m trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasn’t clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess I’m looking for an easy Mcz I can use with the example below?
>>>> 
>>>> Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?
>>>> 
>>>> I guess I’m trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.
>>>> 
>>>> Tim
>>>> 
>>>>> On 2 Aug 2017, at 07:05, Guillermo Polito <guillermopolito at gmail.com <mailto:guillermopolito at gmail.com>> wrote:
>>>>> 
>>>>> Yes, you should be able to load an mcz in that image by doing:
>>>>> 
>>>>> (MCDirectoryRepository new directory: 'where-your-mcz-is')
>>>>>     loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.
>>>>> 
>>>>> Guille
>>>>> 
>>>>> On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <pavel.krivanek at gmail.com <mailto:pavel.krivanek at gmail.com>> wrote:
>>>>> 
>>>>> 2017-08-01 23:13 GMT+02:00 Tim Mackinnon <tim at testit.works <mailto:tim at testit.works>>:
>>>>> Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.
>>>>> 
>>>>> The SUnit baseline doesn’t specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:
>>>>> 
>>>>> baseline: spec
>>>>> 	<baseline>
>>>>> 
>>>>> 	spec for: #common do: [
>>>>> 		spec configuration: 'ZTimestamp' with: [
>>>>> 			spec 
>>>>> 				versionString: #stable;
>>>>> 				repository: 'http://mc.stfx.eu/Neo <http://mc.stfx.eu/Neo>' ].
>>>>> 			
>>>>> 		spec baseline: 'AWS' with: [	
>>>>> 			spec repository: 'github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository <>' ].
>>>>> 						
>>>>> 		spec 
>>>>> 			package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
>>>>> 			
>>>>> 	].		
>>>>> 
>>>>> 
>>>>> The “spec configuration: ….” Specifications cause the error:
>>>>> 
>>>>> 25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
>>>>> 26 MCRepositoryGroup>>addRepository:
>>>>> 
>>>>> If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.
>>>>> 
>>>>> So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - I’m guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting). 
>>>>> 
>>>>> The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.
>>>>> 
>>>>> -- Pavel
>>>>> 
>>>>> 
>>>>>  
>>>>> 
>>>>> Tim
>>>>> 
>>>>>> On 1 Aug 2017, at 10:06, Pavel Krivanek <pavel.krivanek at gmail.com <mailto:pavel.krivanek at gmail.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 2017-07-31 22:51 GMT+02:00 Tim Mackinnon <tim at testit.works <mailto:tim at testit.works>>:
>>>>>> I wasn’t clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip <https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessfulBuild/artifact/Pharo-minimal-64.zip> one still shows as being last updated 7 days ago.
>>>>>> 
>>>>>> The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/ <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-ConfigurationOfMinimalPharo/> one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).
>>>>>> 
>>>>>> The https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuccessfulBuild/artifact/latest-minimal-64.zip <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuccessfulBuild/artifact/latest-minimal-64.zip> one gives me a walkback when trying to run my install script :
>>>>>> 
>>>>>> 25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
>>>>>> 26 MCRepositoryGroup>>addRepository:
>>>>>> 27 createRepository
>>>>>>   | repo |
>>>>>>   repo := self project createRepository: self.
>>>>>>   ^ MCRepositoryGroup default repositories
>>>>>>     detect: [ :each | each = repo ]
>>>>>>     ifNone: [ 
>>>>>>       MCRepositoryGroup default addRepository: repo.
>>>>>>       repo ] in MetacelloRepositorySpec>>createRepository
>>>>>> 
>>>>>> I think this is because the image doesn’t support Metacello? As in:
>>>>>> Metacello new
>>>>>>     repository: 'filetree://../src <>';
>>>>>>     baseline: 'Lambda';
>>>>>>     load.
>>>>>> 
>>>>>> How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)
>>>>>> 
>>>>>> I can use a big 6.1 image fine (as it has Metacello loaded) but I’d really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.
>>>>>> 
>>>>>> Tim
>>>>>> 
>>>>>> 
>>>>>> I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:
>>>>>> 
>>>>>> ./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src <>'; load."
>>>>>>  
>>>>>> ./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"
>>>>>> 
>>>>>> Can you test it too?
>>>>>> 
>>>>>> -- Pavel
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>>    
>>>>> Guille Polito
>>>>> 
>>>>> Research Engineer
>>>>> French National Center for Scientific Research - http://www.cnrs.fr <http://www.cnrs.fr/>
>>>>> 
>>>>> 
>>>>> Web: http://guillep.github.io <http://guillep.github.io/>
>>>>> Phone: +33 06 52 70 66 13 <tel:+33%206%2052%2070%2066%2013>
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
>> -- 
>> Mariano
>> http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20170804/46b64fef/attachment.html>


More information about the Pharo-dev mailing list