Re: [Pharo-dev] Creating the smallest server runtime footprint

Aug. 4, 2017

      Hi Mariano - Iâve looked at this again, and it was my fault - I was compiling in :

fuelAccept: aGeneralMapper
	^self explicitRequirement.

Because when I looked at the Traits for fuel - this is what it had for TClass? - as Iâm not so clear on traits, what does this mean - as I had just assumed that I should flatten the Fuel methods for TCLass, TBehaviour and TClassDescription - but I think Iâve misunderstood what I should do (and I guess from Pavelâs comment - I should really look at why that initial method was missing as it sounds like it should have worked  - so maybe its an order loading thing?).

Thanks, for helping by the way - this is proving very instructional.

Tim
...
On 3 Aug 2017, at 18:26, Mariano Martinez Peck <marianopeck@gmail.com> wrote:
On Thu, Aug 3, 2017 at 2:14 PM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Hi Mariano - I compiled in the trait methods manually and it got further - but now Iâm seeing another error - Error: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirementâ,
mmmm the stack is weird... it looks like if Class >> fuelAccept: would have the wrong implementation, that is, having the one from TClass:
fuelAccept: aGeneralMapper
  ^self explicitRequirement.
Rather than the correct one:
fuelAccept: aGeneralMapper
^aGeneralMapper visitClass: self
Maybe the solution is to compile all those methods from traits BUT NOT those that are "^self explicitRequirement." because those will override the good methods.
If that doesn't work, then maybe can you print to console the each impl (source) of each implementor of #fuelAccept: ?
Thoughts?
Any ideas?
Tim
"--- RUNNING ERROR HANDLER ---",
    "Error: Explicitly required method",
    "",
    "\u001b[0m\u001b[31mError: Explicitly required method",
    "\u001b[0mLambda class(Object)>>error:",
    "explicitRequirement",
    "  self error: 'No decompiler available' in Lambda class(Object)>>explicitRequirement in Block: explicitRequirement...",
    "False>>ifFalse:",
    "Lambda class(Object)>>explicitRequirement",
    "Lambda class(Class)>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "FLSerializer>>serialize:on:",
    "NonInteractiveUIManager>>writeFuelContext:",
    "Lambda class>>processDebug:",
    "Lambda class>>processJSON:",
    "Lambda class>>process:",
    "activate",
...
On 3 Aug 2017, at 17:26, Mariano Martinez Peck <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote:
On Thu, Aug 3, 2017 at 1:17 PM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Hi guys - I think Iâm getting close - Iâm able to load remote packages, and Iâve added fuel to my repo and its loading - however I am getting an error when I run and test writing out a context - MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames
I can see that this is a trait - is trait support missing in the minimal image?
Uff...that's a good question hahahhah 
I don't know if the minimal image supports traits, but what I am almost sure, is the Behavior DOES need the methods defined in TBehavior. So maybe the minimal image flattens the traits. In this case you could do the same for Fuel (flattenDownAllTraits)
(I also seemed to get this far just loading the Fuel package - but just to be sure I added the other platform ones like you suggested and still get the same error). Given how simple that method is, I could just compile it in - but maybe there are many more?
TBehavior, TClass and TClassDescription are the only one that comes to my mind right now...  there are only a few methods from Fuel. So could compile those directly at least on a first step...at least you keep you going.
Tim
"errorMessage": "Command failed: ./pharo Pharo.image --no-default-preferences process Lambda '{\"debug\":true,\"data\":5}'\n
  "errorType": "Error",
  "stackTrace": [
    "Writing fuel context to S3...",
    "Using S3 bucket morethan-technology-projects",
    "\u001b[31m",
    "--- RUNNING ERROR HANDLER ---",
    "MessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "",
    "\u001b[0m\u001b[31mMessageNotUnderstood: Context class>>fuelIgnoredInstanceVariableNames",
    "\u001b[0mContext class(Object)>>doesNotUnderstand: #fuelIgnoredInstanceVariableNames",
    "FLVariablesMapping>>instanceVariableNamesToSerialize",
    "FLVariablesMapping>>initializeAnalyzing",
    "FLVariablesMapping class>>newAnalyzing:references:",
    "FLContextCluster(FLPointerObjectCluster)>>initializeAnalyzing:",
    "FLContextCluster class(FLObjectCluster class)>>newAnalyzing:",
    "clusterKeyedByObjectClass: t1 class: t2",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class: in Block: clusterKeyedByObjectClass: t1 class: t2...",
    "clusterInstanceOf: t1 keyInBucket: t2 factory: t3",
    "  self error: 'No decompiler available' in FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory: in Block: clusterInstanceOf: t1 keyInBucket: t2 factory: t3...",
    "at: t1 ifAbsentPut: t2",
    "  self error: 'No decompiler available' in IdentityDictionary(Dictionary)>>at:ifAbsentPut: in Block: at: t1 ifAbsentPut: t2...",
    "IdentityDictionary(Dictionary)>>at:ifAbsent:",
    "IdentityDictionary(Dictionary)>>at:ifAbsentPut:",
    "FLLightGeneralMapper(FLMapper)>>clusterInstanceOf:keyInBucket:factory:",
    "FLLightGeneralMapper(FLMapper)>>clusterKeyedByObjectClass:class:",
    "FLLightGeneralMapper(FLMapper)>>mapAndTraceByObjectClass:to:",
    "FLLightGeneralMapper>>visitMethodContext:",
    "Context>>fuelAccept:",
    "FLLightGeneralMapper>>mapAndTrace:",
    "FLLightGlobalMapper>>mapAndTrace:",
    "FLPluggableSubstitutionMapper>>mapAndTrace:",
    "FLAnalysis>>mapAndTrace:",
    "FLAnalysis>>run",
    "setDefaultAnalysis",
    "  self error: 'No decompiler available' in FLAnalyzer>>setDefaultAnalysis in Block: setDefaultAnalysis...",
    "FLAnalyzer>>analysisFor:",
    "FLSerialization>>analysisStep",
    "FLSerialization>>run",
    "setDefaultSerialization",
    "  self error: 'No decompiler available' in FLSerializer>>setDefaultSerialization in Block: setDefaultSerialization...",
    "serialize: t1 on: t2",
    "  self error: 'No decompiler available' in FLSerializer>>serialize:on: in Block: serialize: t1 on: t2...",
    "on: t1 globalEnvironment: t2 do: t3",
    "  self error: 'No decompiler available' in FLEncoder class>>on:globalEnvironment:do: in Block: on: t1 globalEnvironment: t2 do: t3...",
    "BlockClosure>>ensure:",
    "FLEncoder class>>on:globalEnvironment:do:",
    "\u001b[0m"
...
On 3 Aug 2017, at 13:16, Mariano Martinez Peck <marianopeck@gmail.com <mailto:marianopeck@gmail.com>> wrote:
On Thu, Aug 3, 2017 at 8:58 AM, Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>> wrote:
Iâm so close now (bit of a diversion with gitlab yml not liking the eval string) - that did load and it got me really close - Iâm just missing FlSerializer now (I have progressed my work to fuel out the context for debugging to S3)â¦ so looking at the repo, there are quite a few fuel packages - but in a big image I can see that FLSerialiser is in Core - so do I just need core? But then there are a few fuel packages with Core?
The easiest way to understand that is either checking ConfigurationOfFuel directly or some of the related Metacello tool.
Anyway, for this case, I think you need the packages (in this order): FuelPlatform and Fuel. Bug again, Metacello allows some postLoad actions (which we use in Fuel)... so...it's not always easy to load a project without Metacello.
Sometimes you can print the result of #record:
Metacello new 
  configuration: 'Fuel';
  smalltalkhubUser: 'Pharo' project: 'Fuel';
  version: #stable;
  record: #('Core').
->
"linear load : 
  linear load : 2.1.10 [ConfigurationOfFuel]
      linear load : baseline [ConfigurationOfFuelPlatform]
      	load : FuelPlatform-Core
      	load : FuelPlatform-Pharo-Core
      	load : FuelPlatform-Pharo-06"
------
Anyway.... I guess I would try to load these packages (in this order):
FuelPlatform-Core
FuelPlatform-Pharo-Core
FuelPlatform-Pharo-06
Fuel
And when you are down loading, then execute:
(Smalltalk at: #FLPlatform) current addHacks
Iâm also not quite clear on how I would load packages without a Baseline (would I create my own baseline for the fuel packages I need?)
Tim
...
On 3 Aug 2017, at 12:07, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:
2017-08-03 13:02 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>:
I really appreciate your patience and help on this - it looks very promising and I am giving it a spin nowâ¦
When you say the pharo repository - are you referring to this: https://github.com/pharo-project/pharo/tree/master/src <https://github.com/pharo-project/pharo/tree/master/src>. ? Just so I know for the future.
yes
-- Pavel
Tim
...
On 3 Aug 2017, at 09:16, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:
2017-08-03 10:14 GMT+02:00 Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>>:
The easiest think you can do is to copy to your repository (or some other) related packages from the Pharo repository (to do not have to clone it all):
Metacello-GitBasedRepository.package
Metacello-GitHub.package
SUnit-Core.package
and create baseline to load them. I already tried it so you can test it:
https://drive.google.com/open?id=0BzSsmZhqtUTeMVJacTZtdW5UQW8 <https://drive.google.com/open?id=0BzSsmZhqtUTeMVJacTZtdW5UQW8>
Then you will do something like:
(accidental message send ;-) )
BASELINE=MetacelloGitBasedRepository
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
  "Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src <>'; load."
BASELINE=Lambda
pharo "$IMAGE_NAME.image" --no-default-preferences eval --save \
  "Metacello new baseline: '$BASELINE'; repository: 'filetree://./PharoLambda/src <>'; load."
As I tried that. The baseline of Lambda is then able to be loaded.
-- Pavel
2017-08-02 15:18 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>:
Ah, I think Iâm starting to get closer to what I needâ¦
So if I want to load in that last piece to enable more general remote loading - how do I figure out how to do that? Iâm trying to work out where the build steps for building up the image (can I see them in Jenkins? It wasnât clear to me if I can look it up? Or the BaselineOfIDE was mentioned - looking there I can see a few metacello and gofer packages - but then I guess Iâm looking for an easy Mcz I can use with the example below?
Or do I just load Metacello as a git submodule and then it will be on my local filesystem to then bootstrap up?
I guess Iâm trying to work out the best sustainable approach to getting to a good server based image that has minimal tools and the ability to easily load remote code for pre-req projects.
Tim
...
On 2 Aug 2017, at 07:05, Guillermo Polito <guillermopolito@gmail.com <mailto:guillermopolito@gmail.com>> wrote:
Yes, you should be able to load an mcz in that image by doing:
(MCDirectoryRepository new directory: 'where-your-mcz-is')
    loadVersionFromFileNamed: 'Metacello-GitBasedRepository-Author.1.mcz') load.
Guille
On Wed, Aug 2, 2017 at 7:57 AM, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:
2017-08-01 23:13 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>:
Hi Pavel - I tried it again and the problem is do with Metacello dependencies in your baseline.
The SUnit baseline doesnât specify any additional dependencies to load (it just loads local packages), whereas my baseline that fails looks like this:
baseline: spec
  <baseline>
spec for: #common do: [
      spec configuration: 'ZTimestamp' with: [
      	spec 
      		versionString: #stable;
      		repository: 'http://mc.stfx.eu/Neo <http://mc.stfx.eu/Neo>' ].
spec baseline: 'AWS' with: [	
      	spec repository: 'github://newapplesho/aws-sdk-smalltalk:v1.10/pharo-repository <>' ].
spec 
      	package: 'Lambda' with: [ spec requires: {'ZTimestamp'. 'AWS'}].
].
The âspec configuration: â¦.â Specifications cause the error:
25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
If I remove those lines from my baseline above (as well the requires: reference to them) I can then successfully load my packages.
So is this something the minimal image should support (otherwise how do you load external packages into your image without checking them in locally?) OR is there something simple I can checkin and load locally to return that behaviour? (Personally I think it would make sense to allow remote loading - Iâm guessing Pharo itself as a core has everything it needs - but if you want to do any experimentation on the core and need remote packages, you would hit this too - so it feels a bit limiting).
The dependencies in th baseline are supported but the support for most of external repositories (like package Metacello-GitBasedRepository) is loaded at the end of bootstrapping process in BaselineOfIDE. We should check/solve dependencies and move it into the minimal Pharo. For now you can try to load it by yourself or work with already prepared local clones of required repositories.
-- Pavel
Tim
...
On 1 Aug 2017, at 10:06, Pavel Krivanek <pavel.krivanek@gmail.com <mailto:pavel.krivanek@gmail.com>> wrote:
2017-07-31 22:51 GMT+02:00 Tim Mackinnon <tim@testit.works <mailto:tim@testit.works>>:
I wasnât clear on which image to retry - the https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf... <https://ci.inria.fr/pharo/job/Pharo-6.0-Update-Step-3.2-Minimal/lastSuccessf...> one still shows as being last updated 7 days ago.
The https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur... <https://ci.inria.fr/pharo/view/6.0-SysConf/job/Pharo-6.0-Step-04-01-Configur...> one gives me a mismatch error: This interpreter (vers. 68021) cannot read image file (vers. 6521).
The https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc... <https://ci.inria.fr/pharo/view/7.0/job/70-Bootstrap-32bit-Conversion/lastSuc...> one gives me a walkback when trying to run my install script :
25 UndefinedObject(Object)>>doesNotUnderstand: #addTo:
26 MCRepositoryGroup>>addRepository:
27 createRepository
  | repo |
  repo := self project createRepository: self.
  ^ MCRepositoryGroup default repositories
    detect: [ :each | each = repo ]
    ifNone: [ 
      MCRepositoryGroup default addRepository: repo.
      repo ] in MetacelloRepositorySpec>>createRepository
I think this is because the image doesnât support Metacello? As in:
Metacello new
    repository: 'filetree://../src <>';
    baseline: 'Lambda';
    load.
How do you guys load things into the minimal images (I thought you used Metacello - but maybe you do it some other way?)
I can use a big 6.1 image fine (as it has Metacello loaded) but Iâd really like a minimal solution - that can load in libraries like AWS S3, or XML parsing etc. and it seems like I should be a good customer for kicking the tires on all of this.
Tim
I checked the 64-bit Pharo 7 minimal image and loading of baseline (of SUnit from pharo-project/pharo) works:
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval --save "Metacello new baseline: 'SUnit'; repository: 'filetree://./pharo-core/src <>'; load."
./pharo Pharo7.0-minimal-64bit-b1625bf.image eval "TestCase suite run"
Can you test it too?
-- Pavel
--
Guille Polito
Research Engineer
French National Center for Scientific Research - http://www.cnrs.fr <http://www.cnrs.fr/>
Web: http://guillep.github.io <http://guillep.github.io/>
Phone: +33 06 52 70 66 13 <tel:+33%206%2052%2070%2066%2013>
-- 
Mariano
http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
-- 
Mariano
http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
-- 
Mariano
http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>