On Sat, Aug 6, 2022, 18:40 Shaping shaping@uurda.org wrote:

Run a script in Pharo to enumerate all classes. Then use that to see which
classes you have in VW with the same name.

A good strategy for problems is, when you find one, look for other examples
that match the pattern. You can eliminate a lot of trouble quickly.

Do you have an algo you like?  Is there a strategic way to load all the

[…]

Right now I’m trying to find all Pharo classes that will be bases for the extensions I’m transforming.  That could take a while.

Run a script in Pharo to enumerate all classes. Then use that to see which classes you have in VW with the same name.

Yes,  but there are classes  that exist in Metacello, and don’t live by default in a Pharo image.  What to do in that case?  I hardly know what I’m missing.  I just loaded ArbitraryPrecisionFloat from Metacello because I happen to know the code is common to VW and Pharo.  .

A good strategy for problems is, when you find one, look for other examples that match the pattern. You can eliminate a lot of trouble quickly.

Shaping

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st http://PDFtalk.Pharo100.st  into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st http://PDFtalk.Pharo100.st .  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

Von: Shaping shaping@uurda.org
Gesendet: Sonntag, 7. August 2022 03:40
An: 'Any question about pharo is welcome' pharo-users@lists.pharo.org
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: what to ignore; how to find Pharo base classes not in the base image

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Pool dictionaries are a special beast. Since the great namespace introduction, there are no pool dictionaries in VW anymore. Instead, pool dictionaries are ordinary namespaces which can be imported by a class or namespace. Imports can be either <namespace>.* (general import, which can be used for pools) and <namespace>.<class> for specific imports.

Unfortunately, I have no way to tell if an import is a pool (or is there?). Conceptionally this is clean thou, since there is no difference between a pool and a namespace. Import are used so that you can use a name directly without its fully qualified namespace path. Since there are no namespaces and your own names in namespaces are automatically renamed, there is no need for those in Pharo. Still there could be real pools…

Anyways, imports are put into the class definition as pool dictionaries by default. I think this is a tools issue where the user should be confronted with useful choices.

To change the definition of a class to be written, you can add #definitionChanges: to a ClassChange. The argument is a Valuemap (Dictionary) with the constructor argument name as key and the new value. Example for inspiration (all parts used in my code):

            ClassChange

                           classReference: #{Valuemap}

                           definitionChanges: (Valuemap

                                           with: #superclass -> #EsOrderedDictionary

                                           with: #comment -> ‘a new class comment’

                                           with: #instanceVariables -> #()

                                           with: #poolDictionaries -> #())

So, if your #ForeignHeap is not really a pool dictionary, you should add

            ClassChange

                           classReference: #{ByteArray}

                           definitionChanges: (Valuemap with: #poolDictionaries -> #())

I get a warning that says proceed only if you know what you are doing.  That’s encouraging.  Lol

Sorry about that… I guess it is about the pool dictionary. But I cannot find where I throw that…

How did this definition get created in the first place?  More generally and more to the point, I’m trying to determine an efficient way to determine which classes I will always need to move from VW (like the ones I know to be uniquely mine) versus those I should never try to move from VW because Pharo already has the same by the same name or the same in function by a different name—but I cannot easily discover what these classes are without web searches to track down class definitions that exist for Pharo but that are not yet in Pharo because, like this one:

Metacello new

            smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

            configuration: 'ArbitraryPrecisionFloat';

            version: #stable;

            load.

I have some extensions to APF in VW.  I want to port those to Pharo, but the base class must already be there.  APF for VW and for Pharo seems like the same code (or very close).  So I tracked down the above Metacello evaluable, did it, and now I have APF in Pharo, and I can extend it with my transformed file-in.

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references). Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

Seems I should block the VW class like this:

ShapingArbitraryPrecisionFloatTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.ArbitraryPrecisionFloat})

Only if you have not extended the class. #ignoredNames: means that nothing of the class will be written out.

And use the APF class that I load into Pharo instead.

…

I found IntegerArray in Pharo.  So I should exclude this one from the transform on the VW side too:

ShapingCollectionTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.IntegerArray})

No, like above. You want your extensions to be written, I suppose.

Right now I’m trying to find all Pharo classes that will be bases for the extensions I’m transforming.  That could take a while.

Do you have an algo you like?  Is there a strategic way to load all the basic repos for Pharo?  I see the nine standard repos listed by default in the virgin Repositories list.  But not all Pharo code is in Repos.  Some, like the APF class above, is in Metacello.

As tedious as this is, it’s still much better than pure manual porting.

:-) thanks.

Shaping

Christian

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Yes I have:

ByteArray >>

asExternal

^ForeignHeap copyOfByteArray: self

Also note that I have a ByteArray class override:

Smalltalk.Core defineClass: #ByteArray

            superclass: #{Core.IntegerArray}

            indexedType: #bytes

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: '

                                            Shaping.ForeignHeap

                                            '

            category: 'Collections-Arrayed'

that imports my own ForeignHeap.

I need to put that method on an existing Pharo class, which should just be its own version of ByteArray.  So how do I tell my Shaping Collection package transform to ignore transformation of ByteArray’s class definition, but preserve any methods in the same package that are extensions?

Pool dictionaries are a special beast. Since the great namespace introduction, there are no pool dictionaries in VW anymore.

Right.

Instead, pool dictionaries are ordinary namespaces which can be imported by a class or namespace. Imports can be either <namespace>.* (general import, which can be used for pools) and <namespace>.<class> for specific imports.

I have several specific <namespace>.<class> imports.

Unfortunately, I have no way to tell if an import is a pool (or is there?). Conceptionally this is clean thou, since there is no difference between a pool and a namespace. Import are used so that you can use a name directly without its fully qualified namespace path. Since there are no namespaces and your own names in namespaces are automatically renamed, there is no need for those in Pharo. Still there could be real pools…

I don’t think I use any pools.

Anyways, imports are put into the class definition as pool dictionaries by default. I think this is a tools issue where the user should be confronted with useful choices.

To change the definition of a class to be written, you can add #definitionChanges: to a ClassChange.

The argument is a Valuemap (Dictionary) with the constructor argument name as key and the new value. Example for inspiration (all parts used in my code):

            ClassChange

                           classReference: #{Valuemap}

                           definitionChanges: (Valuemap

                                           with: #superclass -> #EsOrderedDictionary

                                           with: #comment -> ‘a new class comment’

                                           with: #instanceVariables -> #()

                                           with: #poolDictionaries -> #())

So, if your #ForeignHeap is not really a pool dictionary, you should add

            ClassChange

                           classReference: #{ByteArray}

                           definitionChanges: (Valuemap with: #poolDictionaries -> #())

Okay, done.  I’ll try it.

I get a warning that says proceed only if you know what you are doing.  That’s encouraging.  Lol

Sorry about that… I guess it is about the pool dictionary. But I cannot find where I throw that…

It’s not your stuff.  The warning is on the Pharo side when I’m filing in.  I’m being warned about the dangers of redefining ByteArray.

How did this definition get created in the first place?  More generally and more to the point, I’m trying to determine an efficient way to determine which classes I will always need to move from VW (like the ones I know to be uniquely mine) versus those I should never try to move from VW because Pharo already has the same by the same name or the same in function by a different name—but I cannot easily discover what these classes are without web searches to track down class definitions that exist for Pharo but that are not yet in Pharo, like this one:

Metacello new

            smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

            configuration: 'ArbitraryPrecisionFloat';

            version: #stable;

            load.

I have some extensions to APF in VW.  I want to port those to Pharo, but the base class must already be there.  APF for VW and for Pharo seems like the same code (or very close).  So I tracked down the above Metacello evaluable, did it, and now I have APF in Pharo, and I can extend it with my transformed file-in.

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references).

Okay.

Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

So you want to do the actual Pharo preload of the needed Pharo base class on the VW side, and write the resulting class definition to the file-out stream.  I’d have to build the above class and copy from Pharo into VW the code that does Metacello load.  It may not be worth the hassle.  I might need only this one missing Pharo base class, but I could be wrong about that.  Perhaps there are many such classes.

Seems I should block the VW class like this:

ShapingArbitraryPrecisionFloatTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.ArbitraryPrecisionFloat})

Only if you have not extended the class. #ignoredNames: means that nothing of the class will be written out.

Okay.

And use the APF class that I load into Pharo instead.

…

I found IntegerArray in Pharo.  So I should exclude this one from the transform on the VW side too:

ShapingCollectionTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.IntegerArray})

No, like above. You want your extensions to be written, I suppose.

Okay.

Right now I’m trying to find all Pharo classes that will be bases for the extensions I’m transforming.  That could take a while.

Do you have an algo you like?  Is there a strategic way to load all the basic repos for Pharo?  I see the nine standard repos listed by default in the virgin Repository’s list.  But not all Pharo code is in Repos.  Some, like the APF class above, is in Metacello.

As tedious as this is, it’s still much better than pure manual porting.

:-) thanks.

Okay, let’s try the new addition above and see how it goes….

Running with

TargetSmalltalk class>>

ShapingCollectionTransform

^PackageChange new

 add:

   (ClassChange

     classReference:    #{ByteArray}

     definitionChanges: (Valuemap with: #poolDictionaries -> #())

   )

…

The file-in gets past the APF stuff and is failing on setting the comment of a class that is not present.  I don’t see how that can happen, but the needed class def is not present in the chunk list before or after the comment: send.

Shaping

Christian

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

Correction:

I’m betting another DangerousClassNotifier about redefining ByteArray.

I don’t understand that.  The Changes Browser is given a different result.  The problem with the chunks occurs later and involves a different class

Shaping

From: Shaping shaping@uurda.org
Sent: Sunday, 7 August, 2022 06:33
To: 'Any question about pharo is welcome' pharo-users@lists.pharo.org
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: what to ignore; how to find Pharo base classes not in the base image

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Yes I have:

ByteArray >>

asExternal

^ForeignHeap copyOfByteArray: self

Also note that I have a ByteArray class override:

Smalltalk.Core defineClass: #ByteArray

            superclass: #{Core.IntegerArray}

            indexedType: #bytes

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: '

                                            Shaping.ForeignHeap

                                            '

            category: 'Collections-Arrayed'

that imports my own ForeignHeap.

I need to put that method on an existing Pharo class, which should just be its own version of ByteArray.  So how do I tell my Shaping Collection package transform to ignore transformation of ByteArray’s class definition, but preserve any methods in the same package that are extensions?

Pool dictionaries are a special beast. Since the great namespace introduction, there are no pool dictionaries in VW anymore.

Right.

Instead, pool dictionaries are ordinary namespaces which can be imported by a class or namespace. Imports can be either <namespace>.* (general import, which can be used for pools) and <namespace>.<class> for specific imports.

I have several specific <namespace>.<class> imports.

Unfortunately, I have no way to tell if an import is a pool (or is there?). Conceptionally this is clean thou, since there is no difference between a pool and a namespace. Import are used so that you can use a name directly without its fully qualified namespace path. Since there are no namespaces and your own names in namespaces are automatically renamed, there is no need for those in Pharo. Still there could be real pools…

I don’t think I use any pools.

Anyways, imports are put into the class definition as pool dictionaries by default. I think this is a tools issue where the user should be confronted with useful choices.

To change the definition of a class to be written, you can add #definitionChanges: to a ClassChange.

The argument is a Valuemap (Dictionary) with the constructor argument name as key and the new value. Example for inspiration (all parts used in my code):

            ClassChange

                           classReference: #{Valuemap}

                           definitionChanges: (Valuemap

                                           with: #superclass -> #EsOrderedDictionary

                                           with: #comment -> ‘a new class comment’

                                           with: #instanceVariables -> #()

                                           with: #poolDictionaries -> #())

So, if your #ForeignHeap is not really a pool dictionary, you should add

            ClassChange

                           classReference: #{ByteArray}

                           definitionChanges: (Valuemap with: #poolDictionaries -> #())

Okay, done.  I’ll try it.

I get a warning that says proceed only if you know what you are doing.  That’s encouraging.  Lol

Sorry about that… I guess it is about the pool dictionary. But I cannot find where I throw that…

It’s not your stuff.  The warning is on the Pharo side when I’m filing in.  I’m being warned about the dangers of redefining ByteArray.

How did this definition get created in the first place?  More generally and more to the point, I’m trying to determine an efficient way to determine which classes I will always need to move from VW (like the ones I know to be uniquely mine) versus those I should never try to move from VW because Pharo already has the same by the same name or the same in function by a different name—but I cannot easily discover what these classes are without web searches to track down class definitions that exist for Pharo but that are not yet in Pharo, like this one:

Metacello new

            smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

            configuration: 'ArbitraryPrecisionFloat';

            version: #stable;

            load.

I have some extensions to APF in VW.  I want to port those to Pharo, but the base class must already be there.  APF for VW and for Pharo seems like the same code (or very close).  So I tracked down the above Metacello evaluable, did it, and now I have APF in Pharo, and I can extend it with my transformed file-in.

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references).

Okay.

Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

So you want to do the actual Pharo preload of the needed Pharo base class on the VW side, and write the resulting class definition to the file-out stream.  I’d have to build the above class and copy from Pharo into VW the code that does Metacello load.  It may not be worth the hassle.  I might need only this one missing Pharo base class, but I could be wrong about that.  Perhaps there are many such classes.

Seems I should block the VW class like this:

ShapingArbitraryPrecisionFloatTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.ArbitraryPrecisionFloat})

Only if you have not extended the class. #ignoredNames: means that nothing of the class will be written out.

Okay.

And use the APF class that I load into Pharo instead.

…

I found IntegerArray in Pharo.  So I should exclude this one from the transform on the VW side too:

ShapingCollectionTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.IntegerArray})

No, like above. You want your extensions to be written, I suppose.

Okay.

Right now I’m trying to find all Pharo classes that will be bases for the extensions I’m transforming.  That could take a while.

Do you have an algo you like?  Is there a strategic way to load all the basic repos for Pharo?  I see the nine standard repos listed by default in the virgin Repository’s list.  But not all Pharo code is in Repos.  Some, like the APF class above, is in Metacello.

As tedious as this is, it’s still much better than pure manual porting.

:-) thanks.

Okay, let’s try the new addition above and see how it goes….

Running with

TargetSmalltalk class>>

ShapingCollectionTransform

^PackageChange new

 add:

   (ClassChange

     classReference:    #{ByteArray}

     definitionChanges: (Valuemap with: #poolDictionaries -> #())

   )

…

The file-in gets past the APF stuff and is failing on setting the comment of a class that is not present.  I don’t see how that can happen, but the needed class def is not present in the chunk list before or after the comment: send.

Shaping

Christian

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' pharo-users@lists.pharo.org; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

Correction:

I’m getting another DangerousClassNotifier about redefining ByteArray.

I don’t understand that.  The Changes Browser is given a different result.  The problem with the chunks occurs later and involves a different class

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Sunday, 7 August, 2022 06:33
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: what to ignore; how to find Pharo base classes not in the base image

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Yes I have:

ByteArray >>

asExternal

^ForeignHeap copyOfByteArray: self

Also note that I have a ByteArray class override:

Smalltalk.Core defineClass: #ByteArray

            superclass: #{Core.IntegerArray}

            indexedType: #bytes

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: '

                                            Shaping.ForeignHeap

                                            '

            category: 'Collections-Arrayed'

that imports my own ForeignHeap.

I need to put that method on an existing Pharo class, which should just be its own version of ByteArray.  So how do I tell my Shaping Collection package transform to ignore transformation of ByteArray’s class definition, but preserve any methods in the same package that are extensions?

Pool dictionaries are a special beast. Since the great namespace introduction, there are no pool dictionaries in VW anymore.

Right.

Instead, pool dictionaries are ordinary namespaces which can be imported by a class or namespace. Imports can be either <namespace>.* (general import, which can be used for pools) and <namespace>.<class> for specific imports.

I have several specific <namespace>.<class> imports.

Unfortunately, I have no way to tell if an import is a pool (or is there?). Conceptionally this is clean thou, since there is no difference between a pool and a namespace. Import are used so that you can use a name directly without its fully qualified namespace path. Since there are no namespaces and your own names in namespaces are automatically renamed, there is no need for those in Pharo. Still there could be real pools…

I don’t think I use any pools.

Anyways, imports are put into the class definition as pool dictionaries by default. I think this is a tools issue where the user should be confronted with useful choices.

To change the definition of a class to be written, you can add #definitionChanges: to a ClassChange.

The argument is a Valuemap (Dictionary) with the constructor argument name as key and the new value. Example for inspiration (all parts used in my code):

            ClassChange

                           classReference: #{Valuemap}

                           definitionChanges: (Valuemap

                                           with: #superclass -> #EsOrderedDictionary

                                           with: #comment -> ‘a new class comment’

                                           with: #instanceVariables -> #()

                                           with: #poolDictionaries -> #())

So, if your #ForeignHeap is not really a pool dictionary, you should add

            ClassChange

                           classReference: #{ByteArray}

                           definitionChanges: (Valuemap with: #poolDictionaries -> #())

Okay, done.  I’ll try it.

I get a warning that says proceed only if you know what you are doing.  That’s encouraging.  Lol

Sorry about that… I guess it is about the pool dictionary. But I cannot find where I throw that…

It’s not your stuff.  The warning is on the Pharo side when I’m filing in.  I’m being warned about the dangers of redefining ByteArray.

How did this definition get created in the first place?  More generally and more to the point, I’m trying to determine an efficient way to determine which classes I will always need to move from VW (like the ones I know to be uniquely mine) versus those I should never try to move from VW because Pharo already has the same by the same name or the same in function by a different name—but I cannot easily discover what these classes are without web searches to track down class definitions that exist for Pharo but that are not yet in Pharo, like this one:

Metacello new

            smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

            configuration: 'ArbitraryPrecisionFloat';

            version: #stable;

            load.

I have some extensions to APF in VW.  I want to port those to Pharo, but the base class must already be there.  APF for VW and for Pharo seems like the same code (or very close).  So I tracked down the above Metacello evaluable, did it, and now I have APF in Pharo, and I can extend it with my transformed file-in.

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references).

Okay.

Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

So you want to do the actual Pharo preload of the needed Pharo base class on the VW side, and write the resulting class definition to the file-out stream.  I’d have to build the above class and copy from Pharo into VW the code that does Metacello load.  It may not be worth the hassle.  I might need only this one missing Pharo base class, but I could be wrong about that.  Perhaps there are many such classes.

Seems I should block the VW class like this:

ShapingArbitraryPrecisionFloatTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.ArbitraryPrecisionFloat})

Only if you have not extended the class. #ignoredNames: means that nothing of the class will be written out.

Okay.

And use the APF class that I load into Pharo instead.

…

I found IntegerArray in Pharo.  So I should exclude this one from the transform on the VW side too:

ShapingCollectionTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.IntegerArray})

No, like above. You want your extensions to be written, I suppose.

Okay.

Right now I’m trying to find all Pharo classes that will be bases for the extensions I’m transforming.  That could take a while.

Do you have an algo you like?  Is there a strategic way to load all the basic repos for Pharo?  I see the nine standard repos listed by default in the virgin Repository’s list.  But not all Pharo code is in Repos.  Some, like the APF class above, is in Metacello.

As tedious as this is, it’s still much better than pure manual porting.

:-) thanks.

Okay, let’s try the new addition above and see how it goes….

Running with

TargetSmalltalk class>>

ShapingCollectionTransform

^PackageChange new

 add:

   (ClassChange

     classReference:    #{ByteArray}

     definitionChanges: (Valuemap with: #poolDictionaries -> #())

   )

…

The file-in gets past the APF stuff and is failing on setting the comment of a class that is not present.  I don’t see how that can happen, but the needed class def is not present in the chunk list before or after the comment: send.

Shaping

Christian

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

Von: Shaping shaping@uurda.org
Gesendet: Sonntag, 7. August 2022 13:33
An: 'Any question about pharo is welcome' pharo-users@lists.pharo.org
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: what to ignore; how to find Pharo base classes not in the base image

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Yes I have:

ByteArray >>

asExternal

^ForeignHeap copyOfByteArray: self

Also note that I have a ByteArray class override:

Smalltalk.Core defineClass: #ByteArray

            superclass: #{Core.IntegerArray}

            indexedType: #bytes

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: '

                                            Shaping.ForeignHeap

                                            '

            category: 'Collections-Arrayed'

that imports my own ForeignHeap.

I need to put that method on an existing Pharo class, which should just be its own version of ByteArray.  So how do I tell my Shaping Collection package transform to ignore transformation of ByteArray’s class definition, but preserve any methods in the same package that are extensions?

It should be fine to extend the corresponding Pharo ByteArray with your methods. The problem is your overwritten class definition. Since the definition is in your package, it gets transformed as a defined class and not as an extension. But you don’t need to redefine the Pharo class with a pool dictionary, since the transformations take care of the class renamings.

This is a new situation for me: an override of a system class definition which must be ignored and the methods treated as extensions.

Valid case. I have to think about it.

On the other hand, it is bad style to override system class definitions. It is great that VW allows overrides of this kind which can be cleanly unloaded(!). I am not aware of any other Smalltalk with this capability (except maybe Gemstone). Therefore, it should really be the last resort to override system classes in the target Smalltalks.

Similar cases in my code I resolved by removing the import and spelling out the full name of the referenced classes. (This might be slow when used in tight loops). Also generally bad to change your code just for a port. (ok, not generally).

Or, you could ignore the class and add a SystemClassChange (for classes not defined or extended in the package) and add all your methods like:

            PackageChange

                           ignoredNames: #(#{Core.ByteArray})

                           …

                           extensions: (Array

                                           with: (SystemClassChange

                                                           className: #ByteArray

                                                           instanceChanges: (Array 

with: (Add method: #myMethod code: #_shaping_myMethod)

…)

                                                           classChanges: (Array 

with: (Add method: #myClassMethod code: #_shaping_myClassMethod)

…))

(I think that you could even use the original methods instead of the shaping* copies – nice. Untested though)

Pool dictionaries are a special beast. Since the great namespace introduction, there are no pool dictionaries in VW anymore.

Right.

Instead, pool dictionaries are ordinary namespaces which can be imported by a class or namespace. Imports can be either <namespace>.* (general import, which can be used for pools) and <namespace>.<class> for specific imports.

I have several specific <namespace>.<class> imports.

Unfortunately, I have no way to tell if an import is a pool (or is there?). Conceptionally this is clean thou, since there is no difference between a pool and a namespace. Import are used so that you can use a name directly without its fully qualified namespace path. Since there are no namespaces and your own names in namespaces are automatically renamed, there is no need for those in Pharo. Still there could be real pools…

I don’t think I use any pools.

Anyways, imports are put into the class definition as pool dictionaries by default. I think this is a tools issue where the user should be confronted with useful choices.

To change the definition of a class to be written, you can add #definitionChanges: to a ClassChange.

The argument is a Valuemap (Dictionary) with the constructor argument name as key and the new value. Example for inspiration (all parts used in my code):

            ClassChange

                           classReference: #{Valuemap}

                           definitionChanges: (Valuemap

                                           with: #superclass -> #EsOrderedDictionary

                                           with: #comment -> ‘a new class comment’

                                           with: #instanceVariables -> #()

                                           with: #poolDictionaries -> #())

So, if your #ForeignHeap is not really a pool dictionary, you should add

            ClassChange

                           classReference: #{ByteArray}

                           definitionChanges: (Valuemap with: #poolDictionaries -> #())

Okay, done.  I’ll try it.

I get a warning that says proceed only if you know what you are doing.  That’s encouraging.  Lol

Sorry about that… I guess it is about the pool dictionary. But I cannot find where I throw that…

It’s not your stuff.  The warning is on the Pharo side when I’m filing in.  I’m being warned about the dangers of redefining ByteArray.

How did this definition get created in the first place?  More generally and more to the point, I’m trying to determine an efficient way to determine which classes I will always need to move from VW (like the ones I know to be uniquely mine) versus those I should never try to move from VW because Pharo already has the same by the same name or the same in function by a different name—but I cannot easily discover what these classes are without web searches to track down class definitions that exist for Pharo but that are not yet in Pharo, like this one:

Metacello new

            smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

            configuration: 'ArbitraryPrecisionFloat';

            version: #stable;

            load.

I have some extensions to APF in VW.  I want to port those to Pharo, but the base class must already be there.  APF for VW and for Pharo seems like the same code (or very close).  So I tracked down the above Metacello evaluable, did it, and now I have APF in Pharo, and I can extend it with my transformed file-in.

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references).

Okay.

Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

So you want to do the actual Pharo preload of the needed Pharo base class on the VW side, and write the resulting class definition to the file-out stream.  I’d have to build the above class and copy from Pharo into VW the code that does Metacello load.  It may not be worth the hassle.  I might need only this one missing Pharo base class, but I could be wrong about that.  Perhaps there are many such classes.

No no no. On the VW side, the following string has to be written into the fileout before the package loads (with #writeLoadDoitOn: ):

“Metacello new

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

configuration: 'ArbitraryPrecisionFloat';

version: #stable;

load!”

Since the fileout is in chunk format, you can add any doIts you want. That’s how I can create packages and whatnot, because I can do anything you can do in a workspace. So the DoIt above will load the required package into Pharo. With that you have nothing to do in VW.

Seems I should block the VW class like this:

ShapingArbitraryPrecisionFloatTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.ArbitraryPrecisionFloat})

Only if you have not extended the class. #ignoredNames: means that nothing of the class will be written out.

Okay.

And use the APF class that I load into Pharo instead.

…

I found IntegerArray in Pharo.  So I should exclude this one from the transform on the VW side too:

ShapingCollectionTransform

^PackageChange new

   ignoredNames: #(#{Smalltalk.IntegerArray})

No, like above. You want your extensions to be written, I suppose.

Okay.

Right now I’m trying to find all Pharo classes that will be bases for the extensions I’m transforming.  That could take a while.

Do you have an algo you like?  Is there a strategic way to load all the basic repos for Pharo?  I see the nine standard repos listed by default in the virgin Repository’s list.  But not all Pharo code is in Repos.  Some, like the APF class above, is in Metacello.

As tedious as this is, it’s still much better than pure manual porting.

:-) thanks.

Okay, let’s try the new addition above and see how it goes….

Running with

TargetSmalltalk class>>

ShapingCollectionTransform

^PackageChange new

 add:

   (ClassChange

     classReference:    #{ByteArray}

     definitionChanges: (Valuemap with: #poolDictionaries -> #())

   )

…

The file-in gets past the APF stuff and is failing on setting the comment of a class that is not present.  I don’t see how that can happen, but the needed class def is not present in the chunk list before or after the comment: send.

???

Shaping

Christian

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Yes I have:

ByteArray >>

asExternal

^ForeignHeap copyOfByteArray: self

Also note that I have a ByteArray class override:

Smalltalk.Core defineClass: #ByteArray

            superclass: #{Core.IntegerArray}

            indexedType: #bytes

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: '

                                            Shaping.ForeignHeap

                                            '

            category: 'Collections-Arrayed'

that imports my own ForeignHeap.

I need to put that method on an existing Pharo class, which should just be its own version of ByteArray.  So how do I tell my Shaping Collection package transform to ignore transformation of ByteArray’s class definition, but preserve any methods in the same package that are extensions?

It should be fine to extend the corresponding Pharo ByteArray with your methods. The problem is your overwritten class definition. Since the definition is in your package, it gets transformed as a defined class and not as an extension. But you don’t need to redefine the Pharo class with a pool dictionary, since the transformations take care of the class renamings.

Okay, so I’ll just undo the VW override of ByteArray, so that the transformation, as currently structured, can operate correctly.  I don’t need VW’s or my flavor of VW’s ForeignHeap.  Pharo has its own idea of what that is, and any of my VW ForeignHeap-related method can be moved over separately.  I guess I need to find the right Pharo class and try to attached the methods to that by do code rewrites.  If the class doesn’t exist, I can use something like my current VW ForeignHeap wrapper to reify foreign memory.  I’m not sure how the uFFI does things.  I found this:

http://books.pharo.org/booklet-uffi/pdf/2020-02-12-uFFI-V1.0.1

(p. 25)

“

Obtaining an ExternalAddress The most common way of obtaining an ExternalAddress is to receive it as the return value of a called C function. A good example is the libc function malloc(), which takes an integer specifying the desired size of a heap buffer, tries to allocate a contiguous block of memory of the size requested, and, if successful, returns a pointer to the allocated region of memory. The C declaration of such function reads as follows (from the libc manual): void malloc( size_t size ); We only need create a uFFI binding to it by copy-pasting the declaration as follows: FFITutorial class >> malloc: aSize [ ^ self ffiCall: #( void * malloc ( size_t aSize ) ) library: LibC ] Notice that this function returns a ”generic pointer”, of type void * (meaning, what it points to in memory is untyped). In uFFI, this is marshalled to a ExternalAddress object. In other words, our malloc() binding yields an ExternalAddress in return (if successful). For example, if we use the above binding to ask the system to allocate a buffer of 200 bytes for us, then on return we should have an address that points somewhere in the C heap: FFITutorial malloc: 200 => (void)@ 16r7FFBDE0DE030 But if we ask for more memory than there is currently available, malloc() will fail and return a NULL pointer instead: FFITutorial malloc: SmallInteger maxVal => (void*)@ 16r00000000 Consequently, we must always check the return value in cases like this, lest we invoke the infamous ”null pointer assignment” bug.

“

This is a new situation for me: an override of a system class definition which must be ignored and the methods treated as extensions.

Yup, that’s odd.  I’ll patch around it for now.  Let me know if you publish something to the public repo, or if you have file-in I can apply.

Valid case. I have to think about it.

On the other hand, it is bad style to override system class definitions. It is great that VW allows overrides of this kind which can be cleanly unloaded(!). I am not aware of any other Smalltalk with this capability (except maybe Gemstone). Therefore, it should really be the last resort to override system classes in the target Smalltalks.

It was done for both notational convenience and speed. Otherwise, I would have needed to use the dotted notation.  I’ve timed it, and it costs more microseconds than I wanted.

Similar cases in my code I resolved by removing the import and spelling out the full name of the referenced classes.

Yeah, see above.  It does cost a little in speed.  I don’t recall the extra latency, but it was not insignificant, especially in high-frequency loops.  I did the measurement sometime around 7.5/7.6 and have not done it since.  Yes, I prefer not to crap-up a system class, but I’m flexible when I need speed.

(This might be slow when used in tight loops). Also generally bad to change your code just for a port. (ok, not generally).

Or, you could ignore the class and add a SystemClassChange (for classes not defined or extended in the package) and add all your methods like:

            PackageChange

                           ignoredNames: #(#{Core.ByteArray})

                           …

                           extensions: (Array

                                           with: (SystemClassChange

                                                           className: #ByteArray

                                                           instanceChanges: (Array 

with: (Add method: #myMethod code: #_shaping_myMethod)

…)

                                                           classChanges: (Array 

with: (Add method: #myClassMethod code: #_shaping_myClassMethod)

…))

(I think that you could even use the original methods instead of the shaping* copies – nice. Untested though)

Okay, I put that bit in the code but commented out.  I can’t really move that method anyway, because there is no ForeignHeap equivalent that I know yet in Pharo.    As far as I can tell, Pharo doesn’t try to reify as I do.  It would be a separate task I would do later.  Not a high priority now.

[…]

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references).

Okay.

Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

So you want to do the actual Pharo preload of the needed Pharo base class on the VW side, and write the resulting class definition to the file-out stream.  I’d have to build the above class and copy from Pharo into VW the code that does Metacello load.  It may not be worth the hassle.  I might need only this one missing Pharo base class, but I could be wrong about that.  Perhaps there are many such classes.

No no no. On the VW side, the following string has to be written into the fileout before the package loads (with #writeLoadDoitOn: ):

“Metacello new

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

configuration: 'ArbitraryPrecisionFloat';

version: #stable;

load!”

Since the fileout is in chunk format, you can add any doIts you want. That’s how I can create packages and whatnot, because I can do anything you can do in a workspace. So the DoIt above will load the required package into Pharo. With that you have nothing to do in VW.

I’m trying to find the right spot in the transformation to do this.  The below method is the only sender of #writeLoadDoitOn  for Pharo (there’s one for Squeak too).

PharoPreload>>

writePackageHeadFor: aWriter

            aWriter change hasPreload ifFalse: [

                            ^self].

            aWriter change preload do: [:preload | preload writeLoadDoitOn: aWriter stream]

Working down the stack we have

PackageWriter>>

writeHead

            self target writePackageHeadFor: self

then

Subwriter>>

writeCodeComponent

          self writeBaseClasses “write pharo code that loads essential base classes  <==============”

            self writeHead.

            self writeRunPrereadCode.

            self writeRunPreloadCode.

            self writeContents.

            self writeRunPostloadCode.

            self writeTail

So where indicated we need to write:

“Metacello new

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

configuration: 'ArbitraryPrecisionFloat';

version: #stable;

load!”

at the arrow.

So as a first go I’ve written:

PackageWriter>>

writeBaseClasses

            self target writeBaseClassesFor: self

Pharo class>>

writeBaseClassesFor: aWriter

            aWriter change hasBaseclassLoadEvaluables ifFalse: [

                            ^self].

            aWriter change baseClassLoadEvaluables do: [:preload | preload writeLoadDoitOn: aWriter stream]

PackageChange>>

baseClassLoadEvaluables: baseClassLoadEvaluables

       ignoredNames: ingoredNames



            ^self

    baseClassLoadEvaluables: baseClassLoadEvaluables 

                            preload: nil

                            ignoredNames: ingoredNames

                            bridgeClasses: nil

                            hierarchyChanges: nil

                            localChanges: nil

                            extensions: nil

All the downstream methods have been created.    baseClassLoadEvaluables will be an array of string evaluables like the above (could have a bunch of them).

You can probably do a nicer job than I because you know the code better, but something like this hack could work.  You might just want to do that the way you want nice and clean, and hand it back to me as a file in.  I’ve not run this yet.  I’ll do that in a few minutes and let you know how it goes.  I can send you the complete file-out if/when it works, or you can use the above as a rough template and do it for us, and pass it back to me so that I can test it.  The code will stay cleaner that way.

The file-in gets past the APF stuff and is failing on setting the comment of a class that is not present.  I don’t see how that can happen, but the needed class def is not present in the chunk list before or after the comment: send.

???

Yeah, I don’t understand it either.  I get different behavior from the file in when I just run it versus manually evaluating the chunks form the Changes browser.  I’ll ty it again with the new stuff for the baseClasses.

Shaping

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

I just realized that the baseClassLoadEvaluable (to load AFP for example) needs to be the install parameter in an instance of PharoPreload.

Fixing that now.

Shaping

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Yes I have:

ByteArray >>

asExternal

^ForeignHeap copyOfByteArray: self

Also note that I have a ByteArray class override:

Smalltalk.Core defineClass: #ByteArray

            superclass: #{Core.IntegerArray}

            indexedType: #bytes

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: '

                                            Shaping.ForeignHeap

                                            '

            category: 'Collections-Arrayed'

that imports my own ForeignHeap.

I need to put that method on an existing Pharo class, which should just be its own version of ByteArray.  So how do I tell my Shaping Collection package transform to ignore transformation of ByteArray’s class definition, but preserve any methods in the same package that are extensions?

It should be fine to extend the corresponding Pharo ByteArray with your methods. The problem is your overwritten class definition. Since the definition is in your package, it gets transformed as a defined class and not as an extension. But you don’t need to redefine the Pharo class with a pool dictionary, since the transformations take care of the class renamings.

Okay, so I’ll just undo the VW override of ByteArray, so that the transformation, as currently structured, can operate correctly.  I don’t need VW’s or my flavor of VW’s ForeignHeap.  Pharo has its own idea of what that is, and any of my VW ForeignHeap-related method can be moved over separately.  I guess I need to find the right Pharo class and try to attached the methods to that by do code rewrites.  If the class doesn’t exist, I can use something like my current VW ForeignHeap wrapper to reify foreign memory.  I’m not sure how the uFFI does things.  I found this:

http://books.pharo.org/booklet-uffi/pdf/2020-02-12-uFFI-V1.0.1

(p. 25)

“

Obtaining an ExternalAddress The most common way of obtaining an ExternalAddress is to receive it as the return value of a called C function. A good example is the libc function malloc(), which takes an integer specifying the desired size of a heap buffer, tries to allocate a contiguous block of memory of the size requested, and, if successful, returns a pointer to the allocated region of memory. The C declaration of such function reads as follows (from the libc manual): void malloc( size_t size ); We only need create a uFFI binding to it by copy-pasting the declaration as follows: FFITutorial class >> malloc: aSize [ ^ self ffiCall: #( void * malloc ( size_t aSize ) ) library: LibC ] Notice that this function returns a ”generic pointer”, of type void * (meaning, what it points to in memory is untyped). In uFFI, this is marshalled to a ExternalAddress object. In other words, our malloc() binding yields an ExternalAddress in return (if successful). For example, if we use the above binding to ask the system to allocate a buffer of 200 bytes for us, then on return we should have an address that points somewhere in the C heap: FFITutorial malloc: 200 => (void)@ 16r7FFBDE0DE030 But if we ask for more memory than there is currently available, malloc() will fail and return a NULL pointer instead: FFITutorial malloc: SmallInteger maxVal => (void*)@ 16r00000000 Consequently, we must always check the return value in cases like this, lest we invoke the infamous ”null pointer assignment” bug.

“

This is a new situation for me: an override of a system class definition which must be ignored and the methods treated as extensions.

Yup, that’s odd.  I’ll patch around it for now.  Let me know if you publish something to the public repo, or if you have file-in I can apply.

Valid case. I have to think about it.

On the other hand, it is bad style to override system class definitions. It is great that VW allows overrides of this kind which can be cleanly unloaded(!). I am not aware of any other Smalltalk with this capability (except maybe Gemstone). Therefore, it should really be the last resort to override system classes in the target Smalltalks.

It was done for both notational convenience and speed. Otherwise, I would have needed to use the dotted notation.  I’ve timed it, and it costs more microseconds than I wanted.

Similar cases in my code I resolved by removing the import and spelling out the full name of the referenced classes.

Yeah, see above.  It does cost a little in speed.  I don’t recall the extra latency, but it was not insignificant, especially in high-frequency loops.  I did the measurement sometime around 7.5/7.6 and have not done it since.  Yes, I prefer not to crap-up a system class, but I’m flexible when I need speed.

(This might be slow when used in tight loops). Also generally bad to change your code just for a port. (ok, not generally).

Or, you could ignore the class and add a SystemClassChange (for classes not defined or extended in the package) and add all your methods like:

            PackageChange

                           ignoredNames: #(#{Core.ByteArray})

                           …

                           extensions: (Array

                                           with: (SystemClassChange

                                                           className: #ByteArray

                                                           instanceChanges: (Array 

with: (Add method: #myMethod code: #_shaping_myMethod)

…)

                                                           classChanges: (Array 

with: (Add method: #myClassMethod code: #_shaping_myClassMethod)

…))

(I think that you could even use the original methods instead of the shaping* copies – nice. Untested though)

Okay, I put that bit in the code but commented out.  I can’t really move that method anyway, because there is no ForeignHeap equivalent that I know yet in Pharo.    As far as I can tell, Pharo doesn’t try to reify as I do.  It would be a separate task I would do later.  Not a high priority now.

[…]

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references).

Okay.

Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

So you want to do the actual Pharo preload of the needed Pharo base class on the VW side, and write the resulting class definition to the file-out stream.  I’d have to build the above class and copy from Pharo into VW the code that does Metacello load.  It may not be worth the hassle.  I might need only this one missing Pharo base class, but I could be wrong about that.  Perhaps there are many such classes.

No no no. On the VW side, the following string has to be written into the fileout before the package loads (with #writeLoadDoitOn: ):

“Metacello new

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

configuration: 'ArbitraryPrecisionFloat';

version: #stable;

load!”

Since the fileout is in chunk format, you can add any doIts you want. That’s how I can create packages and whatnot, because I can do anything you can do in a workspace. So the DoIt above will load the required package into Pharo. With that you have nothing to do in VW.

I’m trying to find the right spot in the transformation to do this.  The below method is the only sender of #writeLoadDoitOn  for Pharo (there’s one for Squeak too).

PharoPreload>>

writePackageHeadFor: aWriter

            aWriter change hasPreload ifFalse: [

                            ^self].

            aWriter change preload do: [:preload | preload writeLoadDoitOn: aWriter stream]

Working down the stack we have

PackageWriter>>

writeHead

            self target writePackageHeadFor: self

then

Subwriter>>

writeCodeComponent

          self writeBaseClasses “write pharo code that loads essential base classes  <==============”

            self writeHead.

            self writeRunPrereadCode.

            self writeRunPreloadCode.

            self writeContents.

            self writeRunPostloadCode.

            self writeTail

So where indicated we need to write:

“Metacello new

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

configuration: 'ArbitraryPrecisionFloat';

version: #stable;

load!”

at the arrow.

So as a first go I’ve written:

PackageWriter>>

writeBaseClasses

            self target writeBaseClassesFor: self

Pharo class>>

writeBaseClassesFor: aWriter

            aWriter change hasBaseclassLoadEvaluables ifFalse: [

                            ^self].

            aWriter change baseClassLoadEvaluables do: [:preload | preload writeLoadDoitOn: aWriter stream]

PackageChange>>

baseClassLoadEvaluables: baseClassLoadEvaluables

       ignoredNames: ingoredNames



            ^self

    baseClassLoadEvaluables: baseClassLoadEvaluables 

                            preload: nil

                            ignoredNames: ingoredNames

                            bridgeClasses: nil

                            hierarchyChanges: nil

                            localChanges: nil

                            extensions: nil

All the downstream methods have been created.    baseClassLoadEvaluables will be an array of string evaluables like the above (could have a bunch of them).

You can probably do a nicer job than I because you know the code better, but something like this hack could work.  You might just want to do that the way you want nice and clean, and hand it back to me as a file in.  I’ve not run this yet.  I’ll do that in a few minutes and let you know how it goes.  I can send you the complete file-out if/when it works, or you can use the above as a rough template and do it for us, and pass it back to me so that I can test it.  The code will stay cleaner that way.

The file-in gets past the APF stuff and is failing on setting the comment of a class that is not present.  I don’t see how that can happen, but the needed class def is not present in the chunk list before or after the comment: send.

???

Yeah, I don’t understand it either.  I get different behavior from the file in when I just run it versus manually evaluating the chunks form the Changes browser.  I’ll ty it again with the new stuff for the baseClasses.

Shaping

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian

This

writeLoadDoitOn: aStream

            aStream

                            cr;

                            nextPutAll: 'Installer ss';

                            crtab;

                            nextPutAll: 'project: ';

                            nextPutAll: self project printString;

                            nextPut: $;;

                            crtab;

                            nextPutAll: 'install: ';

                            nextPutAll: self install printString;

                            nextPut: $!;

                            cr

wrote the following to the stream:

            ^ Installer ss

                              project: 'ArbitraryPrecisionFloat';

                              install: 'MetacelloPreload

   smalltalkhubUser: ''Pharo'' project: ''MetaRepoForPharo70''

   configuration:    ''ArbitraryPrecisionFloat''

   version:          #stable'

which doesn’t compile.  We just want the string after install:

Shaping

From: Shaping shaping@uurda.org
Sent: Monday, 8 August, 2022 04:54
To: 'Any question about pharo is welcome' pharo-users@lists.pharo.org
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: what to ignore; how to find Pharo base classes not in the base image

I just realized that the baseClassLoadEvaluable (to load AFP for example) needs to be the install parameter in an instance of PharoPreload.

Fixing that now.

Shaping

This is VW’s definition of IntegerArray:

Smalltalk.Core defineClass: #IntegerArray

            superclass: #{Core.ArrayedCollection}

            indexedType: #none

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: ''

            category: 'Collections-Arrayed'

I’m stepping through the chunks in the Changes Browser because I don’t see an easier way currently.  This is a separate tool from Epicea.  The first chunk whose file-in fails is:

IntegerArray

            variableByteSubclass: #ByteArray

            instanceVariableNames: ''

            classVariableNames: 'LastDecodeMap Decodings LastEncodeMap Encodings Lock'

            poolDictionaries: 'ForeignHeap'

            package: 'Shaping-Collection'

I don’t understand how this definition could have been created. ByteArray is a VW system class. Such classes should not be ported anyways – alone from copyright reasons. The only addition I see is the pool ForeignHeap. I guess, that you extended ByteArray somehow and added an import ForeignHeap.* to ByteArray.

Yes I have:

ByteArray >>

asExternal

^ForeignHeap copyOfByteArray: self

Also note that I have a ByteArray class override:

Smalltalk.Core defineClass: #ByteArray

            superclass: #{Core.IntegerArray}

            indexedType: #bytes

            private: false

            instanceVariableNames: ''

            classInstanceVariableNames: ''

            imports: '

                                            Shaping.ForeignHeap

                                            '

            category: 'Collections-Arrayed'

that imports my own ForeignHeap.

I need to put that method on an existing Pharo class, which should just be its own version of ByteArray.  So how do I tell my Shaping Collection package transform to ignore transformation of ByteArray’s class definition, but preserve any methods in the same package that are extensions?

It should be fine to extend the corresponding Pharo ByteArray with your methods. The problem is your overwritten class definition. Since the definition is in your package, it gets transformed as a defined class and not as an extension. But you don’t need to redefine the Pharo class with a pool dictionary, since the transformations take care of the class renamings.

Okay, so I’ll just undo the VW override of ByteArray, so that the transformation, as currently structured, can operate correctly.  I don’t need VW’s or my flavor of VW’s ForeignHeap.  Pharo has its own idea of what that is, and any of my VW ForeignHeap-related method can be moved over separately.  I guess I need to find the right Pharo class and try to attached the methods to that by do code rewrites.  If the class doesn’t exist, I can use something like my current VW ForeignHeap wrapper to reify foreign memory.  I’m not sure how the uFFI does things.  I found this:

http://books.pharo.org/booklet-uffi/pdf/2020-02-12-uFFI-V1.0.1

(p. 25)

“

Obtaining an ExternalAddress The most common way of obtaining an ExternalAddress is to receive it as the return value of a called C function. A good example is the libc function malloc(), which takes an integer specifying the desired size of a heap buffer, tries to allocate a contiguous block of memory of the size requested, and, if successful, returns a pointer to the allocated region of memory. The C declaration of such function reads as follows (from the libc manual): void malloc( size_t size ); We only need create a uFFI binding to it by copy-pasting the declaration as follows: FFITutorial class >> malloc: aSize [ ^ self ffiCall: #( void * malloc ( size_t aSize ) ) library: LibC ] Notice that this function returns a ”generic pointer”, of type void * (meaning, what it points to in memory is untyped). In uFFI, this is marshalled to a ExternalAddress object. In other words, our malloc() binding yields an ExternalAddress in return (if successful). For example, if we use the above binding to ask the system to allocate a buffer of 200 bytes for us, then on return we should have an address that points somewhere in the C heap: FFITutorial malloc: 200 => (void)@ 16r7FFBDE0DE030 But if we ask for more memory than there is currently available, malloc() will fail and return a NULL pointer instead: FFITutorial malloc: SmallInteger maxVal => (void*)@ 16r00000000 Consequently, we must always check the return value in cases like this, lest we invoke the infamous ”null pointer assignment” bug.

“

This is a new situation for me: an override of a system class definition which must be ignored and the methods treated as extensions.

Yup, that’s odd.  I’ll patch around it for now.  Let me know if you publish something to the public repo, or if you have file-in I can apply.

Valid case. I have to think about it.

On the other hand, it is bad style to override system class definitions. It is great that VW allows overrides of this kind which can be cleanly unloaded(!). I am not aware of any other Smalltalk with this capability (except maybe Gemstone). Therefore, it should really be the last resort to override system classes in the target Smalltalks.

It was done for both notational convenience and speed. Otherwise, I would have needed to use the dotted notation.  I’ve timed it, and it costs more microseconds than I wanted.

Similar cases in my code I resolved by removing the import and spelling out the full name of the referenced classes.

Yeah, see above.  It does cost a little in speed.  I don’t recall the extra latency, but it was not insignificant, especially in high-frequency loops.  I did the measurement sometime around 7.5/7.6 and have not done it since.  Yes, I prefer not to crap-up a system class, but I’m flexible when I need speed.

(This might be slow when used in tight loops). Also generally bad to change your code just for a port. (ok, not generally).

Or, you could ignore the class and add a SystemClassChange (for classes not defined or extended in the package) and add all your methods like:

            PackageChange

                           ignoredNames: #(#{Core.ByteArray})

                           …

                           extensions: (Array

                                           with: (SystemClassChange

                                                           className: #ByteArray

                                                           instanceChanges: (Array 

with: (Add method: #myMethod code: #_shaping_myMethod)

…)

                                                           classChanges: (Array 

with: (Add method: #myClassMethod code: #_shaping_myClassMethod)

…))

(I think that you could even use the original methods instead of the shaping* copies – nice. Untested though)

Okay, I put that bit in the code but commented out.  I can’t really move that method anyway, because there is no ForeignHeap equivalent that I know yet in Pharo.    As far as I can tell, Pharo doesn’t try to reify as I do.  It would be a separate task I would do later.  Not a high priority now.

[…]

For this, a PackageChange has a #preload. It can be filled with a dialect specific specification of what to load beforehand. I defined PharoPreload and use it in Squeak :-) (look for the references).

Okay.

Actually, it should be renamed to MonticelloPreload or so. What you need would be a MetacelloPreload in which you would capture the parameters needed to load that package (and implement “#writeLoadDoitOn: aStream” too :-) ).

I need to learn more about Monticello and Metacello to design the proper preload objects… (or I meet somebody who known everything about it :-) .

Something like

            MetacelloPreload

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70'

configuration: 'ArbitraryPrecisionFloat'

version: #stable

So you want to do the actual Pharo preload of the needed Pharo base class on the VW side, and write the resulting class definition to the file-out stream.  I’d have to build the above class and copy from Pharo into VW the code that does Metacello load.  It may not be worth the hassle.  I might need only this one missing Pharo base class, but I could be wrong about that.  Perhaps there are many such classes.

No no no. On the VW side, the following string has to be written into the fileout before the package loads (with #writeLoadDoitOn: ):

“Metacello new

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

configuration: 'ArbitraryPrecisionFloat';

version: #stable;

load!”

Since the fileout is in chunk format, you can add any doIts you want. That’s how I can create packages and whatnot, because I can do anything you can do in a workspace. So the DoIt above will load the required package into Pharo. With that you have nothing to do in VW.

I’m trying to find the right spot in the transformation to do this.  The below method is the only sender of #writeLoadDoitOn  for Pharo (there’s one for Squeak too).

PharoPreload>>

writePackageHeadFor: aWriter

            aWriter change hasPreload ifFalse: [

                            ^self].

            aWriter change preload do: [:preload | preload writeLoadDoitOn: aWriter stream]

Working down the stack we have

PackageWriter>>

writeHead

            self target writePackageHeadFor: self

then

Subwriter>>

writeCodeComponent

          self writeBaseClasses “write pharo code that loads essential base classes  <==============”

            self writeHead.

            self writeRunPrereadCode.

            self writeRunPreloadCode.

            self writeContents.

            self writeRunPostloadCode.

            self writeTail

So where indicated we need to write:

“Metacello new

smalltalkhubUser: 'Pharo' project: 'MetaRepoForPharo70';

configuration: 'ArbitraryPrecisionFloat';

version: #stable;

load!”

at the arrow.

So as a first go I’ve written:

PackageWriter>>

writeBaseClasses

            self target writeBaseClassesFor: self

Pharo class>>

writeBaseClassesFor: aWriter

            aWriter change hasBaseclassLoadEvaluables ifFalse: [

                            ^self].

            aWriter change baseClassLoadEvaluables do: [:preload | preload writeLoadDoitOn: aWriter stream]

PackageChange>>

baseClassLoadEvaluables: baseClassLoadEvaluables

       ignoredNames: ingoredNames



            ^self

    baseClassLoadEvaluables: baseClassLoadEvaluables 

                            preload: nil

                            ignoredNames: ingoredNames

                            bridgeClasses: nil

                            hierarchyChanges: nil

                            localChanges: nil

                            extensions: nil

All the downstream methods have been created.    baseClassLoadEvaluables will be an array of string evaluables like the above (could have a bunch of them).

You can probably do a nicer job than I because you know the code better, but something like this hack could work.  You might just want to do that the way you want nice and clean, and hand it back to me as a file in.  I’ve not run this yet.  I’ll do that in a few minutes and let you know how it goes.  I can send you the complete file-out if/when it works, or you can use the above as a rough template and do it for us, and pass it back to me so that I can test it.  The code will stay cleaner that way.

The file-in gets past the APF stuff and is failing on setting the comment of a class that is not present.  I don’t see how that can happen, but the needed class def is not present in the chunk list before or after the comment: send.

???

Yeah, I don’t understand it either.  I get different behavior from the file in when I just run it versus manually evaluating the chunks form the Changes browser.  I’ll ty it again with the new stuff for the baseClasses.

Shaping

I got about 6 seconds into the first file-in.  I’m running empty transforms on all packages.

Great. You do need a ProjectChange though. Starting with empty PackageChanges is perfect.

Yes, I have the one big ProjectChange with all the PackageChanges  (about 25)  currently empty.

The problem I’m having is using this Syntax error

to determine where in the file-in the problem occurred so that I can know the package whose transform needs work.  I was expecting more context.  So I’m poking around in STCommandLineHandler next with breakpoints.  Open to suggestions.

Strange. The #{String} syntax should have been automatically transformed to #String for Pharo… strange.

All the namespace related issues (like this literal binding reference) should be taken care of.

I had a bunch of those namespace/shared-variable bad-reference problems, which I cleared up for several hours before I was able to produce the first file-in.  The transformation machinery is very strict, and helps you clean up your code.  I’d forgotten about many of the things that I was forced to fix.

I’ll just debug through the file-in until I find the problem.

Shaping.

I’ll make my first run soon on my first package. I’ll do one package extra each time I run, building up the code.  Each package will have its own <package name>Transform method, per the examples.  I’m not sure how to build these methods, except to assume that rewriting the pharo base methods is never wrong.  I’ll file-in the resulting .st file to see what breaks.  Then I’ll go back to the package whose .st source is not loading completely and add additional fixes (class keeps, base methods rewrites if needed and missing, method code body replacements if needed) to its transform method until it loads completely on the next run.  That could take a while.  Is that what you do in practice?

Perfect! That is exactly how I do it.

I start with an empty PackageChange. Then I add incrementally transforms until the code loads - phase one. This has been achieved for PDFtalk with the first release on GitHub. The second and final phase is to make all tests run (including the new to-write tests). This can take a while, because all syntactic and semantic differences must be addressed. Here, some cross-fertilization is possible.

Yeah, and I have to be thorough finally about all my SUnit tests…

The nice thing is, that the system is always telling you what to do. In fact, at first there are so many issues that it is a lot of fun to browse them and chose a nice one. Always the nicest or easiest bug first :-).

Right.

I’m assuming the code writer is taking into account inter-package dependencies in order to get the load order right.

Yes.

Shaping

From: christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com  <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Friday, 5 August, 2022 05:53
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

The full project in which the mentioned method used looks like this:

PDFtalkProject

            ^ProjectChange

                           name: #PDFtalk

                           source: ((OrderedCollection new)

                                           add: (Package name: #Values);

                                           add: (Bundle name: #PDFtalk);

                                           add: (Package name: #'Values Testing');

                                           add: (Bundle name: #'PDFtalk Testing');

                                           add: (Package name: #'PDFtalk Demonstrations');

                                           yourself)

                           changes: self pdftalkPackageChanges

                           nameMapping: (NameMapping

                                           keep: ((OrderedCollection new)

                                                           add: #{Smalltalk.PDF};

                                                           add: #{PostScript.PSDictionary};

                                                           add: #{PDFtalk.PDFObject};

                                                           add: #{PDFtalk.PDFArray};

                                                           add: #{PDFtalk.PDFDictionary};

                                                           add: #{PDFtalk.PDFStream};

                                                           add: #{PDFtalk.PDFString};

                                                           add: #{PDFtalk.PDFDate};

                                                           add: #{PDFtalk.PDFTypeDefinition};

                                                           add: #{PDFtalk.PDFEncoder};

                                                           yourself)

                                           classToNames: ((Valuemap new)

                                                           add: #{SubscriptOutOfBoundsError} -> #Error;

                                                           add: #{NonIntegerIndexError} -> #Error;

                                                           add: #{NotFoundError} -> #KeyNotFound;

                                                           add: #{KeyNotFoundError} -> #KeyNotFound;

                                                           yourself)

                                           namespaceToPrefixes: ((Valuemap new)

                                                           add: #{Smalltalk.PostScript} -> 'PS';

                                                           add: #{Smalltalk.PDFtalk} -> 'Pt';

                                                           add: #{PDFtalk.Fonts} -> 'PtF';

                                                           add: #{PDFtalk.Fonts.OpenType} -> 'PtOT';

                                                           yourself))

In the #source: are the bundles and packages to be transformed. The #changes: (your method) specify the transforms (PackageChange) for each package explicitly. Only packages contain code and therefore, only packages need the code transformations. Bundles are transformed without transformations. (Well, the code for pre-, post- whatever blocks are transformed with the class name mappings rules).

So, the mapping from packages to the corresponding PackageChange has to be stated somehow. Using a dictionary (Valuemap) for this seems also natural. The only change I might like is to use pragmas to tag the PackageChange returning methods with “their” package like

<package: ‘Values Tools’> or so. Putting the package reference into the PackageChange is not a good idea, because all those Objects need to be created before your can find out which package is affected. (Ok, I am creating all PackageChange objects too…).

Happy hacking,

Christian

Von: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Gesendet: Freitag, 5. August 2022 03:26
An: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >; 'Pharo Development List' <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org >
Betreff: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: pdftalkPackageChanges

This method

pdftalkPackageChanges

            ^(Valuemap new)

                            add: 'Values' -> self ValuesTransform;

                            add: 'PostScript' -> self PostScriptTransform;

                            add: 'PDFtalk Basics' -> self PDFtalkBasicsTransform;

                            add: 'PDFtalk Typing' -> self PDFtalkTypingTransform;

                            add: 'PDFtalk Basic Objects' -> self PDFtalkBasicObjectsTransform;

                            add: 'PDFtalk Streams' -> self PDFtalkStreamsTransform;

                            add: 'PDFtalk Data Structures' -> self PDFtalkDataStructuresTransform;

                            add: 'PDFtalk Parsing' -> self PDFtalkParsingTransform;

                            add: 'PDFtalk Colour' -> self PDFtalkColourTransform;

                            add: 'PostScript Fonts' -> self PostScriptFontsTransform;

                            add: 'PostScript CIDInit' -> self PostScriptCIDInitTransform;

                            add: 'PDFtalk Fonts Basics' -> self PDFtalkFontsBasicsTransform;

                            add: 'PDFtalk Fonts Type1' -> self PDFtalkFontsType1Transform;

                            add: 'PDFtalk Fonts OpenType' -> self PDFtalkFontsOpenTypeTransform;

                            add: 'PDFtalk Fonts' -> self PDFtalkFontsTransform;

                            add: 'PDFtalk Graphics' -> self PDFtalkGraphicsTransform;

                            add: 'PDFtalk Graphics Operations' -> self PDFtalkGraphicsOperationsTransform;

                            add: 'PDFtalk XObjects' -> self PDFtalkXObjectsTransform;

                            add: 'PDFtalk Images' -> self PDFtalkImagesTransform;

                            add: 'PDFtalk Files' -> self PDFtalkFilesTransform;

                            add: 'PDFtalk Document' -> self PDFtalkDocumentTransform;

                            add: 'PDFtalk Rendering' -> self PDFtalkRenderingTransform;

                            add: 'PDFtalk Shading' -> self PDFtalkShadingTransform;

                            add: 'PDFtalk Interactive Features' -> self PDFtalkInteractiveFeaturesTransform;

                            add: 'PDFtalk Deploying' -> self PDFtalkDeployingTransform;

                            add: 'Values Testing' -> self ValuesTestingTransform;

                            add: 'PDFtalk test resources' -> self PDFtalkTestResourcesTransform;

                            add: 'PostScript Testing' -> self PostScriptTestingTransform;

                            add: 'PostScript CIDInit Testing' -> self PostScriptCIDInitTestingTransform;

                            add: 'PDFtalk Fonts tests' -> self PDFtalkFontsTestsTransform;

                            add: 'PDFtalk tests' -> self PDFtalkTestsTransform;

                            add: 'PDFtalk Demonstrations' -> self PDFtalkDemonstrationsTransform;

                            yourself

effectively looks like the head transform structure for a project, in this case all the PDFtalk stuff, which includes Values and Postscript.

This is not exactly a bundle idea, is it?  It’s project spread across potentially many bundles and packages.

I’ll start coding my transform with a similar method, and work down toward the details.  Takes a bit to get used to all the correct, yet dangling VW methods that are useless in VW, but which will become new code in the target image and there no longer appear to be dangling (with syntax highlighting aberrations).  Odd looking but completely by design.

Shaping

From: Shaping <shaping@uurda.org mailto:shaping@uurda.org >
Sent: Thursday, 4 August, 2022 19:18
To: 'Any question about pharo is welcome' <pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org >
Subject: [Pharo-users] Re: [Pharo-users]Porting from VW 8.3 to Pharo: Pharo100 fileOutValues

I will look, although this kind of always available on demand thing is too disruptive for me…

I know what you mean.  I try be disciplined about it.  I really like to be able to fix typos, because there are almost always typos.  But e-mail is okay.

So that little example is a test that shows how the transformation is done.  It converts just package Values to a Pharo-compatible file-in.  My task is then to queue a bunch of ProjectChange instances like this one:

SmalltalkTransform.Pharo100>>

ValuesProject

            ^ProjectChange

                            name: #Values

                            source: (Array with: (Package name: #Values))

                            changes: (Valuemap with: 'Values' -> self ValuesTransform)

Exactly

but for my own packages.  No bundles are transformed (just their contained packages) because Pharo doesn’t have bundles.

Is that right?

No, bundles are handled. For real examples, you need to look at the PDFtalk transforms.

Yes, Pharo does not have a concept of bundles (ordered aggregates of packages). Instead it relies on a naming convention for packages. That convention is honored in the fileout, so that packages will be partly grouped in Pharo according to the category prefix.

For each VW-package, one Pharo package is created. A bundle itself is also represented as Pharo package with one class About<bundlename> with class methods for the metadata of the bundle, including a method giving you the ordered list of component packages. So, all contents and metadata of packages and bundles are transformed for Pharo. No code or info gets lost.

Okay.

Is method

ValuesTransform

            ^PackageChange

                            ignoredNames: #(#{Smalltalk.GeneralBindingReference})

                            bridgeClasses: (Valuemap

                                            with: #{Timestamp} -> #DateAndTime

                                            with: #{Smalltalk.ColorValue} -> #Color)

                            localChanges: self valuesLocalTransform

                            extensions: (Array

                                            with: (SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

                                            with: (SystemClassChange

                                                            className: #TextStream

                                                            instanceChanges: (Array with: (Add method: #nextPutAllText: code: #_ph_nextPutAllText:))))

written specifically for that package?  I would think it applies to all packages.  I see some expected mappings like Timestamp to DateAndTime.

Yes, this method returns a PackageChange Value describing the transformations needed to create the Pharo fileout for this specific package (inspect the return value for the fully expanded Value). Methods exist with the same name for other Smalltalks. Depending on the dialect (or version of a dialect), the transforms are different. Squeak and Pharo are quite similar, because they share a common history, but VA or Gemstone need quite different transforms.

So, in general, for each package, there is one such method/Value for each target Smalltalk/version.

Okay.

I do not dare to extract commonalities before the machinery is really robust and stable. For now everything is neatly separate and self-contained (and probably it will stay that way, although there are lots of duplications).

The mapping of class names is the responsibility of the enclosing ProjectChange Value where you define the list of source bundles/packages to transform, the PackageChanges for all packages and the mapping of “global” names.

(The bridge classes above are no renames, but a subclass relationship (is-a) to avoid renamings. The new class Timestamp will be created as subclass of DateAndTime which has almost the same semantics. Therefore, I can still use Timestamp which will be basically a DateAndTime now.)

Okay.

There is still a technical challenge here. Currently, a ProjectChange need to include all prerequisites (Values is part of the PDFtalk project and will be transformed with it). A ProjectWriter, which coordinates the transform, keeps track of the mappings when they are created (either explicitly or through a namespace renaming – see implementers of #PDFtalkProject).

I would like to have this more modular: the mappings from the Values transformation should be persistently saved, so that other transformation projects can just use them, instead of including the sources into one own project.

For this, I need to have renamings local to a package (where they first occur), not global on the project level.

Right.

For Values and Values Tests and Values Tools this works, because there are no mappings in the Values package.

What about conversion of VW arrays to Pharo literal arrays?  How is that done?

(I think you mean dynamic arrays like {1. ‘abc’ size. 42} in which evaluation happens (in contrast to literal things which can be resolved already by the compiler).

Yes, dynamic arrays.

Not! Since a while, VW also has dynamic arrays, but not in VW 8.3 – the last publicly available version.

Okay, I was wondering when that would happen.

I will not shut out those users, because “open-source” would be quite absurd, if it is only available for paying customers.

In 8.3, the compiler does not accept that syntax and therefore, there is no easy way to represent this in replacement code.

So, no. It is not possible until Cincom releases an public version which can handle that.

Okay.

I recall that one of the Smalltalks (I don’t recall which) had Stream semantics differing from VW’s.

… I just checked.  VW’s #upTo: method includes the object and leaves the index after it, and Pharo’s excludes the object and leaves the index at the object.  So that is some major breakage if we don’t correct it.  Can it be done automatically?

Yes, these are the usual porting challenges and exactly the reason why this library exists :). Thank you for the question :).

Yes, the stream semantics need to be fixed. The idea is that a set of transforms for this issue can be reused by others.

Okay.

has lots of juicy bits.  But this doesn’t look very simple.  We can’t just replace an old method with a new one.  We also have to write the new one to tweak how the indices are used in #upTo:,  and make sure that new method gets filed-in as well into the Pharo target image.  Or, we have to do this kind of change manually.

Naa, it’s very easy, I think :).

A PackageChange specifies transforms for classes used in the package (#localChanges) and #extensions for system classes of the target. For a class, you can have a ClassChange describing the changes to instance or class methods. A MethodChange has 4 subclasses for:

•      Ignore – don’t write this method to the target
  

•      Add – add this new method (not in the source system) to the target
  

•      Replace – replace the body of this method with other code
  

•      Rewrite – rewrite the method source using a rewrite rule.
  

Add and Replace need the target code.

Add then always involves a new name for a method in the target.  Replaces use an old name in the target with a new code body.

This is stored in another method with a derived name like #_ph_upTo: . The method name is not important, because only the body of the method is used. But the name should not be used in the source – it is just a holder for the replacement code. These methods live in the specific [<Smalltalk> Fileout <Package>] package.

Okay.

There are lots of working(!) examples for all of those in the PDFtalk transform project.

This bit

(SystemClassChange

                                                            className: #Color

                                                            instanceChanges: (Array with: (Add method: #asColorValue code: #_ph_asColorValue)))

is replacing #asColorValue with #_ph_asColorValue because some special Pharo-color conversion needs to happen.  But how does #_ph_asColorValue get defined?  It’s neither in VW nor in Pharo 10.

You got bitten by the old version of [Pharo Fileout Values]. Please load [Pharo Fileout PDFtalk]. There, the methods exist.

Yes, I see it now.

Ok.  I don’t have a virgin image.  I have a very non-virgin image, about 27 years of development I’m trying to port to Pharo.  I don’t yet have a specific interest in the PDFtalk, though I do see a need for PDF generation later, and will probably revisit that.  For now, I just want my own stuff to run in Pharo.

Virgin image just means that you don’t need anything else. You can safely load it in you favorite special images :).

I would load PDFtalk, although technically you don’t need to (all the extensions to PDFtalk would be unloadable, but that doesn’t affect Values).

Okay.

is:

            Load {Values Project] bundle

            Load {PDFtalk Project} bundle

            Load {Smalltalk Transform Project} bundle

            Load [Pharo Fileout PDFtalk] package

            Save, done

Okay, so do I understand correctly that I need to include the PDFtalk stuff even if I’m not interested in PDFtalk, because that’s where a lot of the Smalltalk transformation machinery lives?  Or is the PDFtalk just being used as an example for how to do a massive transformation?  Or Both?

No, the transformation machinery is fully independent of PDFtalk. I just tried it. The dependencies are in the specific [Pharo Fileout PDFtalk] package, since I have already quite a few replacement methods which are extensions to PDFtalk classes.

Okay.

PDFtalk is the focus of the project and therefore all issues are solved first with this library in mind. Therefore, bundles and namespaces are handled, for example. When you study the more interesting transformations for PDFtalk, it would be a shame not to be able to browse the methods and classes involved.

So, PDFtalk is the real world reference example.

And Values is the simplest example.

            To transform Values do: “Pharo100 fileOutValues”

The [Pharo Fileout PDFtalk] package includes the latest Values transformations.

I am thinking about a better modularization…

Also, the wiki is a bit out of control. It really needs some restructuring.

In the cites wiki page, there is a link to a blog where I record the changes. This might be informative.

2. Port the Values package. This is easy, since no namespaces are involved.

This first instruction after VW package setup says to port the contents of the Values package from VW to Pharo.  Do you mean manually?  Probably not.

No, no. This has been finished in March.

For each dialect, I have a GitHub repository where I release important versions:  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk. You can find the working port as first release “ https://github.com/PortingPDFtalk/PharoPDFtalk/releases/tag/1.3.0.0 Working version“. There you can download the ported Values fileout with the exact description with which versions of what it was created.

This should be a good starting example.

Why do I need any new code installed in Pharo before I begin the transformation, if I’m transforming code from VW to pharo?  I’m not understanding the basic constraints of the problem, even when the detailed steps are clear.

No, no. You don’t need anything on the Pharo side. The fileouts on GitHub are the end products of a transformation for people who don’t use VisualWorks, but want to use Values in Pharo. Or help with PDFtalk by fixing some issues, so that I can write the transformations.

I’ve done these steps so far:

1. Went to  https://github.com/PortingPDFtalk/PharoPDFtalk https://github.com/PortingPDFtalk/PharoPDFtalk.

Mistake :).
You don’t want to look at the unfinished product of the current version of thincomplete transformations for PDFtalk.

Do you mean “finished” here? Isn’t that file-in the finished result?

I thought the above links was the currently finished result (as good as it can be until the rest of the bugs are goine and tests all run).

Yes, that was confusing.  That’s why I had the early impression that Values was somehow apart of the transformation machinery.

See https://wiki.pdftalk.de/doku.php?id=stateoftheport#pharo-10-0:

Instead, you want to look at the unfinished product of the transformations of your projects :).

2. Saved down PDFtalk.Pharo100.st into <my Pharo 10 image directory>/pharo-local/Smalltalk-Transformation. (I figured that was a good place to save it.  If anyone disagrees, or has a better or more conventional idea about where files should be saved, please say so.  I setup a Pharo Git repo and played with it briefly for the first time yesterday.  I’ve used Pharo off an on for 16 years, but this is the first time I’m making a serious effort to manage source, and not throw away what I’m working on.)

3. Filed-in PDFtalk.Pharo100.st.  This went on for about 7 minutes or so.  I have a hundreds if not over a thousand classes showing in Epicea.  Is there anyway to get Epicea to give me a count of changes with a time-range filter?

4. Deleted (forgot) yesterday’s, old Main practice-repo from both the image and the drive, and made a new one.  I need to add to Main all the packages I just filed-in, but I don’t see an efficient way to do that.  I would like to use the Add Package button, but this gives a filtered list of available packages.  I can filter subgroups, and then individually select each of the checkboxes to the left of each package (there is no Ctrl-A [select all] option here, which seems to be a strange omission given the potentially large number of packages involved).  I see lots of prefixes for the classes just loaded.  I could easily miss something if I filter/select/add one prefix-group at a time.  Is there an easier way?  Over in Epicea I don’t see a way to push the loaded items listed to a specific repo.  The first thought I have is to select all filed-in code artifacts by datetime span.  I did that and saved it as an .ombu file (I have no idea what that is).  I don’t see a way to import that .ombu file into repo Main’s “Working copy” window.  It must be easy, but I don’t see it.  Please suggest the best way.

I’d like to know as well – I am not quite familiar with the source management concepts in Pharo.

I asked in Discord.  I don’t understand why stuff like this is missing.  The only conclusion I can draw is that no one does huge file-ins (but you do).

It is planned to generate Tonel output in the future, but for now I feel safer with the traditional fileout where I can have doIts. I am not sure how Tonel

reacts to crippled sources, which are normal during the development of the transformations.

Shaping

From: christian.haider <christian.haider@smalltalked-visuals.com mailto:christian.haider@smalltalked-visuals.com >
Sent: Wednesday, 22 June, 2022 05:42
To: pharo-users@lists.pharo.org mailto:pharo-users@lists.pharo.org
Subject: [Pharo-users] [PDFtalk] second fileOut for Squeak and Pharo

With help from the community some issues were fixed which improved the test statistics nicely.
Check it out: https://wiki.pdftalk.de/doku.php?id=portingblog#second_pdftalk_fileout_for_squeak_and_pharo

Thanks to everybody involved!

Happy hacking,
Christian