On Sun, Apr 16, 2017 at 11:21:48PM +0800, Ben Coman wrote:

On Sun, Apr 16, 2017 at 10:50 PM, Alistair Grant <akgrant0710@gmail.com> wrote:

...

Do you know how the linux zero conf scripts are / will be built?

My assumption has been that they are part of the image build.

I'm not familiar with the zero-conf implementation. You might learn something here... https://github.com/pharo-project/pharo-zeroconf

Thanks. This looks like it provides the download scripts and CONFIG for the threaded vm, but not the pharo and pharo-ui scripts that actually launch Pharo. I'll have to keep searching... :-) Thanks again, Alistair

I have the VM building under nix from the master branch now. The build is based loosely on the linux.64x64 mvm script so it's building a 64-bit x86-64 VM. The pharo-vm commit I am testing is 1c38b03f (from wednesday.) I have tried opening both Pharo-60465.image and Pharo-50771.image and both give this error: This interpreter (vers. 68021) cannot read image file (vers. 6521). How come? Is this because I built a 64-bit VM? If I want to use those images then should I build a 32-bit image instead? (Or none of the above?) Then I found Pharo64-60465.image and this does open and display a window but it's frozen and prints these messages: pthread_setschedparam failed: Operation not permitted This VM uses a separate heartbeat thread to update its internal clock and handle events. For best operation, this thread should run at a higher priority, however the VM was unable to change the priority. The effect is that heavily loaded systems may experience some latency issues. If this occurs, please create the appropriate configuration file in /etc/security/limits.d/ as shown below: cat <<END | sudo tee /etc/security/limits.d/squeak.conf * hard rtprio 2 * soft rtprio 2 END and report to the squeak mailing list whether this improves behaviour. You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux 'Your VM is too old for this image. Please download the latest VM.' Error: Can't find the requested origin UnixResolver(PlatformResolver)>>cantFindOriginError [ self cantFindOriginError ] in UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed: in Block: [ self cantFindOriginError ] [ ^ aBlock value ] in UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed:or: in Block: [ ^ aBlock value ] BlockClosure>>cull: Context>>evaluateSignal: Context>>handleSignal: Error(Exception)>>signal Error(Exception)>>signal: ExternalLibraryFunction(Object)>>error: ExternalLibraryFunction(Object)>>externalCallFailed ExternalLibraryFunction(ExternalFunction)>>invokeWithArguments: UnixEnvironment(OSEnvironment)>>getEnv: UnixEnvironment(OSEnvironment)>>at:ifAbsent: [ Smalltalk os environment at: aString ifAbsent: [ nil ] ] in UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed:or: in Block: [ Smalltalk os environment at: aString ifAbse\ nt: [...etc... BlockClosure>>on:do: UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed:or: UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed: UnixResolver>>home [ self home / '.config' ] in UnixResolver>>preferences in Block: [ self home / '.config' ] [ ^ aBlock value ] in UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed:or: in Block: [ ^ aBlock value ] BlockClosure>>cull: Context>>evaluateSignal: Context>>handleSignal: Error(Exception)>>signal Error(Exception)>>signal: ExternalLibraryFunction(Object)>>error: ExternalLibraryFunction(Object)>>externalCallFailed ExternalLibraryFunction(ExternalFunction)>>invokeWithArguments: UnixEnvironment(OSEnvironment)>>getEnv: FFICalloutAPI>>function:module: Googling around it sounds like this "origin error" is caused by not finding the Sources file but I have provided the PharoV50.sources and strace suggests that it is opened okay: open("/nix/store/w85ynpp3zm33k8h1hvzh2g7hf7ggwx1k-pharo-share-1.0/lib/PharoV50.sources", O_RDONLY) = 3 open("/nix/store/w85ynpp3zm33k8h1hvzh2g7hf7ggwx1k-pharo-share-1.0/lib/PharoV50.sources", O_RDONLY) = 6 So maybe the problem is that I need a different source file? Or maybe that I should switch to a 32-bit VM? Or something else entirely...? Any help appreciated :) -Luke

On Sun, Apr 16, 2017 at 11:55:27PM +0200, Luke Gorrie wrote:

I have the VM building under nix from the master branch now. The build is based loosely on the linux.64x64 mvm script so it's building a 64-bit x86-64 VM. The pharo-vm commit I am testing is 1c38b03f (from wednesday.)

I have tried opening both Pharo-60465.image and Pharo-50771.image and both give this error:

This interpreter (vers. 68021) cannot read image file (vers. 6521).

How come? Is this because I built a 64-bit VM? If I want to use those images then should I build a 32-bit image instead? (Or none of the above?)

Yes, 32 bit VM will only work with 32 bit images, and 64 bit VM with 64 bit images.

Then I found Pharo64-60465.image and this does open and display a window but it's frozen and prints these messages:

pthread_setschedparam failed: Operation not permitted This VM uses a separate heartbeat thread to update its internal clock and handle events. For best operation, this thread should run at a higher priority, however the VM was unable to change the priority. The effect is that heavily loaded systems may experience some latency issues. If this occurs, please create the appropriate configuration file in /etc/security/limits.d/ as shown below:

cat <<END | sudo tee /etc/security/limits.d/squeak.conf * hard rtprio 2 * soft rtprio 2 END

and report to the squeak mailing list whether this improves behaviour.

You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux

As this says, for best operation you should enable higher priority threads. Have you tried creating the file as described? That said, the message is only a warning, the system should be basically working.

'Your VM is too old for this image. Please download the latest VM.' Error: Can't find the requested origin

UnixResolver(PlatformResolver)>>cantFindOriginError [ self cantFindOriginError ] in UnixResolver(PlatformResolver)>> directoryFromEnvVariableNamed: in Block: [ self cantFindOriginError ] [ ^ aBlock value ] in UnixResolver(PlatformResolver)>> directoryFromEnvVariableNamed:or: in Block: [ ^ aBlock value ] BlockClosure>>cull: Context>>evaluateSignal: Context>>handleSignal: Error(Exception)>>signal Error(Exception)>>signal: ExternalLibraryFunction(Object)>>error: ExternalLibraryFunction(Object)>>externalCallFailed ExternalLibraryFunction(ExternalFunction)>>invokeWithArguments: UnixEnvironment(OSEnvironment)>>getEnv: UnixEnvironment(OSEnvironment)>>at:ifAbsent: [ Smalltalk os environment at: aString ifAbsent: [ nil ] ] in UnixResolver (PlatformResolver)>>directoryFromEnvVariableNamed:or: in Block: [ Smalltalk os environment at: aString ifAbse\ nt: [...etc... BlockClosure>>on:do: UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed:or: UnixResolver(PlatformResolver)>>directoryFromEnvVariableNamed: UnixResolver>>home [ self home / '.config' ] in UnixResolver>>preferences in Block: [ self home / '.config' ] [ ^ aBlock value ] in UnixResolver(PlatformResolver)>> directoryFromEnvVariableNamed:or: in Block: [ ^ aBlock value ] BlockClosure>>cull: Context>>evaluateSignal: Context>>handleSignal: Error(Exception)>>signal Error(Exception)>>signal: ExternalLibraryFunction(Object)>>error: ExternalLibraryFunction(Object)>>externalCallFailed ExternalLibraryFunction(ExternalFunction)>>invokeWithArguments: UnixEnvironment(OSEnvironment)>>getEnv: FFICalloutAPI>>function:module:

Googling around it sounds like this "origin error" is caused by not finding the Sources file but I have provided the PharoV50.sources and strace suggests that it is opened okay:

open("/nix/store/w85ynpp3zm33k8h1hvzh2g7hf7ggwx1k-pharo-share-1.0/lib/ PharoV50.sources", O_RDONLY) = 3 open("/nix/store/w85ynpp3zm33k8h1hvzh2g7hf7ggwx1k-pharo-share-1.0/lib/ PharoV50.sources", O_RDONLY) = 6

If I'm reading the stack correctly, it looks like your HOME environment variable isn't set (or the VM isn't able to access it).

So maybe the problem is that I need a different source file? Or maybe that I should switch to a 32-bit VM? Or something else entirely...?

The 32 bit VM is more stable than the 64 bit one at the moment. If you are still getting the build system stabilised, it might be easier to stick with 32 bits for now. Switching to 64 bits later should be straightforward (certainly easier than switching from building Pharo 5 to Pharo 6). Cheers, Alistair

On Mon, Apr 17, 2017 at 09:34:53AM +0200, Luke Gorrie wrote:

On 17 April 2017 at 07:50, Alistair Grant <akgrant0710@gmail.com> wrote:

Yes, 32 bit VM will only work with 32 bit images, and 64 bit VM with 64 bit images.

I wonder if some DWIM would make sense here. For example, my bash wrapper that starts the VM could inspect the image file and decide which VM is appropriate (spur, non-spur, 32-it, 64-bit, etc.) Otherwise the user needs to guess and decipher cryptic error messages.

That would be great. It doesn't look like the images have a human readable magic number at the start. Hopefully someone can give us a hint.

Has anybody made such a "DWIM" script that automatically selects the right VM? Would love a link to use for reference.

Is there any compatibility matrix to help me understand which VMs are compatible with which images? Now I know that the 64-bit VM can't run 32-bit images but I am wondering whether e.g. the Spur VM can run older images like Pharo 4 / DrGeo / etc or whether different VMs need to be built for those too.

As this says, for best operation you should enable higher priority threads. Have you tried creating the file as described?

I will look for a build option to disable this feature that wants to set realtime priority.

This isn't really a feature, it's a core part of the VM's operation. You could choose to build the itimer VM, but that has its own problems. See: https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux http://forum.world.st/Iceberg-with-SSH-on-Linux-td4940436.html http://forum.world.st/Unix-heartbeat-thread-vs-itimer-td4928943.html#a493845... Cheers, Alistair

I don't think it's reasonable to expect end-users to edit /etc files in order to run GUI applications. I also don't like applications blindly printing instructions that are distro-specific (there is no /etc/security directory on NixOS for example.) I also :-) don't immediately see why it is safe to assume that running pharo at realtime scheduler priority is what the user will want when their system becomes overloaded.

I already know that another problem coming down the pipeline, after making the VM work, is finding hard-coded paths in popular image files that only work on some distributions. For example the Moose image makes some unsafe assumptions about what libcairo will be called, https://github.com/NixOS/nixpkgs/pull/14414

Grumble, grumble, grumble :). But soon all fixed!

If I'm reading the stack correctly, it looks like your HOME environment variable isn't set (or the VM isn't able to access it).

Odd. That should not be the case. I will dig deeper.

> So maybe the problem is that I need a different source file? Or maybe that I > should switch to a 32-bit VM? Or something else entirely...?

The 32 bit VM is more stable than the 64 bit one at the moment. If you are still getting the build system stabilised, it might be easier to stick with 32 bits for now. Switching to 64 bits later should be straightforward (certainly easier than switching from building Pharo 5 to Pharo 6).

Thanks for the tip!

I will try to build the 32-bit VM now.

On 17 April 2017 at 11:01, Luke Gorrie <luke@snabb.co> wrote:

I have only read through once, but isn't it a no-brainer

^^^ please forgive gratuitously combative tone :) Fighting the VM build in another window. Just now I am able to build the 32-bit VM from master and run both the Pharo 5 and Pharo 6 images. The next problem is the issue of accessing $HOME. The actual error is ExternalLibraryFunction>>invokeWithArguments: failing to call getenv() from libc. Simplest way that I can reproduce it here is Smalltalk os environment at: 'HOME'. Checking for what I can have mucked up in the build... Otherwise I need to supply the "smudged" version information in some reasonable way. I don't have access to the .git folder during build so I can't use the perl script directly. I see that I can't just skip these values either because on startup I am also seeing a warning that my "VM is too old" coming from DiskStore class>>checkVMVersion that wants to see that my VM source is from at least July 2012 :).

On 17 April 2017 at 11:59, Alistair Grant <akgrant0710@gmail.com> wrote:

What about the genVersion.sh script I provided earlier in this thread?

The genVersion.sh script has a dependency under the hood that I can't satisfy: it expects to be able to run git commands but I can't do that in the nix environment. The root problem is that nix requires a perfectly reproducible build environment, with the sha256 hash of every dependency declared, and that is hard to do for the .git folder. The .git is really a database that contains many source code versions and not just the one that I have checked out. Which perl script (I'm still learning about the build process), and why

would you need access to git blobs directly?

Check out opensmalltalk-vm/.git_filters/RevDateURL.smudge. This is a perl script that is accessing environmental state that's off-limits in the nix universe e.g. whoami, hostname, date, git, etc. Could be that I should write the version info by hand and apply it as a patch. I have to make my build produce the same result every time it runs. That way it will always work the same, and somebody in 5 years who wants to run an _ancient_ Pharo 6 image will be able to do that with my build script because it's a fixed point.

What does "Smalltalk vm interpreterSourceDate" actually return?

actually it gets an exception because I don't have well-formed values from the strings it is trying to parse.

On Mon, Apr 17, 2017 at 12:47:38PM +0200, Luke Gorrie wrote:

On 17 April 2017 at 11:33, Luke Gorrie <luke@snabb.co> wrote:

Fighting the VM build in another window. Just now I am able to build the 32-bit VM from master and run both the Pharo 5 and Pharo 6 images. The next problem is the issue of accessing $HOME. The actual error is ExternalLibraryFunction>>invokeWithArguments: failing to call getenv() from libc. Simplest way that I can reproduce it here is Smalltalk os environment at: 'HOME'.

I have resolved this now.

The problem was the VM primitive ioLoadModule() not being able to find libc. It has some heuristics for where to look but they don't work on NixOS. I added the relevant folder into $LD_LIBRARY_PATH in my wrapper script and now it is happy.

There will be a lot of this in my future... finding hard-coded paths for things like shared libraries and un-hard-coding them. I am not sure what to do in cases where the paths are in Smalltalk code inside an image, that's harder to patch than the VM sources or ELF headers etc.

I've hit this as well building the snap package, e.g. CairoLibrary>>unixModuleName. I modified the method directly, but probably it would be better to create a UnixPlatform>>findLibrary: method to look after it centrally. Cheers, Alistair

Hi Luke, On Mon, Apr 17, 2017 at 9:50 PM, Luke Gorrie <luke@snabb.co> wrote:

Success!

I'm now running the Pharo 5.0, Pharo 6.0, and Moose 6.1 images without a hitch.

Some hero has even fixed the libcairo name lookup issue in the Moose image. Now it's searching LD_LIBRARY_PATH and so I have added every shared library there in the wrapper script.

I decided to manually write the SCCS version information directly in the package definition and update that with sed. This is the part that I need to maintain manually now:

pharo-spur = pharo-vm-build rec { name = "pharo-vm-${version}"; version = "git.${revision}"; binary-basename = "pharo-vm"; src = fetchFromGitHub { owner = "pharo-project"; repo = "pharo-vm"; rev = revision; sha256 = "114ydx015i8sg4xwy1hkb7vwkcxl93h5vqz4bry46x2gnvsay7yh"; }; # metadata compiled into the image: revision = "1c38b03fb043a2962f30f080db5b1292b5b7badb"; sourceDate = "Wed Apr 12 19:25:16 2017 +0200"; sourceURL = "https://github.com/pharo-project/pharo-vm"; };

I will close my PR on the pharo-vm repo too because I ended up packaging the upstream source as-is without depending on any of those changes.

Have to think about what's left... tidy up, test with the pharo-launcher image, test on OSX, push to nix upstream... something like that.

Thanks for all the help !!

Cheers, -Luke

Great to hear of your success. I was just in the middle of composing the following when your post popped up for me. Some of the points you raise actually would have been best discussed on the vm-dev [1] mail list. I'm going to post some follow up questions there and you may want to listen in. Also, since much of the vm level stuff is shared with Squeak, it would be good for them to benefit from knowing of your solution. Could you post a summary at vm-dev? [1] http://lists.squeakfoundation.org/mailman/listinfo/vm-dev cheers -ben

On Mon, Apr 17, 2017 at 02:07:40PM +0530, K K Subbu wrote:

On Monday 17 April 2017 01:04 PM, Luke Gorrie wrote:

...

I wonder if some DWIM would make sense here. For example, my bash wrapper that starts the VM could inspect the image file and decide which VM is appropriate (spur, non-spur, 32-it, 64-bit, etc.) Otherwise the user needs to guess and decipher cryptic error messages.

You can use the attached magic file with file(1) command:

file -m magic <image files> ...

to detect the image type. or register the patterns with binfmt_misc to match the vm to the image type automatically.

Thanks! This should make it straightforward to update the startup scripts. Cheers, Alistair

On Mon, Apr 17, 2017 at 09:34:53AM +0200, Luke Gorrie wrote:

On 17 April 2017 at 07:50, Alistair Grant <akgrant0710@gmail.com> wrote:

...

Yes, 32 bit VM will only work with 32 bit images, and 64 bit VM with 64 bit images.

I wonder if some DWIM would make sense here. For example, my bash wrapper that starts the VM could inspect the image file and decide which VM is appropriate (spur, non-spur, 32-it, 64-bit, etc.) Otherwise the user needs to guess and decipher cryptic error messages.

Has anybody made such a "DWIM" script that automatically selects the right VM? Would love a link to use for reference.

http://lists.squeakfoundation.org/pipermail/vm-dev/2017-April/024751.html Dave