[Pharo-project] detect if image was saved with Cog
Hi guys Is there a way to find out (from outside of the image), if an image has been saved with Cog? What I would like to do is something like this: $ head -20 myImage.image | grep <some expression> The use case for this is seaside hosting, where we only have squeak VMs to run images, so when an image is uploaded that was saved with Cog that image cause trouble. Any ideas? Cheers, Max
On 1 October 2012 11:09, Max Leske <maxleske@gmail.com> wrote:
Hi guys
Is there a way to find out (from outside of the image), if an image has been saved with Cog? What I would like to do is something like this:
$ head -20 myImage.image | grep <some expression>
The use case for this is seaside hosting, where we only have squeak VMs to run images, so when an image is uploaded that was saved with Cog that image cause trouble. Any ideas?
Cheers, Max
get new queak vms, they now should support new image fromat used by cog -- Best regards, Igor Stasenko.
I believe they have a custom VM with restricted primitives, Still, his question is relevant: how to interpret the image header ? I know there is a header with useful things like the initial extent. On 01 Oct 2012, at 12:15, Igor Stasenko <siguctua@gmail.com> wrote:
On 1 October 2012 11:09, Max Leske <maxleske@gmail.com> wrote:
Hi guys
Is there a way to find out (from outside of the image), if an image has been saved with Cog? What I would like to do is something like this:
$ head -20 myImage.image | grep <some expression>
The use case for this is seaside hosting, where we only have squeak VMs to run images, so when an image is uploaded that was saved with Cog that image cause trouble. Any ideas?
Cheers, Max
get new queak vms, they now should support new image fromat used by cog
-- Best regards, Igor Stasenko.
On Mon, 1 Oct 2012, Max Leske wrote:
Hi guys
Is there a way to find out (from outside of the image), if an image has been saved with Cog? What I would like to do is something like this:
$ head -20 myImage.image | grep <some expression>
The use case for this is seaside hosting, where we only have squeak VMs to run images, so when an image is uploaded that was saved with Cog that image cause trouble. Any ideas?
Currently there are 3 image formats around 6502 (old interpreter format, no closures), 6504 (new interpreter format with closures), 6505 (Cog format). The first two bytes of the image file contain the image format in little-endian (IIRC) format. So you can decide it by checking the first byte (head -c 1). If you only support the old interpreter format, then the following bash expression should work: [ $(head -c 1 $IMAGE_NAME) == 'f' ] Levente
Cheers, Max
On 01 Oct 2012, at 13:30, Levente Uzonyi <leves@elte.hu> wrote:
Currently there are 3 image formats around 6502 (old interpreter format, no closures), 6504 (new interpreter format with closures), 6505 (Cog format). The first two bytes of the image file contain the image format in little-endian (IIRC) format. So you can decide it by checking the first byte (head -c 1). If you only support the old interpreter format, then the following bash expression should work:
[ $(head -c 1 $IMAGE_NAME) == 'f' ]
SmalltalkImage current imagePath asFileReference readStreamDo: [ :stream | stream binary. (stream next: 2) reversed asInteger ]. to express your description in Smalltalk code then. Thanks for the explanation. -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 01.10.2012 13:30, Levente Uzonyi wrote:
On Mon, 1 Oct 2012, Max Leske wrote:
Hi guys
Is there a way to find out (from outside of the image), if an image has been saved with Cog? What I would like to do is something like this:
$ head -20 myImage.image | grep <some expression>
The use case for this is seaside hosting, where we only have squeak VMs to run images, so when an image is uploaded that was saved with Cog that image cause trouble. Any ideas?
Currently there are 3 image formats around 6502 (old interpreter format, no closures), 6504 (new interpreter format with closures), 6505 (Cog format). The first two bytes of the image file contain the image format in little-endian (IIRC) format. Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days.
There is no symmetry in the format which allows you to bit-test endianness in a platform independent manner, afaik, the only "rule" is that format numbers which read as an existing format in the opposite endianness is disallowed. Cheers, Henry
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days.
How can an image be cross-platform, but the header not ? -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
On 1 October 2012 14:43, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days.
How can an image be cross-platform, but the header not ?
Just another artifact of incremental development, i guess. I don't know what reasoning was behind this, but IMO, defining header to have fixed byte order would be much better. Of course, it is always easier to think backwards. However, things like IP header, were existed before squeak image format. So, again, i am clueless why guys decided that doing like it's done is a Good Idea (tm). -- Best regards, Igor Stasenko.
On 01.10.2012 15:44, Igor Stasenko wrote:
On 1 October 2012 14:43, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days. How can an image be cross-platform, but the header not ?
Just another artifact of incremental development, i guess. I don't know what reasoning was behind this, but IMO, defining header to have fixed byte order would be much better. Of course, it is always easier to think backwards. However, things like IP header, were existed before squeak image format. So, again, i am clueless why guys decided that doing like it's done is a Good Idea (tm).
IIRC, the code that writes the image header is in Slang. Using uints for fields there would be a logical choice, I would guess. TBH, I don't think it would be much better if a byte order were enforced for a single header field only. Though, I personally would think it'd be nice if the format field were to be expanded and kept symmetrical. That way, deciphering the image formats numerical value could be done independently of endianness (and knowing what #'s are valid formats). As nice as backwards compatability though, considering there would be no breaking image format changes per se? Probably not. Cheers, Henry
On Mon, Oct 01, 2012 at 05:07:48PM +0200, Henrik Sperre Johansen wrote:
On 01.10.2012 15:44, Igor Stasenko wrote:
On 1 October 2012 14:43, Sven Van Caekenberghe <sven@stfx.eu> wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days. How can an image be cross-platform, but the header not ?
Just another artifact of incremental development, i guess. I don't know what reasoning was behind this, but IMO, defining header to have fixed byte order would be much better. Of course, it is always easier to think backwards. However, things like IP header, were existed before squeak image format. So, again, i am clueless why guys decided that doing like it's done is a Good Idea (tm).
IIRC, the code that writes the image header is in Slang. Using uints for fields there would be a logical choice, I would guess.
Yes, it is done in #checkImageVersionFrom:startingAt:
TBH, I don't think it would be much better if a byte order were enforced for a single header field only.
Though, I personally would think it'd be nice if the format field were to be expanded and kept symmetrical. That way, deciphering the image formats numerical value could be done independently of endianness (and knowing what #'s are valid formats).
As nice as backwards compatability though, considering there would be no breaking image format changes per se? Probably not.
It would be straightforward to design a new object file header, so long as the VM retains the ability to read the old one. The resulting image could not be read by older VMs but otherwise there would be no problem. Dave
On 01.10.2012 14:43, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days. How can an image be cross-platform, but the header not ?
-- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill
IIRC, because the VM checks which endianness the header is written in (by comparing to known image formats in big and little-endian format), and then triggering a swap of all pointers/variableWord objects when loading the image on a differing platform. Cheers, Henry
On 01 Oct 2012, at 16:33, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
On 01.10.2012 14:43, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days. How can an image be cross-platform, but the header not ?
IIRC, because the VM checks which endianness the header is written in (by comparing to known image formats in big and little-endian format), and then triggering a swap of all pointers/variableWord objects when loading the image on a differing platform.
Thanks for the answer, I suspected something like that. But that means that the image format on disk, being binary, is native only to one endianess and not the other, making startup or saving a image faster/slower depending on whether the conversion needs to be done, right ? Or is this automagically fixed with the first save ?
On 01.10.2012 16:56, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 16:33, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
On 01.10.2012 14:43, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days. How can an image be cross-platform, but the header not ?
IIRC, because the VM checks which endianness the header is written in (by comparing to known image formats in big and little-endian format), and then triggering a swap of all pointers/variableWord objects when loading the image on a differing platform. Thanks for the answer, I suspected something like that.
But that means that the image format on disk, being binary, is native only to one endianess and not the other, Yes making startup or saving a image faster/slower depending on whether the conversion needs to be done, right ? Or is this automagically fixed with the first save ?
Swapping happens at startup, so penalty is only when loading a big-endian image on little-endian platform or vice versa. Without saving, that happens every time, it is not automatically saved if a conversion takes place when loading. When saving, it saves in the platform endianness (which is the same as is in memory), and incur no extra penalty. Incidentally, that's the same approach I suggested Mariano use in fuel. IMHO, it really is most optimal approach when platform endianness changes are rare, and the penalty can easily be avoided by simply saving the image on the platform it is intended to be used. Cheers, Henry
On 01.10.2012 17:08, Henrik Sperre Johansen wrote:
IMHO, it really is most optimal approach when platform endianness changes are rare, and the penalty can easily be avoided by simply saving the image on the platform it is intended to be used.
Cheers, Henry
In other words; it's not really something you usually need to think about unless you are developing closed applications shipping with a fixed image. Obviously, one such case is phone applications, John mentioned its impact on iPhone apps here; http://forum.world.st/Where-is-the-heap-tp2195336p2195639.html Interestingly, (at least the way I read it) the real cost was memory access, and endianness swap was really just one of multiple items that fell under the category "Accesses the entire object memory at startup". Cheers, Henry
OK, thanks guys, I learrned new things today. -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill On 01 Oct 2012, at 17:08, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
On 01.10.2012 16:56, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 16:33, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
On 01.10.2012 14:43, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days. How can an image be cross-platform, but the header not ?
IIRC, because the VM checks which endianness the header is written in (by comparing to known image formats in big and little-endian format), and then triggering a swap of all pointers/variableWord objects when loading the image on a differing platform. Thanks for the answer, I suspected something like that.
But that means that the image format on disk, being binary, is native only to one endianess and not the other, Yes making startup or saving a image faster/slower depending on whether the conversion needs to be done, right ? Or is this automagically fixed with the first save ?
Swapping happens at startup, so penalty is only when loading a big-endian image on little-endian platform or vice versa. Without saving, that happens every time, it is not automatically saved if a conversion takes place when loading.
When saving, it saves in the platform endianness (which is the same as is in memory), and incur no extra penalty.
Incidentally, that's the same approach I suggested Mariano use in fuel. IMHO, it really is most optimal approach when platform endianness changes are rare, and the penalty can easily be avoided by simply saving the image on the platform it is intended to be used.
Cheers, Henry
On Mon, Oct 01, 2012 at 04:56:39PM +0200, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 16:33, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
On 01.10.2012 14:43, Sven Van Caekenberghe wrote:
On 01 Oct 2012, at 14:01, Henrik Sperre Johansen <henrik.s.johansen@veloxit.no> wrote:
Endianness in header depends on the platform the image was saved on. Which for all practical purposes, means little-endian these days. How can an image be cross-platform, but the header not ?
IIRC, because the VM checks which endianness the header is written in (by comparing to known image formats in big and little-endian format), and then triggering a swap of all pointers/variableWord objects when loading the image on a differing platform.
Thanks for the answer, I suspected something like that.
But that means that the image format on disk, being binary, is native only to one endianess and not the other, making startup or saving a image faster/slower depending on whether the conversion needs to be done, right ? Or is this automagically fixed with the first save ?
Normally the conversion does not need to be done, because the common use case is for starting an image on the same (or similar) platform. That means that in the common case, object memory can be directly loaded without word swapping, which is a bit performance win when loading the image. Of course the header part of the file is different from the object memory itself, so Igor is correct: The real reason it got done that way is ... that's how it got done ;) Dave
On Mon, Oct 01, 2012 at 11:09:51AM +0200, Max Leske wrote:
Hi guys
Is there a way to find out (from outside of the image), if an image has been saved with Cog? What I would like to do is something like this:
$ head -20 myImage.image | grep <some expression>
The use case for this is seaside hosting, where we only have squeak VMs to run images, so when an image is uploaded that was saved with Cog that image cause trouble. Any ideas?
Cheers, Max
You should definitely update those older VMs if you can (http://squeakvm.org), but if that is not possible, here is how to check the image format from a shell script. 1) Load package ImageFormat from the VMMaker repository at http://source.squeak.org/VMMaker. 2) Evaluate "ImageFormat createCkStatusProgram" to generate source code ckformat.c. 3) Compile ckformat ($ cc -o ckformat ckformat.c). The ckformat program reads an image file header and prints the image format number on standard output. Non-zero exit status means the format could not be read (e.g. if it was not an image file). You can see how it is used in the /usr/local/bin/squeak script in the latest unix interpreter VMs from squeakvm.org. In your script you can do something like this: imagenumber=`ckformat myimage.image` if test $? -eq 0 then echo the image number is $imagenumber else echo could not read the image number fi If you are interested in the meanings of the various image format numbers, browse class ImageFormat. Dave
participants (6)
-
David T. Lewis -
Henrik Sperre Johansen -
Igor Stasenko -
Levente Uzonyi -
Max Leske -
Sven Van Caekenberghe