If this is all you are doing then yes, that is correct. If you expect the format to be different in the new subclass you will have to recalculate it, but since you aren't doing that in this example, this is fine. cheers, Toon On 03/21/2011 11:29 AM, Tudor Girba wrote:

Thanks, Toon!

I changed the code to explicitly set the format, and it seems to fix the problem:

cls := Class new superclass: MooseElement; setFormat: MooseElement format; yourself. cls compileSilently: 'mooseName ^ 1/0'. element := cls new.

Is this correct?

Cheers, Doru

On 21 Mar 2011, at 10:57, Toon Verwaest wrote:

...

This does crash whenever you subclass a class which has instance variables and you try to access those instance variables. The problem is that you don't properly initialize your class, leaving you with a Class that has a wrong format. For example:

cls := Class new superclass: Class; yourself. cls format

returns 2. 2 basically means it's an object with pointers but with 0 instance variables. If you instantiate the 'cls' I just made it also crashes. Why? Well, class has an initialize method that is compiled to write to the fields of the new instance. It puts an empty method dictionary into the class you create as an instance of my cls. This segfaults because you are writing outside of memory.

So just make sure you properly create classes, with a proper format!

This test should crash all VMs btw... at least at some point. Since you are writing in random memory it might take longer to notice it in some cases; especially when padded memory is owned by the garbage collector :)

cheers, Toon

On 03/21/2011 10:24 AM, Tudor Girba wrote:

...

Hi,

Alex recently wrote a test in Moose that seems to crash the Cog VM at least on Mac.

How to reproduce:

- download the following image http://dl.dropbox.com/u/18323746/Tmp/moose-crashing-cog-jit.zip

- execute the following code in the workspace (already provided in the image) cls := Class new superclass: MooseElement; yourself. cls compileSilently: 'mooseName ^ 1/0'. element := cls new.

- I used all of the followings and they all crashed: https://pharo-ic.lille.inria.fr/hudson/view/Cog/job/Cog%20Mac%20Cocoa/4/arti... https://pharo-ic.lille.inria.fr/hudson/view/Cog/job/StackVM%20Mac%20Cocoa/2/... http://www.mirandabanda.org/files/Cog/VM/VM.r2370/Cog.app.tgz http://www.mirandabanda.org/files/Cog/VM/VM.r2361/Cog.app.tgz

The strange thing is that the crash only happens when we subclass MooseElement, but not another class.

Could someone take a look?

Cheers, Doru

-- www.tudorgirba.com

"Every thing should have the right to be different."

-- www.tudorgirba.com

"Every thing has its own flow."

I couldn't agree more! However, here the problem is not that the format isn't a correct format. The problem is that a bytecode is activated that is incompatible with the instance format. So basically on every bytecode accessing an instance you would have to check if this is a valid access. And as we all know, Smalltalk avoids this; it makes things faster :) Toon On 03/21/2011 12:49 PM, Igor Stasenko wrote:

On 21 March 2011 11:29, Tudor Girba<tudor.girba@gmail.com> wrote:

...

Thanks, Toon!

I changed the code to explicitly set the format, and it seems to fix the problem:

cls := Class new superclass: MooseElement; setFormat: MooseElement format; yourself. cls compileSilently: 'mooseName ^ 1/0'. element := cls new.

Is this correct?

Yes, you should pay attention to have correct format. But of course VM could also be more cautios and verify if your class are well formed before creating an instance of it..

...

Cheers, Doru

Yes you are right! Bytecode validation would be very important here. If you ask me, the VM should only allow verified bytecode to be executed. Here it could have a dual model, where bytecodes are "internalized" whenever you give them to the VM first. At that point they are validated. Once it's internalized, it can't be modified anymore from the outside, since the Smalltalk world doesn't have direct access to the internalized version. All changes to the bytecode "source", needs to be propagated to the VM by re-internalizing it. This is exactly the model I'm using in Pinocchio... and that allows us to not use bytecodes in the first place :) You just need a way to map the internal version back onto the highlevel version for debugging. (We don't verify the bytecode yet though). Bytecode verification btw shouldn't be too hard. You just need to make sure that all the codes are safe for the instance at hand, and for the method contexts you create. Anything else is just a runtime semantical error which you can't guarantee, since you don't know what the programmer is trying to do. We don't verify it yet since I'm still designing my register-based bytecodes; and since I atm have too much work at hand :) cheers, Toon On 03/21/2011 02:22 PM, Igor Stasenko wrote:

On 21 March 2011 13:14, Toon Verwaest<toon.verwaest@gmail.com> wrote:

...

I couldn't agree more!

However, here the problem is not that the format isn't a correct format. The problem is that a bytecode is activated that is incompatible with the instance format. So basically on every bytecode accessing an instance you would have to check if this is a valid access. And as we all know, Smalltalk avoids this; it makes things faster :)

In other words, a responsibility of having correct bytecode lies on image side, not on VM side.

Adding a bytecode verifier to VM will be a lot of work.. And of course it could verify it only when you installing a method, not when you activating it.

...

Toon

On 03/21/2011 12:49 PM, Igor Stasenko wrote:

...

On 21 March 2011 11:29, Tudor Girba<tudor.girba@gmail.com> wrote:

...

Thanks, Toon!

I changed the code to explicitly set the format, and it seems to fix the problem:

cls := Class new superclass: MooseElement; setFormat: MooseElement format; yourself. cls compileSilently: 'mooseName ^ 1/0'. element := cls new.

Is this correct?

Yes, you should pay attention to have correct format. But of course VM could also be more cautios and verify if your class are well formed before creating an instance of it..

...

Cheers, Doru

This would be a nice little topic... this bytecode verifier. Stef On Mar 21, 2011, at 2:57 PM, Toon Verwaest wrote:

Yes you are right! Bytecode validation would be very important here. If you ask me, the VM should only allow verified bytecode to be executed. Here it could have a dual model, where bytecodes are "internalized" whenever you give them to the VM first. At that point they are validated. Once it's internalized, it can't be modified anymore from the outside, since the Smalltalk world doesn't have direct access to the internalized version. All changes to the bytecode "source", needs to be propagated to the VM by re-internalizing it.

This is exactly the model I'm using in Pinocchio... and that allows us to not use bytecodes in the first place :) You just need a way to map the internal version back onto the highlevel version for debugging. (We don't verify the bytecode yet though).

Bytecode verification btw shouldn't be too hard. You just need to make sure that all the codes are safe for the instance at hand, and for the method contexts you create. Anything else is just a runtime semantical error which you can't guarantee, since you don't know what the programmer is trying to do. We don't verify it yet since I'm still designing my register-based bytecodes; and since I atm have too much work at hand :)

cheers, Toon

On 03/21/2011 02:22 PM, Igor Stasenko wrote:

...

On 21 March 2011 13:14, Toon Verwaest<toon.verwaest@gmail.com> wrote:

...

I couldn't agree more!

However, here the problem is not that the format isn't a correct format. The problem is that a bytecode is activated that is incompatible with the instance format. So basically on every bytecode accessing an instance you would have to check if this is a valid access. And as we all know, Smalltalk avoids this; it makes things faster :)

In other words, a responsibility of having correct bytecode lies on image side, not on VM side.

Adding a bytecode verifier to VM will be a lot of work.. And of course it could verify it only when you installing a method, not when you activating it.

...

Toon

On 03/21/2011 12:49 PM, Igor Stasenko wrote:

...

On 21 March 2011 11:29, Tudor Girba<tudor.girba@gmail.com> wrote:

...

Thanks, Toon!

I changed the code to explicitly set the format, and it seems to fix the problem:

cls := Class new superclass: MooseElement; setFormat: MooseElement format; yourself. cls compileSilently: 'mooseName ^ 1/0'. element := cls new.

Is this correct?

Yes, you should pay attention to have correct format. But of course VM could also be more cautios and verify if your class are well formed before creating an instance of it..

...

Cheers, Doru

On Mon, Mar 21, 2011 at 12:13 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:

On Mon, Mar 21, 2011 at 7:23 PM, Stéphane Ducasse < stephane.ducasse@inria.fr> wrote:

...

This would be a nice little topic... this bytecode verifier.

Probably on top of Opal

The bytecode verifier must be in the VM. If it is up in the image it can be side-stepped. The VM is the ultimate executor of code and so it must apply verification. It could be written in Smalltalk and verified in Smalltalk and then translated. But it must be part of the VM and used by the VM before running any previously unverified method. best, Eliot

...

Stef

On Mar 21, 2011, at 2:57 PM, Toon Verwaest wrote:

...

Yes you are right! Bytecode validation would be very important here. If you ask me, the VM should only allow verified bytecode to be executed. Here it could have a dual model, where bytecodes are "internalized" whenever you give them to the VM first. At that point they are validated. Once it's internalized, it can't be modified anymore from the outside, since the Smalltalk world doesn't have direct access to the internalized version. All changes to the bytecode "source", needs to be propagated to the VM by re-internalizing it.

This is exactly the model I'm using in Pinocchio... and that allows us to not use bytecodes in the first place :) You just need a way to map the internal version back onto the highlevel version for debugging. (We don't verify the bytecode yet though).

Bytecode verification btw shouldn't be too hard. You just need to make sure that all the codes are safe for the instance at hand, and for the method contexts you create. Anything else is just a runtime semantical error which you can't guarantee, since you don't know what the programmer is trying to do. We don't verify it yet since I'm still designing my register-based bytecodes; and since I atm have too much work at hand :)

cheers, Toon

On 03/21/2011 02:22 PM, Igor Stasenko wrote:

...

On 21 March 2011 13:14, Toon Verwaest<toon.verwaest@gmail.com> wrote:

...

I couldn't agree more!

However, here the problem is not that the format isn't a correct format. The problem is that a bytecode is activated that is incompatible with the instance format. So basically on every bytecode accessing an instance you would have to check if this is a valid access. And as we all know, Smalltalk avoids this; it makes things faster :)

In other words, a responsibility of having correct bytecode lies on image side, not on VM side.

Adding a bytecode verifier to VM will be a lot of work.. And of course it could verify it only when you installing a method, not when you activating it.

...

Toon

On 03/21/2011 12:49 PM, Igor Stasenko wrote:

...

On 21 March 2011 11:29, Tudor Girba<tudor.girba@gmail.com> wrote:

...

Thanks, Toon!

I changed the code to explicitly set the format, and it seems to fix the problem:

cls := Class new superclass: MooseElement; setFormat: MooseElement format; yourself. cls compileSilently: 'mooseName ^ 1/0'. element := cls new.

Is this correct?

Yes, you should pay attention to have correct format. But of course VM could also be more cautios and verify if your class are well formed before creating an instance of it..

...

Cheers, Doru

On Mon, Mar 21, 2011 at 7:23 PM, Stéphane Ducasse <stephane.ducasse@inria.fr <mailto:stephane.ducasse@inria.fr>> wrote:

This would be a nice little topic... this bytecode verifier.

Probably on top of Opal

The bytecode verifier must be in the VM. If it is up in the image it can be side-stepped. The VM is the ultimate executor of code and so it must apply verification. It could be written in Smalltalk and verified in Smalltalk and then translated. But it must be part of the VM and used by the VM before running any previously unverified method.

best, Eliot

Indeed, Mariano's response made me want to scream: NOOOOO! :) HOWEVER!, Thinking about it now, it's actually a very interesting proposal! What if we write this piece of software in Smalltak and give it to the VM as part of the VM definition. The VM decides what the application can see, so it might as well not give access to this smalltalk-written bytecode validator that it knows to be safe. If it's built by a separate classbuilder in a separate environment, it can be clearly separated, invulnerable to become: and other evil-doing. So, in short. It's written in Smalltalk (YAY!), but becomes part of the VM runtime, not of the image running on the VM. If we just make sure that we can write out image segments that the VM can use as part of its core (exactly what I do in Pinocchio incidentally), we can just develop and test it nicely in Pharo. Ain't that super-duper cool? Ok, I think I drank too much Yerba Mate. Bad bad caffeine :) cheers, Toon

But why we could not have a byecode validator at the image level that first make sure that byte code are in sync with the format of the objects. Why this has to be done in the vm.

I agree with Stef. It is not obvious for me why it has to be done at the VM. Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

On 22 March 2011 14:02, Toon Verwaest <toon.verwaest@gmail.com> wrote:

The problem is exactly what you had at hand. Your bytecode WAS valid, but it was used in combination with an incompatible class layout. So validation here wouldn't solve anything. You always need to validate in a closed world to ensure you don't accidentally break everything. Whenever you change something, you need to ensure that you revalidate the relevant parts. In this case you could for example just have validated that your new class is a valid subclass of its superclass; which it was not.

To make the system more secure obviously you would have to check those things, but if you add part of the API that circumvents these checks, like you were doing by calling "Class new" rather than using the ClassBuilder which does do the checks, everything breaks. The only authority that can actually ensure that you don't circumvent these checks is the VM.

Obviously you can make sure already in your image that you have enough checks everywhere. You just don't have a crashproof mechanism that will chain your users down avoiding that they shoot themselves in the foot with segfaults. If you put it inside of the VM you -can- provide such a mechanism, because you don't execute anything unless you know it's safe. And as I said, this piece of code could be a piece of prevalidated Smalltalk code that's immutable from the rest of the image.

Yep.. and that is possible only after you introduce irreversible immutability mechanism into VM, which marking object(s) to be immutable for the rest of their existence.

cheers, Toon

On 03/22/2011 02:37 PM, Alexandre Bergel wrote:

...

...

But why we could not have a byecode validator at the image level that first make sure that byte code are in sync with the format of the objects. Why this has to be done in the vm.

I agree with Stef. It is not obvious for me why it has to be done at the VM.

Alexandre

-- Best regards, Igor Stasenko AKA sig.

Immutability may or may not be paid attention to by oneWayBecome:. You can verify the class format, just to have someone in the image change the class of an object. What should you do when you do a become: of a compiled method? What happens when you get the VM to verify and JIT all the compiled methods for class X, and then you put your badly shaped class Y as a subclass of X? If Y does not provide any messages, then inheritance will find all the already verified methods in class X, so the verifier doesn't run and there you go again. Sooner or later, there will be some decision that will throw out the dynamism of the language, or the extreme safety checks, or performance. On 3/22/11 6:17 , Igor Stasenko wrote:

On 22 March 2011 14:02, Toon Verwaest<toon.verwaest@gmail.com> wrote:

...

The problem is exactly what you had at hand. Your bytecode WAS valid, but it was used in combination with an incompatible class layout. So validation here wouldn't solve anything. You always need to validate in a closed world to ensure you don't accidentally break everything. Whenever you change something, you need to ensure that you revalidate the relevant parts. In this case you could for example just have validated that your new class is a valid subclass of its superclass; which it was not.

To make the system more secure obviously you would have to check those things, but if you add part of the API that circumvents these checks, like you were doing by calling "Class new" rather than using the ClassBuilder which does do the checks, everything breaks. The only authority that can actually ensure that you don't circumvent these checks is the VM.

Obviously you can make sure already in your image that you have enough checks everywhere. You just don't have a crashproof mechanism that will chain your users down avoiding that they shoot themselves in the foot with segfaults. If you put it inside of the VM you -can- provide such a mechanism, because you don't execute anything unless you know it's safe. And as I said, this piece of code could be a piece of prevalidated Smalltalk code that's immutable from the rest of the image.

Yep.. and that is possible only after you introduce irreversible immutability mechanism into VM, which marking object(s) to be immutable for the rest of their existence.

...

cheers, Toon

On 03/22/2011 02:37 PM, Alexandre Bergel wrote:

...

...

But why we could not have a byecode validator at the image level that first make sure that byte code are in sync with the format of the objects. Why this has to be done in the vm.

I agree with Stef. It is not obvious for me why it has to be done at the VM.

Alexandre

Construct me any scenario just based on the image, and I can show you how I can avoid it and segfault your system anyway. Construct me any piece of code that can segfault my approach. I doubt that you'll find any. I can't really say more, since I don't know what you are unconvinced about :) cheers, Toon On 03/22/2011 03:19 PM, Alexandre Bergel wrote:

Hi Toon,

Thanks for having taken the time to reply. I understand your and Eliot's email but I am left unconvinced.

Alexandre

On 22 Mar 2011, at 09:02, Toon Verwaest wrote:

...

The problem is exactly what you had at hand. Your bytecode WAS valid, but it was used in combination with an incompatible class layout. So validation here wouldn't solve anything. You always need to validate in a closed world to ensure you don't accidentally break everything. Whenever you change something, you need to ensure that you revalidate the relevant parts. In this case you could for example just have validated that your new class is a valid subclass of its superclass; which it was not.

To make the system more secure obviously you would have to check those things, but if you add part of the API that circumvents these checks, like you were doing by calling "Class new" rather than using the ClassBuilder which does do the checks, everything breaks. The only authority that can actually ensure that you don't circumvent these checks is the VM.

Obviously you can make sure already in your image that you have enough checks everywhere. You just don't have a crashproof mechanism that will chain your users down avoiding that they shoot themselves in the foot with segfaults. If you put it inside of the VM you -can- provide such a mechanism, because you don't execute anything unless you know it's safe. And as I said, this piece of code could be a piece of prevalidated Smalltalk code that's immutable from the rest of the image.

cheers, Toon

On 03/22/2011 02:37 PM, Alexandre Bergel wrote:

...

...

But why we could not have a byecode validator at the image level that first make sure that byte code are in sync with the format of the objects. Why this has to be done in the vm. I agree with Stef. It is not obvious for me why it has to be done at the VM.

Alexandre

On 03/22/2011 03:35 PM, Alexandre Bergel wrote:

I am not saying that this is feasible with the current VM. But I am doubtful that the bytecode verification can only be done in the VM. If something get wrong, why not to raise a primitiveFailed, as 1/0 will do?

I have the impression that you give a particular status to the bytecodes. I have the feeling that we should not. I can realize a division, and time to time it get wrong (/0). But the VM does not crash. So, why if I evaluate the wrong bytecodes, the VM should crash? You could have the necessary guards in the VM. But the full algorithm for verification could be in the VM. I am not expert on this and I may perfectly be wrong.

Most of virtual machine maker will say that a good debugger and profiler can only be done in the VM. Do you agree with this?

Cheers, Alexandre Well, there is 1 alternative. Check at runtime if what the bytecode wants to do is safe. And that's what you basically suggest. This imposes a high runtime overhead however and that by itself makes that an unwanted approach.

The same is true for debuggers and profilers. Depending on what your task at hand is, you don't want this to interfere with the speed of the system. In other cases however you are just fine with knowing where you spend most of your time based on your model. These are 2 different things. And I do agree very strongly that it's most often more interesting to have the high-level view. If you want the bytecode to run safely but -also- as fast as possible, I don't see a way around precomputing if it is valid. This is how VMs optimize things all the time; that's exactly why register-based bytecodes are faster than stack-based bytecodes. You avoid doing at runtime what you know at compilation time. Obviously it gives the same effect in both cases, but you avoid an overhead with the increased risk of doing things wrongly. However, your solution of having a VM that safely executes bytecodes doesn't bring the verification to the image! It keeps it in the VM, just in a different location. Now every execution of a bytecode needs to perform the check, while beforehand you only would do it once for the whole method. Even worse! My approach would allow you to write the verification in Smalltalk code; it's just Smalltalk code running in a privileged sector of the VM. Your code will be pure C code. And why would you need Smalltalk-level verification anyway if your bytecodes already do the checks anyway? This seems duplicate work with no gain. (it basically would just check that your compiler is surely not broken :)) cheers, Toon

On 22 Mar 2011, at 09:24, Toon Verwaest wrote:

...

Construct me any scenario just based on the image, and I can show you how I can avoid it and segfault your system anyway.

Construct me any piece of code that can segfault my approach. I doubt that you'll find any.

I can't really say more, since I don't know what you are unconvinced about :)

cheers, Toon

On 03/22/2011 03:19 PM, Alexandre Bergel wrote:

...

Hi Toon,

Thanks for having taken the time to reply. I understand your and Eliot's email but I am left unconvinced.

Alexandre

On 22 Mar 2011, at 09:02, Toon Verwaest wrote:

...

The problem is exactly what you had at hand. Your bytecode WAS valid, but it was used in combination with an incompatible class layout. So validation here wouldn't solve anything. You always need to validate in a closed world to ensure you don't accidentally break everything. Whenever you change something, you need to ensure that you revalidate the relevant parts. In this case you could for example just have validated that your new class is a valid subclass of its superclass; which it was not.

To make the system more secure obviously you would have to check those things, but if you add part of the API that circumvents these checks, like you were doing by calling "Class new" rather than using the ClassBuilder which does do the checks, everything breaks. The only authority that can actually ensure that you don't circumvent these checks is the VM.

Obviously you can make sure already in your image that you have enough checks everywhere. You just don't have a crashproof mechanism that will chain your users down avoiding that they shoot themselves in the foot with segfaults. If you put it inside of the VM you -can- provide such a mechanism, because you don't execute anything unless you know it's safe. And as I said, this piece of code could be a piece of prevalidated Smalltalk code that's immutable from the rest of the image.

cheers, Toon

On 03/22/2011 02:37 PM, Alexandre Bergel wrote:

...

...

But why we could not have a byecode validator at the image level that first make sure that byte code are in sync with the format of the objects. Why this has to be done in the vm. I agree with Stef. It is not obvious for me why it has to be done at the VM.

Alexandre

On 24 March 2011 19:34, Eliot Miranda <eliot.miranda@gmail.com> wrote:

On Tue, Mar 22, 2011 at 5:34 AM, Stéphane Ducasse <stephane.ducasse@inria.fr> wrote:

...

But why we could not have a byecode validator at the image level that first make sure that byte code are in sync with the format of the objects.

Because it can be compromised.  An in-image verifier is subject to attack, and could be disabled by an attack that got past the in-image verifier before it got a chance to run.  An in-VM verifier is not possible to side-step because it is the only way to execute code.  So an in-VM verifier can be secure but an in-image one can't and so is pointless.

For real hacker there's nothing impossible :) Right now its not possible to split image to layered onion (like operating system does, where you have kernel level, and user level), but i think (at least in theory) such composition could be implemented, except that sure thing we don't have resources to invest in this direction. It is actually nice field for research (hello guys from academy :) -- Best regards, Igor Stasenko AKA sig.

...

<stephane.ducasse@inria.fr> wrote:

...

But why we could not have a byecode validator at the image level that first make sure that byte code are in sync with the format of the objects.

Because it can be compromised. An in-image verifier is subject to attack, and could be disabled by an attack that got past the in-image verifier before it got a chance to run. An in-VM verifier is not possible to side-step because it is the only way to execute code. So an in-VM verifier can be secure but an in-image one can't and so is pointless.

For real hacker there's nothing impossible :)

Right now its not possible to split image to layered onion (like operating system does, where you have kernel level, and user level), but i think (at least in theory) such composition could be implemented, except that sure thing we don't have resources to invest in this direction.

It is actually nice field for research (hello guys from academy :)

we have three phds on the topics and we will see.

-- Best regards, Igor Stasenko AKA sig.