Here it is. Let me know if you want to see the Linux crash report. Cheers, Hernán 2017-10-05 3:23 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

H Hernani,

Most probably is a problem in the library and not UFFI, but I could not know without a crash report.

Esteban

...

On 5 Oct 2017, at 06:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi,

I ported Nacl (a libsodium wrapper) from the old FFI apicall: format to use the UFFI ffiCall:, but there should be something terribly wrong because is crashing the VM, in both Windows 8.1 and Linux.

How to reproduce in Pharo 6.1

Metacello new smalltalkhubUser: 'tonyg' project: 'Crypto-Nacl'; configuration: 'Nacl'; version: #development; load.

(Nacl hashString: 'The quick brown fox jumps over the lazy dog') hex

This one does deserve a bug entry?

Cheers,

Hernán

On 5 Oct 2017, at 14:55, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Forgot to comment that Nacl worked in Pharo 5.

yes but that was with NB and there are some minimum differences. I do not have the library and I lack the time to try more, but seems to me that here: apiCryptoHashSha512Output: outByteArray input: inByteArray inputLength: inByteArrayLength ^ self ffiCall: #(long crypto_hash_sha512_ref (byte * outByteArray, byte * inByteArray, ulonglong * inByteArrayLength)) module: 'libsodium’. instead "byte * outByteArray”, you want "byte **outByteArray” can you try? Esteban

Cheers,

Hernán

2017-10-05 3:23 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

...

H Hernani,

Most probably is a problem in the library and not UFFI, but I could not know without a crash report.

Esteban

...

On 5 Oct 2017, at 06:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi,

I ported Nacl (a libsodium wrapper) from the old FFI apicall: format to use the UFFI ffiCall:, but there should be something terribly wrong because is crashing the VM, in both Windows 8.1 and Linux.

How to reproduce in Pharo 6.1

Metacello new smalltalkhubUser: 'tonyg' project: 'Crypto-Nacl'; configuration: 'Nacl'; version: #development; load.

(Nacl hashString: 'The quick brown fox jumps over the lazy dog') hex

This one does deserve a bug entry?

Cheers,

Hernán

then replacing "byte *” with ByteArray ?

On 5 Oct 2017, at 15:42, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

I tried, now I get an exception "Use ExternalAddress instead!"

I guess the message means Use ExternalAddres in the "out" parameter. But replacing byte with ExternalAddress also crashes the VM (crash.dmp attached).

2017-10-05 10:03 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

...

...

On 5 Oct 2017, at 14:55, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Forgot to comment that Nacl worked in Pharo 5.

yes but that was with NB and there are some minimum differences. I do not have the library and I lack the time to try more, but seems to me that here:

apiCryptoHashSha512Output: outByteArray input: inByteArray inputLength: inByteArrayLength

^ self ffiCall: #(long crypto_hash_sha512_ref (byte * outByteArray, byte * inByteArray, ulonglong * inByteArrayLength)) module: 'libsodium’.

instead "byte * outByteArray”, you want "byte **outByteArray”

can you try?

Esteban

...

Cheers,

Hernán

2017-10-05 3:23 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

...

H Hernani,

Most probably is a problem in the library and not UFFI, but I could not know without a crash report.

Esteban

...

On 5 Oct 2017, at 06:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi,

I ported Nacl (a libsodium wrapper) from the old FFI apicall: format to use the UFFI ffiCall:, but there should be something terribly wrong because is crashing the VM, in both Windows 8.1 and Linux.

How to reproduce in Pharo 6.1

Metacello new smalltalkhubUser: 'tonyg' project: 'Crypto-Nacl'; configuration: 'Nacl'; version: #development; load.

(Nacl hashString: 'The quick brown fox jumps over the lazy dog') hex

This one does deserve a bug entry?

Cheers,

Hernán

<crash.dmp>

Hi Nicolas 2017-10-05 16:04 GMT-03:00 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:

2017-10-05 15:42 GMT+02:00 Hernán Morales Durand <hernan.morales@gmail.com>:

...

I tried, now I get an exception "Use ExternalAddress instead!"

I guess the message means Use ExternalAddres in the "out" parameter. But replacing byte with ExternalAddress also crashes the VM (crash.dmp attached).

2017-10-05 10:03 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

...

...

On 5 Oct 2017, at 14:55, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Forgot to comment that Nacl worked in Pharo 5.

yes but that was with NB and there are some minimum differences. I do not have the library and I lack the time to try more, but seems to me that here:

apiCryptoHashSha512Output: outByteArray input: inByteArray inputLength: inByteArrayLength

^ self ffiCall: #(long crypto_hash_sha512_ref (byte * outByteArray, byte * inByteArray, ulonglong * inByteArrayLength)) module: 'libsodium’.

instead "byte * outByteArray”, you want "byte **outByteArray”

can you try?

Esteban

Hmm no, internet reference says:

extern int crypto_hash_sha512_ref(unsigned char *,const unsigned char *,unsigned long long);

The error is rather that you pass the address of length rather than the length... So you pass an erroneous length that may lead to a segfault...

Yes, that extra * was the error indeed. Thank you Nicolas. I do not know the VM internals, however I wonder if it is understandable to expect a VM crash for a typo like this? Maybe it is too difficult to catch in the VM side? Just asking, not criticizing anyone's work. Hernán

...

...

...

Cheers,

Hernán

2017-10-05 3:23 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

...

H Hernani,

Most probably is a problem in the library and not UFFI, but I could not know without a crash report.

Esteban

...

On 5 Oct 2017, at 06:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi,

I ported Nacl (a libsodium wrapper) from the old FFI apicall: format to use the UFFI ffiCall:, but there should be something terribly wrong because is crashing the VM, in both Windows 8.1 and Linux.

How to reproduce in Pharo 6.1

Metacello new smalltalkhubUser: 'tonyg' project: 'Crypto-Nacl'; configuration: 'Nacl'; version: #development; load.

(Nacl hashString: 'The quick brown fox jumps over the lazy dog') hex

This one does deserve a bug entry?

Cheers,

Hernán

2017-10-06 2:39 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>:

...

On 6 Oct 2017, at 05:34, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi Nicolas

2017-10-05 16:04 GMT-03:00 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:

...

2017-10-05 15:42 GMT+02:00 Hernán Morales Durand <hernan.morales@gmail.com>:

...

I tried, now I get an exception "Use ExternalAddress instead!"

I guess the message means Use ExternalAddres in the "out" parameter. But replacing byte with ExternalAddress also crashes the VM (crash.dmp attached).

2017-10-05 10:03 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

...

...

On 5 Oct 2017, at 14:55, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Forgot to comment that Nacl worked in Pharo 5.

yes but that was with NB and there are some minimum differences. I do not have the library and I lack the time to try more, but seems to me that here:

apiCryptoHashSha512Output: outByteArray input: inByteArray inputLength: inByteArrayLength

^ self ffiCall: #(long crypto_hash_sha512_ref (byte * outByteArray, byte * inByteArray, ulonglong * inByteArrayLength)) module: 'libsodium’.

instead "byte * outByteArray”, you want "byte **outByteArray”

can you try?

Esteban

Hmm no, internet reference says:

extern int crypto_hash_sha512_ref(unsigned char *,const unsigned char *,unsigned long long);

The error is rather that you pass the address of length rather than the length... So you pass an erroneous length that may lead to a segfault...

Yes, that extra * was the error indeed. Thank you Nicolas.

I do not know the VM internals, however I wonder if it is understandable to expect a VM crash for a typo like this? Maybe it is too difficult to catch in the VM side?

Just asking, not criticizing anyone's work.

Of course that is normal. This is C code, any error has the potential to crash your program. In C there are no runtime safe guards.

There is a reason why we like/use Pharo.

However there are a couple of lightweight libraries which claim to provide exceptions for C: https://github.com/guillermocalvo/exceptions4c or this one https://github.com/psevon/exceptions-and-raii-in-c I wonder if something like that could be used in UFFI

...

Hernán

...

...

...

...

Cheers,

Hernán

2017-10-05 3:23 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

...

H Hernani,

Most probably is a problem in the library and not UFFI, but I could not know without a crash report.

Esteban

...

On 5 Oct 2017, at 06:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi,

I ported Nacl (a libsodium wrapper) from the old FFI apicall: format to use the UFFI ffiCall:, but there should be something terribly wrong because is crashing the VM, in both Windows 8.1 and Linux.

How to reproduce in Pharo 6.1

Metacello new smalltalkhubUser: 'tonyg' project: 'Crypto-Nacl'; configuration: 'Nacl'; version: #development; load.

(Nacl hashString: 'The quick brown fox jumps over the lazy dog') hex

This one does deserve a bug entry?

Cheers,

Hernán

Hi Esteban 2017-10-07 5:51 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

On 7 Oct 2017, at 00:11, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

2017-10-06 2:39 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>:

On 6 Oct 2017, at 05:34, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi Nicolas

2017-10-05 16:04 GMT-03:00 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:

2017-10-05 15:42 GMT+02:00 Hernán Morales Durand <hernan.morales@gmail.com>:

I tried, now I get an exception "Use ExternalAddress instead!"

I guess the message means Use ExternalAddres in the "out" parameter. But replacing byte with ExternalAddress also crashes the VM (crash.dmp attached).

2017-10-05 10:03 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

On 5 Oct 2017, at 14:55, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Forgot to comment that Nacl worked in Pharo 5.

yes but that was with NB and there are some minimum differences. I do not have the library and I lack the time to try more, but seems to me that here:

apiCryptoHashSha512Output: outByteArray input: inByteArray inputLength: inByteArrayLength

^ self ffiCall: #(long crypto_hash_sha512_ref (byte * outByteArray, byte * inByteArray, ulonglong * inByteArrayLength)) module: 'libsodium’.

instead "byte * outByteArray”, you want "byte **outByteArray”

can you try?

Esteban

Hmm no, internet reference says:

extern int crypto_hash_sha512_ref(unsigned char *,const unsigned char *,unsigned long long);

The error is rather that you pass the address of length rather than the length... So you pass an erroneous length that may lead to a segfault...

Yes, that extra * was the error indeed. Thank you Nicolas.

I do not know the VM internals, however I wonder if it is understandable to expect a VM crash for a typo like this? Maybe it is too difficult to catch in the VM side?

Just asking, not criticizing anyone's work.

Of course that is normal. This is C code, any error has the potential to crash your program. In C there are no runtime safe guards.

There is a reason why we like/use Pharo.

However there are a couple of lightweight libraries which claim to provide exceptions for C: https://github.com/guillermocalvo/exceptions4c or this one https://github.com/psevon/exceptions-and-raii-in-c

I wonder if something like that could be used in UFFI

Nothing in C *ever* will save you from accessing bad memory blocks. And yes, it will always crash your program.

Let me see if I get this right. If there is nothing to prevent accessing bad memory blocks, assuming accessing or executing invalid memory, then how the memory debuggers like mpatrol or valgrind, do their work? Just a rethoric question to make a point, please keep reading... Memory debuggers don't prevent segmentation violation of the debuggee, but they don't crash either. I know they slow down application, but it wouldn't be preferable to gracefully recover from crashes during coding time? My (maybe crazy) idea was while coding sensitive parts (like using UFFI) one can use a "debugger vm", and use the "normal vm" when safer. So I think we are talking about different things here. I don't want to save "bad memory block" errors nor dream about bullet proof VM, but if we know the bullet then let's use a nice bulletproof vest :) What do you think about having some self-healing feedback loop to "hide" the crash? like the described in https://link.springer.com/article/10.1007/s00607-010-0107-y In general, how the VM developers view the resilient systems approach? Are they too far away to get it right given the current VM complexity?

those links you point are exception mechanisms, a set of tools to easy the catch of programming exceptions. But, if you try to read a chunk of memory in, say 0x8000 and the data/function that should be there is now garbage, you’re screw.

as Sven pointed, that’s why we use higher level languages, with GC and all that.

Thank you for the clarifications BTW Hernán

Esteban

Hernán

Cheers,

Hernán

2017-10-05 3:23 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

H Hernani,

Most probably is a problem in the library and not UFFI, but I could not know without a crash report.

Esteban

On 5 Oct 2017, at 06:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi,

I ported Nacl (a libsodium wrapper) from the old FFI apicall: format to use the UFFI ffiCall:, but there should be something terribly wrong because is crashing the VM, in both Windows 8.1 and Linux.

How to reproduce in Pharo 6.1

Metacello new smalltalkhubUser: 'tonyg' project: 'Crypto-Nacl'; configuration: 'Nacl'; version: #development; load.

(Nacl hashString: 'The quick brown fox jumps over the lazy dog') hex

This one does deserve a bug entry?

Cheers,

Hernán

On 8 Oct 2017, at 10:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi Esteban

2017-10-07 5:51 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com <mailto:estebanlm@gmail.com>>:

...

On 7 Oct 2017, at 00:11, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

2017-10-06 2:39 GMT-03:00 Sven Van Caekenberghe <sven@stfx.eu>:

On 6 Oct 2017, at 05:34, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi Nicolas

2017-10-05 16:04 GMT-03:00 Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com>:

2017-10-05 15:42 GMT+02:00 Hernán Morales Durand <hernan.morales@gmail.com>:

I tried, now I get an exception "Use ExternalAddress instead!"

I guess the message means Use ExternalAddres in the "out" parameter. But replacing byte with ExternalAddress also crashes the VM (crash.dmp attached).

2017-10-05 10:03 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

On 5 Oct 2017, at 14:55, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Forgot to comment that Nacl worked in Pharo 5.

yes but that was with NB and there are some minimum differences. I do not have the library and I lack the time to try more, but seems to me that here:

apiCryptoHashSha512Output: outByteArray input: inByteArray inputLength: inByteArrayLength

^ self ffiCall: #(long crypto_hash_sha512_ref (byte * outByteArray, byte * inByteArray, ulonglong * inByteArrayLength)) module: 'libsodium’.

instead "byte * outByteArray”, you want "byte **outByteArray”

can you try?

Esteban

Hmm no, internet reference says:

extern int crypto_hash_sha512_ref(unsigned char *,const unsigned char *,unsigned long long);

The error is rather that you pass the address of length rather than the length... So you pass an erroneous length that may lead to a segfault...

Yes, that extra * was the error indeed. Thank you Nicolas.

I do not know the VM internals, however I wonder if it is understandable to expect a VM crash for a typo like this? Maybe it is too difficult to catch in the VM side?

Just asking, not criticizing anyone's work.

Of course that is normal. This is C code, any error has the potential to crash your program. In C there are no runtime safe guards.

There is a reason why we like/use Pharo.

However there are a couple of lightweight libraries which claim to provide exceptions for C: https://github.com/guillermocalvo/exceptions4c or this one https://github.com/psevon/exceptions-and-raii-in-c

I wonder if something like that could be used in UFFI

Nothing in C *ever* will save you from accessing bad memory blocks. And yes, it will always crash your program.

Let me see if I get this right.

If there is nothing to prevent accessing bad memory blocks, assuming accessing or executing invalid memory, then how the memory debuggers like mpatrol or valgrind, do their work? Just a rethoric question to make a point, please keep reading…

they fire a signal and give you an amount of ms to do something before you crash (like writing a stack). that’s how OS work.

Memory debuggers don't prevent segmentation violation of the debuggee, but they don't crash either. I know they slow down application, but it wouldn't be preferable to gracefully recover from crashes during coding time? My (maybe crazy) idea was while coding sensitive parts (like using UFFI) one can use a "debugger vm", and use the "normal vm" when safer.

maybe, you could run Pharo inside one of that platforms.That’s out of the scope of Pharo itself and I don’t know how hard would be, but since Pharo is “just another app”, it can be possible. (side note: I used that kind of stuff years ago and programs still crash, just less and producing more information when they do).

So I think we are talking about different things here. I don't want to save "bad memory block" errors nor dream about bullet proof VM, but if we know the bullet then let's use a nice bulletproof vest :)

if you want to contribute on that area, I will be very grateful. Sadly I do not have time to work on it right now (nor in the near year(s), if I watch my ever-increasing todo list :( ), and all I can suggest to people is some good practices when programming C interfaces.

What do you think about having some self-healing feedback loop to "hide" the crash? like the described in https://link.springer.com/article/10.1007/s00607-010-0107-y <https://link.springer.com/article/10.1007/s00607-010-0107-y>

In general, how the VM developers view the resilient systems approach?

they don't

Are they too far away to get it right given the current VM complexity? infinite far, since there are no plans at all to tackle this :)

...

those links you point are exception mechanisms, a set of tools to easy the catch of programming exceptions. But, if you try to read a chunk of memory in, say 0x8000 and the data/function that should be there is now garbage, you’re screw.

as Sven pointed, that’s why we use higher level languages, with GC and all that.

Thank you for the clarifications BTW

cheers! Esteban

Hernán

...

Esteban

Hernán

Cheers,

Hernán

2017-10-05 3:23 GMT-03:00 Esteban Lorenzano <estebanlm@gmail.com>:

H Hernani,

Most probably is a problem in the library and not UFFI, but I could not know without a crash report.

Esteban

On 5 Oct 2017, at 06:00, Hernán Morales Durand <hernan.morales@gmail.com> wrote:

Hi,

I ported Nacl (a libsodium wrapper) from the old FFI apicall: format to use the UFFI ffiCall:, but there should be something terribly wrong because is crashing the VM, in both Windows 8.1 and Linux.

How to reproduce in Pharo 6.1

Metacello new smalltalkhubUser: 'tonyg' project: 'Crypto-Nacl'; configuration: 'Nacl'; version: #development; load.

(Nacl hashString: 'The quick brown fox jumps over the lazy dog') hex

This one does deserve a bug entry?

Cheers,

Hernán

Yeh, the most important thing is what Jan says there. What you want to ensure is that you don't corrupt your memory (and particularly the heap). If you do so (i.e., corruption) the only way you can come back is that you remember the old uncorrupted value and, given that everything is still working, you put it back where it belongs. In any case, Hernan, I think this discussion belongs to the vm-dev list. On Sun, Oct 8, 2017 at 1:32 PM, Jan Vrany <jan.vrany@fit.cvut.cz> wrote:

...

So I think we are talking about different things here. I don't want to save "bad memory block" errors nor dream about bullet proof VM, but if we know the bullet then let's use a nice bulletproof vest :)

This can be done and has been done.

Following code would clearly result in segmentation violation:

bytes := ExternalBytes address: 16r10 size: 100. bytes byteAt: 1 put: 10.

There are smalltalk implementations out there that handle this and open a (smalltalk) debugger (see the screenshot). Of course, you can just abort, fix your code and try again.

Of course, if you happen to overwrite your object memory or VM internal structures, you're screwed.

As you said, you can still be shot and die, but the west can save your life in some cases :-)

Best, Jan

-- Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - *http://www.cnrs.fr <http://www.cnrs.fr>* *Web:* *http://guillep.github.io* <http://guillep.github.io> *Phone: *+33 06 52 70 66 13