FFI Struct Argument Pass By Value Fails on Mac 64 bit
I've been trying to track this down for a couple weeks now. I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd. I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct. I've carved out the wee bit of clang to reproduce this in a tiny library. The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side. The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c I'm stuck. The FFI64 plugin is way beyond my comprehension. // microclang.c typedef struct { const void *ptr_data[2]; unsigned int_data; } CXSourceLocation; /** * \brief Identifies a half-open character range in the source code. * * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the * starting and end locations from a source range, respectively. */ typedef struct { const void *ptr_data[2]; unsigned begin_int_data; unsigned end_int_data; } CXSourceRange; const char* first = "first_pointer"; const char* second = "second_pointer"; // return a fake range with non zero data CXSourceRange clang_getArbitraryRange() { CXSourceRange range = {0}; range.ptr_data[0] = (void*)first; range.ptr_data[1] = (void*)second; range.begin_int_data = 17; range.end_int_data = 24; return range; } // Actual clang function - range is always zero'd here despite having values in the image CXSourceLocation clang_getRangeStart(CXSourceRange range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; return Result; } CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, range.begin_int_data }; return Result; }
Hi Todd, Could you advise the the Playground code to use to test this? And maybe a C program that links to the microclang shared library that does the equivalent test? cheers -ben On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
I'm stuck. The FFI64 plugin is way beyond my comprehension.
// microclang.c
typedef struct { const void *ptr_data[2]; unsigned int_data; } CXSourceLocation;
/** * \brief Identifies a half-open character range in the source code. * * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the * starting and end locations from a source range, respectively. */ typedef struct { const void *ptr_data[2]; unsigned begin_int_data; unsigned end_int_data; } CXSourceRange;
const char* first = "first_pointer"; const char* second = "second_pointer";
// return a fake range with non zero data CXSourceRange clang_getArbitraryRange() { CXSourceRange range = {0}; range.ptr_data[0] = (void*)first; range.ptr_data[1] = (void*)second; range.begin_int_data = 17; range.end_int_data = 24; return range; }
// Actual clang function - range is always zero'd here despite having values in the image CXSourceLocation clang_getRangeStart(CXSourceRange range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; return Result; }
CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, range.begin_int_data }; return Result; }
Playground code: LibCClang startFromRange:(LibCClang defaultRange) it worked if the resulting struct is not all zeroes. A tiny C program is the equivalent. int main() { if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) printf("That failed\n"); else printf("That worked\n") return 0; }
On Nov 21, 2017, at 11:18 PM, Ben Coman <btc@openinworld.com> wrote:
Hi Todd,
Could you advise the the Playground code to use to test this? And maybe a C program that links to the microclang shared library that does the equivalent test?
cheers -ben
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
I'm stuck. The FFI64 plugin is way beyond my comprehension.
// microclang.c
typedef struct { const void *ptr_data[2]; unsigned int_data; } CXSourceLocation;
/** * \brief Identifies a half-open character range in the source code. * * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the * starting and end locations from a source range, respectively. */ typedef struct { const void *ptr_data[2]; unsigned begin_int_data; unsigned end_int_data; } CXSourceRange;
const char* first = "first_pointer"; const char* second = "second_pointer";
// return a fake range with non zero data CXSourceRange clang_getArbitraryRange() { CXSourceRange range = {0}; range.ptr_data[0] = (void*)first; range.ptr_data[1] = (void*)second; range.begin_int_data = 17; range.end_int_data = 24; return range; }
// Actual clang function - range is always zero'd here despite having values in the image CXSourceLocation clang_getRangeStart(CXSourceRange range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; return Result; }
CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, range.begin_int_data }; return Result; }
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c $ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing... $ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed So I'm not sure how to proceed. I was expecting that would work while Pharo failed. Now interestingly... $ clang test.c microlang.c $ ./a.out That worked So it seems a similar problem exists outside our FFI. cheers -ben P.S. I refactored you code to extract a header file (attached)
I'm stuck. The FFI64 plugin is way beyond my comprehension.
// microclang.c
typedef struct { const void *ptr_data[2]; unsigned int_data; } CXSourceLocation;
/** * \brief Identifies a half-open character range in the source code. * * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the * starting and end locations from a source range, respectively. */ typedef struct { const void *ptr_data[2]; unsigned begin_int_data; unsigned end_int_data; } CXSourceRange;
const char* first = "first_pointer"; const char* second = "second_pointer";
// return a fake range with non zero data CXSourceRange clang_getArbitraryRange() { CXSourceRange range = {0}; range.ptr_data[0] = (void*)first; range.ptr_data[1] = (void*)second; range.begin_int_data = 17; range.end_int_data = 24; return range; }
// Actual clang function - range is always zero'd here despite having values in the image CXSourceLocation clang_getRangeStart(CXSourceRange range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 }; return Result; }
CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] }, range.begin_int_data }; return Result; }
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference... ___microlang.h___ typedef unsigned uintptr_t; typedef struct { const void *ptr_data[2]; } CXSourceRange_; CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range); ___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer"; CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; } int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } } ___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer"; void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } } int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string"); CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); } ___Makefile___ default: clean static shared clean: rm -f *so *App @echo shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo static: clang -g -o staticApp test.c microclang.c ./staticApp @echo Now running... $ make > report gives... ___report___ rm -f *so *App clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string) clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string) cheers -ben
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...
___microlang.h___ typedef unsigned uintptr_t;
typedef struct { const void *ptr_data[2]; } CXSourceRange_;
CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);
___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";
CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }
int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";
void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");
CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); }
___Makefile___ default: clean static shared
clean: rm -f *so *App @echo
shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo
static: clang -g -o staticApp test.c microclang.c ./staticApp @echo
Now running... $ make > report
gives... ___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)
Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way) ___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer"; const char * lib_getLibraryString() { return myLibraryString; } int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } } ___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer"; void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } } int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string"); test(lib_testString(localString), "local string"); } $ make > report ___report___ rm -f *so *App clang -g -o staticApp test.c microclang.c ./staticApp test=0, aString-->4196150 That worked (library string) test=0, aString-->4196068 That worked (local string) clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp test=1, aString-->-792512599 That failed (library string) test=0, aString-->4196484 That worked (local string) cheers -ben
i'm getting the idea that we should probably write a test suite/library for FFI
On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...
___microlang.h___ typedef unsigned uintptr_t;
typedef struct { const void *ptr_data[2]; } CXSourceRange_;
CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);
___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";
CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }
int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";
void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");
CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); }
___Makefile___ default: clean static shared
clean: rm -f *so *App @echo
shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo
static: clang -g -o staticApp test.c microclang.c ./staticApp @echo
Now running... $ make > report
gives... ___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)
Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way)
___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer";
const char * lib_getLibraryString() { return myLibraryString; }
int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer";
void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string");
test(lib_testString(localString), "local string"); }
$ make > report
___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp
test=0, aString-->4196150 That worked (library string)
test=0, aString-->4196068 That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp
test=1, aString-->-792512599 That failed (library string)
test=0, aString-->4196484 That worked (local string)
cheers -ben
<sharedLibString.zip><clang3.zip>
It would be wonderful! On Sun, Nov 26, 2017 at 5:24 PM, Todd Blanchard <tblanchard@mac.com> wrote:
i'm getting the idea that we should probably write a test suite/library for FFI
On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...
___microlang.h___ typedef unsigned uintptr_t;
typedef struct { const void *ptr_data[2]; } CXSourceRange_;
CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);
___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";
CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }
int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";
void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");
CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); }
___Makefile___ default: clean static shared
clean: rm -f *so *App @echo
shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo
static: clang -g -o staticApp test.c microclang.c ./staticApp @echo
Now running... $ make > report
gives... ___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)
Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way)
___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer";
const char * lib_getLibraryString() { return myLibraryString; }
int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer";
void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string");
test(lib_testString(localString), "local string"); }
$ make > report
___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp
test=0, aString-->4196150 That worked (library string)
test=0, aString-->4196068 That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp
test=1, aString-->-792512599 That failed (library string)
test=0, aString-->4196484 That worked (local string)
cheers -ben
<sharedLibString.zip><clang3.zip>
On 27 November 2017 at 00:24, Todd Blanchard <tblanchard@mac.com> wrote:
i'm getting the idea that we should probably write a test suite/library for FFI
I noticed these... * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/ platforms/unix/plugins/SqueakFFIPrims/ffi-test-main.c * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/ Cog/platforms/Cross/plugins/SqueakFFIPrims/sqFFITestFuncs.c * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/ Cog/platforms/Cross/plugins/IA32ABI/AlienSUnitTestProcedures.c These look like c-code test frames for FFI. Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them? And also some general info... * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/ Cog/platforms/unix/plugins/SqueakFFIPrims/00README
On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole. I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases. The attached zipfile containing libstruct.c a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers. There is a Makefile with three main targets: 1. make layout - statically compiles libstruct.c and runs produced a.out to display structure layouts 2. make run - downloads Pharo, starts it loading LibStruct.st , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console 3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs. Note Pharo will freeze and you need to move to LLDB. Try these commands... frame variable call print_struct(&GoodStruct1_fmt, &aStruct) continue That works on Ubuntu 16.04 64 bit. You will need to tune it for OSX. clang and lldb are required. ================================ The offsets configured in the class variables of all ExternalStructs correctly matches that reported by the C code test frame results here... $ make layout clang -g libstruct.c ./a.out GoodStruct1: uint32_t:int1: 01 4 uint32_t:int2: 05 4 GoodStruct2: uint32_t:int1: 01 4 uint32_t:int2: 05 4 void*:ptr_data: 09 8 GoodStruct3: void*:ptr_data: 01 8 uint32_t:int1: 09 4 uint32_t:int2: 13 4 BadStruct: void*:ptr_data1: 01 8 void*:ptr_data2: 09 8 uint32_t:int1: 17 4 uint32_t:int2: 21 4 ================================ $ make run clang -g -o libstruct.so -shared -fPIC libstruct.c getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st Now manually browse to and run LibStruct>>>LibStructTest>>#testStructs LibStructTest>>testStructs |good1Struct good2Struct good3Struct badStruct | good1Struct := GoodStruct1 new int1: 2; int2: 3. good2Struct := GoodStruct2 new int1: 2; int2: 3; ptr_force_int: 4. good3Struct := GoodStruct3 new int1: 2; int2: 3; ptr_force_int: 4. badStruct := BadStruct new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5. self assert: (LibStruct tryGood1: good1Struct) equals: 6. self assert: (LibStruct tryGood2: good2Struct) equals: 6. self assert: (LibStruct tryGood3: good3Struct) equals: 6. "Problem exposed in next line" self assert: (LibStruct tryBad: badStruct) equals: 6. which on console produces... GoodStruct1: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 GoodStruct2: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 void*:ptr_data: 09 8 = 04 00 00 00 00 00 00 00 GoodStruct3: void*:ptr_data: 01 8 = 04 00 00 00 00 00 00 00 uint32_t:int1: 09 4 = 02 00 00 00 uint32_t:int2: 13 4 = 03 00 00 00 BadStruct: void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00 void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 17 4 = 29 03 03 00 uint32_t:int2: 21 4 = 00 00 00 00 Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two. The first time this is run after Image boots seems like BadStruct gets some random data. ================================ In same image, subsequent runs of LibStruct>>>LibStructTest>>testStructs give only all zeros for BadStruct, as Todd observed. GoodStruct1: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 GoodStruct2: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 void*:ptr_data: 09 8 = 00 00 00 00 00 00 00 00 GoodStruct3: void*:ptr_data: 01 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 09 4 = 02 00 00 00 uint32_t:int2: 13 4 = 03 00 00 00 BadStruct: void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00 void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 17 4 = 00 00 00 00 uint32_t:int2: 21 4 = 00 00 00 00 cheers -ben
I got pulled off to do some $ work and haven't got back to this but I would like to get back to it. Has there been any progress on this in the publicly available 64bit VM? I see there is a completely separate FFI plugin for 32 vs 64 and worry 64 is still not quite ready for real work.
On Dec 3, 2017, at 7:18 AM, Ben Coman <btc@openinworld.com> wrote:
On 27 November 2017 at 00:24, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
i'm getting the idea that we should probably write a test suite/library for FFI
I noticed these... * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/pl... <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/pl...> * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/p... <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/p...> * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/p... <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/p...>
These look like c-code test frames for FFI. Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them?
And also some general info... * https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/pl... <https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/pl...>
On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole.
I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases. The attached zipfile containing libstruct.c a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers. There is a Makefile with three main targets: 1. make layout - statically compiles libstruct.c and runs produced a.out to display structure layouts
2. make run - downloads Pharo, starts it loading LibStruct.st <http://libstruct.st/> , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console
3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs. Note Pharo will freeze and you need to move to LLDB. Try these commands... frame variable call print_struct(&GoodStruct1_fmt, &aStruct) continue
That works on Ubuntu 16.04 64 bit. You will need to tune it for OSX. clang and lldb are required.
================================ The offsets configured in the class variables of all ExternalStructs correctly matches that reported by the C code test frame results here...
$ make layout clang -g libstruct.c ./a.out
GoodStruct1: uint32_t:int1: 01 4 uint32_t:int2: 05 4 GoodStruct2: uint32_t:int1: 01 4 uint32_t:int2: 05 4 void*:ptr_data: 09 8 GoodStruct3: void*:ptr_data: 01 8 uint32_t:int1: 09 4 uint32_t:int2: 13 4 BadStruct: void*:ptr_data1: 01 8 void*:ptr_data2: 09 8 uint32_t:int1: 17 4 uint32_t:int2: 21 4
================================
$ make run clang -g -o libstruct.so -shared -fPIC libstruct.c getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st <http://libstruct.st/>
Now manually browse to and run LibStruct>>>LibStructTest>>#testStructs LibStructTest>>testStructs |good1Struct good2Struct good3Struct badStruct |
good1Struct := GoodStruct1 new int1: 2; int2: 3. good2Struct := GoodStruct2 new int1: 2; int2: 3; ptr_force_int: 4. good3Struct := GoodStruct3 new int1: 2; int2: 3; ptr_force_int: 4. badStruct := BadStruct new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5.
self assert: (LibStruct tryGood1: good1Struct) equals: 6. self assert: (LibStruct tryGood2: good2Struct) equals: 6. self assert: (LibStruct tryGood3: good3Struct) equals: 6. "Problem exposed in next line" self assert: (LibStruct tryBad: badStruct) equals: 6.
which on console produces...
GoodStruct1: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 GoodStruct2: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 void*:ptr_data: 09 8 = 04 00 00 00 00 00 00 00 GoodStruct3: void*:ptr_data: 01 8 = 04 00 00 00 00 00 00 00 uint32_t:int1: 09 4 = 02 00 00 00 uint32_t:int2: 13 4 = 03 00 00 00 BadStruct: void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00 void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 17 4 = 29 03 03 00 uint32_t:int2: 21 4 = 00 00 00 00
Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two. The first time this is run after Image boots seems like BadStruct gets some random data.
================================ In same image, subsequent runs of LibStruct>>>LibStructTest>>testStructs give only all zeros for BadStruct, as Todd observed.
GoodStruct1: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 GoodStruct2: uint32_t:int1: 01 4 = 02 00 00 00 uint32_t:int2: 05 4 = 03 00 00 00 void*:ptr_data: 09 8 = 00 00 00 00 00 00 00 00 GoodStruct3: void*:ptr_data: 01 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 09 4 = 02 00 00 00 uint32_t:int2: 13 4 = 03 00 00 00 BadStruct: void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00 void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00 uint32_t:int1: 17 4 = 00 00 00 00 uint32_t:int2: 21 4 = 00 00 00 00
cheers -ben
<FFI64StructTest.zip>
Hi Todd, On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <tblanchard@mac.com> wrote:
i'm getting the idea that we should probably write a test suite/library for FFI
See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c
On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...
___microlang.h___ typedef unsigned uintptr_t;
typedef struct { const void *ptr_data[2]; } CXSourceRange_;
CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);
___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";
CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }
int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";
void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");
CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); }
___Makefile___ default: clean static shared
clean: rm -f *so *App @echo
shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo
static: clang -g -o staticApp test.c microclang.c ./staticApp @echo
Now running... $ make > report
gives... ___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)
Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way)
___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer";
const char * lib_getLibraryString() { return myLibraryString; }
int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer";
void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string");
test(lib_testString(localString), "local string"); }
$ make > report
___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp
test=0, aString-->4196150 That worked (library string)
test=0, aString-->4196068 That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp
test=1, aString-->-792512599 That failed (library string)
test=0, aString-->4196484 That worked (local string)
cheers -ben
<sharedLibString.zip><clang3.zip>
-- _,,,^..^,,,_ best, Eliot
Todd, On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Hi Todd,
On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <tblanchard@mac.com> wrote:
i'm getting the idea that we should probably write a test suite/library for FFI
See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c
I suggest you extend these with an example that demonstrates your failing case. One thing to be very careful about is the platform. Are you on Mac OS X, linux or Win64? Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-ab...) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147...). In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list. The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register. The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. HTH On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...
___microlang.h___ typedef unsigned uintptr_t;
typedef struct { const void *ptr_data[2]; } CXSourceRange_;
CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);
___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";
CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }
int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";
void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");
CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); }
___Makefile___ default: clean static shared
clean: rm -f *so *App @echo
shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo
static: clang -g -o staticApp test.c microclang.c ./staticApp @echo
Now running... $ make > report
gives... ___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)
Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way)
___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer";
const char * lib_getLibraryString() { return myLibraryString; }
int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer";
void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string");
test(lib_testString(localString), "local string"); }
$ make > report
___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp
test=0, aString-->4196150 That worked (library string)
test=0, aString-->4196068 That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp
test=1, aString-->-792512599 That failed (library string)
test=0, aString-->4196484 That worked (local string)
cheers -ben
<sharedLibString.zip><clang3.zip>
-- _,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
I'm mac osx I will update the test library and ping back
On Mar 1, 2018, at 10:11 AM, Eliot Miranda <eliot.miranda@gmail.com> wrote:
Todd,
On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>> wrote: Hi Todd,
On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
i'm getting the idea that we should probably write a test suite/library for FFI
See the package FFI-Tests at http://source.squeak.org/FFI <http://source.squeak.org/FFI> and the file sqFFITestFuncs.c <http://sqffitestfuncs.c/>
I suggest you extend these with an example that demonstrates your failing case. One thing to be very careful about is the platform. Are you on Mac OS X, linux or Win64? Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-ab... <https://software.intel.com/sites/default/files/article/402129/mpx-linux64-ab...>) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147... <https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147...>). In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list. The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register. The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.
HTH
On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...
___microlang.h___ typedef unsigned uintptr_t;
typedef struct { const void *ptr_data[2]; } CXSourceRange_;
CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);
___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";
CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }
int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";
void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");
CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); }
___Makefile___ default: clean static shared
clean: rm -f *so *App @echo
shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo
static: clang -g -o staticApp test.c microclang.c ./staticApp @echo
Now running... $ make > report
gives... ___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)
Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way)
___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer";
const char * lib_getLibraryString() { return myLibraryString; }
int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer";
void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string");
test(lib_testString(localString), "local string"); }
$ make > report
___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp
test=0, aString-->4196150 That worked (library string)
test=0, aString-->4196068 That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp
test=1, aString-->-792512599 That failed (library string)
test=0, aString-->4196484 That worked (local string)
cheers -ben
<sharedLibString.zip><clang3.zip>
-- _,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
Silly me. I have no idea how to update this library. I thought it would be a file. I've used Monticello years ago in Squeak - however trying to load this repository into Pharo 6.1-64 is impossible. Invariably I end up with an undefined symbol and a couple of undismissable progress dialogs. Also, its .c files that compile into a library, right? Where would such files be in this repository? I love Smalltalk. I'm really starting to hate this though.
On Mar 1, 2018, at 10:25 AM, todd blanchard <tblanchard@mac.com> wrote:
I'm mac osx
I will update the test library and ping back
On Mar 1, 2018, at 10:11 AM, Eliot Miranda <eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>> wrote:
Todd,
On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <eliot.miranda@gmail.com <mailto:eliot.miranda@gmail.com>> wrote: Hi Todd,
On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
i'm getting the idea that we should probably write a test suite/library for FFI
See the package FFI-Tests at http://source.squeak.org/FFI <http://source.squeak.org/FFI> and the file sqFFITestFuncs.c <http://sqffitestfuncs.c/>
I suggest you extend these with an example that demonstrates your failing case. One thing to be very careful about is the platform. Are you on Mac OS X, linux or Win64? Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-ab... <https://software.intel.com/sites/default/files/article/402129/mpx-linux64-ab...>) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147... <https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147...>). In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list. The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register. The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.
HTH
On Nov 24, 2017, at 12:54 AM, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 24 November 2017 at 13:16, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 21:59, Ben Coman <btc@openinworld.com <mailto:btc@openinworld.com>> wrote:
On 22 November 2017 at 13:38, Todd Blanchard <tblanchard@mac.com <mailto:tblanchard@mac.com>> wrote:
I've been trying to track this down for a couple weeks now.
I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.
I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.
I've carved out the wee bit of clang to reproduce this in a tiny library.
The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.
The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c
On Ubuntu 16.04 I used... $ clang -shared -fPIC -o libmicroclang.so microclang.c
$ clang test.c -L. -l microclang test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation' if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)
I presume you meant... if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0) so correcting and continuing...
$ clang test.c -L. -l microclang $ LD_LIBRARY_PATH=. ./a.out That failed
So I'm not sure how to proceed. I was expecting that would work while Pharo failed.
Now interestingly... $ clang test.c microlang.c $ ./a.out That worked
So it seems a similar problem exists outside our FFI.
cheers -ben
P.S. I refactored you code to extract a header file (attached)
The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue. Attached zip exploded here for easy reference...
___microlang.h___ typedef unsigned uintptr_t;
typedef struct { const void *ptr_data[2]; } CXSourceRange_;
CXSourceRange_ clang_getArbitraryRange_(); int clang_getRangeEnd_(CXSourceRange_ range);
___microclang.c___ #include "microclang.h" const char* libraryString = "library_pointer";
CXSourceRange_ clang_getArbitraryRange_() { CXSourceRange_ range = {0}; range.ptr_data[0] = (void*)libraryString; return range; }
int clang_getRangeEnd_(CXSourceRange_ range) { // Special decoding for CXSourceLocations for CXLoadedDiagnostics. if ((uintptr_t)range.ptr_data[0] & 0x1) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> #include "microclang.h" const char* localString = "local_pointer";
void test( CXSourceRange_ range, char *note ) { int result = clang_getRangeEnd_(range); if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { CXSourceRange_ range1 = clang_getArbitraryRange_(); test(range1, "library string");
CXSourceRange_ range2 = {0}; range2.ptr_data[0] = (void*)localString; test(range2, "local string"); }
___Makefile___ default: clean static shared
clean: rm -f *so *App @echo
shared: clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp @echo
static: clang -g -o staticApp test.c microclang.c ./staticApp @echo
Now running... $ make > report
gives... ___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp That worked (library string) That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp That failed (library string) That worked (local string)
Further simplification dealing *only* with strings (see attached sharedLibString.zip) (also attached is clang3.zip as a step along the way)
___microclang.c___ typedef unsigned uintptr_t; const char* myLibraryString = "library_pointer";
const char * lib_getLibraryString() { return myLibraryString; }
int lib_testString( const char *aString ) { unsigned test = (uintptr_t)aString & 0x1; printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString); if (test) { return 0; } else { return 1; } }
___test.c___ #include <stdio.h> int lib_testString( const char *aString ); const char *lib_getLibraryString(); const char *localString = "local_pointer";
void test( int result, char *note ) { if(result == 0) { printf("That failed (%s)\n", note); } else { printf("That worked (%s)\n", note); } }
int main() { const char * libraryString = lib_getLibraryString(); test(lib_testString(libraryString), "library string");
test(lib_testString(localString), "local string"); }
$ make > report
___report___ rm -f *so *App
clang -g -o staticApp test.c microclang.c ./staticApp
test=0, aString-->4196150 That worked (library string)
test=0, aString-->4196068 That worked (local string)
clang -g -o libmicroclang.so -shared -fPIC microclang.c clang -g -o sharedApp test.c -L. -lmicroclang LD_LIBRARY_PATH=. ./sharedApp
test=1, aString-->-792512599 That failed (library string)
test=0, aString-->4196484 That worked (local string)
cheers -ben
<sharedLibString.zip><clang3.zip>
-- _,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
participants (4)
-
Ben Coman -
Eliot Miranda -
Stephane Ducasse -
Todd Blanchard