April 2014 - Pharo-users - lists.pharo.org

Re: [Pharo-users] Error: No factory specified

by Esteban A. Maringolo

Renaming Cache to AbstractCache (or whatever else, it has no global references) I was able to load Glorp and have it running with the NativePostgresDriver. However there are a mix of things that are innecesary interdependant. For instance: * Running the tests the default glorp login doesn't have any encoding strategy, but it is required by the Glorp Session. So many tests fail. * The Encoders are in the DBXTalk packages :) I'm willing to help here, but what approach should we take here? I look forward for advice and guidance to accomplish this. Regards, Esteban A. Maringolo 2014-04-09 6:22 GMT-03:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > Thanks, Stephan. > > I will have a look and try to load/use Glorp myself towards the end of the week. > > On 09 Apr 2014, at 11:08, Stephan Eggermont <stephan(a)stack.nl> wrote: > >> Ok, there is a bug in The ConfigurationOfDBXDriver, >> the baseline has a versionString entry. That should be removed >> >> After removing that (and renaming Cache) it loads. >> >> Glorp loads already after renaming Cache. >> I'm not sure the isKindOf: on ProtoObject is safe. >> There is a renaming issue after loading Glorp (13188) >> >> Stephan > >

April 9, 2014

Re: [Pharo-users] Socket Handles to C

by Igor Stasenko

On 9 April 2014 16:59, Sean P. DeNigris <sean(a)clipperadams.com> wrote: > How do Socket handles map to C socket IDs? I'm wrapping libssh2 via Native > Boost and I'd much rather create sockets via Smalltalk than C, but I'm not > sure how to pass the handle to a C function expecting a C socket ID. Maybe > I'm missing something really simple? Thanks. > > socket plugin using own data structure for socket handles, it includes different kind of internal information, including OS-specific socket handle. typedef struct { int sessionID; int socketType; /* 0 = TCP, 1 = UDP */ void *privateSocketPtr; } SQSocket, *SocketPtr; extracting the OS-specific handle could be problematic, since it is in that opaque void *privateSocketPtr; field. Then: sqUnixSocket.c typedef struct privateSocketStruct { int s; /* Unix socket */ int connSema; /* connection io notification semaphore */ int readSema; /* read io notification semaphore */ int writeSema; /* write io notification semaphore */ int sockState; /* connection + data state */ int sockError; /* errno after socket error */ union sockaddr_any peer; /* default send/recv address for UDP */ socklen_t peerSize; /* dynamic sizeof(peer) */ union sockaddr_any sender; /* sender address for last UDP receive */ socklen_t senderSize; /* dynamic sizeof(sender) */ int multiListen; /* whether to listen for multiple connections */ int acceptedSock; /* a connection that has been accepted */ } privateSocketStruct; so, to extract OS-level socked handle *on unix*, you have to handle = ( (privateSocketStruct*) sqsocket.privateSocketPtr) -> s. where sqsocket is SQSocket. there's even macros for that: /*** Accessors for private socket members from a Squeak socket pointer ***/ #define _PSP(S) (((S)->privateSocketPtr)) #define PSP(S) ((privateSocketStruct *)((S)->privateSocketPtr)) #define SOCKET(S) (PSP(S)->s) #define SOCKETSTATE(S) (PSP(S)->sockState) #define SOCKETERROR(S) (PSP(S)->sockError) #define SOCKETPEER(S) (PSP(S)->peer) #define SOCKETPEERSIZE(S) (PSP(S)->peerSize) > > ----- > Cheers, > Sean > -- > View this message in context: > http://forum.world.st/Socket-Handles-to-C-tp4753619.html > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > -- Best regards, Igor Stasenko.

April 9, 2014

Re: [Pharo-users] Zinc HTTP server seems to convert always ...

by itlists＠schrievkrom.de

Ok, forget the JSON stuff - it has nothing to do with the "problem". Other way round: My whole database and internal processing is done in UTF8. This is the most important point here to mention. Now the request comes into Zinc as mentioned below (the content of the request is a JSON string only): HTML-Request (charset=UTF-8) =(sends)=> ZINC HTTP Now Zinc sees the content of the body, knows that it is coded in UTF8 and creates a ZnStringEntity with UTF8Encoder. Zinc HTTP =(builds)=> ZnStringEntity (with UTF8Encoder) The instance of ZnRequest and its entity value is an instance of ZnStringEntity (with its encoder attribute is set to an instance to ZnUTF8Encoder). I checked the content of the string attribute of the ZnStringEntity and this string is NOT encoded in UTF8 any more, but in either ISO8859-? or WIN1252. I think, that this is ok for almost all people, because they work with some CodePages - but my internal processing assumes UTF8. I just fixed this for me by changing ZnStringEntity>>initializeEncoder to ALWAYS set the encoder attribute to ZnNullEncoder and now everthing is ok again. This means of course, that all apllication running with that source code work in UTF 8 only ... Marten Am 09.04.2014 18:42, schrieb Sven Van Caekenberghe: > Marten, > > On 09 Apr 2014, at 18:25, itlists(a)schrievkrom.de wrote: > >> Ok, if the browser sends POST/PUT request with a JSON structure it also >> sends charset = utf8 (in my case). That's ok, because for JSON this is >> more or less the default charset. >> >> Zinc now seems to notice, that UTF8 charset is needed and creates a >> ZnStringEntity with an UTF8Encoder. >> >> Now when my application tries to get the JSON string of that >> ZnStringEntity and builds the structure out of that string - and the >> strings are NOT UTF8, but converted to (?) ISO8859 ? > > (NeoJSONReader fromString: > (ZnEntity with: (NeoJSONWriter toString: { #message -> 'An der schÃ¶nen blauen Donau' } asDictionary))) > at: #message. > > You must be doing something possibly wrong when you <<get the JSON string of that ZnStringEntity and builds the structure out of that string>> (how do you do that, BTW), so please write some code that demonstrates what is not right according to you. > > Sven > -- Marten Feldtmann

April 9, 2014

Re: [Pharo-users] NeoJSON Parsing Nested Objects

by Sean P. DeNigris

Sean P. DeNigris wrote > I'd like to convert them to DropletSize objects Duh :-P reader for: DoResponse do: [ :m | m mapInstVar: #status. (m mapInstVar: #contents to: #sizes) valueSchema: #ArrayOfDropletSizes ]. ----- Cheers, Sean -- View this message in context: http://forum.world.st/NeoJSON-Parsing-Nested-Objects-tp4753695p4753696.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

April 9, 2014

NeoJSON Parsing Nested Objects

by Sean P. DeNigris

I'm wrapping the Digital Ocean API. This particular response has a status, and an array of droplet sizes. For example: '{"status":"OK","sizes":[{"id":66,"name":"512MB","slug":"512mb","memory":512,"cpu":1,"disk":20,"cost_per_hour":0.00744,"cost_per_month":"5.0"},{"id":63,"name":"1GB","slug":"1gb","memory":1024,"cpu":1,"disk":30,"cost_per_hour":0.01488,"cost_per_month":"10.0"}]}' As a start, I did: reader := NeoJSONReader on: jsonString readStream. reader for: DoResponse customDo: [ :m | m decoder: [ :dict | DoResponse new status: (dict at: 'status'); contents: (dict at: 'sizes') ] ]. response := reader nextAs: DoResponse. response isOk ifFalse: [ self error: 'Query failed!' ]. ^ response contents. However, the size objects are still plain dictionaries. I'd like to convert them to DropletSize objects. And I'd rather leverage NeoJSON than implement a custom DropletSize fromDictionary: if possible. What's the best way to handle this? Thanks. ----- Cheers, Sean -- View this message in context: http://forum.world.st/NeoJSON-Parsing-Nested-Objects-tp4753695.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

April 9, 2014

Re: [Pharo-users] Zinc HTTP server seems to convert always ...

by Sven Van Caekenberghe

Marten, On 09 Apr 2014, at 18:25, itlists(a)schrievkrom.de wrote: > Ok, if the browser sends POST/PUT request with a JSON structure it also > sends charset = utf8 (in my case). That's ok, because for JSON this is > more or less the default charset. > > Zinc now seems to notice, that UTF8 charset is needed and creates a > ZnStringEntity with an UTF8Encoder. > > Now when my application tries to get the JSON string of that > ZnStringEntity and builds the structure out of that string - and the > strings are NOT UTF8, but converted to (?) ISO8859 ? (NeoJSONReader fromString: (ZnEntity with: (NeoJSONWriter toString: { #message -> 'An der schÃ¶nen blauen Donau' } asDictionary))) at: #message. You must be doing something possibly wrong when you <<get the JSON string of that ZnStringEntity and builds the structure out of that string>> (how do you do that, BTW), so please write some code that demonstrates what is not right according to you. Sven

April 9, 2014

Re: [Pharo-users] Zinc HTTP server seems to convert always ...

by itlists＠schrievkrom.de

Ok, if the browser sends POST/PUT request with a JSON structure it also sends charset = utf8 (in my case). That's ok, because for JSON this is more or less the default charset. Zinc now seems to notice, that UTF8 charset is needed and creates a ZnStringEntity with an UTF8Encoder. Now when my application tries to get the JSON string of that ZnStringEntity and builds the structure out of that string - and the strings are NOT UTF8, but converted to (?) ISO8859 ? Marten -- Marten Feldtmann

April 9, 2014

Re: [Pharo-users] Memory profiling: usage per class - instance in old memory

by Sven Van Caekenberghe

On 09 Apr 2014, at 18:01, Thomas Bany <mun.sysdev(a)gmail.com> wrote: > Thanks ! > > That's exactly what I was looking for. There is a compare method I dont quite understand but I think I found what is going on. > > I failed to grasp that an Array reply to #sizeInMemory with it's own size, without the sizes of its references. A single position object weight 96 bytes, which make the whole Array weight arround 8Mb, and the 32 objects arround 250 Mb. Yes, #sizeInMemory is confusing, a recursive version would be nice, but also dangerous as it could get in a loop. Here is something related: http://www.humane-assessment.com/blog/traversal-enabled-pharo-objects/ If you want to use a huge data structure, you have to think carefully about your representations. There are tricks you can use to conserve memory: use more primitive types (SmallIntegers, bit flags, Symbols), use shared instances, use alternatives like ZTimestamp which is half the size of DateAndTime, or use your own integer time, sparse data structures, and so on - and you can hide these optimisations behind your standard API. > I'm not sure I can get arround that aspect since the computation is costly and I need its output multiple times. > > I will make further testing to see why the memory is not released at the end of the execution. Good luck. > Thanks again ! > > > > 2014-04-09 15:19 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > Hi Thomas, > > Fixing memory consumption problems is hard, but important: memory efficient code is automatically faster in the long run as well. > > Your issue sounds serious. However, I would start by trying to figure out what is happening at your coding level: somehow you (or something you use) must be holding on too much memory. Questioning low level memory management functionality should be the last resort, not the first. > > There is SpaceTally that you could use before and after running part of your code. Once something unexpected survives GC, there is the PointerFinder functionality (Inspector > Explore Pointers) to find what holds onto objects. But no matter what, it is hard. > > If you have some public code that you could share to demonstrate your problem, then we could try to help. > > Sven > > On 09 Apr 2014, at 12:54, Thomas Bany <mun.sysdev(a)gmail.com> wrote: > > > Hi, > > > > My app is a parser/filter for binary files, that produces a bunch of ascii files. > > > > At the begining of the parsing, the filtering step involves the storage of the positions of 32 objects, each second for a whole day. So that's 32 Arrays with 86400 elements each. > > > > During this step, the memory used by my image grows from 50Mb to ~500Mb. I find it far too large since I'm pretty sure my arrays are the largest objects I create and only weight something like 300kb. > > > > The profiling of the app shows that hte footprint of the "old memory" went up by 350Mb. Which I'm pretty sure is super bad. Maybe as a consequence, after the parsing is finished, the memory footprint of the image stays at ~500Mb > > > > What are the tools I have to find where precisely the memory usage explodes ? For example, is it possible to browse the "old memory" objects to see which one fails to get GC'ed ? > > > > Thanks in advance, > > > > Thomas. > -- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org

April 9, 2014

Re: [Pharo-users] Zinc HTTP server seems to convert always ...

by Sven Van Caekenberghe

On 09 Apr 2014, at 17:29, itlists(a)schrievkrom.de wrote: > The browser sends UTF8 data and in my application code I get instances > of ZnStringEntity and the contained string is converted to (?) ISO8859-1 > (?) or CP-1252 (?). This seems to be due to the fact, that the entity > instance always has a ZnUTF8Encoder to do the conversion. I still don't understand the problem, but consider this: ZnServer startDefaultOn: 1701. ZnClient new url: 'http://localhost:1701/echo'; entity: (ZnEntity with: 'An der schÃ¶nen blauen Donau'); post. ZnClient new url: 'http://localhost:1701/echo'; entity: (ZnEntity with: 'An der schÃ¶nen blauen Donau' type: (ZnMimeType textPlain charSet: #'iso-8859-1'; yourself)); post; yourself. In the first case, a UTF-8 encoded string is POST-ed and correctly returned (in a UTF-8 encoded response). In the second case, an ISO-8859-1 encoded string is POST-ed and correctly returned (in a UTF-8 encoded response). In both cases the decoding was done correctly, using the specified charset (if that is missing, the ZnNullEncoder is used). Now, Ã¶ is not a perfect test example because its encoding value in Unicode, 246 decimal, U+00F6 hex, still fits in 1 byte and hence survives null encoding/decoding. That is why the following still works, although it is wrong to drop the charset. ZnClient new url: 'http://localhost:1701/echo'; entity: (ZnEntity with: 'An der schÃ¶nen blauen Donau' type: (ZnMimeType textPlain clearCharSet; yourself)); post; yourself. HTH, Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill

April 9, 2014

Re: [Pharo-users] Memory profiling: usage per class - instance in old memory

by Thomas Bany

Thanks ! That's exactly what I was looking for. There is a compare method I dont quite understand but I think I found what is going on. I failed to grasp that an Array reply to #sizeInMemory with it's own size, without the sizes of its references. A single position object weight 96 bytes, which make the whole Array weight arround 8Mb, and the 32 objects arround 250 Mb. I'm not sure I can get arround that aspect since the computation is costly and I need its output multiple times. I will make further testing to see why the memory is not released at the end of the execution. Thanks again ! 2014-04-09 15:19 GMT+02:00 Sven Van Caekenberghe <sven(a)stfx.eu>: > Hi Thomas, > > Fixing memory consumption problems is hard, but important: memory > efficient code is automatically faster in the long run as well. > > Your issue sounds serious. However, I would start by trying to figure out > what is happening at your coding level: somehow you (or something you use) > must be holding on too much memory. Questioning low level memory management > functionality should be the last resort, not the first. > > There is SpaceTally that you could use before and after running part of > your code. Once something unexpected survives GC, there is the > PointerFinder functionality (Inspector > Explore Pointers) to find what > holds onto objects. But no matter what, it is hard. > > If you have some public code that you could share to demonstrate your > problem, then we could try to help. > > Sven > > On 09 Apr 2014, at 12:54, Thomas Bany <mun.sysdev(a)gmail.com> wrote: > > > Hi, > > > > My app is a parser/filter for binary files, that produces a bunch of > ascii files. > > > > At the begining of the parsing, the filtering step involves the storage > of the positions of 32 objects, each second for a whole day. So that's 32 > Arrays with 86400 elements each. > > > > During this step, the memory used by my image grows from 50Mb to ~500Mb. > I find it far too large since I'm pretty sure my arrays are the largest > objects I create and only weight something like 300kb. > > > > The profiling of the app shows that hte footprint of the "old memory" > went up by 350Mb. Which I'm pretty sure is super bad. Maybe as a > consequence, after the parsing is finished, the memory footprint of the > image stays at ~500Mb > > > > What are the tools I have to find where precisely the memory usage > explodes ? For example, is it possible to browse the "old memory" objects > to see which one fails to get GC'ed ? > > > > Thanks in advance, > > > > Thomas. > > >

April 9, 2014