November 2014 - Pharo-users

Re: [Pharo-users] running out of memory while processing a 220MB csv file with NeoCSVReader - tips?

by Stephan Eggermont

Alain wrote: >you are saying that zip ratio is somewhat related to normalized data, >interesting view, and certainly true :) I find it a nice heuristic to help me get started. Just sort the tables on size, start compressing them and start with ones compressing best. >About DateTimes, I think this is not different than with other values, >using a pointer to an interned value should be equivalent to using an >int, as it would be a 32 bits pointer, and with this approach, using >compact records should not make a big difference too if there is not a >lot of different values. Combining multiple booleans in one word still helps a lot, and introducing extra objects for highly correlated fields. >The key I mentioned here is that in real life, this "normalizing ratio" >is very high for almost every kind of data and that's what puzzles me >(not the technique). My impression is that the a lot of the design decisions for relational databases are cargo cult based on the time where most database did not fit into ram and the query optimizers were not good at dealing with lots of joins. Stephan

Nov. 18, 2014

Re: [Pharo-users] [ANN]: NewExternalWebBrowser

by jannik laval

Thank you, I added the code. Cheers, Jannik 2014-11-11 22:34 GMT+01:00 Sean P. DeNigris <sean(a)clipperadams.com>: > jannik laval wrote > > Could you give me access to the repository ? > > Added user JLaval as contributor > > > > ----- > Cheers, > Sean > -- > View this message in context: > http://forum.world.st/ANN-NewExternalWebBrowser-tp4788403p4789740.html > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > -- ~~Jannik Laval~~ Ãcole des Mines de Douai Enseignant-chercheur http://www.jannik-laval.eu http://www.phratch.com http://www.approchealpes.info http://car.mines-douai.fr/

Nov. 18, 2014

Re: [Pharo-users] Citizen example for manipulating a bibtex file

by Offray Vladimir Luna CÃ¡rdenas

Hi Sven, Sorry for my late response. The constructive comments on the list and yours in particular are very valuable to my. I was finishing some details, so only until now I have the time to implement your suggestions. The new code for custom keys on bibtex files from pharo is published at [1] (by the grace of Doru's easy publishing of playgrounds on your stfx server) [1] http://ws.stfx.eu/3CEKQQQ3NL2E By the way the article I'm writing is about a tool for open, citizen, garage research and science developed in Pharo. It is published at [2] and was stored nicely using STON[3] and is superb. To test the tool, the article was wrote on it. [2] http://mutabit.com/deltas/repos.fossil/grafoscopio/doc/tip/Docs/Es/Articulo… [3] http://mutabit.com/deltas/repos.fossil/grafoscopio/doc/tip/Docs/Es/Articulo… [4] http://mutabit.com/deltas/repos.fossil/grafoscopio/doc/tip/Docs/Es/Articulo… The only thing I would add to STON would be an option to support line breaks so long character sequences can be broken to make the format DVCS friendly (git, fossil, etc) and support collaboration and changes tracking. At this moment, because of the long lines in STON, the files are treated as binaries by fossil :-/. Thanks for STON and your lessons, Offray El 22/10/14 a las #4, Sven Van Caekenberghe escribiÃ³: > >> On 22 Oct 2014, at 18:42, Offray Vladimir Luna CÃ¡rdenas <offray(a)riseup.net> wrote: >> >> Hi, >> >> Thanks again. I have a small script, using Citezen which does the trick. I can explore and modify the BibTeX File from the playground with this: >> >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> | bibFile bibliography bibStream bibOutputer | >> bibFile := ((FileLocator documents / 'U/Libertadores/Grafoscopio') children >> detect: [:each | each basename endsWith: 'bib' ]). >> bibliography := CZBibParser parse: bibFile contents. >> bibStream := '' writeStream. >> 1 to: (bibliography size) do: [:index | >> (((bibliography entries at: index) fields at: 2) key = 'shorttitle') >> ifTrue: [ >> (bibliography entries at: index) >> key: ((bibliography entries at: index) fields at: 2) value]. >> bibOutputer := CZBibtexOutputer new. >> bibStream nextPutAll: >> (bibOutputer entryToBibtexString: >> (bibliography entries at: index)); cr.]. >> bibliography. >> bibFile writeStreamDo: [:stream | >> stream nextPutAll: bibStream contents withUnixLineEndings ]. >> bibStream contents. >> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > > Some constructive comments about your code (smaller is always better, especially for interactive snippets): > > - you can change the #detect: to [ :each | each extension = #bib ] > - you can iterate directly over the entries with #do: as in bibliography entries do: [ :each | .. ] which saves you the #at: index > - there are handy unary shortcuts for accessing elements, like #first, #second and so on (up to #ninth) which also save you parenthesis > - you can also construct strings using the idiom String streamContents: [ :bibStream | .. ] > > Sorry, these jumped to me when I saw your code, I hope you don't mind ;-) > >> I will put some functionality inspired by this on my prototype this weekend. >> >> Cheers, >> >> Offray >> >> On 10/21/2014 01:20 AM, stepharo wrote: >>> Check in the tools there is a bib writer. >>> >>> Stef >>> >>> On 21/10/14 03:33, Offray Vladimir Luna CÃ¡rdenas wrote: >>>> Thanks Stef and Damien, >>>> >>>> I have this small script as a proof of concept: >>>> >>>> =================================== >>>> | bibFile bibliography | >>>> bibFile := ((FileLocator documents / 'U/Libertadores/Grafoscopio') >>>> children >>>> detect: [:each | each basename endsWith: 'bib' ]) contents. >>>> bibliography := CZBibParser parse: bibFile. >>>> 1 to: (bibliography size) do: [:index | >>>> (((bibliography entries at: index) fields at: 2) key = 'shorttitle') >>>> ifTrue: [ >>>> (bibliography entries at: index) >>>> key: ((bibliography entries at: index) fields at: 2) value >>>> ]]. >>>> bibliography. >>>> =================================== >>>> >>>> Now I want to write back the corrected file to the .bib ... just >>>> having problems finding which message does the job. I'll keep searching. >>>> >>>> Cheers, >>>> >>>> Offray >>>> >>>> On 10/16/2014 06:40 AM, Damien Cassou wrote: >>>>> from Damien Pollet: >>>>> >>>>> You will need to load the .bib file from zotero (read the file however >>>>> you like, then pass the stream to the CZ parser). You'll get a >>>>> CZBibSet (I don't recall the name exactly) which represents the >>>>> contents of the file. A Set is composed of entries, each of which has >>>>> a key and a set of fields. Finally, fields accept a few different >>>>> kinds of values. >>>>> >>>>> Your processing is just iterating a set then setting the key of each >>>>> entry (or possibly removing and re-adding the entry, I don't recall if >>>>> it's implemented like a dictionary or more like a list). >>>>> >>>>> >>>>> On Mon, Oct 13, 2014 at 2:57 AM, Offray Vladimir Luna CÃ¡rdenas >>>>> <offray(a)riseup.net <mailto:offray@riseup.net>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I'm using a Zotero collection for keeping track of several >>>>> references I have >>>>> found for my article about the experience of the >>>>> outline/tree-like metaphor >>>>> for writing inside Pharo (as soon as I have a presentable >>>>> working draft I >>>>> hope to share it with you). >>>>> >>>>> Now I want to make a post-processing of the bibtex file exported >>>>> from >>>>> Zotero. The idea is to use "shorttitle" field instead to replace >>>>> the Zotero >>>>> auto-generated one and have custom keys. So for example instead of: >>>>> >>>>> ======= >>>>> @misc{_holistic_????, >>>>> title = {Holistic software assessment (Uni Zurich - >>>>> 2011) on Vimeo}, >>>>> shorttitle = {Girba-holistic-2011}, >>>>> url = {http://vimeo.com/42073344?__from=outro-local >>>>> <http://vimeo.com/42073344?from=outro-local>}, >>>>> urldate = {2014-08-19}, >>>>> note = {00000} >>>>> } >>>>> >>>>> ======= >>>>> >>>>> >>>>> I would like to have: >>>>> >>>>> >>>>> ======= >>>>> >>>>> @misc{Girba-holistic-2011, >>>>> title = {Holistic software assessment (Uni Zurich - >>>>> 2011) on Vimeo}, >>>>> shorttitle = {Girba-holistic-2011}, >>>>> url = {http://vimeo.com/42073344?__from=outro-local >>>>> <http://vimeo.com/42073344?from=outro-local>}, >>>>> urldate = {2014-08-19}, >>>>> note = {00000} >>>>> } >>>>> >>>>> ======= >>>>> >>>>> >>>>> I have already installed Citizen and open it on the browser to >>>>> see the code, >>>>> but I can find any place to start with examples. >>>>> >>>>> Any advice on how to solve this issue will be appreciated. >>>>> >>>>> Cheers, >>>>> >>>>> Offray >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Damien Cassou >>>>> http://damiencassou.seasidehosting.st >>>>> >>>>> "Success is the ability to go from one failure to another without losing >>>>> enthusiasm." >>>>> Winston Churchill >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > >

Nov. 18, 2014

Re: [Pharo-users] 'function unavailable' when calling OpenDBXDriver for MySQL

by Markus Fritsche

On 2014-11-17 23:14, Brad wrote: > I think it's "(Smalltalk at: #NBPharoOpenDBX) installAsCurrent. That sounds familiar. But since I didn't read a "hooray", I guess that didn't do the trick either? Off for some sleep

Nov. 17, 2014

Re: [Pharo-users] 'function unavailable' when calling OpenDBXDriver for MySQL

by Brad

I think it's "(Smalltalk at: #NBPharoOpenDBX) installAsCurrent. Brad Selfridge 913-269-2385 > On Nov 17, 2014, at 3:35 PM, Markus Fritsche <mfritsche(a)reauktion.de> wrote: > > Hi, > > So as far as I can see, you have all dependencies... > > I remember that I had to reinitialize the OpenDBX libraries to change the adaptor to the native library calls (from FFI to NBPharoOpenDBX or similar). Unfortuanetly, I don't have the image at hand. > > >

Nov. 17, 2014

Re: [Pharo-users] 'function unavailable' when calling OpenDBXDriver for MySQL

by Brad

How would one re-initialize OpenDBX libraries? Brad Selfridge 913-269-2385 > On Nov 17, 2014, at 3:35 PM, Markus Fritsche <mfritsche(a)reauktion.de> wrote: > > Hi, > > So as far as I can see, you have all dependencies... > > I remember that I had to reinitialize the OpenDBX libraries to change the adaptor to the native library calls (from FFI to NBPharoOpenDBX or similar). Unfortuanetly, I don't have the image at hand. > > >

Nov. 17, 2014

Re: [Pharo-users] 'function unavailable' when calling OpenDBXDriver for MySQL

by Markus Fritsche

Hi, So as far as I can see, you have all dependencies... I remember that I had to reinitialize the OpenDBX libraries to change the adaptor to the native library calls (from FFI to NBPharoOpenDBX or similar). Unfortuanetly, I don't have the image at hand.

Nov. 17, 2014

Re: [Pharo-users] running out of memory while processing a 220MB csv file with NeoCSVReader - tips?

by Alain Rastoul

our referential world is very restricted, whatever area we are talking about Le 17/11/2014 21:04, Alain Rastoul a Ã©crit : > you are saying that zip ratio is somewhat related to normalized data, > interesting view, and certainly true :) > And right, this somewhat normalize all fields, a technique used in > specialized columnstore databases (monetdb and others), often BI > databases with id representing values (that were my experiments). > About DateTimes, I think this is not different than with other values, > using a pointer to an interned value should be equivalent to using an > int, as it would be a 32 bits pointer, and with this approach, using > compact records should not make a big difference too if there is not a > lot of different values. > The key I mentioned here is that in real life, this "normalizing ratio" > is very high for almost every kind of data and that's what puzzles me > (not the technique). > > Regards, > Alain > > Le 17/11/2014 10:47, Stephan Eggermont a Ã©crit : >> Open package contents on your vm, >> open Contents, >> take a look at the info.plist >> >> <key>SqueakMaxHeapSize</key> >> <integer>541065216</integer> >> >> That value needs to be increased to be able to use more than ~512 MB. >> >> Alain wrote: >>> Let say it's your current requirement, and you want to do it like that, >>> a trick that may help you : during personal experiments about loading >>> data in memory and statistics from databases, I found that most often 70 >>> to 80 % of real data is the same. >> >> It is easy to confirm if this is the case in your data: just zip the >> csv file. >> Reasonably structured relational database output often reduces to 10% >> of size. >> With explicitly denormalized data I've seen 99% reduction. >> >> In addition, DateAndTime has a rather wasteful representation for your >> purpose. >> Just reduce to one SmallInt, or with Pharo4, use slots to get a more >> compact >> record representation. >> >> Stephan >> > > > >

Nov. 17, 2014

Re: [Pharo-users] running out of memory while processing a 220MB csv file with NeoCSVReader - tips?

by Alain Rastoul

you are saying that zip ratio is somewhat related to normalized data, interesting view, and certainly true :) And right, this somewhat normalize all fields, a technique used in specialized columnstore databases (monetdb and others), often BI databases with id representing values (that were my experiments). About DateTimes, I think this is not different than with other values, using a pointer to an interned value should be equivalent to using an int, as it would be a 32 bits pointer, and with this approach, using compact records should not make a big difference too if there is not a lot of different values. The key I mentioned here is that in real life, this "normalizing ratio" is very high for almost every kind of data and that's what puzzles me (not the technique). Regards, Alain Le 17/11/2014 10:47, Stephan Eggermont a Ã©crit : > Open package contents on your vm, > open Contents, > take a look at the info.plist > > <key>SqueakMaxHeapSize</key> > <integer>541065216</integer> > > That value needs to be increased to be able to use more than ~512 MB. > > Alain wrote: >> Let say it's your current requirement, and you want to do it like that, >> a trick that may help you : during personal experiments about loading >> data in memory and statistics from databases, I found that most often 70 >> to 80 % of real data is the same. > > It is easy to confirm if this is the case in your data: just zip the csv file. > Reasonably structured relational database output often reduces to 10% of size. > With explicitly denormalized data I've seen 99% reduction. > > In addition, DateAndTime has a rather wasteful representation for your purpose. > Just reduce to one SmallInt, or with Pharo4, use slots to get a more compact > record representation. > > Stephan >

Nov. 17, 2014

Re: [Pharo-users] Opentalk or remoting libraries

by Alain Rastoul

For a machine to machine connection, there will be no direct memory reference but under the hood some socket connection with a tcp server and client, and some marshalling of arguments that is to say some serialization. I'm not sure it would be really faster than websocket and Fuel, because it will be a generic api that have to be able to deal with complex cases. I don't see xml and webservice here ? Le 17/11/2014 11:05, Annick Fron a Ã©crit : > Thank you, > My need is on a local network from machine to machine, in real time is possible, so no security involved. > I would have preferred something more performant than web services or XMLRPC, since both serialization and XML serialization are slow. > Annick > Le 13 nov. 2014 Ã 20:17, Sven Van Caekenberghe <sven(a)stfx.eu> a Ã©crit : > >> Nice. >> >> It is of course also important to note the security risks involved: the client can execute absolutely anything. >> >> One partial solution is to bind the server only to the localhost. >> >>> On 13 Nov 2014, at 19:17, Alain Rastoul <alf.mmm.cat(a)gmail.com> wrote: >>> >>> Hi, >>> CORBA main focus is about interoperability between systems, languages (don't know about opentalk). >>> If you want smalltalk only remote execution, you can very easily do your own on Pharo with Zinc http components and Fuel serializer: a small server that reads smalltalk blocks, evaluates them and return the result (see below an example of server and client). >>> I don't know if Zinc and Fuel run on Squeak, but I guess if not, they are equivalents ones or you could do the same. >>> Beware that Fuel builds a graph of the objects touched by the object it serializes (the block) and serialize them along the way, that may be a problem if you have references to other big objects of your image. >>> >>> " ================================= RPC server" >>> " start Zn server in background" >>> ZnServer startDefaultOn: 1701. >>> ZnServer default delegate: (ZnWebSocketDelegate handler: >>> ([ :webSocket | >>> [ [ | serializedRequest request response serializedResponse | >>> serializedRequest := webSocket readMessage. >>> request := FLMaterializer materializeFromByteArray: serializedRequest. >>> "execute block and it's value to response byte array" >>> response := [ request value ] >>> on: Exception do: [ :ex | ex resume: 'Exception ',ex class name , ' - ', ex messageText ]. >>> serializedResponse := FLSerializer serializeToByteArray: response . >>> webSocket sendMessage: serializedResponse . >>> ] repeat. >>> ] >>> on: ConnectionClosed, ConnectionTimedOut >>> do: [ self crLog: 'Ignoring connection or timeout' ] >>> ]) ). >>> "=========================================" >>> >>> " to stop the server" >>> ZnServer stopDefault . >>> >>> >>> >>> "========================================= RPC CLIENT" >>> >>> " Client call to server: calculate factorial 100 on the other side" >>> | webSocket | >>> webSocket := ZnWebSocket to: 'ws://127.0.0.1:1701'. >>> [ | request serializedRequest serializedResponse response | >>> request := [ 100 factorial ] . " the request is the block to evaluate" >>> serializedRequest := FLSerializer serializeToByteArray: request . "serialize the block" >>> webSocket sendMessage: serializedRequest . "send it" >>> serializedResponse := webSocket readMessage . " read response" >>> response := FLMaterializer materializeFromByteArray: serializedResponse. "deserialize to object" >>> Transcript show: response asString ; cr ] >>> on: ConnectionClosed >>> do: [ self crLog: 'Ignoring connection close, done' ] >>> >>> >>> regards, >>> >>> Alain >>> >>> >>> Le 12/11/2014 18:17, Annick Fron a Ã©crit : >>>> Hi, >>>> >>>> Are there some libraries in pharo to do remote calls like in Opentalk or Corba ? >>>> >>>> Annick >>>> >>> >>> >>> >> >> > > >

Nov. 17, 2014