[Pharo-users] Dynamic graph exploration - seeking input on idiomatic smalltalk [LONG]

S Krish krishnamachari.sudhakar at gmail.com
Tue Jul 24 14:14:37 EDT 2012


I doubt I can understand the whole of your idea and would aver that a
working model / code is lot better for anyone equally interested to load ,
try and comment on it.

The goal is big and nice, utility not yet comprehensible fully, but perhaps
a working code will clarify a bit more again to some one in the similar
line of action.

Go ahead and let the code + screen shots do the talk.. you might have good
feedback

On Tue, Jul 24, 2012 at 11:20 PM, Patrik Sundberg <patrik.sundberg at gmail.com
> wrote:

> On Tue, Jul 10, 2012 at 8:15 PM, Patrik Sundberg wrote:
>
>> I've done a bit more studying and I think a combination of
>> SystemAnnouncer events to catch method changes, pragmas to mark "graph
>> nodes" and use of method wrappers could create a very nice way to shield
>> the user of my graph API from thinking of identity caches and graph node
>> details.
>>
>> The other cool feature of use is the new class builder and custom slots.
>> For nodes that are not dynamic/calculations (what I call properties or
>> relationship nodes) I think custom slots could be interesting but need to
>> test out some ideas to organize thoughts. Obviously won't work for
>> calculated/dynamic nodes since I need ability to take arguments for those
>> and can't map to a slot. In principle all I'm after is to have an identity
>> map and a graph (DAG) in the background whose management is hidden away
>> from user so he can focus on domain logic and not graph stuff.
>>
>> Going on holiday for a week and a bit, shall start experimenting after
>> that.
>>
>
> Ok, so I'm back on this. My first idea is to do this:
>
> - Every node in my graph is either a piece of data (I call it a leaf or a
> property node) or a node depending on other nodes that perform a
> calculation (I call it a calc node for now)
>
> - All the nodes are part of an identity map which means a node can only
> exist in one copy and any relations and dependencies will sit between these
> unique nodes
>
> - I'd like to shield the user from all the identify map stuff etc as much
> as possible.
>
> - A smalltalk object is an instance of a class, and it can participate in
> the graph by providing leaf nodes and calc nodes. Those nodes are
> associated with the messages of the object.
>
> - I have some ideas for how to bootstrap the dependency graph between
> nodes etc and I'm not worried about that bit (on first calculation create
> structure via reflection and a dependency stack)
>
> - The explicit and uggly way I can do things are like this:
> Foo>>addOneTo: aNumberNode
>    ^GraphManager instance
>        getCalcNode: #addOneTo:
>        for: self
>        withArgs: #( aNumberNode )
>        block: [ :node |
>           node value + 1
>        ]
>
>   - pretend this goes and asks the identity cache and either gets an
> existing leaf node for the addOneTo: for this object for a particular
> argument, or if none exist it creates it and attaches the block as the
> calculation to perform when #value is sent to the node
>
>   - let's say 'foo' is an instance of Foo class, then 'foo addOneTo: baz'
> refers to the calc node, and '(foo addOneTo: baz) value' refers to the
> value of the calculation
>
> - What I was thinking here was that I could use method wrappers to hide
> all the junk, like this:
> Foo>>addOneTo: aNode
>    <calcNode>
>    aNode value + 1
>
> - I listen to all method add/update/remove announcements, and when a
> method that is marked with the <calcNode> pragma is involved, what I do is
> create a MethodWrapper that takes care of all the uggly stuff to get/create
> the graph node via the identity cache and as #value method for the node
> uses the method implementation the user originally provided (and replaces
> the entry in the method dictionary)
>
> Does that sound doable? The browsers etc wont go nuts?
>
> If I do that, will system browsers still show the original source for the
> method even though I've replaced it via the method dictionary and
> MethodWrapper?
>
> I'm assuming stack traces and debugging etc will be just fine, will just
> have the "hidden" node logic via the MethodWrappers included in the call
> stack.
>
> I'll give that type of setup a go, just wanted to throw it out there and
> hear if it's an obviously terrible idea first.
>
>
> P.S. for leaf nodes (or any node not requiring an argument to be provided)
> I think I could use custom slots instead of method wrappers and do the same
> kind of logic as the method wrapper would do in the slot instead. I'll
> worry about that later.
>
>
>
>  On Jul 6, 2012 1:16 PM, "Patrik Sundberg" wrote:
>
>>  On Fri, Jul 6, 2012 at 12:38 PM, S Krish  wrote:
>>>
>>> You should read as much as you can about Kapital of JPMC. It does more
>>>> or less what you describe for esoteric Prime Interest Derivatives and very
>>>> scalably ..
>>>>
>>>>
>>> I've heard of that one. Never worked at JPM so haven't seen it first
>>> hand though. I'll see what I can find. In terms of concepts I'm fully on
>>> top of it (have a background with other companies using similar ideas, but
>>> not in smalltalk but their own in-house language). So the good part is that
>>> I know exactly what I want conceptually, but I'm not stuck into smalltalk
>>> enough to know the natural way to implement the concepts. It may sound like
>>> a huge endeavour but I've done a lot of it in ruby and even as a side
>>> project to my real job we're talking about a couple of months to have
>>> something useful.
>>>
>>>
>>>> I doubt if there is much in the public domain, but the fact that
>>>> Kapital is the golden standard in this product line talks highly of why
>>>> Smalltalk is THE platform for this kind of product.
>>>>
>>>>
>>> I wouldn't say it's the golden standard, that's taking it a bit far,
>>> there are others that I rate higher bit I'm also biased :) But I do agree
>>> that smalltalk is a reasonably natural platform for these types of
>>> concepts, hence why I'm exploring it.
>>>
>>>
>>>> It would be nice to have a Pharo based derivative product that can
>>>> easily beat Ruby/ Java / .Net at this dynamic visualization using standard
>>>> browsers, where Nautilus too can serve your purpose.. or a home grown tree
>>>> based browser.../ Grids..
>>>>
>>>>
>>> That's my hunch, that the tooling work would be much cut down. In my
>>> current ruby mockup I'm kind of creating a poor man's image like experience
>>> - it's jruby on the JVM, vim + vim-slime to send code fragments to the
>>> "runtime", git, and most likely homegrown GUI to navigate the graph and
>>> play with it (not done anything on that yet). I'll be keeping jruby as a
>>> dear tool as right now it's a fantastic glue tool in terms of interacting
>>> with anything out there in the world (i.e. I can easily work with the
>>> bloomberg java API), but I'm very keen to explore pharo for my graph +
>>> tooling around that.
>>>
>>> (slight tangent - coming from outside the smalltalk world it'd be HUGE
>>> to me to have git backend for packages and use of github for collaboration.
>>> seen that mentioned in other threads. I think it'd be HUGE for community's
>>> ability to attract more people).
>>>
>>>
>>>
>>>> Once I have completed my run at a PharoTabletIDE/ PharoMorphicView
>>>> framework, I would love to collaborate on your endeavour to see if it can
>>>> be modelled on it and exposed..
>>>>
>>>>
>>> Cool. Early days yet and we'll see where I end up. Not familiar with
>>> those projects, I shall take a look.
>>>
>>>
>>> Patrik
>>>
>>>
>>>
>>>
>>>> On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> First of all, sorry for the long email as my first email to the list.
>>>>> Hopefully some people will find it an interesting discussion :)
>>>>>
>>>>> I'm a long time programmer that has been studying smalltalk and pharo
>>>>> over
>>>>> the last year. It's a beautiful language I'm liking the image style of
>>>>> development a whole lot. I've been looking for a good test case to try
>>>>> out
>>>>> in pharo and I've got something in mind. I've used ruby since around
>>>>> 2000
>>>>> and given how much it has been inspired by smalltalk it's not a big
>>>>> leap.
>>>>>
>>>>> My real job is being a commodities trader but I build my own tools and
>>>>> something I'm always working on is improved risk and pricing tools.
>>>>> Lots of
>>>>> that comes down to 2 things:
>>>>> 1. evaluating a "function" (in a general sense) with inputs that are
>>>>> picked
>>>>> depending on time. e.g. I want to price a security and I want to pick
>>>>> up
>>>>> prices for time X as inputs to the pricing model.
>>>>> 2. calculating a derivative of the "function" above with respect to
>>>>> inputs
>>>>>
>>>>> What I tend to do in order to keep things consistent by design, plus
>>>>> add
>>>>> natural caching, is create a calculation graph. I define all the
>>>>> dependencies and create an in-memory graph where each node is unique
>>>>> using
>>>>> an identity map. I then perform a calculation where the calculations
>>>>> trickle
>>>>> down the graph starting from the node I'm interested in down all it's
>>>>> dependencies. To calculate the derivative I then find the node
>>>>> representing
>>>>> the input I want the derivative with respect to and use finite
>>>>> differences,
>>>>> e.g. move it's value, and recalculate the top level node. On the second
>>>>> valuation of the graph only the parts that are affected by me changing
>>>>> that
>>>>> 1 value will need to be evaluated, the rest is cached an unaffected. It
>>>>> makes it very easy to ask questions like "How much will I be affected
>>>>> by X
>>>>> changing by Y?" since I can take the top level node of the graph,
>>>>> search for
>>>>> X, and if found change it by Y and recalculate.
>>>>>
>>>>> How the graph is constructed in memory depends on time (and a few other
>>>>> things). I always store all previous states of the world so that I can
>>>>> recreate any calculation I did in the past. Very useful for diagnosing
>>>>> problem. Hence if I for example change what model I use to model
>>>>> something,
>>>>> that change is recorded in a way such that if I set my "time" to
>>>>> before the
>>>>> change happened the graph will get created as it would have with the
>>>>> old
>>>>> model, and for a time after the change it'll create a different graph
>>>>> reflecting the new model. Hence the graph can only be known at runtime
>>>>> when
>>>>> the "time" etc is known.
>>>>>
>>>>> I currently have a ruby mockup of this. It's a DSL that looks like
>>>>> this:
>>>>>
>>>>> ----------- EXAMPLE
>>>>>
>>>>> class ExampleObject
>>>>>   include GraphEntity # includes a module with functionality for
>>>>> objects
>>>>> participating in the graph, including #property and #calc used below
>>>>>
>>>>>   property :foo do
>>>>>     123 # this is the default value for this property which will be
>>>>> used
>>>>> before a value has been set and saved
>>>>>   end
>>>>>
>>>>>   calc :bar do |arg|
>>>>>     foo.value + arg     # take the value of the arg property and add
>>>>> the
>>>>> argument given to the calculation node
>>>>>   end
>>>>> end
>>>>>
>>>>> o = ExampleObject.new
>>>>> # this will kick of the graph being built in memory and setup the
>>>>> dependency
>>>>> between bar and foo nodes
>>>>> o.bar(1).value  # -> 124
>>>>> # this will be a "cache lookup" since nothing in the graph that bar
>>>>> depends
>>>>> on has changed (i.e. the "expensive" calculation is not performed again
>>>>> o.bar(1).value  # -> 124
>>>>>
>>>>> o.foo.set_value(2)
>>>>> o.bar(1).value  # -> 125, realizes that foo changed and performs a
>>>>> recalc
>>>>> --------------------------------------------------------
>>>>>
>>>>> To accomplish this I use dynamic code generation like the below for
>>>>> #property and similar for other types of nodes:
>>>>>
>>>>>         # NOTE: it's a class level method, hence how I can call it when
>>>>> defining the class in the example
>>>>>
>>>>>         def property(name, time = CdrEnv.instance.time, &block)
>>>>>           clear_property(name)
>>>>>           getter_method_body = <<-EOF
>>>>>             def #{name}
>>>>>               init_block = self.class.property_init_block_for(:#{name})
>>>>>               time_for_property =
>>>>> self.class.property_time_dependency_for(:#{name})
>>>>>               if init_block.nil?
>>>>>                 CdrEnv.instance.get_property_node(self, :#{name},
>>>>> time_for_property)
>>>>>               else
>>>>>                 CdrEnv.instance.get_property_node(self, :#{name},
>>>>> time_for_property, &init_block)
>>>>>               end
>>>>>             end
>>>>>           EOF
>>>>>           setter_method_body = <<-EOF
>>>>>             def #{name.to_s}=(value)
>>>>>               #{name.to_s}.mutate(value)
>>>>>             end
>>>>>           EOF
>>>>>           class_eval getter_method_body
>>>>>           class_eval setter_method_body
>>>>>           register_property_node(name, time, &block)
>>>>>         end
>>>>>
>>>>> Don't worry about the details or all the unfamiliar ruby, the main
>>>>> point is
>>>>> that it creates an instance method that uses the singleton
>>>>> CdrEnv.instance
>>>>> to either get or create the node representing the property from the
>>>>> graph
>>>>> identity cache depending on if it exists or not.
>>>>>
>>>>> From a tooling point of view I think I'd love to work with this kind of
>>>>> thing in pharo. Building my own browsers for inspecting and debugging
>>>>> my
>>>>> dynamic graph should be very good fit. However, I'd appreciate some
>>>>> pointers
>>>>> to idiomatic smalltalk to attack this kind of problem in terms of
>>>>> implementing the graph itself - I obviously want the user (even if
>>>>> it's me)
>>>>> to just have to focus on the domain model and hide as much as possible
>>>>> of
>>>>> the graph bits under the covers the same way I've done with the code
>>>>> generation stuff in my ruby example.
>>>>>
>>>>> Any input into this would be much appreciated,
>>>>> Patrik
>>>>>
>>>>> P.S.
>>>>> Btw, the persistence backend for this is the neo4j graph database, but
>>>>> it's
>>>>> fronted by a service slotting into my own service framework built using
>>>>> ZeroMQ and services exchanging messages in protobuf format. One can
>>>>> use it
>>>>> from any language as long as one can send protobuf messages over
>>>>> zeromq. I
>>>>> see there's a zeromq ffi library available on SS that I'll check out,
>>>>> but
>>>>> I'm not finding a protobuf implementation. It'd be easy enough for me
>>>>> to
>>>>> port a ruby protobuf implementation but I may as well ask if someone
>>>>> has
>>>>> already done any work on protobuf for smalltalk?
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-smalltalk-LONG-tp4638702.html
>>>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pharo.org/pipermail/pharo-users_lists.pharo.org/attachments/20120724/1c1da8c8/attachment.html>


More information about the Pharo-users mailing list