Dynamic graph exploration - seeking input on idiomatic smalltalk [LONG]
Hi, First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :) I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap. My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate. How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known. I currently have a ruby mockup of this. It's a DSL that looks like this: ----------- EXAMPLE class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124 o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc -------------------------------------------------------- To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes: # NOTE: it's a class level method, hence how I can call it when defining the class in the example def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk? -- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
Hi Patrik, Don't know about all of your specifics but reading through made me think of: 1) Fuel, which has to do with graphs of course (persistence) 2) Moose, which does all kind of fancy things with graphs and visualizations (check Mondrian Easel and the new Roassal based things) 3) Teachable http://lists.squeakfoundation.org/pipermail/squeak-dev/2002-April/038279.htm... Example: | teachable | teachable := self new. teachable whenSend: #help return: 'ok'; whenSend: #doit evaluate: [1 inspect]; acceptSend: #noDebugger; whenSend: #negate: evaluate: [:num | num negated]. teachable help. teachable doit. teachable noDebugger. teachable negate: 120 Teachable may be interesting for adding some stuff dynamically for your models. For the record, I am interested in gold and silver futures in my spare time. Philippe 2012/7/6 Patrik Sundberg <patrik.sundberg@gmail.com>
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be| Web: http://philippeback.eu | Blog: http://philippeback.be High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
On Fri, Jul 6, 2012 at 12:10 PM, phil@highoctane.be <phil@highoctane.be>wrote:
Hi Patrik,
Don't know about all of your specifics but reading through made me think of:
1) Fuel, which has to do with graphs of course (persistence)
Did you mean to look at code for ideas of working with graphs or actual persistence? Actual persistence I've thought long and hard about and considered every type of DB etc out there (incl Magma and GemStone and every NoSQL out there). A domain specific persistence API with a neo4j backend has tunred out to be the best fit so far.
2) Moose, which does all kind of fancy things with graphs and visualizations (check Mondrian Easel and the new Roassal based things)
Yep, visualization wise I've already made a note of Mondrian.
3) Teachable
http://lists.squeakfoundation.org/pipermail/squeak-dev/2002-April/038279.htm...
Example: | teachable | teachable := self new. teachable whenSend: #help return: 'ok'; whenSend: #doit evaluate: [1 inspect]; acceptSend: #noDebugger; whenSend: #negate: evaluate: [:num | num negated]. teachable help. teachable doit. teachable noDebugger. teachable negate: 120
Teachable may be interesting for adding some stuff dynamically for your models.
Ah, I shall definitely have a look at this one!
For the record, I am interested in gold and silver futures in my spare time.
Cool - I've traded most of the commods except base metals. Precious metals is a weird world :) Thanks a lot for the pointers! Patrik
2012/7/6 Patrik Sundberg <patrik.sundberg@gmail.com>
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be| Web: http://philippeback.eu | Blog: http://philippeback.be
High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably .. I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product. It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids.. Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed.. On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg <patrik.sundberg@gmail.com>wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On Fri, Jul 6, 2012 at 12:38 PM, S Krish <krishnamachari.sudhakar@gmail.com> wrote:
You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I've heard of that one. Never worked at JPM so haven't seen it first hand though. I'll see what I can find. In terms of concepts I'm fully on top of it (have a background with other companies using similar ideas, but not in smalltalk but their own in-house language). So the good part is that I know exactly what I want conceptually, but I'm not stuck into smalltalk enough to know the natural way to implement the concepts. It may sound like a huge endeavour but I've done a lot of it in ruby and even as a side project to my real job we're talking about a couple of months to have something useful.
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
I wouldn't say it's the golden standard, that's taking it a bit far, there are others that I rate higher bit I'm also biased :) But I do agree that smalltalk is a reasonably natural platform for these types of concepts, hence why I'm exploring it.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
That's my hunch, that the tooling work would be much cut down. In my current ruby mockup I'm kind of creating a poor man's image like experience - it's jruby on the JVM, vim + vim-slime to send code fragments to the "runtime", git, and most likely homegrown GUI to navigate the graph and play with it (not done anything on that yet). I'll be keeping jruby as a dear tool as right now it's a fantastic glue tool in terms of interacting with anything out there in the world (i.e. I can easily work with the bloomberg java API), but I'm very keen to explore pharo for my graph + tooling around that. (slight tangent - coming from outside the smalltalk world it'd be HUGE to me to have git backend for packages and use of github for collaboration. seen that mentioned in other threads. I think it'd be HUGE for community's ability to attract more people).
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
Cool. Early days yet and we'll see where I end up. Not familiar with those projects, I shall take a look. Patrik
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg <patrik.sundberg@gmail.com
wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
I've done a bit more studying and I think a combination of SystemAnnouncer events to catch method changes, pragmas to mark "graph nodes" and use of method wrappers could create a very nice way to shield the user of my graph API from thinking of identity caches and graph node details. The other cool feature of use is the new class builder and custom slots. For nodes that are not dynamic/calculations (what I call properties or relationship nodes) I think custom slots could be interesting but need to test out some ideas to organize thoughts. Obviously won't work for calculated/dynamic nodes since I need ability to take arguments for those and can't map to a slot. In principle all I'm after is to have an identity map and a graph (DAG) in the background whose management is hidden away from user so he can focus on domain logic and not graph stuff. Going on holiday for a week and a bit, shall start experimenting after that. P.S found some friend of friend who worked on kapital, will explore if they use a graph etc when I get a chance. On Jul 6, 2012 1:16 PM, "Patrik Sundberg" <patrik.sundberg@gmail.com> wrote:
On Fri, Jul 6, 2012 at 12:38 PM, S Krish < krishnamachari.sudhakar@gmail.com> wrote:
You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I've heard of that one. Never worked at JPM so haven't seen it first hand though. I'll see what I can find. In terms of concepts I'm fully on top of it (have a background with other companies using similar ideas, but not in smalltalk but their own in-house language). So the good part is that I know exactly what I want conceptually, but I'm not stuck into smalltalk enough to know the natural way to implement the concepts. It may sound like a huge endeavour but I've done a lot of it in ruby and even as a side project to my real job we're talking about a couple of months to have something useful.
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
I wouldn't say it's the golden standard, that's taking it a bit far, there are others that I rate higher bit I'm also biased :) But I do agree that smalltalk is a reasonably natural platform for these types of concepts, hence why I'm exploring it.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
That's my hunch, that the tooling work would be much cut down. In my current ruby mockup I'm kind of creating a poor man's image like experience - it's jruby on the JVM, vim + vim-slime to send code fragments to the "runtime", git, and most likely homegrown GUI to navigate the graph and play with it (not done anything on that yet). I'll be keeping jruby as a dear tool as right now it's a fantastic glue tool in terms of interacting with anything out there in the world (i.e. I can easily work with the bloomberg java API), but I'm very keen to explore pharo for my graph + tooling around that.
(slight tangent - coming from outside the smalltalk world it'd be HUGE to me to have git backend for packages and use of github for collaboration. seen that mentioned in other threads. I think it'd be HUGE for community's ability to attract more people).
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
Cool. Early days yet and we'll see where I end up. Not familiar with those projects, I shall take a look.
Patrik
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg < patrik.sundberg@gmail.com> wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On Tue, Jul 10, 2012 at 8:15 PM, Patrik Sundberg wrote:
I've done a bit more studying and I think a combination of SystemAnnouncer events to catch method changes, pragmas to mark "graph nodes" and use of method wrappers could create a very nice way to shield the user of my graph API from thinking of identity caches and graph node details.
The other cool feature of use is the new class builder and custom slots. For nodes that are not dynamic/calculations (what I call properties or relationship nodes) I think custom slots could be interesting but need to test out some ideas to organize thoughts. Obviously won't work for calculated/dynamic nodes since I need ability to take arguments for those and can't map to a slot. In principle all I'm after is to have an identity map and a graph (DAG) in the background whose management is hidden away from user so he can focus on domain logic and not graph stuff.
Going on holiday for a week and a bit, shall start experimenting after that.
Ok, so I'm back on this. My first idea is to do this: - Every node in my graph is either a piece of data (I call it a leaf or a property node) or a node depending on other nodes that perform a calculation (I call it a calc node for now) - All the nodes are part of an identity map which means a node can only exist in one copy and any relations and dependencies will sit between these unique nodes - I'd like to shield the user from all the identify map stuff etc as much as possible. - A smalltalk object is an instance of a class, and it can participate in the graph by providing leaf nodes and calc nodes. Those nodes are associated with the messages of the object. - I have some ideas for how to bootstrap the dependency graph between nodes etc and I'm not worried about that bit (on first calculation create structure via reflection and a dependency stack) - The explicit and uggly way I can do things are like this: Foo>>addOneTo: aNumberNode ^GraphManager instance getCalcNode: #addOneTo: for: self withArgs: #( aNumberNode ) block: [ :node | node value + 1 ] - pretend this goes and asks the identity cache and either gets an existing leaf node for the addOneTo: for this object for a particular argument, or if none exist it creates it and attaches the block as the calculation to perform when #value is sent to the node - let's say 'foo' is an instance of Foo class, then 'foo addOneTo: baz' refers to the calc node, and '(foo addOneTo: baz) value' refers to the value of the calculation - What I was thinking here was that I could use method wrappers to hide all the junk, like this: Foo>>addOneTo: aNode <calcNode> aNode value + 1 - I listen to all method add/update/remove announcements, and when a method that is marked with the <calcNode> pragma is involved, what I do is create a MethodWrapper that takes care of all the uggly stuff to get/create the graph node via the identity cache and as #value method for the node uses the method implementation the user originally provided (and replaces the entry in the method dictionary) Does that sound doable? The browsers etc wont go nuts? If I do that, will system browsers still show the original source for the method even though I've replaced it via the method dictionary and MethodWrapper? I'm assuming stack traces and debugging etc will be just fine, will just have the "hidden" node logic via the MethodWrappers included in the call stack. I'll give that type of setup a go, just wanted to throw it out there and hear if it's an obviously terrible idea first. P.S. for leaf nodes (or any node not requiring an argument to be provided) I think I could use custom slots instead of method wrappers and do the same kind of logic as the method wrapper would do in the slot instead. I'll worry about that later. On Jul 6, 2012 1:16 PM, "Patrik Sundberg" wrote:
On Fri, Jul 6, 2012 at 12:38 PM, S Krish wrote:
You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I've heard of that one. Never worked at JPM so haven't seen it first hand though. I'll see what I can find. In terms of concepts I'm fully on top of it (have a background with other companies using similar ideas, but not in smalltalk but their own in-house language). So the good part is that I know exactly what I want conceptually, but I'm not stuck into smalltalk enough to know the natural way to implement the concepts. It may sound like a huge endeavour but I've done a lot of it in ruby and even as a side project to my real job we're talking about a couple of months to have something useful.
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
I wouldn't say it's the golden standard, that's taking it a bit far, there are others that I rate higher bit I'm also biased :) But I do agree that smalltalk is a reasonably natural platform for these types of concepts, hence why I'm exploring it.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
That's my hunch, that the tooling work would be much cut down. In my current ruby mockup I'm kind of creating a poor man's image like experience - it's jruby on the JVM, vim + vim-slime to send code fragments to the "runtime", git, and most likely homegrown GUI to navigate the graph and play with it (not done anything on that yet). I'll be keeping jruby as a dear tool as right now it's a fantastic glue tool in terms of interacting with anything out there in the world (i.e. I can easily work with the bloomberg java API), but I'm very keen to explore pharo for my graph + tooling around that.
(slight tangent - coming from outside the smalltalk world it'd be HUGE to me to have git backend for packages and use of github for collaboration. seen that mentioned in other threads. I think it'd be HUGE for community's ability to attract more people).
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
Cool. Early days yet and we'll see where I end up. Not familiar with those projects, I shall take a look.
Patrik
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
I doubt I can understand the whole of your idea and would aver that a working model / code is lot better for anyone equally interested to load , try and comment on it. The goal is big and nice, utility not yet comprehensible fully, but perhaps a working code will clarify a bit more again to some one in the similar line of action. Go ahead and let the code + screen shots do the talk.. you might have good feedback On Tue, Jul 24, 2012 at 11:20 PM, Patrik Sundberg <patrik.sundberg@gmail.com
wrote:
On Tue, Jul 10, 2012 at 8:15 PM, Patrik Sundberg wrote:
I've done a bit more studying and I think a combination of SystemAnnouncer events to catch method changes, pragmas to mark "graph nodes" and use of method wrappers could create a very nice way to shield the user of my graph API from thinking of identity caches and graph node details.
The other cool feature of use is the new class builder and custom slots. For nodes that are not dynamic/calculations (what I call properties or relationship nodes) I think custom slots could be interesting but need to test out some ideas to organize thoughts. Obviously won't work for calculated/dynamic nodes since I need ability to take arguments for those and can't map to a slot. In principle all I'm after is to have an identity map and a graph (DAG) in the background whose management is hidden away from user so he can focus on domain logic and not graph stuff.
Going on holiday for a week and a bit, shall start experimenting after that.
Ok, so I'm back on this. My first idea is to do this:
- Every node in my graph is either a piece of data (I call it a leaf or a property node) or a node depending on other nodes that perform a calculation (I call it a calc node for now)
- All the nodes are part of an identity map which means a node can only exist in one copy and any relations and dependencies will sit between these unique nodes
- I'd like to shield the user from all the identify map stuff etc as much as possible.
- A smalltalk object is an instance of a class, and it can participate in the graph by providing leaf nodes and calc nodes. Those nodes are associated with the messages of the object.
- I have some ideas for how to bootstrap the dependency graph between nodes etc and I'm not worried about that bit (on first calculation create structure via reflection and a dependency stack)
- The explicit and uggly way I can do things are like this: Foo>>addOneTo: aNumberNode ^GraphManager instance getCalcNode: #addOneTo: for: self withArgs: #( aNumberNode ) block: [ :node | node value + 1 ]
- pretend this goes and asks the identity cache and either gets an existing leaf node for the addOneTo: for this object for a particular argument, or if none exist it creates it and attaches the block as the calculation to perform when #value is sent to the node
- let's say 'foo' is an instance of Foo class, then 'foo addOneTo: baz' refers to the calc node, and '(foo addOneTo: baz) value' refers to the value of the calculation
- What I was thinking here was that I could use method wrappers to hide all the junk, like this: Foo>>addOneTo: aNode <calcNode> aNode value + 1
- I listen to all method add/update/remove announcements, and when a method that is marked with the <calcNode> pragma is involved, what I do is create a MethodWrapper that takes care of all the uggly stuff to get/create the graph node via the identity cache and as #value method for the node uses the method implementation the user originally provided (and replaces the entry in the method dictionary)
Does that sound doable? The browsers etc wont go nuts?
If I do that, will system browsers still show the original source for the method even though I've replaced it via the method dictionary and MethodWrapper?
I'm assuming stack traces and debugging etc will be just fine, will just have the "hidden" node logic via the MethodWrappers included in the call stack.
I'll give that type of setup a go, just wanted to throw it out there and hear if it's an obviously terrible idea first.
P.S. for leaf nodes (or any node not requiring an argument to be provided) I think I could use custom slots instead of method wrappers and do the same kind of logic as the method wrapper would do in the slot instead. I'll worry about that later.
On Jul 6, 2012 1:16 PM, "Patrik Sundberg" wrote:
On Fri, Jul 6, 2012 at 12:38 PM, S Krish wrote:
You should read as much as you can about Kapital of JPMC. It does more
or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I've heard of that one. Never worked at JPM so haven't seen it first hand though. I'll see what I can find. In terms of concepts I'm fully on top of it (have a background with other companies using similar ideas, but not in smalltalk but their own in-house language). So the good part is that I know exactly what I want conceptually, but I'm not stuck into smalltalk enough to know the natural way to implement the concepts. It may sound like a huge endeavour but I've done a lot of it in ruby and even as a side project to my real job we're talking about a couple of months to have something useful.
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
I wouldn't say it's the golden standard, that's taking it a bit far, there are others that I rate higher bit I'm also biased :) But I do agree that smalltalk is a reasonably natural platform for these types of concepts, hence why I'm exploring it.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
That's my hunch, that the tooling work would be much cut down. In my current ruby mockup I'm kind of creating a poor man's image like experience - it's jruby on the JVM, vim + vim-slime to send code fragments to the "runtime", git, and most likely homegrown GUI to navigate the graph and play with it (not done anything on that yet). I'll be keeping jruby as a dear tool as right now it's a fantastic glue tool in terms of interacting with anything out there in the world (i.e. I can easily work with the bloomberg java API), but I'm very keen to explore pharo for my graph + tooling around that.
(slight tangent - coming from outside the smalltalk world it'd be HUGE to me to have git backend for packages and use of github for collaboration. seen that mentioned in other threads. I think it'd be HUGE for community's ability to attract more people).
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
Cool. Early days yet and we'll see where I end up. Not familiar with those projects, I shall take a look.
Patrik
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
On Tue, Jul 24, 2012 at 7:14 PM, S Krish <krishnamachari.sudhakar@gmail.com>wrote:
I doubt I can understand the whole of your idea and would aver that a working model / code is lot better for anyone equally interested to load , try and comment on it.
The goal is big and nice, utility not yet comprehensible fully, but perhaps a working code will clarify a bit more again to some one in the similar line of action.
Go ahead and let the code + screen shots do the talk.. you might have good feedback
yeah, you're right. it's very abstract for someone not in my head - I've been thinking about these concepts for years and i'm "just" trying to translate it into a good smalltalk implementation. i've got a ruby implementation from before but have a hunch smalltalk will be the better fit/experience as soon as I have grokked enough to turn concepts to reality and work effectively in pharo. i'll just start poking at it and see how it goes. thanks On Tue, Jul 24, 2012 at 11:20 PM, Patrik Sundberg <patrik.sundberg@gmail.com
wrote:
On Tue, Jul 10, 2012 at 8:15 PM, Patrik Sundberg wrote:
I've done a bit more studying and I think a combination of SystemAnnouncer events to catch method changes, pragmas to mark "graph nodes" and use of method wrappers could create a very nice way to shield the user of my graph API from thinking of identity caches and graph node details.
The other cool feature of use is the new class builder and custom slots. For nodes that are not dynamic/calculations (what I call properties or relationship nodes) I think custom slots could be interesting but need to test out some ideas to organize thoughts. Obviously won't work for calculated/dynamic nodes since I need ability to take arguments for those and can't map to a slot. In principle all I'm after is to have an identity map and a graph (DAG) in the background whose management is hidden away from user so he can focus on domain logic and not graph stuff.
Going on holiday for a week and a bit, shall start experimenting after that.
Ok, so I'm back on this. My first idea is to do this:
- Every node in my graph is either a piece of data (I call it a leaf or a property node) or a node depending on other nodes that perform a calculation (I call it a calc node for now)
- All the nodes are part of an identity map which means a node can only exist in one copy and any relations and dependencies will sit between these unique nodes
- I'd like to shield the user from all the identify map stuff etc as much as possible.
- A smalltalk object is an instance of a class, and it can participate in the graph by providing leaf nodes and calc nodes. Those nodes are associated with the messages of the object.
- I have some ideas for how to bootstrap the dependency graph between nodes etc and I'm not worried about that bit (on first calculation create structure via reflection and a dependency stack)
- The explicit and uggly way I can do things are like this: Foo>>addOneTo: aNumberNode ^GraphManager instance getCalcNode: #addOneTo: for: self withArgs: #( aNumberNode ) block: [ :node | node value + 1 ]
- pretend this goes and asks the identity cache and either gets an existing leaf node for the addOneTo: for this object for a particular argument, or if none exist it creates it and attaches the block as the calculation to perform when #value is sent to the node
- let's say 'foo' is an instance of Foo class, then 'foo addOneTo: baz' refers to the calc node, and '(foo addOneTo: baz) value' refers to the value of the calculation
- What I was thinking here was that I could use method wrappers to hide all the junk, like this: Foo>>addOneTo: aNode <calcNode> aNode value + 1
- I listen to all method add/update/remove announcements, and when a method that is marked with the <calcNode> pragma is involved, what I do is create a MethodWrapper that takes care of all the uggly stuff to get/create the graph node via the identity cache and as #value method for the node uses the method implementation the user originally provided (and replaces the entry in the method dictionary)
Does that sound doable? The browsers etc wont go nuts?
If I do that, will system browsers still show the original source for the method even though I've replaced it via the method dictionary and MethodWrapper?
I'm assuming stack traces and debugging etc will be just fine, will just have the "hidden" node logic via the MethodWrappers included in the call stack.
I'll give that type of setup a go, just wanted to throw it out there and hear if it's an obviously terrible idea first.
P.S. for leaf nodes (or any node not requiring an argument to be provided) I think I could use custom slots instead of method wrappers and do the same kind of logic as the method wrapper would do in the slot instead. I'll worry about that later.
On Jul 6, 2012 1:16 PM, "Patrik Sundberg" wrote:
On Fri, Jul 6, 2012 at 12:38 PM, S Krish wrote:
You should read as much as you can about Kapital of JPMC. It does more
or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I've heard of that one. Never worked at JPM so haven't seen it first hand though. I'll see what I can find. In terms of concepts I'm fully on top of it (have a background with other companies using similar ideas, but not in smalltalk but their own in-house language). So the good part is that I know exactly what I want conceptually, but I'm not stuck into smalltalk enough to know the natural way to implement the concepts. It may sound like a huge endeavour but I've done a lot of it in ruby and even as a side project to my real job we're talking about a couple of months to have something useful.
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
I wouldn't say it's the golden standard, that's taking it a bit far, there are others that I rate higher bit I'm also biased :) But I do agree that smalltalk is a reasonably natural platform for these types of concepts, hence why I'm exploring it.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
That's my hunch, that the tooling work would be much cut down. In my current ruby mockup I'm kind of creating a poor man's image like experience - it's jruby on the JVM, vim + vim-slime to send code fragments to the "runtime", git, and most likely homegrown GUI to navigate the graph and play with it (not done anything on that yet). I'll be keeping jruby as a dear tool as right now it's a fantastic glue tool in terms of interacting with anything out there in the world (i.e. I can easily work with the bloomberg java API), but I'm very keen to explore pharo for my graph + tooling around that.
(slight tangent - coming from outside the smalltalk world it'd be HUGE to me to have git backend for packages and use of github for collaboration. seen that mentioned in other threads. I think it'd be HUGE for community's ability to attract more people).
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
Cool. Early days yet and we'll see where I end up. Not familiar with those projects, I shall take a look.
Patrik
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
What's that PharoTabletIDE thing? I am doing some work with Pharo on iOS. Is it the platform you use? Phil 2012/7/6 S Krish <krishnamachari.sudhakar@gmail.com>
You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg <patrik.sundberg@gmail.com
wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be| Web: http://philippeback.eu | Blog: http://philippeback.be High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
This is a little old, I have not been able to devote the time reqd to really bring this to usable state. But I am pushing now through weekends to get it out.. The MorphicView: https://plus.google.com/photos/101656219013170129431/albums/5721907895325589... * Trying to upgrade the wizard UI Generator, make it capable of remoting , work with DB based modelling and has a lot of other stuff of usual enterprise apps: localization, singleton/ factory creator, configuration file , logger integrated... Will need a more active few weeks to run these through to usable state.. The PharoTabletIDE: https://plus.google.com/photos/101656219013170129431/albums/5717749739476813... More like a Ubuntu Unity attempt on Pharo..but that is runnable on a tiny core linux with 128 MB RAM. Again as above need to devote few actively concentrated weeks on this.. which has been tough to spare last few months.. On Fri, Jul 6, 2012 at 5:53 PM, phil@highoctane.be <phil@highoctane.be>wrote:
What's that PharoTabletIDE thing?
I am doing some work with Pharo on iOS. Is it the platform you use?
Phil
2012/7/6 S Krish <krishnamachari.sudhakar@gmail.com>
You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg < patrik.sundberg@gmail.com> wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be| Web: http://philippeback.eu | Blog: http://philippeback.be
High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
I like the way you show the code. Is there a configuration for trying that out ? Phil 2012/7/6 S Krish <krishnamachari.sudhakar@gmail.com>
This is a little old, I have not been able to devote the time reqd to really bring this to usable state. But I am pushing now through weekends to get it out..
The MorphicView:
https://plus.google.com/photos/101656219013170129431/albums/5721907895325589...
* Trying to upgrade the wizard UI Generator, make it capable of remoting , work with DB based modelling and has a lot of other stuff of usual enterprise apps: localization, singleton/ factory creator, configuration file , logger integrated...
Will need a more active few weeks to run these through to usable state..
The PharoTabletIDE:
https://plus.google.com/photos/101656219013170129431/albums/5717749739476813...
More like a Ubuntu Unity attempt on Pharo..but that is runnable on a tiny core linux with 128 MB RAM.
Again as above need to devote few actively concentrated weeks on this.. which has been tough to spare last few months..
On Fri, Jul 6, 2012 at 5:53 PM, phil@highoctane.be <phil@highoctane.be>wrote:
What's that PharoTabletIDE thing?
I am doing some work with Pharo on iOS. Is it the platform you use?
Phil
2012/7/6 S Krish <krishnamachari.sudhakar@gmail.com>
You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg < patrik.sundberg@gmail.com> wrote:
Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be | Web: http://philippeback.eu | Blog: http://philippeback.be
High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be| Web: http://philippeback.eu | Blog: http://philippeback.be High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
There is in the PharoGoodies site, b Sudhakar krishnamachari Extn 91-40403012 Cell 9902104814 On Jul 7, 2012, at 12:39 AM, "phil@highoctane.be" <phil@highoctane.be> wrote:
I like the way you show the code. Is there a configuration for trying that out ?
Phil
2012/7/6 S Krish <krishnamachari.sudhakar@gmail.com> This is a little old, I have not been able to devote the time reqd to really bring this to usable state. But I am pushing now through weekends to get it out..
The MorphicView:
https://plus.google.com/photos/101656219013170129431/albums/5721907895325589...
* Trying to upgrade the wizard UI Generator, make it capable of remoting , work with DB based modelling and has a lot of other stuff of usual enterprise apps: localization, singleton/ factory creator, configuration file , logger integrated...
Will need a more active few weeks to run these through to usable state..
The PharoTabletIDE:
https://plus.google.com/photos/101656219013170129431/albums/5717749739476813...
More like a Ubuntu Unity attempt on Pharo..but that is runnable on a tiny core linux with 128 MB RAM.
Again as above need to devote few actively concentrated weeks on this.. which has been tough to spare last few months..
On Fri, Jul 6, 2012 at 5:53 PM, phil@highoctane.be <phil@highoctane.be> wrote: What's that PharoTabletIDE thing?
I am doing some work with Pharo on iOS. Is it the platform you use?
Phil
2012/7/6 S Krish <krishnamachari.sudhakar@gmail.com> You should read as much as you can about Kapital of JPMC. It does more or less what you describe for esoteric Prime Interest Derivatives and very scalably ..
I doubt if there is much in the public domain, but the fact that Kapital is the golden standard in this product line talks highly of why Smalltalk is THE platform for this kind of product.
It would be nice to have a Pharo based derivative product that can easily beat Ruby/ Java / .Net at this dynamic visualization using standard browsers, where Nautilus too can serve your purpose.. or a home grown tree based browser.../ Grids..
Once I have completed my run at a PharoTabletIDE/ PharoMorphicView framework, I would love to collaborate on your endeavour to see if it can be modelled on it and exposed..
On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg <patrik.sundberg@gmail.com> wrote: Hi,
First of all, sorry for the long email as my first email to the list. Hopefully some people will find it an interesting discussion :)
I'm a long time programmer that has been studying smalltalk and pharo over the last year. It's a beautiful language I'm liking the image style of development a whole lot. I've been looking for a good test case to try out in pharo and I've got something in mind. I've used ruby since around 2000 and given how much it has been inspired by smalltalk it's not a big leap.
My real job is being a commodities trader but I build my own tools and something I'm always working on is improved risk and pricing tools. Lots of that comes down to 2 things: 1. evaluating a "function" (in a general sense) with inputs that are picked depending on time. e.g. I want to price a security and I want to pick up prices for time X as inputs to the pricing model. 2. calculating a derivative of the "function" above with respect to inputs
What I tend to do in order to keep things consistent by design, plus add natural caching, is create a calculation graph. I define all the dependencies and create an in-memory graph where each node is unique using an identity map. I then perform a calculation where the calculations trickle down the graph starting from the node I'm interested in down all it's dependencies. To calculate the derivative I then find the node representing the input I want the derivative with respect to and use finite differences, e.g. move it's value, and recalculate the top level node. On the second valuation of the graph only the parts that are affected by me changing that 1 value will need to be evaluated, the rest is cached an unaffected. It makes it very easy to ask questions like "How much will I be affected by X changing by Y?" since I can take the top level node of the graph, search for X, and if found change it by Y and recalculate.
How the graph is constructed in memory depends on time (and a few other things). I always store all previous states of the world so that I can recreate any calculation I did in the past. Very useful for diagnosing problem. Hence if I for example change what model I use to model something, that change is recorded in a way such that if I set my "time" to before the change happened the graph will get created as it would have with the old model, and for a time after the change it'll create a different graph reflecting the new model. Hence the graph can only be known at runtime when the "time" etc is known.
I currently have a ruby mockup of this. It's a DSL that looks like this:
----------- EXAMPLE
class ExampleObject include GraphEntity # includes a module with functionality for objects participating in the graph, including #property and #calc used below
property :foo do 123 # this is the default value for this property which will be used before a value has been set and saved end
calc :bar do |arg| foo.value + arg # take the value of the arg property and add the argument given to the calculation node end end
o = ExampleObject.new # this will kick of the graph being built in memory and setup the dependency between bar and foo nodes o.bar(1).value # -> 124 # this will be a "cache lookup" since nothing in the graph that bar depends on has changed (i.e. the "expensive" calculation is not performed again o.bar(1).value # -> 124
o.foo.set_value(2) o.bar(1).value # -> 125, realizes that foo changed and performs a recalc --------------------------------------------------------
To accomplish this I use dynamic code generation like the below for #property and similar for other types of nodes:
# NOTE: it's a class level method, hence how I can call it when defining the class in the example
def property(name, time = CdrEnv.instance.time, &block) clear_property(name) getter_method_body = <<-EOF def #{name} init_block = self.class.property_init_block_for(:#{name}) time_for_property = self.class.property_time_dependency_for(:#{name}) if init_block.nil? CdrEnv.instance.get_property_node(self, :#{name}, time_for_property) else CdrEnv.instance.get_property_node(self, :#{name}, time_for_property, &init_block) end end EOF setter_method_body = <<-EOF def #{name.to_s}=(value) #{name.to_s}.mutate(value) end EOF class_eval getter_method_body class_eval setter_method_body register_property_node(name, time, &block) end
Don't worry about the details or all the unfamiliar ruby, the main point is that it creates an instance method that uses the singleton CdrEnv.instance to either get or create the node representing the property from the graph identity cache depending on if it exists or not.
From a tooling point of view I think I'd love to work with this kind of thing in pharo. Building my own browsers for inspecting and debugging my dynamic graph should be very good fit. However, I'd appreciate some pointers to idiomatic smalltalk to attack this kind of problem in terms of implementing the graph itself - I obviously want the user (even if it's me) to just have to focus on the domain model and hide as much as possible of the graph bits under the covers the same way I've done with the code generation stuff in my ruby example.
Any input into this would be much appreciated, Patrik
P.S. Btw, the persistence backend for this is the neo4j graph database, but it's fronted by a service slotting into my own service framework built using ZeroMQ and services exchanging messages in protobuf format. One can use it from any language as long as one can send protobuf messages over zeromq. I see there's a zeromq ffi library available on SS that I'll check out, but I'm not finding a protobuf implementation. It'd be easy enough for me to port a ruby protobuf implementation but I may as well ask if someone has already done any work on protobuf for smalltalk?
-- View this message in context: http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-s... Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be | Web: http://philippeback.eu | Blog: http://philippeback.be
High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
-- Philippe Back Dramatic Performance Improvements Mob: +32(0) 478 650 140 | Fax: +32 (0) 70 408 027 Mail: phil@highoctane.be | Web: http://philippeback.eu | Blog: http://philippeback.be
High Octane SPRL rue cour Boisacq 101 1301 Bierges Belgium
participants (4)
-
Krishsmalltalk -
Patrik Sundberg -
phil＠highoctane.be -
S Krish