July 2012 - Pharo-users - lists.pharo.org

Re: [Pharo-users] Dynamic graph exploration - seeking input on idiomatic smalltalk [LONG]

by Patrik Sundberg

On Tue, Jul 24, 2012 at 7:14 PM, S Krish <krishnamachari.sudhakar(a)gmail.com>wrote: > I doubt I can understand the whole of your idea and would aver that a > working model / code is lot better for anyone equally interested to load , > try and comment on it. > > The goal is big and nice, utility not yet comprehensible fully, but > perhaps a working code will clarify a bit more again to some one in the > similar line of action. > > Go ahead and let the code + screen shots do the talk.. you might have good > feedback > > yeah, you're right. it's very abstract for someone not in my head - I've been thinking about these concepts for years and i'm "just" trying to translate it into a good smalltalk implementation. i've got a ruby implementation from before but have a hunch smalltalk will be the better fit/experience as soon as I have grokked enough to turn concepts to reality and work effectively in pharo. i'll just start poking at it and see how it goes. thanks On Tue, Jul 24, 2012 at 11:20 PM, Patrik Sundberg <patrik.sundberg(a)gmail.com > > wrote: > >> On Tue, Jul 10, 2012 at 8:15 PM, Patrik Sundberg wrote: >> >>> I've done a bit more studying and I think a combination of >>> SystemAnnouncer events to catch method changes, pragmas to mark "graph >>> nodes" and use of method wrappers could create a very nice way to shield >>> the user of my graph API from thinking of identity caches and graph node >>> details. >>> >>> The other cool feature of use is the new class builder and custom slots. >>> For nodes that are not dynamic/calculations (what I call properties or >>> relationship nodes) I think custom slots could be interesting but need to >>> test out some ideas to organize thoughts. Obviously won't work for >>> calculated/dynamic nodes since I need ability to take arguments for those >>> and can't map to a slot. In principle all I'm after is to have an identity >>> map and a graph (DAG) in the background whose management is hidden away >>> from user so he can focus on domain logic and not graph stuff. >>> >>> Going on holiday for a week and a bit, shall start experimenting after >>> that. >>> >> >> Ok, so I'm back on this. My first idea is to do this: >> >> - Every node in my graph is either a piece of data (I call it a leaf or a >> property node) or a node depending on other nodes that perform a >> calculation (I call it a calc node for now) >> >> - All the nodes are part of an identity map which means a node can only >> exist in one copy and any relations and dependencies will sit between these >> unique nodes >> >> - I'd like to shield the user from all the identify map stuff etc as much >> as possible. >> >> - A smalltalk object is an instance of a class, and it can participate in >> the graph by providing leaf nodes and calc nodes. Those nodes are >> associated with the messages of the object. >> >> - I have some ideas for how to bootstrap the dependency graph between >> nodes etc and I'm not worried about that bit (on first calculation create >> structure via reflection and a dependency stack) >> >> - The explicit and uggly way I can do things are like this: >> Foo>>addOneTo: aNumberNode >> ^GraphManager instance >> getCalcNode: #addOneTo: >> for: self >> withArgs: #( aNumberNode ) >> block: [ :node | >> node value + 1 >> ] >> >> - pretend this goes and asks the identity cache and either gets an >> existing leaf node for the addOneTo: for this object for a particular >> argument, or if none exist it creates it and attaches the block as the >> calculation to perform when #value is sent to the node >> >> - let's say 'foo' is an instance of Foo class, then 'foo addOneTo: baz' >> refers to the calc node, and '(foo addOneTo: baz) value' refers to the >> value of the calculation >> >> - What I was thinking here was that I could use method wrappers to hide >> all the junk, like this: >> Foo>>addOneTo: aNode >> <calcNode> >> aNode value + 1 >> >> - I listen to all method add/update/remove announcements, and when a >> method that is marked with the <calcNode> pragma is involved, what I do is >> create a MethodWrapper that takes care of all the uggly stuff to get/create >> the graph node via the identity cache and as #value method for the node >> uses the method implementation the user originally provided (and replaces >> the entry in the method dictionary) >> >> Does that sound doable? The browsers etc wont go nuts? >> >> If I do that, will system browsers still show the original source for the >> method even though I've replaced it via the method dictionary and >> MethodWrapper? >> >> I'm assuming stack traces and debugging etc will be just fine, will just >> have the "hidden" node logic via the MethodWrappers included in the call >> stack. >> >> I'll give that type of setup a go, just wanted to throw it out there and >> hear if it's an obviously terrible idea first. >> >> >> P.S. for leaf nodes (or any node not requiring an argument to be >> provided) I think I could use custom slots instead of method wrappers and >> do the same kind of logic as the method wrapper would do in the slot >> instead. I'll worry about that later. >> >> >> >> On Jul 6, 2012 1:16 PM, "Patrik Sundberg" wrote: >> >>> On Fri, Jul 6, 2012 at 12:38 PM, S Krish wrote: >>>> >>>> You should read as much as you can about Kapital of JPMC. It does more >>>>> or less what you describe for esoteric Prime Interest Derivatives and very >>>>> scalably .. >>>>> >>>>> >>>> I've heard of that one. Never worked at JPM so haven't seen it first >>>> hand though. I'll see what I can find. In terms of concepts I'm fully on >>>> top of it (have a background with other companies using similar ideas, but >>>> not in smalltalk but their own in-house language). So the good part is that >>>> I know exactly what I want conceptually, but I'm not stuck into smalltalk >>>> enough to know the natural way to implement the concepts. It may sound like >>>> a huge endeavour but I've done a lot of it in ruby and even as a side >>>> project to my real job we're talking about a couple of months to have >>>> something useful. >>>> >>>> >>>>> I doubt if there is much in the public domain, but the fact that >>>>> Kapital is the golden standard in this product line talks highly of why >>>>> Smalltalk is THE platform for this kind of product. >>>>> >>>>> >>>> I wouldn't say it's the golden standard, that's taking it a bit far, >>>> there are others that I rate higher bit I'm also biased :) But I do agree >>>> that smalltalk is a reasonably natural platform for these types of >>>> concepts, hence why I'm exploring it. >>>> >>>> >>>>> It would be nice to have a Pharo based derivative product that can >>>>> easily beat Ruby/ Java / .Net at this dynamic visualization using standard >>>>> browsers, where Nautilus too can serve your purpose.. or a home grown tree >>>>> based browser.../ Grids.. >>>>> >>>>> >>>> That's my hunch, that the tooling work would be much cut down. In my >>>> current ruby mockup I'm kind of creating a poor man's image like experience >>>> - it's jruby on the JVM, vim + vim-slime to send code fragments to the >>>> "runtime", git, and most likely homegrown GUI to navigate the graph and >>>> play with it (not done anything on that yet). I'll be keeping jruby as a >>>> dear tool as right now it's a fantastic glue tool in terms of interacting >>>> with anything out there in the world (i.e. I can easily work with the >>>> bloomberg java API), but I'm very keen to explore pharo for my graph + >>>> tooling around that. >>>> >>>> (slight tangent - coming from outside the smalltalk world it'd be HUGE >>>> to me to have git backend for packages and use of github for collaboration. >>>> seen that mentioned in other threads. I think it'd be HUGE for community's >>>> ability to attract more people). >>>> >>>> >>>> >>>>> Once I have completed my run at a PharoTabletIDE/ PharoMorphicView >>>>> framework, I would love to collaborate on your endeavour to see if it can >>>>> be modelled on it and exposed.. >>>>> >>>>> >>>> Cool. Early days yet and we'll see where I end up. Not familiar with >>>> those projects, I shall take a look. >>>> >>>> >>>> Patrik >>>> >>>> >>>> >>>> >>>>> On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> First of all, sorry for the long email as my first email to the list. >>>>>> Hopefully some people will find it an interesting discussion :) >>>>>> >>>>>> I'm a long time programmer that has been studying smalltalk and pharo >>>>>> over >>>>>> the last year. It's a beautiful language I'm liking the image style of >>>>>> development a whole lot. I've been looking for a good test case to >>>>>> try out >>>>>> in pharo and I've got something in mind. I've used ruby since around >>>>>> 2000 >>>>>> and given how much it has been inspired by smalltalk it's not a big >>>>>> leap. >>>>>> >>>>>> My real job is being a commodities trader but I build my own tools and >>>>>> something I'm always working on is improved risk and pricing tools. >>>>>> Lots of >>>>>> that comes down to 2 things: >>>>>> 1. evaluating a "function" (in a general sense) with inputs that are >>>>>> picked >>>>>> depending on time. e.g. I want to price a security and I want to pick >>>>>> up >>>>>> prices for time X as inputs to the pricing model. >>>>>> 2. calculating a derivative of the "function" above with respect to >>>>>> inputs >>>>>> >>>>>> What I tend to do in order to keep things consistent by design, plus >>>>>> add >>>>>> natural caching, is create a calculation graph. I define all the >>>>>> dependencies and create an in-memory graph where each node is unique >>>>>> using >>>>>> an identity map. I then perform a calculation where the calculations >>>>>> trickle >>>>>> down the graph starting from the node I'm interested in down all it's >>>>>> dependencies. To calculate the derivative I then find the node >>>>>> representing >>>>>> the input I want the derivative with respect to and use finite >>>>>> differences, >>>>>> e.g. move it's value, and recalculate the top level node. On the >>>>>> second >>>>>> valuation of the graph only the parts that are affected by me >>>>>> changing that >>>>>> 1 value will need to be evaluated, the rest is cached an unaffected. >>>>>> It >>>>>> makes it very easy to ask questions like "How much will I be affected >>>>>> by X >>>>>> changing by Y?" since I can take the top level node of the graph, >>>>>> search for >>>>>> X, and if found change it by Y and recalculate. >>>>>> >>>>>> How the graph is constructed in memory depends on time (and a few >>>>>> other >>>>>> things). I always store all previous states of the world so that I can >>>>>> recreate any calculation I did in the past. Very useful for diagnosing >>>>>> problem. Hence if I for example change what model I use to model >>>>>> something, >>>>>> that change is recorded in a way such that if I set my "time" to >>>>>> before the >>>>>> change happened the graph will get created as it would have with the >>>>>> old >>>>>> model, and for a time after the change it'll create a different graph >>>>>> reflecting the new model. Hence the graph can only be known at >>>>>> runtime when >>>>>> the "time" etc is known. >>>>>> >>>>>> I currently have a ruby mockup of this. It's a DSL that looks like >>>>>> this: >>>>>> >>>>>> ----------- EXAMPLE >>>>>> >>>>>> class ExampleObject >>>>>> include GraphEntity # includes a module with functionality for >>>>>> objects >>>>>> participating in the graph, including #property and #calc used below >>>>>> >>>>>> property :foo do >>>>>> 123 # this is the default value for this property which will be >>>>>> used >>>>>> before a value has been set and saved >>>>>> end >>>>>> >>>>>> calc :bar do |arg| >>>>>> foo.value + arg # take the value of the arg property and add >>>>>> the >>>>>> argument given to the calculation node >>>>>> end >>>>>> end >>>>>> >>>>>> o = ExampleObject.new >>>>>> # this will kick of the graph being built in memory and setup the >>>>>> dependency >>>>>> between bar and foo nodes >>>>>> o.bar(1).value # -> 124 >>>>>> # this will be a "cache lookup" since nothing in the graph that bar >>>>>> depends >>>>>> on has changed (i.e. the "expensive" calculation is not performed >>>>>> again >>>>>> o.bar(1).value # -> 124 >>>>>> >>>>>> o.foo.set_value(2) >>>>>> o.bar(1).value # -> 125, realizes that foo changed and performs a >>>>>> recalc >>>>>> -------------------------------------------------------- >>>>>> >>>>>> To accomplish this I use dynamic code generation like the below for >>>>>> #property and similar for other types of nodes: >>>>>> >>>>>> # NOTE: it's a class level method, hence how I can call it >>>>>> when >>>>>> defining the class in the example >>>>>> >>>>>> def property(name, time = CdrEnv.instance.time, &block) >>>>>> clear_property(name) >>>>>> getter_method_body = <<-EOF >>>>>> def #{name} >>>>>> init_block = >>>>>> self.class.property_init_block_for(:#{name}) >>>>>> time_for_property = >>>>>> self.class.property_time_dependency_for(:#{name}) >>>>>> if init_block.nil? >>>>>> CdrEnv.instance.get_property_node(self, :#{name}, >>>>>> time_for_property) >>>>>> else >>>>>> CdrEnv.instance.get_property_node(self, :#{name}, >>>>>> time_for_property, &init_block) >>>>>> end >>>>>> end >>>>>> EOF >>>>>> setter_method_body = <<-EOF >>>>>> def #{name.to_s}=(value) >>>>>> #{name.to_s}.mutate(value) >>>>>> end >>>>>> EOF >>>>>> class_eval getter_method_body >>>>>> class_eval setter_method_body >>>>>> register_property_node(name, time, &block) >>>>>> end >>>>>> >>>>>> Don't worry about the details or all the unfamiliar ruby, the main >>>>>> point is >>>>>> that it creates an instance method that uses the singleton >>>>>> CdrEnv.instance >>>>>> to either get or create the node representing the property from the >>>>>> graph >>>>>> identity cache depending on if it exists or not. >>>>>> >>>>>> From a tooling point of view I think I'd love to work with this kind >>>>>> of >>>>>> thing in pharo. Building my own browsers for inspecting and debugging >>>>>> my >>>>>> dynamic graph should be very good fit. However, I'd appreciate some >>>>>> pointers >>>>>> to idiomatic smalltalk to attack this kind of problem in terms of >>>>>> implementing the graph itself - I obviously want the user (even if >>>>>> it's me) >>>>>> to just have to focus on the domain model and hide as much as >>>>>> possible of >>>>>> the graph bits under the covers the same way I've done with the code >>>>>> generation stuff in my ruby example. >>>>>> >>>>>> Any input into this would be much appreciated, >>>>>> Patrik >>>>>> >>>>>> P.S. >>>>>> Btw, the persistence backend for this is the neo4j graph database, >>>>>> but it's >>>>>> fronted by a service slotting into my own service framework built >>>>>> using >>>>>> ZeroMQ and services exchanging messages in protobuf format. One can >>>>>> use it >>>>>> from any language as long as one can send protobuf messages over >>>>>> zeromq. I >>>>>> see there's a zeromq ffi library available on SS that I'll check out, >>>>>> but >>>>>> I'm not finding a protobuf implementation. It'd be easy enough for me >>>>>> to >>>>>> port a ruby protobuf implementation but I may as well ask if someone >>>>>> has >>>>>> already done any work on protobuf for smalltalk? >>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-… >>>>>> Sent from the Pharo Smalltalk Users mailing list archive at >>>>>> Nabble.com. >>>>>> >>>>>> >>>>> >>>> >> >

July 24, 2012

Re: [Pharo-users] Dynamic graph exploration - seeking input on idiomatic smalltalk [LONG]

by S Krish

I doubt I can understand the whole of your idea and would aver that a working model / code is lot better for anyone equally interested to load , try and comment on it. The goal is big and nice, utility not yet comprehensible fully, but perhaps a working code will clarify a bit more again to some one in the similar line of action. Go ahead and let the code + screen shots do the talk.. you might have good feedback On Tue, Jul 24, 2012 at 11:20 PM, Patrik Sundberg <patrik.sundberg(a)gmail.com > wrote: > On Tue, Jul 10, 2012 at 8:15 PM, Patrik Sundberg wrote: > >> I've done a bit more studying and I think a combination of >> SystemAnnouncer events to catch method changes, pragmas to mark "graph >> nodes" and use of method wrappers could create a very nice way to shield >> the user of my graph API from thinking of identity caches and graph node >> details. >> >> The other cool feature of use is the new class builder and custom slots. >> For nodes that are not dynamic/calculations (what I call properties or >> relationship nodes) I think custom slots could be interesting but need to >> test out some ideas to organize thoughts. Obviously won't work for >> calculated/dynamic nodes since I need ability to take arguments for those >> and can't map to a slot. In principle all I'm after is to have an identity >> map and a graph (DAG) in the background whose management is hidden away >> from user so he can focus on domain logic and not graph stuff. >> >> Going on holiday for a week and a bit, shall start experimenting after >> that. >> > > Ok, so I'm back on this. My first idea is to do this: > > - Every node in my graph is either a piece of data (I call it a leaf or a > property node) or a node depending on other nodes that perform a > calculation (I call it a calc node for now) > > - All the nodes are part of an identity map which means a node can only > exist in one copy and any relations and dependencies will sit between these > unique nodes > > - I'd like to shield the user from all the identify map stuff etc as much > as possible. > > - A smalltalk object is an instance of a class, and it can participate in > the graph by providing leaf nodes and calc nodes. Those nodes are > associated with the messages of the object. > > - I have some ideas for how to bootstrap the dependency graph between > nodes etc and I'm not worried about that bit (on first calculation create > structure via reflection and a dependency stack) > > - The explicit and uggly way I can do things are like this: > Foo>>addOneTo: aNumberNode > ^GraphManager instance > getCalcNode: #addOneTo: > for: self > withArgs: #( aNumberNode ) > block: [ :node | > node value + 1 > ] > > - pretend this goes and asks the identity cache and either gets an > existing leaf node for the addOneTo: for this object for a particular > argument, or if none exist it creates it and attaches the block as the > calculation to perform when #value is sent to the node > > - let's say 'foo' is an instance of Foo class, then 'foo addOneTo: baz' > refers to the calc node, and '(foo addOneTo: baz) value' refers to the > value of the calculation > > - What I was thinking here was that I could use method wrappers to hide > all the junk, like this: > Foo>>addOneTo: aNode > <calcNode> > aNode value + 1 > > - I listen to all method add/update/remove announcements, and when a > method that is marked with the <calcNode> pragma is involved, what I do is > create a MethodWrapper that takes care of all the uggly stuff to get/create > the graph node via the identity cache and as #value method for the node > uses the method implementation the user originally provided (and replaces > the entry in the method dictionary) > > Does that sound doable? The browsers etc wont go nuts? > > If I do that, will system browsers still show the original source for the > method even though I've replaced it via the method dictionary and > MethodWrapper? > > I'm assuming stack traces and debugging etc will be just fine, will just > have the "hidden" node logic via the MethodWrappers included in the call > stack. > > I'll give that type of setup a go, just wanted to throw it out there and > hear if it's an obviously terrible idea first. > > > P.S. for leaf nodes (or any node not requiring an argument to be provided) > I think I could use custom slots instead of method wrappers and do the same > kind of logic as the method wrapper would do in the slot instead. I'll > worry about that later. > > > > On Jul 6, 2012 1:16 PM, "Patrik Sundberg" wrote: > >> On Fri, Jul 6, 2012 at 12:38 PM, S Krish wrote: >>> >>> You should read as much as you can about Kapital of JPMC. It does more >>>> or less what you describe for esoteric Prime Interest Derivatives and very >>>> scalably .. >>>> >>>> >>> I've heard of that one. Never worked at JPM so haven't seen it first >>> hand though. I'll see what I can find. In terms of concepts I'm fully on >>> top of it (have a background with other companies using similar ideas, but >>> not in smalltalk but their own in-house language). So the good part is that >>> I know exactly what I want conceptually, but I'm not stuck into smalltalk >>> enough to know the natural way to implement the concepts. It may sound like >>> a huge endeavour but I've done a lot of it in ruby and even as a side >>> project to my real job we're talking about a couple of months to have >>> something useful. >>> >>> >>>> I doubt if there is much in the public domain, but the fact that >>>> Kapital is the golden standard in this product line talks highly of why >>>> Smalltalk is THE platform for this kind of product. >>>> >>>> >>> I wouldn't say it's the golden standard, that's taking it a bit far, >>> there are others that I rate higher bit I'm also biased :) But I do agree >>> that smalltalk is a reasonably natural platform for these types of >>> concepts, hence why I'm exploring it. >>> >>> >>>> It would be nice to have a Pharo based derivative product that can >>>> easily beat Ruby/ Java / .Net at this dynamic visualization using standard >>>> browsers, where Nautilus too can serve your purpose.. or a home grown tree >>>> based browser.../ Grids.. >>>> >>>> >>> That's my hunch, that the tooling work would be much cut down. In my >>> current ruby mockup I'm kind of creating a poor man's image like experience >>> - it's jruby on the JVM, vim + vim-slime to send code fragments to the >>> "runtime", git, and most likely homegrown GUI to navigate the graph and >>> play with it (not done anything on that yet). I'll be keeping jruby as a >>> dear tool as right now it's a fantastic glue tool in terms of interacting >>> with anything out there in the world (i.e. I can easily work with the >>> bloomberg java API), but I'm very keen to explore pharo for my graph + >>> tooling around that. >>> >>> (slight tangent - coming from outside the smalltalk world it'd be HUGE >>> to me to have git backend for packages and use of github for collaboration. >>> seen that mentioned in other threads. I think it'd be HUGE for community's >>> ability to attract more people). >>> >>> >>> >>>> Once I have completed my run at a PharoTabletIDE/ PharoMorphicView >>>> framework, I would love to collaborate on your endeavour to see if it can >>>> be modelled on it and exposed.. >>>> >>>> >>> Cool. Early days yet and we'll see where I end up. Not familiar with >>> those projects, I shall take a look. >>> >>> >>> Patrik >>> >>> >>> >>> >>>> On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg wrote: >>>> >>>>> Hi, >>>>> >>>>> First of all, sorry for the long email as my first email to the list. >>>>> Hopefully some people will find it an interesting discussion :) >>>>> >>>>> I'm a long time programmer that has been studying smalltalk and pharo >>>>> over >>>>> the last year. It's a beautiful language I'm liking the image style of >>>>> development a whole lot. I've been looking for a good test case to try >>>>> out >>>>> in pharo and I've got something in mind. I've used ruby since around >>>>> 2000 >>>>> and given how much it has been inspired by smalltalk it's not a big >>>>> leap. >>>>> >>>>> My real job is being a commodities trader but I build my own tools and >>>>> something I'm always working on is improved risk and pricing tools. >>>>> Lots of >>>>> that comes down to 2 things: >>>>> 1. evaluating a "function" (in a general sense) with inputs that are >>>>> picked >>>>> depending on time. e.g. I want to price a security and I want to pick >>>>> up >>>>> prices for time X as inputs to the pricing model. >>>>> 2. calculating a derivative of the "function" above with respect to >>>>> inputs >>>>> >>>>> What I tend to do in order to keep things consistent by design, plus >>>>> add >>>>> natural caching, is create a calculation graph. I define all the >>>>> dependencies and create an in-memory graph where each node is unique >>>>> using >>>>> an identity map. I then perform a calculation where the calculations >>>>> trickle >>>>> down the graph starting from the node I'm interested in down all it's >>>>> dependencies. To calculate the derivative I then find the node >>>>> representing >>>>> the input I want the derivative with respect to and use finite >>>>> differences, >>>>> e.g. move it's value, and recalculate the top level node. On the second >>>>> valuation of the graph only the parts that are affected by me changing >>>>> that >>>>> 1 value will need to be evaluated, the rest is cached an unaffected. It >>>>> makes it very easy to ask questions like "How much will I be affected >>>>> by X >>>>> changing by Y?" since I can take the top level node of the graph, >>>>> search for >>>>> X, and if found change it by Y and recalculate. >>>>> >>>>> How the graph is constructed in memory depends on time (and a few other >>>>> things). I always store all previous states of the world so that I can >>>>> recreate any calculation I did in the past. Very useful for diagnosing >>>>> problem. Hence if I for example change what model I use to model >>>>> something, >>>>> that change is recorded in a way such that if I set my "time" to >>>>> before the >>>>> change happened the graph will get created as it would have with the >>>>> old >>>>> model, and for a time after the change it'll create a different graph >>>>> reflecting the new model. Hence the graph can only be known at runtime >>>>> when >>>>> the "time" etc is known. >>>>> >>>>> I currently have a ruby mockup of this. It's a DSL that looks like >>>>> this: >>>>> >>>>> ----------- EXAMPLE >>>>> >>>>> class ExampleObject >>>>> include GraphEntity # includes a module with functionality for >>>>> objects >>>>> participating in the graph, including #property and #calc used below >>>>> >>>>> property :foo do >>>>> 123 # this is the default value for this property which will be >>>>> used >>>>> before a value has been set and saved >>>>> end >>>>> >>>>> calc :bar do |arg| >>>>> foo.value + arg # take the value of the arg property and add >>>>> the >>>>> argument given to the calculation node >>>>> end >>>>> end >>>>> >>>>> o = ExampleObject.new >>>>> # this will kick of the graph being built in memory and setup the >>>>> dependency >>>>> between bar and foo nodes >>>>> o.bar(1).value # -> 124 >>>>> # this will be a "cache lookup" since nothing in the graph that bar >>>>> depends >>>>> on has changed (i.e. the "expensive" calculation is not performed again >>>>> o.bar(1).value # -> 124 >>>>> >>>>> o.foo.set_value(2) >>>>> o.bar(1).value # -> 125, realizes that foo changed and performs a >>>>> recalc >>>>> -------------------------------------------------------- >>>>> >>>>> To accomplish this I use dynamic code generation like the below for >>>>> #property and similar for other types of nodes: >>>>> >>>>> # NOTE: it's a class level method, hence how I can call it when >>>>> defining the class in the example >>>>> >>>>> def property(name, time = CdrEnv.instance.time, &block) >>>>> clear_property(name) >>>>> getter_method_body = <<-EOF >>>>> def #{name} >>>>> init_block = self.class.property_init_block_for(:#{name}) >>>>> time_for_property = >>>>> self.class.property_time_dependency_for(:#{name}) >>>>> if init_block.nil? >>>>> CdrEnv.instance.get_property_node(self, :#{name}, >>>>> time_for_property) >>>>> else >>>>> CdrEnv.instance.get_property_node(self, :#{name}, >>>>> time_for_property, &init_block) >>>>> end >>>>> end >>>>> EOF >>>>> setter_method_body = <<-EOF >>>>> def #{name.to_s}=(value) >>>>> #{name.to_s}.mutate(value) >>>>> end >>>>> EOF >>>>> class_eval getter_method_body >>>>> class_eval setter_method_body >>>>> register_property_node(name, time, &block) >>>>> end >>>>> >>>>> Don't worry about the details or all the unfamiliar ruby, the main >>>>> point is >>>>> that it creates an instance method that uses the singleton >>>>> CdrEnv.instance >>>>> to either get or create the node representing the property from the >>>>> graph >>>>> identity cache depending on if it exists or not. >>>>> >>>>> From a tooling point of view I think I'd love to work with this kind of >>>>> thing in pharo. Building my own browsers for inspecting and debugging >>>>> my >>>>> dynamic graph should be very good fit. However, I'd appreciate some >>>>> pointers >>>>> to idiomatic smalltalk to attack this kind of problem in terms of >>>>> implementing the graph itself - I obviously want the user (even if >>>>> it's me) >>>>> to just have to focus on the domain model and hide as much as possible >>>>> of >>>>> the graph bits under the covers the same way I've done with the code >>>>> generation stuff in my ruby example. >>>>> >>>>> Any input into this would be much appreciated, >>>>> Patrik >>>>> >>>>> P.S. >>>>> Btw, the persistence backend for this is the neo4j graph database, but >>>>> it's >>>>> fronted by a service slotting into my own service framework built using >>>>> ZeroMQ and services exchanging messages in protobuf format. One can >>>>> use it >>>>> from any language as long as one can send protobuf messages over >>>>> zeromq. I >>>>> see there's a zeromq ffi library available on SS that I'll check out, >>>>> but >>>>> I'm not finding a protobuf implementation. It'd be easy enough for me >>>>> to >>>>> port a ruby protobuf implementation but I may as well ask if someone >>>>> has >>>>> already done any work on protobuf for smalltalk? >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-… >>>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>> >

July 24, 2012

Re: [Pharo-users] Dynamic graph exploration - seeking input on idiomatic smalltalk [LONG]

by Patrik Sundberg

On Tue, Jul 10, 2012 at 8:15 PM, Patrik Sundberg wrote: > I've done a bit more studying and I think a combination of SystemAnnouncer > events to catch method changes, pragmas to mark "graph nodes" and use of > method wrappers could create a very nice way to shield the user of my graph > API from thinking of identity caches and graph node details. > > The other cool feature of use is the new class builder and custom slots. > For nodes that are not dynamic/calculations (what I call properties or > relationship nodes) I think custom slots could be interesting but need to > test out some ideas to organize thoughts. Obviously won't work for > calculated/dynamic nodes since I need ability to take arguments for those > and can't map to a slot. In principle all I'm after is to have an identity > map and a graph (DAG) in the background whose management is hidden away > from user so he can focus on domain logic and not graph stuff. > > Going on holiday for a week and a bit, shall start experimenting after > that. > Ok, so I'm back on this. My first idea is to do this: - Every node in my graph is either a piece of data (I call it a leaf or a property node) or a node depending on other nodes that perform a calculation (I call it a calc node for now) - All the nodes are part of an identity map which means a node can only exist in one copy and any relations and dependencies will sit between these unique nodes - I'd like to shield the user from all the identify map stuff etc as much as possible. - A smalltalk object is an instance of a class, and it can participate in the graph by providing leaf nodes and calc nodes. Those nodes are associated with the messages of the object. - I have some ideas for how to bootstrap the dependency graph between nodes etc and I'm not worried about that bit (on first calculation create structure via reflection and a dependency stack) - The explicit and uggly way I can do things are like this: Foo>>addOneTo: aNumberNode ^GraphManager instance getCalcNode: #addOneTo: for: self withArgs: #( aNumberNode ) block: [ :node | node value + 1 ] - pretend this goes and asks the identity cache and either gets an existing leaf node for the addOneTo: for this object for a particular argument, or if none exist it creates it and attaches the block as the calculation to perform when #value is sent to the node - let's say 'foo' is an instance of Foo class, then 'foo addOneTo: baz' refers to the calc node, and '(foo addOneTo: baz) value' refers to the value of the calculation - What I was thinking here was that I could use method wrappers to hide all the junk, like this: Foo>>addOneTo: aNode <calcNode> aNode value + 1 - I listen to all method add/update/remove announcements, and when a method that is marked with the <calcNode> pragma is involved, what I do is create a MethodWrapper that takes care of all the uggly stuff to get/create the graph node via the identity cache and as #value method for the node uses the method implementation the user originally provided (and replaces the entry in the method dictionary) Does that sound doable? The browsers etc wont go nuts? If I do that, will system browsers still show the original source for the method even though I've replaced it via the method dictionary and MethodWrapper? I'm assuming stack traces and debugging etc will be just fine, will just have the "hidden" node logic via the MethodWrappers included in the call stack. I'll give that type of setup a go, just wanted to throw it out there and hear if it's an obviously terrible idea first. P.S. for leaf nodes (or any node not requiring an argument to be provided) I think I could use custom slots instead of method wrappers and do the same kind of logic as the method wrapper would do in the slot instead. I'll worry about that later. On Jul 6, 2012 1:16 PM, "Patrik Sundberg" wrote: > On Fri, Jul 6, 2012 at 12:38 PM, S Krish wrote: >> >>> You should read as much as you can about Kapital of JPMC. It does more >>> or less what you describe for esoteric Prime Interest Derivatives and very >>> scalably .. >>> >>> >> I've heard of that one. Never worked at JPM so haven't seen it first hand >> though. I'll see what I can find. In terms of concepts I'm fully on top of >> it (have a background with other companies using similar ideas, but not in >> smalltalk but their own in-house language). So the good part is that I know >> exactly what I want conceptually, but I'm not stuck into smalltalk enough >> to know the natural way to implement the concepts. It may sound like a >> huge endeavour but I've done a lot of it in ruby and even as a side project >> to my real job we're talking about a couple of months to have something >> useful. >> >> >>> I doubt if there is much in the public domain, but the fact that Kapital >>> is the golden standard in this product line talks highly of why Smalltalk >>> is THE platform for this kind of product. >>> >>> >> I wouldn't say it's the golden standard, that's taking it a bit far, >> there are others that I rate higher bit I'm also biased :) But I do agree >> that smalltalk is a reasonably natural platform for these types of >> concepts, hence why I'm exploring it. >> >> >>> It would be nice to have a Pharo based derivative product that can >>> easily beat Ruby/ Java / .Net at this dynamic visualization using standard >>> browsers, where Nautilus too can serve your purpose.. or a home grown tree >>> based browser.../ Grids.. >>> >>> >> That's my hunch, that the tooling work would be much cut down. In my >> current ruby mockup I'm kind of creating a poor man's image like experience >> - it's jruby on the JVM, vim + vim-slime to send code fragments to the >> "runtime", git, and most likely homegrown GUI to navigate the graph and >> play with it (not done anything on that yet). I'll be keeping jruby as a >> dear tool as right now it's a fantastic glue tool in terms of interacting >> with anything out there in the world (i.e. I can easily work with the >> bloomberg java API), but I'm very keen to explore pharo for my graph + >> tooling around that. >> >> (slight tangent - coming from outside the smalltalk world it'd be HUGE to >> me to have git backend for packages and use of github for collaboration. >> seen that mentioned in other threads. I think it'd be HUGE for community's >> ability to attract more people). >> >> >> >>> Once I have completed my run at a PharoTabletIDE/ PharoMorphicView >>> framework, I would love to collaborate on your endeavour to see if it can >>> be modelled on it and exposed.. >>> >>> >> Cool. Early days yet and we'll see where I end up. Not familiar with >> those projects, I shall take a look. >> >> >> Patrik >> >> >> >> >>> On Fri, Jul 6, 2012 at 4:08 PM, Patrik Sundberg wrote: >>> >>>> Hi, >>>> >>>> First of all, sorry for the long email as my first email to the list. >>>> Hopefully some people will find it an interesting discussion :) >>>> >>>> I'm a long time programmer that has been studying smalltalk and pharo >>>> over >>>> the last year. It's a beautiful language I'm liking the image style of >>>> development a whole lot. I've been looking for a good test case to try >>>> out >>>> in pharo and I've got something in mind. I've used ruby since around >>>> 2000 >>>> and given how much it has been inspired by smalltalk it's not a big >>>> leap. >>>> >>>> My real job is being a commodities trader but I build my own tools and >>>> something I'm always working on is improved risk and pricing tools. >>>> Lots of >>>> that comes down to 2 things: >>>> 1. evaluating a "function" (in a general sense) with inputs that are >>>> picked >>>> depending on time. e.g. I want to price a security and I want to pick up >>>> prices for time X as inputs to the pricing model. >>>> 2. calculating a derivative of the "function" above with respect to >>>> inputs >>>> >>>> What I tend to do in order to keep things consistent by design, plus add >>>> natural caching, is create a calculation graph. I define all the >>>> dependencies and create an in-memory graph where each node is unique >>>> using >>>> an identity map. I then perform a calculation where the calculations >>>> trickle >>>> down the graph starting from the node I'm interested in down all it's >>>> dependencies. To calculate the derivative I then find the node >>>> representing >>>> the input I want the derivative with respect to and use finite >>>> differences, >>>> e.g. move it's value, and recalculate the top level node. On the second >>>> valuation of the graph only the parts that are affected by me changing >>>> that >>>> 1 value will need to be evaluated, the rest is cached an unaffected. It >>>> makes it very easy to ask questions like "How much will I be affected >>>> by X >>>> changing by Y?" since I can take the top level node of the graph, >>>> search for >>>> X, and if found change it by Y and recalculate. >>>> >>>> How the graph is constructed in memory depends on time (and a few other >>>> things). I always store all previous states of the world so that I can >>>> recreate any calculation I did in the past. Very useful for diagnosing >>>> problem. Hence if I for example change what model I use to model >>>> something, >>>> that change is recorded in a way such that if I set my "time" to before >>>> the >>>> change happened the graph will get created as it would have with the old >>>> model, and for a time after the change it'll create a different graph >>>> reflecting the new model. Hence the graph can only be known at runtime >>>> when >>>> the "time" etc is known. >>>> >>>> I currently have a ruby mockup of this. It's a DSL that looks like this: >>>> >>>> ----------- EXAMPLE >>>> >>>> class ExampleObject >>>> include GraphEntity # includes a module with functionality for objects >>>> participating in the graph, including #property and #calc used below >>>> >>>> property :foo do >>>> 123 # this is the default value for this property which will be used >>>> before a value has been set and saved >>>> end >>>> >>>> calc :bar do |arg| >>>> foo.value + arg # take the value of the arg property and add the >>>> argument given to the calculation node >>>> end >>>> end >>>> >>>> o = ExampleObject.new >>>> # this will kick of the graph being built in memory and setup the >>>> dependency >>>> between bar and foo nodes >>>> o.bar(1).value # -> 124 >>>> # this will be a "cache lookup" since nothing in the graph that bar >>>> depends >>>> on has changed (i.e. the "expensive" calculation is not performed again >>>> o.bar(1).value # -> 124 >>>> >>>> o.foo.set_value(2) >>>> o.bar(1).value # -> 125, realizes that foo changed and performs a >>>> recalc >>>> -------------------------------------------------------- >>>> >>>> To accomplish this I use dynamic code generation like the below for >>>> #property and similar for other types of nodes: >>>> >>>> # NOTE: it's a class level method, hence how I can call it when >>>> defining the class in the example >>>> >>>> def property(name, time = CdrEnv.instance.time, &block) >>>> clear_property(name) >>>> getter_method_body = <<-EOF >>>> def #{name} >>>> init_block = self.class.property_init_block_for(:#{name}) >>>> time_for_property = >>>> self.class.property_time_dependency_for(:#{name}) >>>> if init_block.nil? >>>> CdrEnv.instance.get_property_node(self, :#{name}, >>>> time_for_property) >>>> else >>>> CdrEnv.instance.get_property_node(self, :#{name}, >>>> time_for_property, &init_block) >>>> end >>>> end >>>> EOF >>>> setter_method_body = <<-EOF >>>> def #{name.to_s}=(value) >>>> #{name.to_s}.mutate(value) >>>> end >>>> EOF >>>> class_eval getter_method_body >>>> class_eval setter_method_body >>>> register_property_node(name, time, &block) >>>> end >>>> >>>> Don't worry about the details or all the unfamiliar ruby, the main >>>> point is >>>> that it creates an instance method that uses the singleton >>>> CdrEnv.instance >>>> to either get or create the node representing the property from the >>>> graph >>>> identity cache depending on if it exists or not. >>>> >>>> From a tooling point of view I think I'd love to work with this kind of >>>> thing in pharo. Building my own browsers for inspecting and debugging my >>>> dynamic graph should be very good fit. However, I'd appreciate some >>>> pointers >>>> to idiomatic smalltalk to attack this kind of problem in terms of >>>> implementing the graph itself - I obviously want the user (even if it's >>>> me) >>>> to just have to focus on the domain model and hide as much as possible >>>> of >>>> the graph bits under the covers the same way I've done with the code >>>> generation stuff in my ruby example. >>>> >>>> Any input into this would be much appreciated, >>>> Patrik >>>> >>>> P.S. >>>> Btw, the persistence backend for this is the neo4j graph database, but >>>> it's >>>> fronted by a service slotting into my own service framework built using >>>> ZeroMQ and services exchanging messages in protobuf format. One can use >>>> it >>>> from any language as long as one can send protobuf messages over >>>> zeromq. I >>>> see there's a zeromq ffi library available on SS that I'll check out, >>>> but >>>> I'm not finding a protobuf implementation. It'd be easy enough for me to >>>> port a ruby protobuf implementation but I may as well ask if someone has >>>> already done any work on protobuf for smalltalk? >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://forum.world.st/Dynamic-graph-exploration-seeking-input-on-idiomatic-… >>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>>> >>>> >>> >>

July 24, 2012

Debugging block with ensure block

by kurs.jan

Hi, I observe a strange behaviour (in Pharo-1.4-14557-one-click.app) while debugging the following code. [ | a | self halt. a := (1 == 1). a ifTrue: [ ^ 'A'] . Transcript crShow: 'foo'. ^ 'B' ] ensure: [ Transcript crShow: 'bar'. ] If I comment out the halt statement and if I inspect the result of the attached code, the 'A' is returned and there is a 'bar' text in the transcript. If I inspect the result of the attached code (with halt enabled) and if I click step over in a debugger, I can see, that the statement ^ 'A' does NOT return, the program flow continues on the next line and the 'foo' is shown in the transcript. Now, I can click proceed and the 'B' is returned as a result. I tried the same process in Pharo-1.3-13328-OneClick.app and I got 'A' result in both cases. I tried on this system: System Version: Mac OS X 10.7.3 (11D2001) Kernel Version: Darwin 11.3.0 Regards, Jan -- View this message in context: http://forum.world.st/Debugging-block-with-ensure-block-tp4641279.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

July 24, 2012

Re: [Pharo-users] Recovery log thoughts

by StÃ©phane Ducasse

On Jul 23, 2012, at 10:34 PM, Chris wrote: > On 23/07/2012 07:58, StÃ©phane Ducasse wrote: >> On Jul 22, 2012, at 9:23 PM, Chris wrote: >> >>> One area I see some beginners struggle with image development is with regards to losing work, and I feel that Pharo's recovery log could be improved to help with this. >> yes :) >> Now that we have ring we should start rethinking source management but this will not be for 2.0 >> except if somebody come up with a good and working solution. >> >>> Two scenarios that come to mind are firstly where methods are filed in that created an instance variable at method save the first time round. These log to the transcript that the variable is now undeclared, but that can go unseen, and the result is things appearing to work while you only have one instance of that object. Could this bring up a similar prompt maybe? Another difficulty is that you can not simply file in all selections as any do-it's which were done in a different context will fail. Is there any way we can differentiate the important do-it's such as class definitions, selector removes and so on from the rest? >> I would really like to have a structured logger and not just a flow of text. >> > > That sounds great for the long term, but was wondering if there were any good ideas in the meantime! I could only think of applying do-it's if it starts with a class name for example. yes if you have some hacks like that send them and we will probably introduce them. > >

July 23, 2012

Re: [Pharo-users] Recovery log thoughts

by Chris

On 23/07/2012 07:58, StÃ©phane Ducasse wrote: > On Jul 22, 2012, at 9:23 PM, Chris wrote: > >> One area I see some beginners struggle with image development is with regards to losing work, and I feel that Pharo's recovery log could be improved to help with this. > yes :) > Now that we have ring we should start rethinking source management but this will not be for 2.0 > except if somebody come up with a good and working solution. > >> Two scenarios that come to mind are firstly where methods are filed in that created an instance variable at method save the first time round. These log to the transcript that the variable is now undeclared, but that can go unseen, and the result is things appearing to work while you only have one instance of that object. Could this bring up a similar prompt maybe? Another difficulty is that you can not simply file in all selections as any do-it's which were done in a different context will fail. Is there any way we can differentiate the important do-it's such as class definitions, selector removes and so on from the rest? > I would really like to have a structured logger and not just a flow of text. > That sounds great for the long term, but was wondering if there were any good ideas in the meantime! I could only think of applying do-it's if it starts with a class name for example.

July 23, 2012

Re: [Pharo-users] Recovery log thoughts

by StÃ©phane Ducasse

On Jul 22, 2012, at 9:23 PM, Chris wrote: > One area I see some beginners struggle with image development is with regards to losing work, and I feel that Pharo's recovery log could be improved to help with this. yes :) Now that we have ring we should start rethinking source management but this will not be for 2.0 except if somebody come up with a good and working solution. > Two scenarios that come to mind are firstly where methods are filed in that created an instance variable at method save the first time round. These log to the transcript that the variable is now undeclared, but that can go unseen, and the result is things appearing to work while you only have one instance of that object. Could this bring up a similar prompt maybe? Another difficulty is that you can not simply file in all selections as any do-it's which were done in a different context will fail. Is there any way we can differentiate the important do-it's such as class definitions, selector removes and so on from the rest? I would really like to have a structured logger and not just a flow of text. > > Regards, > Chris >

July 23, 2012

Recovery log thoughts

by Chris

One area I see some beginners struggle with image development is with regards to losing work, and I feel that Pharo's recovery log could be improved to help with this. Two scenarios that come to mind are firstly where methods are filed in that created an instance variable at method save the first time round. These log to the transcript that the variable is now undeclared, but that can go unseen, and the result is things appearing to work while you only have one instance of that object. Could this bring up a similar prompt maybe? Another difficulty is that you can not simply file in all selections as any do-it's which were done in a different context will fail. Is there any way we can differentiate the important do-it's such as class definitions, selector removes and so on from the rest? Regards, Chris

July 22, 2012

Re: [Pharo-users] The opposite of encodeForHTTP

by StÃ©phane Ducasse

On Jul 21, 2012, at 2:13 PM, Davide Varvello wrote: > Right, I seconded urlEncoded and urlDecoded go for it. we will deprecate or let encodeForHTTP for backward compat. My point is: let us steadily improve the situation. And if tomorrow something nicer exists then we just replace and throw away what we did :). Stef > Davide > > > Norbert Hartl wrote >> >> IMHO that would worsen the problem :) >> >> encodeForHTTP is not a good name. The encoding is defined for URLs and has >> nothing to do with HTTP. It is mostly called "url safe encoded" or just >> "url encoded". Doing it similar as base64 I would propose >> >> urlEncoded >> urlDecoded >> >> or >> >> urlSafeEncoded >> urlSafeDecoded >> >> my 2 cents, >> >> Norbert >> >> Am 19.07.2012 um 21:47 schrieb StÃ©phane Ducasse: >> >>> Let us fix it and propose a decodeFromHTTP method >>> >>> Stef >>> >>> On Jul 18, 2012, at 2:02 PM, Davide Varvello wrote: >>> >>>> Thanks Sven, >>>> I was looking for String>>decode..whatever... with no luck :-) >>>> Cheers >>>> >>>> -- >>>> View this message in context: >>>> http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640510.html >>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>>> >>> >>> >> > > > > > -- > View this message in context: http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4641004.html > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >

July 22, 2012

Re: [Pharo-users] The opposite of encodeForHTTP

by StÃ©phane Ducasse

On Jul 20, 2012, at 6:25 PM, Brenda Larcom wrote: > I suppose I could unlurk at this point. :) > > I'm a security geek (specifically, a secure development geek focusing on security architecture) in my day job, and I have a long unmaintained architecture security analysis tool written in Squeak (http://www.octotrike.org/ for the curious), which I have been unmothballing. We are considering switching to Pharo, partly because we are planning to add some P2P collaboration features we think have an HTTP layer in there somewhere & partly because we like it small, tidy, and self-compatible. Hence my lurking. Welcome and I would love to have more people working on these areas :). > I've done some work on how data validation should be done for security purposes, for my day job. This includes output encoding and decoding, like what Davide is talking about. It's pretty tricky to get right because of the large number of contexts, with subtly different rules. E.g. I would expect encodeForHTTP to be appropriate for HTTP headers, except that e.g. two things you usually want to put in HTTP headers are URIs and cookies, each of which have different rules (for different subparts, even) for what should be encoded. The differences don't seem like much, but in the wild, my coworkers & I see these sorts of differences lead to vulnerabilities on a daily basis. > > From a security architecture perspective, the absolute best way to handle encoding & decoding for a structured object like an HTTP request or response (or a URI, or a cookie, or an HTML document, or..) is to use a validating parser. Basically, when you get an HTTP request, parse it & put it in an object structured like the request. At that time, you know the meaning of each portion of the string you are parsing, so you can interpret the bits correctly/safely. The object(s) should store the individual strings that are actually content (vs. structure & constants) in a decoded state. The developer should get everything from the objects, in decoded form, and put everything into the objects in decoded form. Then, when it is time to send the response, the objects encode everything safely/canonically based on the exact type of objects they are. This design concentrates the hard stuff (encoding, decoding, canonicalization, layering encodings on top of each other) near the interfaces, at the first/last possible moment enough context is known to interpret the information accurately. It separates the mechanics of using a protocol or format from the intent of using the protocol. It lets someone like me easily QA both the library and application code for security. It is also simple for the developer to use safely (all the dev needs to think about is what objects/content they want to assemble, and the data validation at that layer is taken care of automatically) & is therefore the only design pattern I have seen consistently avoid all encoding-related vulnerabilities in the wild. > > So what does this mean? Basically, from a security perspective, encoding & decoding methods should live in the objects they encode and decode, and never be called from outside code. That is, there should be an HTTPHeader>>fromString: or fromStream: method, which is called from an HTTPResponse >>fromString: or fromStream: method, and no String>>decodeFromHTTP. Adding a String>>decodeFromHTTP method is easy from the library maintainer's point of view, approximately correct (way more correct than no method at all), and it matches what most languages are doing these days, but it shifts the burden of all that thought about the specific HTTP header & context to the application developer, who is usually just trying to write an application, not learn every single detail of the HTTP & gazillion other standards he would need to do this safely. > > Since this is a suggestion for substantial architecture change that would cause significant backwards compatibility issues throughout the entire Web application stack, and I'm new to Pharo to boot, I am expecting some interesting discussion to occur next. Or maybe profound silence. :) Thanks for the explanation. It makes sense. String is a dead object just counting and assembling characters. So Now what I would love to see is if you interested: - how can we improve the infrastructure of Pharo? step by step or via a big refactoring :) - I would add a simple decodeFromHTTP as a convenience method and in the future point to the validators. > In my back pocket somewhere amongst the code I am unmothballing, I have 95% of a thouroughly documented URI implementation and test suite that follows this pattern and is pedantically compliant with one or another of the URI RFCs (it's old, may not be the most recent). Bring it to life. We were discussing internally that we would like to have a decent URI implementation and we would like to massively clean the URL/URI â¦. with ZnURL whatever. So it would be great to have a good part. Now what I see from your mail :) is that you are a kind of perfectionist and you should pay attention (I know some of them) and you should force yourself to be happy with 80% and release it - 1 your 80% may be the 95% of somebody else - 2 release often, make progress is the best way to finish. :) > I believe Spoon & Slate are using a previous version of it or its derivatives. I'll need a fully pedantic HTTP parsing stack to feel comfortable releasing a P2P architecture security analysis tool (high value target, large attack surface, potentially very large professional embarrassment), so whatever isn't available, I expect we'll end up writing. If Pharo folks are interested in this pattern, Yes I'm. I will let the other reply to you because I'm far down in south of france but I'm quite sure that we are all interested. > I would love to contribute my libraries/changes as I finish them, get advice on backward compatibility, performance, and APIs people would like to see, review whatever related code you'd like for security issues, and/or collaborate with any other developer who is interested. I would love to learn from your expertise. Stef > > Brenda > > > On Jul 20, 2012, at 1:47 AM, Davide Varvello <varvello(a)yahoo.com> wrote: > >> Good Stef, I opened a new feature as reminder here: http://code.google.com/p/pharo/issues/detail?id=6430 >> >> Davide >> >> ---- >> - Cerchi un bravo Dentista, Avvocato, Commercialista? Un buon Hotel, Ristorante, Pizzeria? Io l'ho trovato su Oltre il Passaparola >> >> - Blog: Cambia il Tempo >> >> From: StÃ©phane Ducasse [via Smalltalk] <[hidden email]> >> To: Davide Varvello <[hidden email]> >> Sent: Thursday, July 19, 2012 10:43 PM >> Subject: Re: The opposite of encodeForHTTP >> >> Let us fix it and propose a decodeFromHTTP method >> >> Stef >> >> On Jul 18, 2012, at 2:02 PM, Davide Varvello wrote: >> >> > Thanks Sven, >> > I was looking for String>>decode..whatever... with no luck :-) >> > Cheers >> > >> > -- >> > View this message in context: http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640510.html >> > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >> > >> >> >> >> >> If you reply to this email, your message will be added to the discussion below: >> http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640822.html >> To unsubscribe from The opposite of encodeForHTTP, click here. >> NAML >> >> >> >> View this message in context: Re: The opposite of encodeForHTTP >> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

July 22, 2012

Re: [Pharo-users] WaveFront File importer on Pharo

by Jean Baptiste Arnaud

On Jul 18, 2012, at 7:34 AM, StÃ©phane Ducasse wrote: > > On Jul 17, 2012, at 2:21 PM, Jean Baptiste Arnaud wrote: > >> Hi, >> Under the pressure of Camillo i publish the code. > > Thanks camillo :) > > >> I beginning to do a Wavefront (human readable standard for 3d Obj Model ) importer on Pharo. >> >> So i made a importer. >> Not all the case are manage, only the case I need to import the current Obj model. > > you relax from PhD writing :). Exactly ^^. > >> >> And a drawer which is in Alpha version. >> face and normal vector are manage. >> Missing color (in progress), and texturing. >> >> You need .obj file and all the .mtl related (open you blender and make it nice). >> >> If you are not running on Mac os comment this code in GLWorldTest>>#render, the >> >> display makeCurrent. >> >> else if you running on mac os implement it in >> NBMSAAOffscreenDisplay>>#makeCurrent >> ^driver makeCurrent. >> >> My code can be find on JBARepo on squeaksource >> ObjModel Package but you need NBOpenGL (i do a configuration in same place i just need to be motivated for keep it up to date). > > Keep it up to date. > > >> >> So >> put on same folder of the .image your .obj and .mtl file (i am lazy) (maybe it is VM folder i do not know). >> open a workspace and do >> >> "s := ObjImporter importFrom: 'xwing-map.obj'. >> s parse. >> object := s objects. >> >> w := GLWorldTest new. >> object do: [:e | w addElement: e]. >> w openInWorld." >> >> 3d result with normal >> <Screen Shot 2012-07-17 at 1.59.22 PM.png> >> >> >> With normal and color but not texture : >> >> <Screen Shot 2012-07-17 at 1.58.18 PM.png> >> >> Enjoy >> >> >> >> >> Best Regards >> Jean Baptiste Arnaud >> jbaptiste.arnaud(a)gmail.com >> >> >> >> >> >> >> > > Best Regards Jean Baptiste Arnaud jbaptiste.arnaud(a)gmail.com

July 21, 2012

Re: [Pharo-users] WaveFront File importer on Pharo

by Jean Baptiste Arnaud

I will check tomorrow, because i don't manage all the dispatch case g Object001 in file is not manage is do that tomorow fast. On Jul 21, 2012, at 9:44 AM, Luc Fabresse wrote: > Hi JB, > > Excellent! > I tried to give it a try for the fun. > I couldn't make it work. > > I updated some stuff (find it attached): > - ConfigurationOfObjModel-LucFabresse.2.mcz > added repository for package OBJModel in baseline > - OBJModel-LucFabresse.7 > add support for empty lines and "g" lines in wavefront format parser (ObjImporter>>dispach:) > > Here the snippet I tried: > > filename := 'teapot.obj'. > response := ZnEasy get: 'http://people.sc.fsu.edu/~jburkardt/data/obj/',filename. > (FileSystem disk workingDirectory / filename) > writeStreamDo: [ :stream | stream nextPutAll: response contents]. > > s := ObjImporter importFrom: filename. > s parse. > object := s objects. > > w := GLWorldTest new. > object do: [:e | w addElement: e]. > w openInWorld. > > Parsing seems to be ok. > But, creating the GL context failed: invalid pixel format. > On a mac OSX 10.6.8. > > I am eager to play deeper with that ;-) > > #Luc > > > > 2012/7/18 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> > > On Jul 17, 2012, at 2:21 PM, Jean Baptiste Arnaud wrote: > > > Hi, > > Under the pressure of Camillo i publish the code. > > Thanks camillo :) > > > > I beginning to do a Wavefront (human readable standard for 3d Obj Model ) importer on Pharo. > > > > So i made a importer. > > Not all the case are manage, only the case I need to import the current Obj model. > > you relax from PhD writing :). > > > > > And a drawer which is in Alpha version. > > face and normal vector are manage. > > Missing color (in progress), and texturing. > > > > You need .obj file and all the .mtl related (open you blender and make it nice). > > > > If you are not running on Mac os comment this code in GLWorldTest>>#render, the > > > > display makeCurrent. > > > > else if you running on mac os implement it in > > NBMSAAOffscreenDisplay>>#makeCurrent > > ^driver makeCurrent. > > > > My code can be find on JBARepo on squeaksource > > ObjModel Package but you need NBOpenGL (i do a configuration in same place i just need to be motivated for keep it up to date). > > Keep it up to date. > > > > > > So > > put on same folder of the .image your .obj and .mtl file (i am lazy) (maybe it is VM folder i do not know). > > open a workspace and do > > > > "s := ObjImporter importFrom: 'xwing-map.obj'. > > s parse. > > object := s objects. > > > > w := GLWorldTest new. > > object do: [:e | w addElement: e]. > > w openInWorld." > > > > 3d result with normal > > <Screen Shot 2012-07-17 at 1.59.22 PM.png> > > > > > > With normal and color but not texture : > > > > <Screen Shot 2012-07-17 at 1.58.18 PM.png> > > > > Enjoy > > > > > > > > > > Best Regards > > Jean Baptiste Arnaud > > jbaptiste.arnaud(a)gmail.com > > > > > > > > > > > > > > > > > > <OBJModel-LucFabresse.7.mcz><ConfigurationOfObjModel-LucFabresse.2.mcz> Best Regards Jean Baptiste Arnaud jbaptiste.arnaud(a)gmail.com

July 21, 2012

Re: [Pharo-users] [Esug-list] [ANN] humane assessment / moose courses

by HwaJongOh

Pity! I already fixed all my travel reservations. HwaJong Oh 2012. 7. 22., ìì¹¨ 5:46, Tudor Girba ìì±: > Hi, > > Humane assessment is a method for making software engineering decisions. Assessing software systems to make decisions is a critical activity that needs to be approached explicitly during development. Humane assessment is made possible by the Moose analysis platform. > > > I am organizing a couple of courses in Bern that might be of interest to people on this list: > > Humane Assessment Primer (August 17) > - This course is relevant for both managers and engineers. It covers assessment economics and the means to integrate humane assessment in the development process and in the organization > - http://www.humane-assessment.com/courses/humane-assessment-primer > > Moose Apprentice (September 6-7) > - This course is relevant for engineers. This is an introductory hands-on course on using the Moose analysis platform for putting humane assessment into practice > - http://www.humane-assessment.com/courses/moose-apprentice > > > If you are interested in participating, please contact me directly. > > > Cheers, > Doru > > -- > www.tudorgirba.com > > "In a world where everything is moving ever faster, > one might have better chances to win by moving slower." > > > > > _______________________________________________ > Esug-list mailing list > Esug-list(a)lists.esug.org > http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org

July 21, 2012

[ANN] humane assessment / moose courses

by Tudor Girba

Hi, Humane assessment is a method for making software engineering decisions. Assessing software systems to make decisions is a critical activity that needs to be approached explicitly during development. Humane assessment is made possible by the Moose analysis platform. I am organizing a couple of courses in Bern that might be of interest to people on this list: Humane Assessment Primer (August 17) - This course is relevant for both managers and engineers. It covers assessment economics and the means to integrate humane assessment in the development process and in the organization - http://www.humane-assessment.com/courses/humane-assessment-primer Moose Apprentice (September 6-7) - This course is relevant for engineers. This is an introductory hands-on course on using the Moose analysis platform for putting humane assessment into practice - http://www.humane-assessment.com/courses/moose-apprentice If you are interested in participating, please contact me directly. Cheers, Doru -- www.tudorgirba.com "In a world where everything is moving ever faster, one might have better chances to win by moving slower."

July 21, 2012

Re: [Pharo-users] Zinc downloadTo: help required

by Chris

On 20/07/2012 18:30, Sven Van Caekenberghe wrote: > On 20 Jul 2012, at 18:22, Chris wrote: > >> Thanks for that. I'm actually still having a bit of trouble when the file is bigger than the chunk size. ZnChunkedReadStream>>#readInto:startingAt:count: uses a limit variable which is bigger than the collection and requestedCount. > I won't be able to do anything until the beginning of August. > > It would be really helpful if you could provide me with an actual test case that fails. > > Okay thanks Sven, I'll see what I can do. I am just working with a third party server which is sending chunks of 1mb (which I assume is valid) and I think Zinc just needs to support chunks bigger than the 16k in the above method Regards, Chris

July 21, 2012

glamorous inspector - extensions

by Tudor Girba

Hi, I see that there is a bit of an interest in the GTInspector, so here are some extra info about it. Beyond the things you can do out-of-the-box with it, the true power of the inspector comes from its extensibility. The idea is to offer each object an easy way to exhibit its own characteristics. This changes the way you can think of interacting with objects. Some more details here: http://www.humane-assessment.com/blog/equal-browsing-opportunity-for-every-… A new feature is the ability to edit code directly in the methods presentation. Combined with the possibility of executing a method in place and previewing the result, this opens the door for a prototype-like programming style. The GTInspector is part of the Glamorous Toolkit. It is still in its infancy, but I think it shows that there is a great deal of possibilities to improve the IDE and to depart from the what we might think is a given. I am still looking for people interested in joining this effort (remember the idea of the ide taskforce). It would be great to get more people interested in improving this part of Pharo. Cheers, Doru p.s. You can get the Glamorous Inspector in the following ways: - download the Moose image: http://www.moosetechnology.org/download/4.7 - download Pharo 1.4 (summer) with the Glamorous Inspector: http://ci.moosetechnology.org/job/glamorous-toolkit-latest-dev/lastSuccessf… - load it in Pharo 1.4: Gofer new squeaksource: 'glamoroust'; package: 'ConfigurationOfGlamoroust'; load. ((Smalltalk at: #ConfigurationOfGlamoroust) project version: #development) load: 'GT-Inspector' -- www.tudorgirba.com "Live like you mean it."

July 21, 2012

Re: [Pharo-users] The opposite of encodeForHTTP

by Davide Varvello

Right, I seconded urlEncoded and urlDecoded Davide Norbert Hartl wrote > > IMHO that would worsen the problem :) > > encodeForHTTP is not a good name. The encoding is defined for URLs and has > nothing to do with HTTP. It is mostly called "url safe encoded" or just > "url encoded". Doing it similar as base64 I would propose > > urlEncoded > urlDecoded > > or > > urlSafeEncoded > urlSafeDecoded > > my 2 cents, > > Norbert > > Am 19.07.2012 um 21:47 schrieb StÃ©phane Ducasse: > >> Let us fix it and propose a decodeFromHTTP method >> >> Stef >> >> On Jul 18, 2012, at 2:02 PM, Davide Varvello wrote: >> >>> Thanks Sven, >>> I was looking for String>>decode..whatever... with no luck :-) >>> Cheers >>> >>> -- >>> View this message in context: >>> http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640510.html >>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>> >> >> > -- View this message in context: http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4641004.html Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

July 21, 2012

Re: [Pharo-users] The opposite of encodeForHTTP

by Norbert Hartl

Am 20.07.2012 um 20:53 schrieb Brenda Larcom: > Thanks, Norbert; I'll take a look at Zinc, see how my existing code might integrate, and propose something specific. I personally think having an insecure option for things like URIs and HTTP that are inherently on borders almost all the time is unwise, but I'm happy to resolve my personal issues via documentation. :) > You said "almost" yourself :) I just wanted to say that different people have different ideas. Restricting software to what we can imagine is like avoiding that other people realize amazing things we couldn't imagine. > One reason validating parsers are so powerful is that when layers stack as you mentioned, the security starts working as soon as the functional part does. I agree, such a parser definitely belongs at the borders of interpretation schemes, not inside them. Inside them it'll just use up time without providing value. Conveniently, the tool people naturally reach for at interpretation borders usually has a parser in it someplace. > > And yes, there do seem to be a particular lot of fiddly bits in URIs. So fiddly a few of the examples in the RFCs (as usual) don't match the rest of the spec. > agreed. I'm eager to see what you'll come up with. Norbert > > On Jul 20, 2012, at 10:50 AM, Norbert Hartl <norbert(a)hartl.name> wrote: > >> Brenda, >> >> these are all good points as you said from a "security architecture perspective" and we should improve on that. The zinc http components do already a good job in structuring the entities as they should be. I think security add-ons can hook onto what is already there. There is a huge amount of things to consider. Even for a single URL the different components of an url have different encoding needs. >> On the other hand security is not a major target in a lot of use cases I can imagine. There is at least (for me) a triangle of security - performance - usability that makes it hard to have a single approach to fit them all. And we smalltalkers tend to judge freedom very high if it comes to program. In other words I would say we like to preserve the freedom of designing an insecure application at will :) The best way to solve those issues is by being modular, meaning a layer that can be put on top of the existing stuff to fulfill a particular use case. >> The things you describe are present in a lot of environments. I mostly call this a "at the border of a system" problem. Things like strings inside of an environment are harmless. Problems appear if you cross system borders, meaning you cross interpretation schemes. And this a topic more broad then only HTTP. >> If we look at a widely known problem like sql injection there is not only the need for proper entity handling but for stacking validators and converters for different problems. It is such a big thing because you have an URL that goes through middleware and ends in a storage system like an SQL database. Here you cross at least two borders: HTTP to middleware and middleware to database. So you need to stack up converters and validators for HTTP, probably shell escapes in a middleware and finally for SQL. I think if you can assemble those things by the layers you use a security approach is doable. And for the same reason it goes so terribly wrong everywhere. >> So what does this modular thing mean? To have a lot of possibilities to fulfill certain needs without restricting everyone to a single scheme. >> My advice would be to have a look at the zinc components and propose things to improve from your perspective. Then publish your results here and there will be a lot of clever people finding a good way to integrate it in a modular way. >> >> I hope this helps, >> >> Norbert >> >> Am 20.07.2012 um 18:25 schrieb Brenda Larcom: >> >>> I suppose I could unlurk at this point. :) >>> >>> I'm a security geek (specifically, a secure development geek focusing on security architecture) in my day job, and I have a long unmaintained architecture security analysis tool written in Squeak (http://www.octotrike.org/ for the curious), which I have been unmothballing. We are considering switching to Pharo, partly because we are planning to add some P2P collaboration features we think have an HTTP layer in there somewhere & partly because we like it small, tidy, and self-compatible. Hence my lurking. >>> >>> I've done some work on how data validation should be done for security purposes, for my day job. This includes output encoding and decoding, like what Davide is talking about. It's pretty tricky to get right because of the large number of contexts, with subtly different rules. E.g. I would expect encodeForHTTP to be appropriate for HTTP headers, except that e.g. two things you usually want to put in HTTP headers are URIs and cookies, each of which have different rules (for different subparts, even) for what should be encoded. The differences don't seem like much, but in the wild, my coworkers & I see these sorts of differences lead to vulnerabilities on a daily basis. >>> >>> From a security architecture perspective, the absolute best way to handle encoding & decoding for a structured object like an HTTP request or response (or a URI, or a cookie, or an HTML document, or..) is to use a validating parser. Basically, when you get an HTTP request, parse it & put it in an object structured like the request. At that time, you know the meaning of each portion of the string you are parsing, so you can interpret the bits correctly/safely. The object(s) should store the individual strings that are actually content (vs. structure & constants) in a decoded state. The developer should get everything from the objects, in decoded form, and put everything into the objects in decoded form. Then, when it is time to send the response, the objects encode everything safely/canonically based on the exact type of objects they are. This design concentrates the hard stuff (encoding, decoding, canonicalization, layering encodings on top of each other) near the interfaces, at the first/last possible moment enough context is known to interpret the information accurately. It separates the mechanics of using a protocol or format from the intent of using the protocol. It lets someone like me easily QA both the library and application code for security. It is also simple for the developer to use safely (all the dev needs to think about is what objects/content they want to assemble, and the data validation at that layer is taken care of automatically) & is therefore the only design pattern I have seen consistently avoid all encoding-related vulnerabilities in the wild. >>> >>> So what does this mean? Basically, from a security perspective, encoding & decoding methods should live in the objects they encode and decode, and never be called from outside code. That is, there should be an HTTPHeader>>fromString: or fromStream: method, which is called from an HTTPResponse >>fromString: or fromStream: method, and no String>>decodeFromHTTP. Adding a String>>decodeFromHTTP method is easy from the library maintainer's point of view, approximately correct (way more correct than no method at all), and it matches what most languages are doing these days, but it shifts the burden of all that thought about the specific HTTP header & context to the application developer, who is usually just trying to write an application, not learn every single detail of the HTTP & gazillion other standards he would need to do this safely. >>> >>> Since this is a suggestion for substantial architecture change that would cause significant backwards compatibility issues throughout the entire Web application stack, and I'm new to Pharo to boot, I am expecting some interesting discussion to occur next. Or maybe profound silence. :) >>> >>> In my back pocket somewhere amongst the code I am unmothballing, I have 95% of a thouroughly documented URI implementation and test suite that follows this pattern and is pedantically compliant with one or another of the URI RFCs (it's old, may not be the most recent). I believe Spoon & Slate are using a previous version of it or its derivatives. I'll need a fully pedantic HTTP parsing stack to feel comfortable releasing a P2P architecture security analysis tool (high value target, large attack surface, potentially very large professional embarrassment), so whatever isn't available, I expect we'll end up writing. If Pharo folks are interested in this pattern, I would love to contribute my libraries/changes as I finish them, get advice on backward compatibility, performance, and APIs people would like to see, review whatever related code you'd like for security issues, and/or collaborate with any other developer who is interested. >>> >>> Brenda >>> >>> >>> On Jul 20, 2012, at 1:47 AM, Davide Varvello <varvello(a)yahoo.com> wrote: >>> >>>> Good Stef, I opened a new feature as reminder here: http://code.google.com/p/pharo/issues/detail?id=6430 >>>> >>>> Davide >>>> >>>> ---- >>>> - Cerchi un bravo Dentista, Avvocato, Commercialista? Un buon Hotel, Ristorante, Pizzeria? Io l'ho trovato su Oltre il Passaparola >>>> >>>> - Blog: Cambia il Tempo >>>> >>>> From: StÃ©phane Ducasse [via Smalltalk] <[hidden email]> >>>> To: Davide Varvello <[hidden email]> >>>> Sent: Thursday, July 19, 2012 10:43 PM >>>> Subject: Re: The opposite of encodeForHTTP >>>> >>>> Let us fix it and propose a decodeFromHTTP method >>>> >>>> Stef >>>> >>>> On Jul 18, 2012, at 2:02 PM, Davide Varvello wrote: >>>> >>>> > Thanks Sven, >>>> > I was looking for String>>decode..whatever... with no luck :-) >>>> > Cheers >>>> > >>>> > -- >>>> > View this message in context: http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640510.html >>>> > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>>> > >>>> >>>> >>>> >>>> >>>> If you reply to this email, your message will be added to the discussion below: >>>> http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640822.html >>>> To unsubscribe from The opposite of encodeForHTTP, click here. >>>> NAML >>>> >>>> >>>> >>>> View this message in context: Re: The opposite of encodeForHTTP >>>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>

July 21, 2012

Re: [Pharo-users] WaveFront File importer on Pharo

by Luc Fabresse

Hi JB, Excellent! I tried to give it a try for the fun. I couldn't make it work. I updated some stuff (find it attached): - ConfigurationOfObjModel-LucFabresse.2.mcz added repository for package OBJModel in baseline - OBJModel-LucFabresse.7 add support for empty lines and "g" lines in wavefront format parser (ObjImporter>>dispach:) Here the snippet I tried: filename := 'teapot.obj'. response := ZnEasy get: 'http://people.sc.fsu.edu/~jburkardt/data/obj/ ',filename. (FileSystem disk workingDirectory / filename) writeStreamDo: [ :stream | stream nextPutAll: response contents]. s := ObjImporter importFrom: filename. s parse. object := s objects. w := GLWorldTest new. object do: [:e | w addElement: e]. w openInWorld. Parsing seems to be ok. But, creating the GL context failed: invalid pixel format. On a mac OSX 10.6.8. I am eager to play deeper with that ;-) #Luc 2012/7/18 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> > > On Jul 17, 2012, at 2:21 PM, Jean Baptiste Arnaud wrote: > > > Hi, > > Under the pressure of Camillo i publish the code. > > Thanks camillo :) > > > > I beginning to do a Wavefront (human readable standard for 3d Obj Model > ) importer on Pharo. > > > > So i made a importer. > > Not all the case are manage, only the case I need to import the current > Obj model. > > you relax from PhD writing :). > > > > > And a drawer which is in Alpha version. > > face and normal vector are manage. > > Missing color (in progress), and texturing. > > > > You need .obj file and all the .mtl related (open you blender and make > it nice). > > > > If you are not running on Mac os comment this code in > GLWorldTest>>#render, the > > > > display makeCurrent. > > > > else if you running on mac os implement it in > > NBMSAAOffscreenDisplay>>#makeCurrent > > ^driver makeCurrent. > > > > My code can be find on JBARepo on squeaksource > > ObjModel Package but you need NBOpenGL (i do a configuration in same > place i just need to be motivated for keep it up to date). > > Keep it up to date. > > > > > > So > > put on same folder of the .image your .obj and .mtl file (i am lazy) > (maybe it is VM folder i do not know). > > open a workspace and do > > > > "s := ObjImporter importFrom: 'xwing-map.obj'. > > s parse. > > object := s objects. > > > > w := GLWorldTest new. > > object do: [:e | w addElement: e]. > > w openInWorld." > > > > 3d result with normal > > <Screen Shot 2012-07-17 at 1.59.22 PM.png> > > > > > > With normal and color but not texture : > > > > <Screen Shot 2012-07-17 at 1.58.18 PM.png> > > > > Enjoy > > > > > > > > > > Best Regards > > Jean Baptiste Arnaud > > jbaptiste.arnaud(a)gmail.com > > > > > > > > > > > > > > > > >

July 21, 2012

Re: [Pharo-users] The opposite of encodeForHTTP

by Brenda Larcom

Thanks, Norbert; I'll take a look at Zinc, see how my existing code might integrate, and propose something specific. I personally think having an insecure option for things like URIs and HTTP that are inherently on borders almost all the time is unwise, but I'm happy to resolve my personal issues via documentation. :) One reason validating parsers are so powerful is that when layers stack as you mentioned, the security starts working as soon as the functional part does. I agree, such a parser definitely belongs at the borders of interpretation schemes, not inside them. Inside them it'll just use up time without providing value. Conveniently, the tool people naturally reach for at interpretation borders usually has a parser in it someplace. And yes, there do seem to be a particular lot of fiddly bits in URIs. So fiddly a few of the examples in the RFCs (as usual) don't match the rest of the spec. Brenda On Jul 20, 2012, at 10:50 AM, Norbert Hartl <norbert(a)hartl.name> wrote: > Brenda, > > these are all good points as you said from a "security architecture perspective" and we should improve on that. The zinc http components do already a good job in structuring the entities as they should be. I think security add-ons can hook onto what is already there. There is a huge amount of things to consider. Even for a single URL the different components of an url have different encoding needs. > On the other hand security is not a major target in a lot of use cases I can imagine. There is at least (for me) a triangle of security - performance - usability that makes it hard to have a single approach to fit them all. And we smalltalkers tend to judge freedom very high if it comes to program. In other words I would say we like to preserve the freedom of designing an insecure application at will :) The best way to solve those issues is by being modular, meaning a layer that can be put on top of the existing stuff to fulfill a particular use case. > The things you describe are present in a lot of environments. I mostly call this a "at the border of a system" problem. Things like strings inside of an environment are harmless. Problems appear if you cross system borders, meaning you cross interpretation schemes. And this a topic more broad then only HTTP. > If we look at a widely known problem like sql injection there is not only the need for proper entity handling but for stacking validators and converters for different problems. It is such a big thing because you have an URL that goes through middleware and ends in a storage system like an SQL database. Here you cross at least two borders: HTTP to middleware and middleware to database. So you need to stack up converters and validators for HTTP, probably shell escapes in a middleware and finally for SQL. I think if you can assemble those things by the layers you use a security approach is doable. And for the same reason it goes so terribly wrong everywhere. > So what does this modular thing mean? To have a lot of possibilities to fulfill certain needs without restricting everyone to a single scheme. > My advice would be to have a look at the zinc components and propose things to improve from your perspective. Then publish your results here and there will be a lot of clever people finding a good way to integrate it in a modular way. > > I hope this helps, > > Norbert > > Am 20.07.2012 um 18:25 schrieb Brenda Larcom: > >> I suppose I could unlurk at this point. :) >> >> I'm a security geek (specifically, a secure development geek focusing on security architecture) in my day job, and I have a long unmaintained architecture security analysis tool written in Squeak (http://www.octotrike.org/ for the curious), which I have been unmothballing. We are considering switching to Pharo, partly because we are planning to add some P2P collaboration features we think have an HTTP layer in there somewhere & partly because we like it small, tidy, and self-compatible. Hence my lurking. >> >> I've done some work on how data validation should be done for security purposes, for my day job. This includes output encoding and decoding, like what Davide is talking about. It's pretty tricky to get right because of the large number of contexts, with subtly different rules. E.g. I would expect encodeForHTTP to be appropriate for HTTP headers, except that e.g. two things you usually want to put in HTTP headers are URIs and cookies, each of which have different rules (for different subparts, even) for what should be encoded. The differences don't seem like much, but in the wild, my coworkers & I see these sorts of differences lead to vulnerabilities on a daily basis. >> >> From a security architecture perspective, the absolute best way to handle encoding & decoding for a structured object like an HTTP request or response (or a URI, or a cookie, or an HTML document, or..) is to use a validating parser. Basically, when you get an HTTP request, parse it & put it in an object structured like the request. At that time, you know the meaning of each portion of the string you are parsing, so you can interpret the bits correctly/safely. The object(s) should store the individual strings that are actually content (vs. structure & constants) in a decoded state. The developer should get everything from the objects, in decoded form, and put everything into the objects in decoded form. Then, when it is time to send the response, the objects encode everything safely/canonically based on the exact type of objects they are. This design concentrates the hard stuff (encoding, decoding, canonicalization, layering encodings on top of each other) near the interfaces, at the first/last possible moment enough context is known to interpret the information accurately. It separates the mechanics of using a protocol or format from the intent of using the protocol. It lets someone like me easily QA both the library and application code for security. It is also simple for the developer to use safely (all the dev needs to think about is what objects/content they want to assemble, and the data validation at that layer is taken care of automatically) & is therefore the only design pattern I have seen consistently avoid all encoding-related vulnerabilities in the wild. >> >> So what does this mean? Basically, from a security perspective, encoding & decoding methods should live in the objects they encode and decode, and never be called from outside code. That is, there should be an HTTPHeader>>fromString: or fromStream: method, which is called from an HTTPResponse >>fromString: or fromStream: method, and no String>>decodeFromHTTP. Adding a String>>decodeFromHTTP method is easy from the library maintainer's point of view, approximately correct (way more correct than no method at all), and it matches what most languages are doing these days, but it shifts the burden of all that thought about the specific HTTP header & context to the application developer, who is usually just trying to write an application, not learn every single detail of the HTTP & gazillion other standards he would need to do this safely. >> >> Since this is a suggestion for substantial architecture change that would cause significant backwards compatibility issues throughout the entire Web application stack, and I'm new to Pharo to boot, I am expecting some interesting discussion to occur next. Or maybe profound silence. :) >> >> In my back pocket somewhere amongst the code I am unmothballing, I have 95% of a thouroughly documented URI implementation and test suite that follows this pattern and is pedantically compliant with one or another of the URI RFCs (it's old, may not be the most recent). I believe Spoon & Slate are using a previous version of it or its derivatives. I'll need a fully pedantic HTTP parsing stack to feel comfortable releasing a P2P architecture security analysis tool (high value target, large attack surface, potentially very large professional embarrassment), so whatever isn't available, I expect we'll end up writing. If Pharo folks are interested in this pattern, I would love to contribute my libraries/changes as I finish them, get advice on backward compatibility, performance, and APIs people would like to see, review whatever related code you'd like for security issues, and/or collaborate with any other developer who is interested. >> >> Brenda >> >> >> On Jul 20, 2012, at 1:47 AM, Davide Varvello <varvello(a)yahoo.com> wrote: >> >>> Good Stef, I opened a new feature as reminder here: http://code.google.com/p/pharo/issues/detail?id=6430 >>> >>> Davide >>> >>> ---- >>> - Cerchi un bravo Dentista, Avvocato, Commercialista? Un buon Hotel, Ristorante, Pizzeria? Io l'ho trovato su Oltre il Passaparola >>> >>> - Blog: Cambia il Tempo >>> >>> From: StÃ©phane Ducasse [via Smalltalk] <[hidden email]> >>> To: Davide Varvello <[hidden email]> >>> Sent: Thursday, July 19, 2012 10:43 PM >>> Subject: Re: The opposite of encodeForHTTP >>> >>> Let us fix it and propose a decodeFromHTTP method >>> >>> Stef >>> >>> On Jul 18, 2012, at 2:02 PM, Davide Varvello wrote: >>> >>> > Thanks Sven, >>> > I was looking for String>>decode..whatever... with no luck :-) >>> > Cheers >>> > >>> > -- >>> > View this message in context: http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640510.html >>> > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >>> > >>> >>> >>> >>> >>> If you reply to this email, your message will be added to the discussion below: >>> http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640822.html >>> To unsubscribe from The opposite of encodeForHTTP, click here. >>> NAML >>> >>> >>> >>> View this message in context: Re: The opposite of encodeForHTTP >>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >

July 20, 2012

Re: [Pharo-users] The opposite of encodeForHTTP

by Norbert Hartl

Brenda, these are all good points as you said from a "security architecture perspective" and we should improve on that. The zinc http components do already a good job in structuring the entities as they should be. I think security add-ons can hook onto what is already there. There is a huge amount of things to consider. Even for a single URL the different components of an url have different encoding needs. On the other hand security is not a major target in a lot of use cases I can imagine. There is at least (for me) a triangle of security - performance - usability that makes it hard to have a single approach to fit them all. And we smalltalkers tend to judge freedom very high if it comes to program. In other words I would say we like to preserve the freedom of designing an insecure application at will :) The best way to solve those issues is by being modular, meaning a layer that can be put on top of the existing stuff to fulfill a particular use case. The things you describe are present in a lot of environments. I mostly call this a "at the border of a system" problem. Things like strings inside of an environment are harmless. Problems appear if you cross system borders, meaning you cross interpretation schemes. And this a topic more broad then only HTTP. If we look at a widely known problem like sql injection there is not only the need for proper entity handling but for stacking validators and converters for different problems. It is such a big thing because you have an URL that goes through middleware and ends in a storage system like an SQL database. Here you cross at least two borders: HTTP to middleware and middleware to database. So you need to stack up converters and validators for HTTP, probably shell escapes in a middleware and finally for SQL. I think if you can assemble those things by the layers you use a security approach is doable. And for the same reason it goes so terribly wrong everywhere. So what does this modular thing mean? To have a lot of possibilities to fulfill certain needs without restricting everyone to a single scheme. My advice would be to have a look at the zinc components and propose things to improve from your perspective. Then publish your results here and there will be a lot of clever people finding a good way to integrate it in a modular way. I hope this helps, Norbert Am 20.07.2012 um 18:25 schrieb Brenda Larcom: > I suppose I could unlurk at this point. :) > > I'm a security geek (specifically, a secure development geek focusing on security architecture) in my day job, and I have a long unmaintained architecture security analysis tool written in Squeak (http://www.octotrike.org/ for the curious), which I have been unmothballing. We are considering switching to Pharo, partly because we are planning to add some P2P collaboration features we think have an HTTP layer in there somewhere & partly because we like it small, tidy, and self-compatible. Hence my lurking. > > I've done some work on how data validation should be done for security purposes, for my day job. This includes output encoding and decoding, like what Davide is talking about. It's pretty tricky to get right because of the large number of contexts, with subtly different rules. E.g. I would expect encodeForHTTP to be appropriate for HTTP headers, except that e.g. two things you usually want to put in HTTP headers are URIs and cookies, each of which have different rules (for different subparts, even) for what should be encoded. The differences don't seem like much, but in the wild, my coworkers & I see these sorts of differences lead to vulnerabilities on a daily basis. > > From a security architecture perspective, the absolute best way to handle encoding & decoding for a structured object like an HTTP request or response (or a URI, or a cookie, or an HTML document, or..) is to use a validating parser. Basically, when you get an HTTP request, parse it & put it in an object structured like the request. At that time, you know the meaning of each portion of the string you are parsing, so you can interpret the bits correctly/safely. The object(s) should store the individual strings that are actually content (vs. structure & constants) in a decoded state. The developer should get everything from the objects, in decoded form, and put everything into the objects in decoded form. Then, when it is time to send the response, the objects encode everything safely/canonically based on the exact type of objects they are. This design concentrates the hard stuff (encoding, decoding, canonicalization, layering encodings on top of each other) near the interfaces, at the first/last possible moment enough context is known to interpret the information accurately. It separates the mechanics of using a protocol or format from the intent of using the protocol. It lets someone like me easily QA both the library and application code for security. It is also simple for the developer to use safely (all the dev needs to think about is what objects/content they want to assemble, and the data validation at that layer is taken care of automatically) & is therefore the only design pattern I have seen consistently avoid all encoding-related vulnerabilities in the wild. > > So what does this mean? Basically, from a security perspective, encoding & decoding methods should live in the objects they encode and decode, and never be called from outside code. That is, there should be an HTTPHeader>>fromString: or fromStream: method, which is called from an HTTPResponse >>fromString: or fromStream: method, and no String>>decodeFromHTTP. Adding a String>>decodeFromHTTP method is easy from the library maintainer's point of view, approximately correct (way more correct than no method at all), and it matches what most languages are doing these days, but it shifts the burden of all that thought about the specific HTTP header & context to the application developer, who is usually just trying to write an application, not learn every single detail of the HTTP & gazillion other standards he would need to do this safely. > > Since this is a suggestion for substantial architecture change that would cause significant backwards compatibility issues throughout the entire Web application stack, and I'm new to Pharo to boot, I am expecting some interesting discussion to occur next. Or maybe profound silence. :) > > In my back pocket somewhere amongst the code I am unmothballing, I have 95% of a thouroughly documented URI implementation and test suite that follows this pattern and is pedantically compliant with one or another of the URI RFCs (it's old, may not be the most recent). I believe Spoon & Slate are using a previous version of it or its derivatives. I'll need a fully pedantic HTTP parsing stack to feel comfortable releasing a P2P architecture security analysis tool (high value target, large attack surface, potentially very large professional embarrassment), so whatever isn't available, I expect we'll end up writing. If Pharo folks are interested in this pattern, I would love to contribute my libraries/changes as I finish them, get advice on backward compatibility, performance, and APIs people would like to see, review whatever related code you'd like for security issues, and/or collaborate with any other developer who is interested. > > Brenda > > > On Jul 20, 2012, at 1:47 AM, Davide Varvello <varvello(a)yahoo.com> wrote: > >> Good Stef, I opened a new feature as reminder here: http://code.google.com/p/pharo/issues/detail?id=6430 >> >> Davide >> >> ---- >> - Cerchi un bravo Dentista, Avvocato, Commercialista? Un buon Hotel, Ristorante, Pizzeria? Io l'ho trovato su Oltre il Passaparola >> >> - Blog: Cambia il Tempo >> >> From: StÃ©phane Ducasse [via Smalltalk] <[hidden email]> >> To: Davide Varvello <[hidden email]> >> Sent: Thursday, July 19, 2012 10:43 PM >> Subject: Re: The opposite of encodeForHTTP >> >> Let us fix it and propose a decodeFromHTTP method >> >> Stef >> >> On Jul 18, 2012, at 2:02 PM, Davide Varvello wrote: >> >> > Thanks Sven, >> > I was looking for String>>decode..whatever... with no luck :-) >> > Cheers >> > >> > -- >> > View this message in context: http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640510.html >> > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. >> > >> >> >> >> >> If you reply to this email, your message will be added to the discussion below: >> http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640822.html >> To unsubscribe from The opposite of encodeForHTTP, click here. >> NAML >> >> >> >> View this message in context: Re: The opposite of encodeForHTTP >> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

July 20, 2012

Re: [Pharo-users] Zinc downloadTo: help required

by Sven Van Caekenberghe

On 20 Jul 2012, at 18:22, Chris wrote: > Thanks for that. I'm actually still having a bit of trouble when the file is bigger than the chunk size. ZnChunkedReadStream>>#readInto:startingAt:count: uses a limit variable which is bigger than the collection and requestedCount. I won't be able to do anything until the beginning of August. It would be really helpful if you could provide me with an actual test case that fails. Thanks, Sven

July 20, 2012

Re: [Pharo-users] The opposite of encodeForHTTP

by Brenda Larcom

I suppose I could unlurk at this point. :) I'm a security geek (specifically, a secure development geek focusing on security architecture) in my day job, and I have a long unmaintained architecture security analysis tool written in Squeak (http://www.octotrike.org/ for the curious), which I have been unmothballing. We are considering switching to Pharo, partly because we are planning to add some P2P collaboration features we think have an HTTP layer in there somewhere & partly because we like it small, tidy, and self-compatible. Hence my lurking. I've done some work on how data validation should be done for security purposes, for my day job. This includes output encoding and decoding, like what Davide is talking about. It's pretty tricky to get right because of the large number of contexts, with subtly different rules. E.g. I would expect encodeForHTTP to be appropriate for HTTP headers, except that e.g. two things you usually want to put in HTTP headers are URIs and cookies, each of which have different rules (for different subparts, even) for what should be encoded. The differences don't seem like much, but in the wild, my coworkers & I see these sorts of differences lead to vulnerabilities on a daily basis. From a security architecture perspective, the absolute best way to handle encoding & decoding for a structured object like an HTTP request or response (or a URI, or a cookie, or an HTML document, or..) is to use a validating parser. Basically, when you get an HTTP request, parse it & put it in an object structured like the request. At that time, you know the meaning of each portion of the string you are parsing, so you can interpret the bits correctly/safely. The object(s) should store the individual strings that are actually content (vs. structure & constants) in a decoded state. The developer should get everything from the objects, in decoded form, and put everything into the objects in decoded form. Then, when it is time to send the response, the objects encode everything safely/canonically based on the exact type of objects they are. This design concentrates the hard stuff (encoding, decoding, canonicalization, layering encodings on top of each other) near the interfaces, at the first/last possible moment enough context is known to interpret the information accurately. It separates the mechanics of using a protocol or format from the intent of using the protocol. It lets someone like me easily QA both the library and application code for security. It is also simple for the developer to use safely (all the dev needs to think about is what objects/content they want to assemble, and the data validation at that layer is taken care of automatically) & is therefore the only design pattern I have seen consistently avoid all encoding-related vulnerabilities in the wild. So what does this mean? Basically, from a security perspective, encoding & decoding methods should live in the objects they encode and decode, and never be called from outside code. That is, there should be an HTTPHeader>>fromString: or fromStream: method, which is called from an HTTPResponse >>fromString: or fromStream: method, and no String>>decodeFromHTTP. Adding a String>>decodeFromHTTP method is easy from the library maintainer's point of view, approximately correct (way more correct than no method at all), and it matches what most languages are doing these days, but it shifts the burden of all that thought about the specific HTTP header & context to the application developer, who is usually just trying to write an application, not learn every single detail of the HTTP & gazillion other standards he would need to do this safely. Since this is a suggestion for substantial architecture change that would cause significant backwards compatibility issues throughout the entire Web application stack, and I'm new to Pharo to boot, I am expecting some interesting discussion to occur next. Or maybe profound silence. :) In my back pocket somewhere amongst the code I am unmothballing, I have 95% of a thouroughly documented URI implementation and test suite that follows this pattern and is pedantically compliant with one or another of the URI RFCs (it's old, may not be the most recent). I believe Spoon & Slate are using a previous version of it or its derivatives. I'll need a fully pedantic HTTP parsing stack to feel comfortable releasing a P2P architecture security analysis tool (high value target, large attack surface, potentially very large professional embarrassment), so whatever isn't available, I expect we'll end up writing. If Pharo folks are interested in this pattern, I would love to contribute my libraries/changes as I finish them, get advice on backward compatibility, performance, and APIs people would like to see, review whatever related code you'd like for security issues, and/or collaborate with any other developer who is interested. Brenda On Jul 20, 2012, at 1:47 AM, Davide Varvello <varvello(a)yahoo.com> wrote: > Good Stef, I opened a new feature as reminder here: http://code.google.com/p/pharo/issues/detail?id=6430 > > Davide > > ---- > - Cerchi un bravo Dentista, Avvocato, Commercialista? Un buon Hotel, Ristorante, Pizzeria? Io l'ho trovato su Oltre il Passaparola > > - Blog: Cambia il Tempo > > From: StÃ©phane Ducasse [via Smalltalk] <[hidden email]> > To: Davide Varvello <[hidden email]> > Sent: Thursday, July 19, 2012 10:43 PM > Subject: Re: The opposite of encodeForHTTP > > Let us fix it and propose a decodeFromHTTP method > > Stef > > On Jul 18, 2012, at 2:02 PM, Davide Varvello wrote: > > > Thanks Sven, > > I was looking for String>>decode..whatever... with no luck :-) > > Cheers > > > > -- > > View this message in context: http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640510.html > > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com. > > > > > > > If you reply to this email, your message will be added to the discussion below: > http://forum.world.st/The-opposite-of-encodeForHTTP-tp4640491p4640822.html > To unsubscribe from The opposite of encodeForHTTP, click here. > NAML > > > > View this message in context: Re: The opposite of encodeForHTTP > Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

July 20, 2012

Re: [Pharo-users] Pharo 1.4 "Summer" released!

by Markus Schlager

On Wed, 18 Jul 2012, Esteban Lorenzano wrote: > - Integrated ProfStef. Again, useful for newcomers. ... and for teachers :) thanks a lot Markus

July 20, 2012

Re: [Pharo-users] Zinc downloadTo: help required

by Chris

On 20/07/2012 12:45, Sven Van Caekenberghe wrote: > On 19 Jul 2012, at 21:09, Sven Van Caekenberghe wrote: > >> Yes, this seems to be a problem: your analysis is correct, since the size is unknown upfront - the whole idea of chunked transfer -, #streamFrom:to:size: in its current version cannot work. I will look at this tomorrow, I am pretty sure this can quite easily be fixed (minus the progress bar). > I made some changes and commits. You can load the lastest version of Zn with the following load script > > Gofer it > squeaksource: 'ZincHTTPComponents'; > package: 'Zinc-HTTP'; > package: 'Zinc-FileSystem'; > package: 'Zinc-Tests'; > load > > If you are not on Pharo 2.0, use the following > > Gofer it > squeaksource: 'ZincHTTPComponents'; > package: 'Zinc-HTTP'; > package: 'Zinc-FileSystem-Legacy'; > package: 'Zinc-Tests'; > load > > Now, your example should work fine > > '/tmp/foo.txt' asFileReference ensureDeleted. > ZnClient new url: 'http://www.google.com'; downloadTo: '/tmp/foo.txt'. > Thanks for that. I'm actually still having a bit of trouble when the file is bigger than the chunk size. ZnChunkedReadStream>>#readInto:startingAt:count: uses a limit variable which is bigger than the collection and requestedCount. Regards, Chris

July 20, 2012