Dynamic Reception

Handle messages without defining them in the receiving class.

Also Known As: Overriding method_missing, Overriding doesNotUnderstand

Any object has a limited set of methods that are defined for it. A client of an object may attempt to invoke a method that isn't defined on the receiver. A statically typed language will spot this at compile-time and cause a compilation error. As a result you know you won't get this kind of error at run-time (unless you do some clever fiddling to get around the type system). With a dynmacially typed language you can invoke a non-existent method at run time and you usually get a run-time error.

Dynamic Reception allows you to adjust this behavior, allowing you respond differently to an unknown message.

How it Works

Many dynamic languages react to an unknown method invocation by calling a special error handling method at the top of the object hierarchy. There is no standard name for this method: in smalltalk it's doesNotUnderstand and ruby it's method_missing. You can introduce your own processing for an unknown method by sub-classing these methods in your own class. When you do this you are essentially dynmically altering the rules for the reception of method calls.

There are many reasons why Dynamic Reception is useful in general programming. One excellent example is supporting automatic delegation to another object. To do this you define the methods that you wish to handle in the original receiver and use Dynamic Reception to send any unknown messages to a delegate object.

Dynamic Reception can feature in several ways in DSL work. One common use is to convert what might otherwise be method parameters into the name of the method. A good example of this is Rails's Active Record dynamic finders. Say you have a Person class with firstname and lastname properties. With these defined you can call find_by_firstname("martin") or find_by_firstname_and_lastname("martin", "fowler") without having the define these methods. The code in the active record super-class overrides Ruby's method_missing and checks to see if the method call begins with "find_by". If so it then parses the method name to find the property names and uses them to construct a query. You could do this by passing in multiple arguments such as find_by("firstname", "martin", "lastname", "fowler"), but putting the property names into the method name is more readable as it mimics what you would actually do if you explicitly defined methods like that.

A method like find_by_name works by taking a single method name and parsing it. Essentially you are embedding an external DSL in the method name. Another approach is to use a sequence of Dynamic Receptions: something like find_by.firstname("martin").and.lastname("fowler) or find_by.firstname.martin.and.lastname.fowler. In this case the find_by method would return an Expression Builder that you can use to build up a query using Method Chaining and Dynamic Reception.

One of the advantages of doing this is that can avoid quoting the various parameters using martin rather than "martin" to reduce noise. If you are using Object Scoping you can use this mechanism to allow you to use bare symbols for arguments, using state idle instead of state :idle. You can do this by implementing Dynamic Reception in the super-class so that once the object has had state invoked it will override the next unknown method call to capture the name of the state.

A further variation on Dynamic Reception is to combine it with a series of regular expression substitutions to implement a simple external DSL. [TBD: Jay's bnl ideas - need separate pattern for this.].

When to use it

Using Dynamic Reception to move parameters into method names is appealing for a couple of reasons. Firstly it can mimic what you would genuinely do with a method with less effort. It's quite reasonable to imagine a person class having a find_by_firstname_and_lastname method, by using Dynamic Reception you are providing this method without having to actually program it. This can be a significant time-saver, particularly if you are using lots of combinations. Certainly there are other ways you can do it. You can put attribute name into the parameters find(:firstname, "martin", :lastname, "fowler"), use a closure find {|p| p.firstname == "martin" and p.lastname = "fowler"}, or even a fragmentary external DSL in a string find("firstname == martin lastname == fowler"). The point of embedding the field names into the method name is that many people find that the most fluent way to express the call.

Another benefit of replacing parameters by method names is that it can give you better consistency in punctuation. An expression like find.by.firstname.martin.and.lastname.fowler uses dots as the only form of punctuation. This has an advantage in that people won't get confused between when they should use a dot and when they should use a parenthesis or when need to quote. For many others this consistency isn't a virtue, I like separating what is schema from what is data, so I like the way that find_by.firstname("martin").and.lastname("fowler") puts field names as method calls and the data as parameters.

Above all of this, it's important to remember that Dynamic Reception only pays its way when it allows you to build these structures in general without any special case handling. This means that it's only worthwhile when you can clearly translate from the dynamic methods to methods that are needed for other purposes. Conditions are good examples of this because they usually call attributes on domain model objects. find_by_firstname_and_lastname is effective because I have a person class that has firstname and lastname attributes. If you need to write special methods to handle particular cases of Dynamic Reception then that usually means you shouldn't be using Dynamic Reception.

Dynamic Reception comes with many problems and limitations. The biggest one, of course, is that you can't do this at all with static languages. But even in a dynamic language you need to be wary about using it. Once you override the handler for unknown method invocations, any mistake can lead you into deep debugging trouble. Stack traces often become impenetrable.

There are also limitations on what you can express. You usually can't put in something like find_by.age.greater_than.2 because most dynamic languages won't allow "2" to be a method name. You can dodge around it with something like find_by.age.greater_than.n2 but that obstructs much of the fluency that you're doing this for.

Since I've focused a lot on Boolean expressions here, I should also point out that this kind of method call composition for Boolean expressions is not a good way of describing complex Boolean conditions. The approach is fine for something simple like find_by.firstname("martin").and.lastname("fowler") but once you get to statements like find_by.firstname.like.("m*").and.age.greater_than(40).and.not.employer.like("thought*") you're running down a road that forces you to implement a kludgy parser in an environment not well suited for it.

The fact that expressions using Dynamic Reception don't work well for complex conditionals isn't a reason to avoid them for simple cases. Active Record uses Dynamic Reception to provide dynamic finders for simple cases, but deliberately does not support more complex expressions, encouraging you to use a different mechanism instead. Some people don't like that, preferring a single mechanism, but I think it's good to realize that different solutions work better at different complexities and provide more than one.

Example: Promotion Points using parsed method names (Ruby)

For this example, let's consider a scheme that assigns points to travel itineraries. Our domain model is an itinerary that consists of items, where each item might be a flight, hotel stay, car hire, etc. We want to allow a flexible way for people to score frequent travel points, such as scoring 300 points for taking a flight out of Boston.

Using Dynamic Reception I'll show this to support the following cases. Firstly a simple case of one simple promotion rule

    @builder = PromotionBuilder.new
    @builder.score(300).when_from("BOS")

Also a case where we can have multiple promotion rules matching different kinds of elements. Here we score for flying out of a particular airport and staying an a particular hotel brand within the same itinerary

    @builder = PromotionBuilder.new
    @builder.score(350).when_from("BOS")
    @builder.score(100).when_brand("hyatt")

And finally a compound flight rule, where we score for flying out of Boston on a particular airline (which may not be around any more when you read this)

    @builder = PromotionBuilder.new
    @builder.score(140).when_from_and_airline("BOS","NW")

Model

The model here has two parts: itineraries and promotions. The itinerary is just a collection of items, which could be all sorts of things. For this simple case I just have flights and hotels.

class Itinerary
  def initialize
    @items = []
  end
  def << arg
    @items << arg
  end
  def items
    return @items.dup
  end
end

class Flight
  attr_reader :from, :to, :airline
  def initialize airline, from, to
    @from, @to, @airline = from, to, airline
  end
end

class Hotel
  attr_accessor :nights, :brand
  def initialize  brand, nights
    @nights, @brand = nights, brand
  end
end

Promotions are a set of rules, where each rule has a score and a list of conditions.

class Itinerary...
  def initialize rules
    @rules = rules
  end

class PromotionRule...
  def initialize anInteger
    @score = anInteger
    @conditions = []
  end
  def add_condition aPromotionCondition
    @conditions << aPromotionCondition
  end

The approach here scores an itinerary against a promotion. It does this by scoring each rule against the itinerary and summing up the results

class Itinerary...
  def score_of anItinerary
    return @rules.inject(0) {|sum, r| sum += r.score_of(anItinerary)}
  end

The rule scores an itinerary by looking to see if all its conditions match the itinerary, if so it returns its score.

class PromotionRule...
  def score_of anItinerary
    return (@conditions.all?{|c| c.match(anItinerary)}) ? @score : 0
  end

Each score line in the DSL is a separate rule. So

    @builder = PromotionBuilder.new
    @builder.score(350).when_from("BOS")
    @builder.score(100).when_brand("hyatt")

is one promotion with two rules. Either or both of them could match a given itinerary. In contrast

    @builder = PromotionBuilder.new
    @builder.score(140).when_from_and_airline("BOS","NW")

is one rule with two conditions. Both conditions have to match to score the points.

To handle this I have an equality condition object that I can set with appropriate names and values.

class EqualityCondition
  def initialize aSymbol, value
    @attribute, @value = aSymbol, value
  end
  def match anItinerary
    return anItinerary.items.any?{|i| match_item i}
  end
  def match_item anItem
    return false unless anItem.respond_to?(@attribute)
    return @value == anItem.send(@attribute)
  end     
end

Using equality conditions in the method name like this is very limited. However the underlying model allows me to have any kind of condition as long as it know how to match to an itinerary. Some of these conditions could be added through the DSL, others through other means, such as a closure.

example......
    rule = PromotionRule.newWithBlock(520) do |itinerary|
      flights = itinerary.items.select{|i| i.kind_of? Flight}
      flights.any? {|f| f.from == "LAX"} and
        flights.any? {|f| f.to == "LAX"} and
        flights.all? {|f| %w[NW CO DL].include?(f.airline)}
    end
    promotion = Promotion.new([rule])

class BlockCondition
  def initialize aBlock
    @block = aBlock
  end
  def match anItinerary
    @block.call(anItinerary)
  end
end

This kind of flexibility can be quite important. It allows people to use the DSL to handle simple cases simply, but provides an alternative mechanism to handle more complicated cases.

Builder

The basic builder wraps a collection of promotion rules that it builds up returning a new promotion object as needed.

class PromotionBuilder...
  def initialize
    @rules = []
  end
  def content
    return Promotion.new(@rules)
  end

The score method creates one of these rules which it holds in a Context Variable. It also creates a particular builder for the condition.

class PromotionBuilder...
  def score anInteger
    @rules << PromotionRule.new(anInteger)
    return PromotionConditionBuilder.new(self)
  end 

The condition builder is the class that uses Dynamic Reception. In ruby you do Dynamic Reception by overriding method_missing.

class PromotionConditionBuilder...
  def initialize parent
    @parent = parent
  end
  def method_missing(method_id, *args)
    if match = /when_([_a-zA-Z]\w*)/.match(method_id.to_s)
      attribute_names = extract_attribute_names_from_match(match)
      check_number_of_attributes(attribute_names, args)
      populate_rules(attribute_names, args)
    else
      super
    end
  end

The method_missing hook looks to see if the caller begins with "when_", if not it forwards to the super-class, which will throw an exception. Assuming we have the right kind of method it pulls the attribute names out of the method call, checks they match the arguments, and then creates the appropriate rules.

class PromotionConditionBuilder...
  def extract_attribute_names_from_match(match)
    match.captures.last.split('_and_')
  end
  def check_number_of_attributes(names, values)
    unless names.size == values.size
      throw "There are %d attribute names but %d arguments" % 
        [names.size, values.size]
    end
  end
  def populate_rules names, args
    names.zip(args).each do |name, value|
      @parent.add_condition(EqualityCondition.new(name, value))
    end
  end

This approach is unsurprisingly similar to Active Record's dynamic finders, if you're curious about those take a look at Jamis Buck's description.

Example: Promotion Points using Chaining (Ruby)

Now I'll take the same example and work it using chaining. I'll use the same model and (mostly) the same example conditions. As the DSL is different the conditions are formulated differently. Here's the simple single selection of flights out of Boston.

    @builder.score(300).when.from.equals.BOS

In this case I'm passing all the arguments to the condition as methods rather than parameters (although I'm keeping the score as a parameter just to be inconsistent). I'm also indicating the operator for the condition as a method.

Here's the case with two separate scores.

    @builder.score(350).when.from.equals.BOS
    @builder.score(100).when.brand.equals.hyatt

Finally here I have a compound condition.

    @builder.score(170).when.from.equals.BOS.and.nights.at.least._3

The compound condition is more involved than the one I used in the previous example. For this one I'm taking advantage of the ability to use other operators as well as showing the kind of smudge you need to make in order to pass a numeric parameter as a method name.

Model

The model is almost identical to the one I used in the previous example. The only change is that I've added an extra condition.

class AtLeastCondition...
  def initialize aSymbol, value
    @attribute, @value = aSymbol, value
  end
  def match anItinerary
    return anItinerary.items.any?{|i| match_item i}
  end
  def match_item anItem
    return false unless anItem.respond_to?(@attribute)
    return @value <= anItem.send(@attribute)
  end

Builder

The differences to the previous example lie in the builder. As before I have a promotion builder object that holds a bunch of rules and produces a promotion when needed.

class PromotionBuilder...
  def initialize
    @rules = []
  end
  def content
    return Promotion.new(@rules)
  end

The score method adds a rule to the rules list.

class PromotionBuilder...
  def score anInteger
    @rules << PromotionRule.new(anInteger)
    return self
  end 

The when method returns a more specific builder to capture the attribute name.

class PromotionBuilder...
  def when
    return ConditionAtributeNameBuilder.new(self)
  end

class ConditionAtributeNameBuilder < Builder
  def initialize parent
    @parent = PromotionConditionBuilder.new(parent)
    @parent.name = self
  end

class Builder
  attr_accessor :content, :parent
  def initialize parentBuilder = nil
    @parent = parentBuilder
  end
end

class PromotionConditionBuilder < Builder
  attr_accessor :name, :operator, :value

To build the condition I create a little parse tree. Each condition in an expression has three parts: name, operator, and condition. So I make a builder for each of the parts as well as a parent builder to tie the condition together. As a result when I create the name builder, I also create the condition builder parent to prepare the tree.

The attribute name builder will look for a suitable name for the attribute we are testing, since this name will vary depending on the model class's attributes I use Dynamic Reception

class ConditionAtributeNameBuilder...
  def method_missing method_id, *args
    @content = method_id.to_s
    return ConditionOperatorBuilder.new(@parent)
  end

This captures the name, and returns the operator builder to capture the operator.

The operator builder only will have a fixed set of operators to work off, so it doesn't need to use Dynamic Reception.

class ConditionOperatorBuilder < Builder
  attr_reader :condition_class
  def initialize parent
    super
    @parent.operator = self
  end
  def equals 
    @content = EqualityCondition
    return next_builder
   end
  def at
    return self
  end
  def least
    @content = AtLeastCondition
    return next_builder
  end
  def next_builder
    return ConditionValueBuilder.new(@parent)
  end  

The basic behavior of the operator builder is similar to the name builder: capture the operator and return a new builder for the final part (the value). There are a couple of interesting points. Firstly the content of this builder is the appropriate condition class from the model. Secondly the at method just returns itself as it's just syntactic sugar - only there to make the expression readable.

The final builder is the value builder which captures the value using Dynamic Reception.

class ConditionValueBuilder < Builder
  def initialize parent
    super
    @parent.value = self
  end
  def method_missing method_id, *args
    @content = method_id.to_s
    @content = @content.to_i if @content =~ /^_\d+$/
    @parent.end_condition
  end  
end

If the value is a number, I need some jiggery-pokery

This method also ends this part of the expression so it tells it's parent to populate the model.

class PromotionConditionBuilder...
  def end_condition
    content = @operator.build_content(@name.content, @value.content)
    @parent.add_condition content
    return @parent
  end

class ConditionOperatorBuilder...
  def build_content name, value
    return @content.new(name, value)
  end

class PromotionBuilder...
  def add_condition cond
    current_rule.add_condition cond
  end
  def current_rule
    @rules.last
  end

At this point I've consumed the little parse tree and created the condition object in the model. If I have a compound condition, I repeat the process.

class PromotionBuilder...
  def and
    return ConditionAtributeNameBuilder.new(self)
  end

Making little parse trees like this isn't a common way to do an internal DSL, it's usually easier to just build the model up as you go. But with an conditional expression like this, it makes sense.

Overall, however, I'm not too keen on building up expressions using this approach. It seems to me that once you start parsing sequences of method calls like this, you might as well just switch to an external DSL where you get more flexibility. The desire to build up parse trees is a smell indicating that the internal DSL is doing too much work.

Example: Removing quoting in the Secret Panel Controller (JRuby)

In the introduction I showed an example of how you could use Ruby as an internal DSL for the secret panel controller. The code looks like this

event :doorClosed, "D1CL"
event :drawOpened,  "D2OP"
event :lightOn, "L1ON"
event :doorOpened,  "D1OP"
event :panelClosed, "PNCL"

command  :unlockPanel, "PNUL"
command  :lockPanel,   "PNLK"
command  :lockDoor,    "D1LK"
command  :unlockDoor,  "D1UL"

resetEvents :doorOpened

state :idle do
  actions :unlockDoor, :lockPanel
  transitions :doorClosed => :active
end

state :active do
  transitions :drawOpened => :waitingForLight,
              :lightOn => :waitingForDraw
end

state :waitingForLight do
  transitions :lightOn => :unlockedPanel
end

state :waitingForDraw do
  transitions :drawOpened => :unlockedPanel
end

state :unlockedPanel do
  actions :unlockPanel, :lockDoor
  transitions :panelClosed => :idle
end

In this example code I don't use any Dynamic Reception, relying on simple function calls. One of the disadvantages of the this script is there's a lot of quoting, in particular every reference to an identifier needs Ruby's symbol marker (the initial ':' in the names). Compared to an external DSL this feels like noise. If I use Dynamic Reception it's possible to get rid of all of the symbol quoting and produce a script like this.

events do
  doorClosed  "D1CL"
  drawOpened  "D2OP"
  lightOn     "L1ON"
  doorOpened  "D1OP"
  panelClosed "PNCL"
end

commands do
  unlockPanel "PNUL"
  lockPanel   "PNLK"
  lockDoor    "D1LK"
  unlockDoor  "D1UL"
end

reset_events do
  doorOpened
end

state.idle do
  actions.unlockDoor.lockPanel
  doorClosed.to.active
end

state.active do
  drawOpened.to.waitingForLight
  lightOn.to.waitingForDraw
end

state.waitingForLight do
  lightOn.to.unlockedPanel
end

state.waitingForDraw do
  drawOpened.to.unlockedPanel
end

state.unlockedPanel do
  panelClosed.to.idle
  actions.unlockPanel.lockDoor
end

The starting point for implementing this is a state machine builder class. This class uses Object Scoping using instance_eval. The build occurs in two stages, first evaluating the script and then some post-processing.

class StateMachineBuilder...
  attr_reader :machine
  def initialize
    @states = {}
    @events = {}
    @commands = {}
    @state_blocks = {}
    @reset_events = []
  end
  def load aString
    instance_eval aString
    build_machine
    return self
  end

To evaluate the script the builder has methods that correspond to the main clauses of the DSL script. I'm using the same Semantic Model that I used in the introduction, the JRuby builder populates Java objects.

The first clause to look at is the event declarations, which I make by calling the events method on the state machine builder passing in a block that contains the individual event declarations.

class StateMachineBuilder...
  def events &block
    EventBuilder.new(self).instance_eval(&block)
    self
  end

  def add_event name, code
    @events[name] = Event.new(name.to_s, code)
  end

class EventBuilder < Builder
  def method_missing name, *args
    @parent.add_event(name, args[0])
  end
end

class Builder
  def initialize parent
    @parent = parent
  end
end

The events method evaluates the block immediately in the context of a separate builder which uses Dynamic Reception to process every method call as an event declaration. With each event declaration I create an event from the Semantic Model and put it into a Symbol Table.

I use the same basic technique for commands and reset events. By using a different builder I can keep each one simple and clearly scope what each builder is recognizing.

The state declaration is more interesting. Again I use a closure to capture the body of the declaration, but there are a couple of differences. The obvious one from the script is that I indicate the name with Dynamic Reception.

class StateMachineBuilder...
  def state
    return StateNameBuilder.new(self)
  end

  def addState name, block
    @states[name] = State.new(name.to_s)
    @state_blocks[name] = block
    @start_state ||= @states[name]
  end

class StateNameBuilder < Builder
  def method_missing name, *args, &block
    @parent.addState(name, block)
    return @parent
  end
end

The second difference is in the implementation, rather than evaluate the Nested Closure right away, I squirrel it away in a map. By deferring the evaluation till later I can avoid worrying about the forward references between states. I can wait to deal with the state bodies until I've declared all the states and fully populated the Symbol Table with them.

The last point is that I treat the first-named state as the start state by using an additional variable which I populate only if it's already nil - which means that the first state will be in there.

Populating this data finishes evaluating the script, now the second stage is the post-processing.

class StateMachineBuilder...
  def build_machine
    @state_blocks.each do |key, value|
      if value
        sb = StateBodyBuilder.new(self, @states[key])
        sb.instance_eval(&value) 
      end
    end
    @machine =  StateMachine.new(@start_state)
    @machine.addResetEvents(@reset_events.collect{|e| @events[e]}.to_java("gothic.model.Event"))
  end

class StateBodyBuilder < Builder...
  def initialize parent, state
    super parent
    @state = state
  end

The first step of post-processing is to evaluate the bodies of the state declarations, again by creating a specific builder and instance_evaling the block with it.

The body can contain two kinds of statements: declaring actions and declaring transitions. The action case is handled by a specific method.

class StateBodyBuilder...
  def actions
    return ActionListBuilder.new(self)
  end
  def add_action name
    @state.addAction(@parent.command_at(name))
  end

class ActionListBuilder < Builder
  def method_missing name, *args
    @parent.add_action name
    return self
  end
end

class StateMachineBuilder...
  def command_at name
    return @commands[name]
  end

The actions creates another builder which absorbs all method calls as command names. This allows you to specify multiple actions in a single line with chaining.

While actions uses a special method, analogous to a keyword in an external DSL, transitions use Dynamic Reception.

class StateBodyBuilder...
  def method_missing name, *args
    return TransitionBuilder.new(self, name)
  end
  def add_transition event, target
    @state.addTransition(@parent.event_at(event), @parent.state_at(target))
  end

class TransitionBuilder < Builder
  def initialize parent, event
    super parent
    @event = event
  end
  def to
    return self
  end
  def method_missing name, *args
    @target = name
    @parent.add_transition @event, @target
    return @parent
  end
end

class StateMachineBuilder...
  def event_at name
    return @events[name]
  end
  def state_at name
    return @states[name]
  end

Here I use an unknown method to start a specific builder to capture the target state with a further use of Dynamic Reception. I also allow to as syntactic sugar.

By doing all of this, I can get rid of all the ":" on symbols. The questions, of course, is whether it's worth the trouble. To my eye I like the way the event and command list turn out, but I'm not so keen on the states. I could, of course, use a hybrid approach using Dynamic Reception for the things I like and using symbol references where Dynamic Reception isn't helping. A mixture of techniques is often the best bet.