Designing for Pluggability

Joe McMahon

Yahoo!

Reminder: Wireless access

We seem to be out of addresses at the moment

- Launch a web browser and follow the instructions. -- Login is 'iit-conference' -- Password is 'guest9900'

why plugins?

<% %>

Technical reasons

Inheritance trouble

  • Some problems handled badly by standard inheritance
  • Sleepy pauses between gets to keep the load down
  • Cached saves a copy of fetched pages
  • Can't do both: sleepy mech has no cache; cached mech doesn't sleep
  • could build a new class to inherit from both, but you have to keep doing this
  • This problem is the real genesis of this talk. I needed to keep adding features to mech, without knowing what might be needed, and I didn't want one gigantic class with 57 unrealted functions in it, which leads into ...

Convenience reasons

do the absolute minimum in your base class, implement functionality in plugins. Mech was already working for the basics; why redo any of that?

People reasons

Some problems are not well-defined, but you need to implement them anyway
  • "we need a simple web-testing framework"
  • "no, we don't know what all we want to test"

What has to happen?

root class knows how to find plugins and load them

dispatch calls to plugins as appropriate

Module::Pluggable

  package CoreClass;
  use Module::Pluggable require=>1;
  ...

  package main;
  use base qw(CoreClass);
  my $obj = CoreClass->new();
  for my $plugin ($obj->plugins) {
    ...
  }
  

great way to find and load your plugins

Very simple interface, very flexible.

base class uses Module::Pluggable

  • creates plugins() instance method
  • CoreClass::Plugin namespace is a plugin.

require=>1 means we've loaded the modules too

Later call to plugins() method returns list of plugin names.

Loaded, but ...

  • ... only half the problem

half-solved: can find and load plugins, can get a list of the plugins that exist. Then what?

There are a bunch of different ways that we can proceed.

Plugin patterns

pass the same data to every plugin

plugin decides whether or not to act

great for a single stream of data

Pipeline example

  package MyApp;
  use Module::Pluggable require=>1;
  sub new { 
    return bless {}, shift;
  }
 
  package main;
  my $app = new MyApp;
  while(defined ($_ = <>)) {
    foreach my $plugin ($app->plugins) {
      $plugin->process_input($_);
    }
  }
  foreach my $plugin ($app->plugins) {
    $plugin->report();
  }

dummy class loads plugins, gives object to call plugins() on.

reads the magic input filehandle and sends every line to every plugin, and then calls the report() method on every plugin at the end.

obviously best for fairly simple plugins that can work independently of each other; plugins handle all execution logic.

Plugin patterns

can "advertise" interfaces to the parent class.

POEST, Casey West's POE-based SMTP server

set of required methods that tell the parent about each plugin:

  1. EVENTS() tells the parent what events this plugin will handle;
  2. CONFIG() tells the parent what configuration data it needs.

parent calls these to figure out when to call and what to pass

can also place data in agreed-upon package global array or hash for each plugin, like Exporter. Just need agreement on what's where

Advertising in POEST

package POEST::Plugin::Check::Hostname;
...

sub EVENTS () { [ qw[ HELO ELOH] ] }
sub CONFIG () { [ qw[ requirehost allowedhost ] ] }
*HELO = *ELOH = sub {
  ...
  if ( $self->{requirehost} ) {
    my (@hosts) = ref $self->{allowedhost} 
      ? @{ $self->{allowedhost} } 
      : $self->{allowedhost};
  ...
  }
};

1;

Note the EVENTS and CONFIG methods

plugin creates subs to handle the event calls.

sub checks the required data as part of its execution.

We'll look at array-based advertising later.

Plugin patterns

Template::Toolkit's plugins are small classes for specific extensions.

plugin creates and returns an object which implements new functions that TT users can call.

one-to-one mapping between a plugin's name and the name of the function it implements.

Delegation in Template Toolkit

   [% USE Image(filename) %]
   [% Image.width %]
   [% Image.height %]
   [% Image.size.join(', ') %]
   [% Image.attr %]
   [% Image.tag %]

Template::Plugin::Image turn USE Image() into Template::Plugin::Image->new()

gives us an Image object

other calls handled through standard method resolution.

Plugin patterns

Mason plugins implement pre and post "hooks" to request processing and component execution.

alter the incoming parameters or return values

Hooks in Mason

package MasonX::Plugin::Timer;
   use base qw(HTML::Mason::Plugin);
   use Time::HiRes;
   sub start_component_hook {
       my ($self, $context) = @_;
       push @{$self->{ timers }}, Time::HiRes::time;
   }
   sub end_component_hook {
       my ($self, $context) = @_;
       my $elapsed = Time::HiRes::time - pop @{$self->{ timers }};
       printf STDERR "Component '%s' took %.1f seconds\n",
           $context->comp->title, $elapsed;
   }
   1;

This plugin grabs the time and stacks it (could be multiple calls to components before this one finishes) in the start hook.

In the end hook, it pops the time and calculates the elapsed time for the component to run.

Plugin patterns

WWW::Mechanize::Pluggable plugins export their methods directly into the parent namespace via a glob assignment.

This is of course the logical equivalent of brain surgery, since we're messing dynamically with the parent class's symbol table.

WWW::Mechanize::Pluggable

sub init {
  no strict 'refs';
  *{caller() . "::snapshots_to"}     = \&snapshots_to;
  *{caller() . "::snapshot"}         = \&snapshot;
  *{caller() . "::_suffix"}          = \&_suffix;
  *{caller() . "::snapshot_comment"} = \&snapshot_comment;
  ...
}
init() gets called for each plugin during WWW::Mech::Pluggable's new(). It gets a copy of the object and exports some of the subs defined here into the caller's namespace. In this architecture, if two plugins export the same method, endless fun results.

Plugin patterns

Lots of ways to do it

Many existing implementation of plugin patterns, but not as many as you might expect.

No way to not have to

DEEP HURTING!

Unfortunately these are all one-shot implementations. They're very well suited to their particular applications, but they aren't useful outside of the particular classes.

The various plugin classes handle the

Don't Repeat Yourself is good practice; so is Don't Repeat Somebody Else

Impatience, Laziness, and Hubris

We'll fix it

Create a class that lets you make anything pluggable

Need a parent base class, and a plugin base class

Take a page from Catalyst/Rails/etc.: minimize code that has to be written

data-structure advertising for plugins

Class::Pluggability

Attributes as interface

	package Vacuum::Plugin::Screamer;
	use base qw(Class::Pluggability::PlugBase);

	sub foo :PluggedMethod(foo) { 
		print "RUNNING!!!!!\n"; 
	}

Damian's rule that the best interface is none at all

To get everything that Mech::Pluggable supports, we'll need a way to define methods, prehooks, and posthooks.

Attributes are a nice way to do this. Only the code for doing things, not making them work.

Attributes as interface

	sub starting :Prehook(bar) {
	  my ($self, $args) = @_;
	  print "About to call ...\n";

	  if ($args eq "skip me") {
	    print "Asked to skip\n";
	    return 1, "SKIPPED!";
	  } else {
	    return 0, @_;
	  }
	}

	sub stopping :Posthook(bar) { 
		print "Call complete ...\n";
	}
    1;

To get everything that Mech::Pluggable supports, we'll need a way to define methods, prehooks, and posthooks.

Attributes are a nice way to do this. Only the code for doing things, not making them work.

Attribute::Handlers

sub PluggedMethod :ATTR(CODE, BEGIN) {
  my ($package, $symbol, $referent, $attr, $data, $phase) = @_;
  _advertise("METHODS", $package, $symbol, $referent, $attr, $data);
  return;
}

sub _advertise {
  my ($queue_name, $package, $symbol, $referent, $attr, $data) = @_;

  no strict 'refs';
  my $target_var = "${package}::$queue_name";
  my $queue = *{$target_var}{ARRAY} || [];
  push @$queue, $data=>$referent;
  *{$target_var} = $queue;
}

CODE - only call this on code

BEGIN - For some reason I haven't yet investigated, the architecture I'm using needs BEGIN instead of CHECK. If this was run during CHECK, the sub we're calling this ATTR on would be defined; it's not yet, so we have to use the DATA to link this back to the code.

Class::Pluggability::Base

	package Class::Pluggability::Base;
	our $AUTOLOAD;
	my @init_methods = qw(_load_methods _set_hooks);
	
	sub new {
	  my ($class, %args) = @_;
	  (my $base_class = $class) =~ s/::Pluggable//;

	  eval "use $base_class";
	  die "$base_class not loadable: $@\n" if $@;

	  my $pluggable_name = $class;
	  $pluggable_name =~ s/Pluggable$/Plugin/ 
	    or die qq(Module name "$class" should end in "::Pluggable"\n);

	  eval <<EOS;
	package $class; 
	use Module::Pluggable search_path=>[qw($pluggable_name)]
	EOS

This is the plugin base class. It contains the code needed to advertise the plugins to the base class.

only basic stuff

  • check to make sure we can load the base class
  • validate this class name
  • get plugins() in this class
  • Aside: the dynamic 'use' of Module::Pluggable seems to be the reason why I need to process the attributes at BEGIN time.
  • Class::Pluggability::Base

    	  my $self = {};
    	  bless $self, $class;
    
    	  $self->{PreHooks} = {};
    	  $self->{PostHooks} = {};
    
    	  local $_;
    	  delete $args{$_} foreach $self->_init(%args);
    	
    	  $self->_consume(@init_methods);	
    	
    	  $self->base_obj($base_class->new(%args));
    	  return $self;
    	}
    

    empty hook queues

    initialize plugins and get rid of extra parameters

    load methods, set hooks

    create and save base class

    Class::Pluggability::Base

    	sub _init {
    	  my ($self, %args) = @_;
    	  # call all the inits (if defined) in all our 
    	  # plugins so they can all set up their defaults
    	  my @deletes;
    
    	  foreach my $plugin ($self->plugins) {
    	    eval "use $plugin";
    	    if ($plugin->can('init')) {
    	      push @deletes, $plugin->init($self, %args);
    	    }
    	  }
    	  @deletes;
    	}
    

    init() - initialize plugins, get list of stuff to lose

    magical extension of new without base class seeing new params

    Class::Pluggability::Base

    	# called with @init_methods = qw(_load_methods _set_hooks);
    	sub _consume {
    	  my ($self, @method_names) = @_;
    	  foreach my $plugin ( $self->plugins) {
    	    foreach my $method (@methods_names) {
    	      $self->$method($plugin);
    	    }
    	  }
    	}
    
    

    consume() - call load_methods and set_hooks for every plugin

    Class::Pluggability::Base

    	sub _load_methods {
    	  my ($self, $plugin) = @_;
    	  my @methods = eval "\@${plugin}::METHODS";
    	  while (@methods) {
    	    (my($method_name, $method_code), @methods) = @methods;
    	    if ($self->can($method_name)) {
    	      warn "$method_name redefined by plugin\n";
    	    }
    	    my $class = ref $self;
    	    my $name = "${class}::". $method_name;
    	    eval "*$name = \$method_code";
    	    die "method import of $method_name failed: $@\n" if $@;
    	   }
    	}
    
    _load_methods installs all the exported methods into the local namespace; we do this here rather than in import() because we're dynamically loading the plugins during new()

    Class::Pluggability::Base

    	sub _set_hooks {
    	  my($self, $plugin) = @_;
    	  my $class = ref $self;
    	  foreach my $hook_type (qw(PRE POST)) {
    	     # Find the hook queue for this plugin
    	     no strict 'refs';
    	     my @hook_queue = eval "\@${plugin}::${hook_type}HOOKS";
    	     my $method_name = lc($hook_type)."_hook";
    	     while ( (my($hooked_method, $hook_sub), @hook_queue) = 
    	              @hook_queue) {
    	       # Install the hooks
    	       $self->$method_name($hooked_method, $hook_sub);
    	     }
    	  }
    	}
    
    set_hooks works very like _install_methods - but it instead hangs things off the prehook and posthook hashes as anonymous arrays of code references, which AUTOLOAD call at the appropriate time.

    Class::Pluggability::Base

    	sub AUTOLOAD { 
    	  return if $AUTOLOAD =~ /DESTROY/;
    
    	  my $self = $_[0];
    
    	  (my $super_sub = $AUTOLOAD) =~ s/::Pluggable//;
    	  my ($plain_sub) = ($AUTOLOAD =~ /.*::(.*)$/);
    
    	  $self->last_method($plain_sub) if ref $self;
    
    	  if (scalar @_ == 0 or !defined $_[0] or !ref $_[0]) {
    	    no strict 'refs';
    	    $super_sub->(@_);
    	  }
          ...
    

    plugin exported methods found by standard resolution (added to class by _load_methods)

    Dispatches to contained object (not a subclass, but has-as relationship)

    wrapped class methods all go through AUTLOLOAD

    calls prehooks (which can modify argument list pre-call, handle the call itself and skip the call, or just do a little something extra), then the method, then posthooks (which can modify the results, or do their own processing)

    Class::Pluggability::Base

    	  ...
    	  else {
    	    my ($result, $argref, $ret, @ret);
    	    shift @_;
    	    my @args = @_;
    	    my $skip;
    	    if (my $pre_hook = $self->{PreHooks}->{$plain_sub}) {
    	      foreach my $hook (@$pre_hook) {
    	        if (wantarray) {
    	          ($result, $argref, @ret) = $hook->($self, @_);
    	        }
    	        else {
    	          ($result, $argref, $ret) = $hook->($self, @_);
    	        }
    	        @args = @$argref if defined $argref;
    	        $skip ||=  (defined $result) && ($result == 1);
    	      }
    	    }
    		...
    

    plugin exported methods found by standard resolution (added to class by _load_methods)

    Dispatches to contained object (not a subclass, but has-as relationship)

    wrapped class methods all go through AUTLOLOAD

    calls prehooks (which can modify argument list pre-call, handle the call itself and skip the call, or just do a little something extra), then the method, then posthooks (which can modify the results, or do their own processing)

    Class::Pluggability::Base

    	  ...
            unless ($skip) {
    	      if (! $self->base_obj->can($plain_sub) ) {
    	        die "can't call $plain_sub() (did your plugins load?)\n";
    	      } 
    	
    	      if (wantarray) {
    	        @ret = $self->base_obj->$plain_sub(@_);
    	      }
    	      else {
    	        $ret = $self->base_obj->$plain_sub(@_);
    	      }
    	    }
    	...
    

    plugin exported methods found by standard resolution (added to class by _load_methods)

    Dispatches to contained object (not a subclass, but has-as relationship)

    wrapped class methods all go through AUTLOLOAD

    calls prehooks (which can modify argument list pre-call, handle the call itself and skip the call, or just do a little something extra), then the method, then posthooks (which can modify the results, or do their own processing)

    Class::Pluggability::Base

    	  ...
    	    if (my $post_hook = $self->{PostHooks}->{$plain_sub}) {
    	      my($maybe_ret, @maybe_ret);
    	      foreach my $hook (@$post_hook) {
    	        if (wantarray) {
    	          my ($result, @maybe_ret) = $hook->($self, @_);
    	          @ret = @maybe_ret if int @maybe_ret;
    	        }
    	        else {
    	          my ($result, $maybe_ret) = $hook->($self, @_);
    	          $ret - $maybe_ret if defined $maybe_ret;
    	        }
    	      }
    	    }
    	    wantarray ? @ret : $ret;
    	  }
    	}
    
    

    plugin exported methods found by standard resolution (added to class by _load_methods)

    Dispatches to contained object (not a subclass, but has-as relationship)

    wrapped class methods all go through AUTLOLOAD

    calls prehooks (which can modify argument list pre-call, handle the call itself and skip the call, or just do a little something extra), then the method, then posthooks (which can modify the results, or do their own processing)

    Now BREATHE

    Seeing is believing

    Let's look at an actual working example.

    Handling the patterns

    standard architecture uses advertising and namespace hacking, provides hooks

    pipelining can be handled by using prehook-only plugins that do the processing plus one plugin that defines the methods that are going to be pipelined.

    Plugin-based architecture concepts

    The real key to a successful plugin-based architecture is to make the main class (or program) as close to an empty shell as possible. The more that the function lives in the plugins, the more flexible you'll be able to be. The best balance is probably to have the basic runloop (read lines and print, accept incoming connections and send back results, fork processes, whatever) be either completely empty, or support the one basic function you want (e.g, WWW::Mechanize::Pluggable is completely empty; it doesn't know how to do anything except pass method calls to WWW::mechanize. App::SimpleScan can read lines, do variable substitutions, and turn test specs into simple tst code).

    A delegation, pipeline, or other redispatch mechanism. Mech::PLuggable's redispatch is a "let Mech eat it".

    It's possible to have multiple types of plugin: one set extends skeleton database functions, while another extends the command set.

    Let yourself consider making anything pluggable that has more than one option, and that you might want to extend.

    Plans

    standard architecture uses advertising and namespace hacking, provides hooks

    pipelining can be handled by using prehook-only plugins that do the processing plus one plugin that defines the methods that are going to be pipelined.

    Thank you!