Comments, REST, Interactions, and Extensible Hypermedia

This post is to mostly keep track of the numerous blog threads going on about IDLs and schemas for REST. I find myself with more to say that wit to organize it... :(

First my summaries of opinions, then links to my comments here and there.

Some of my opinions on a RESTful Client Engine, and what types of server-side changes would NOT break a client:

• URIs are discovered (except the first)
• Server-provided data extensions (hidden fields with defaults in a form) are treated like "does not understand" but still submitted properly
• Changes to which HTTP verb are used. The server can swap PUT for POST without issue (verb is discovered in form not spec'ed in a schema)
• Changes in state path. "checkout" could directly be a single form with POST, a GET link to a single for with POST, a POST to a form with reliable PUT semantics, .......
  

Links,opinions on the changing landscape of content (and predictions of the future of hypermedia content):

• Sam Ruby's suggestions for HTML5 Distributed Extensibility is a fabulous starting point.
• URI Templates, HTML Forms, XForms, Web Forms, WADL
  
• XSD is being extended to better support versioning
• Data schemas designed for extensibility should allow everything possible to be optional
  
• Microformats enable opportunistic clients to machine process
• GRDDL leaps from microformats, xml, some json up to RDF
• HTML+Microformats can do a ton of this already, but with no machine processable anything
• Prediction: in 10 years all of this this will have exploded/merged into one or a few really cool and evolvable data/interaction schema systems
  

References to programming and type systems that address these issues more appropriately than any IDL (as far as I can get into them so far... :)

• from Stu Charlton comes a reference to interaction machines
• from Todd Hoff comes a reference to two NASA reference on Mision Planning and Closed Loop Execution and Recovery
• papers from Luis Caires, for example Spatial-Behavioral Types, Distributed Services, and Resources
  (forget where I found this material from... can't understand it yet...)
  

The following is a guide to my recent comments on these threads from around the web:

Subbu says:

Should this idea be extended to the rest of non-user facing resource-oriented applications? I don't think so. Here is why.

The idea of hypermedia embedding all the action controls necessary to interact with the server works well for an arbitrary number of universal clients interacting with a given server. In this case, the server offering a set of resources specifies all the ordering/interaction rules within the representation. Most application clients, on the other hand, interact with more than one server, and the ordering constraints can not be set by any given server. The clients know how to compose applications out of resources offered by various servers, and each client needs to be able to exercise control over composition. To be able to exercise such a control, client applications can not be universal, and the benefits that John lists above cannot be completely realized.

My comments is that clients should be opportunistic: if they understand some shared semantic (like a RESTful shopping API or task manager) then they can automate some interactions.

On JJ Dubray's blog I make several comments.

In response to "REST creates strong coupling", I say:

• JJ is ignoring the shared definitions of MIME types
• The globally shared "Provider external" (Pe) semantics in REST are (URI, GET representation, hyperlinks)
• in WS-* the Pe globally is 0 (zero), only particular shared uses have a shared semantic.
  

In response to "REST and inter-actions", I say (and say and say):

• For me REST in the enterprise isn't about scalability, but rather independent evolution and support for partial failure.
• In maybe 10 years there will be an XML schema language that properly supports versioning and extensibility

In Steve Vinoski's blog on "IDLs vs Human Documentation" I comment that:

• I side with Patrick Mueller: there is something of value in more than just prose to document RESTful systems
• Hypermedia MIME types are _more_ than just a data schema (embedded forms to signify actions)
• A list of RESTful interactions that _shouldn't_ break a programmed client (listed below)

Premature Conclusions with Services and REST

Damn, I wish we were at a white board. Oh well.

JJ, you've drawn a lot of conclusions from talking with a few people in the REST community. I think you've gotten ahead of yourself.

To defend the "REST community": What we are talking about isn't where most of the value is. Not yet anyway. Maybe someday this issue will really be an issue, but right now just GETing info with declarative representation data is so unusual that we're on the fringe.

Really, I believe this on the internet at large and in many organizations as well (enterprisey doesn't even get funny when sharing a database is so common.) . How much value does a definition language really have? Well, some. Just like I appreciate static languages for documentation and _some_ checking, having a WSDL or something is useful. None of the things being thrown around for "interface definition language" are much more useful than being able to organize documentation, generate code (a bad idea), and create pictures (sometimes useful for humans). That last point, the human part, is where you started this conversation... Let's not pretend that WSDL allows machines to process things for us.

To your recent post. I don't have much time tonight, but hear is my reaction.

Shared Understanding.
This isn't actually denied by the REST. It's just relegated to the media type.

The "contract" that is specific to a particular domain in a RESTful system is buried in the Representation. That means a) we all collectively don't know much about it, b) every system can use the standard VERBS to explore and partially integrate. The first point means we need to have this very conversation, but the value of that struggle is more common ground between all services.

Result Sets aren't Resources
Huh? Why not? I'll need to read up on the posts, but everything interesting is a resource. I don't understand why you conclude this.

Some of your recent posts have indicated what I think may be a confusion regarding the "uniform interface" and resources. Not every resource must support and expose each method. Part of the discovery of REST is learning (at runtime) what methods are available for each resources. Results sets don't need to expose PUT.

How many people are in this discussion?
Not many, I'm afraid. You're assuming that people won't discuss these issues with you, but it's probably that it doesn't matter to most of the RESTful systems out there.

As an example: the Amazon service exposed this weekend. It is people (not machine processes) that are writing the wrapper layers around the new API. It almost doesn't matter what they had exposed it in, people need to spend thought understanding it and wrapping it. Then, other people, need to consume the wrappers to write applications.

Had Amazon used some magically fantastic definition language it wouldn't change but a part of the total human/computer processing cost and value proposition. Again, I think there is value in understanding and exposing that fantastic definition language. I'm not assuming it will magically compose my systems. (I'm not necessarily claiming you make that declaration either, but the value you place on the formality seems too strong. I will read your book on composition to really understand what you are saying.)

REST and actions
There is something critically important about the distinction between "action interfaces" and "document exchange". You took Subbu to task for trying to remove actions, but I think there is a truly subtle reason that REST motivates that thinking.

I'm not sure I can express it well, but this is my intuitive view:

• contract negotiation is an (endless?) cycle of document exchange
• reserving a _good_ hotel room is an cycle of question/answer...
  
• bartering is a cycle of bid exchange
• resolving a speeding ticket is ...
  

what these real ways of getting things done tell me is that an interface contract doesn't model the real world.

REST models "action" as one or more transfer of document representation. That is more like the world than interface actions.

REST and versioning.
Really bad at versioning? Seriously? If you'd said "efficiency", or "tool support", or "machine clients" I'd be right there saying "Yeah, maybe this would improve that..."

Versioning is hard because something wants to change - and the rest of the system isn't yet ready to change as well. REST has from the beginning been about supporting that change (across organizations, and between the client and server). The very reason that interface definition languages are so hard to reconcile with REST is because of the radically different view on how to support evolution.

(I'm posting this without all the links...)

Shared Understanding and/or Evolvability

(This is belated and too short. All you fast-typing bloggers out there can just be patient :) Sorry for the delay!)

How do distributed systems both cooperate and evolve?

That is a subtly different question than JJ asks:

when no human is in the loop, you need a shared understanding between the resource consumer and the resource (provider)

The only viable choices JJ then describes (WSDL or WADL) do indeed provide a partial solution to coding shared understanding, but not evolution.

The short (flip?) answer to my leading question is to create a new media type that describes the semantics necessary for both consumers and providers to communicate.

In the case of exposing a Job Application Service as a RESTful provider that could mean the following:

• for a human provide an HTML response representation with forms for review, cancel, submit interview.
• for a machine provide an XML or RDF (or something) response representation.

That machine representation must have a lot of things going for it. It must provide just enough shared information to do useful things, but not prevent service evolution. In particular, the response must:

• provide links/conformance to some shared schema type(s) (to share semantics)
• be extensible (in the "does not understand" sense)
• the shared schema can't pre-define URIs
• the shared schema can't pre-constrain all of the transition paths
  

That last one is really fuzzy, but I'm trying to express the idea with my "signposts" metaphor. Getting something done (i.e. changing the state of something) isn't alway a single step, and how many steps have to be followed isn't a very stable property. In the case of job applications, getting from "offered" to "accepted" might take a few loops in the 1.1 release of the service, or maybe the "review" state gets split into 2 steps, what happens then?

The current conception of "shared understanding" is "shared interface". In the Job Application example that means the client service is encoded to expect after "Submit Review" that the service moved the job app to "reviewed". If the client service instead was coded to have a pair of (current job app data, desired state) when the service changed to a 2-stage review process the clients would likely continue to work:

• Client: Get job app 123
• Service: job app 123, submitted, "Submit Review", "Cancel"
  
• Client: "Submit Review" for job app 123
• Service: job app 123, review1, "Submit Review", "Cancel", "Reject"
• Client: "Submit Review" for job app 123
• Service: job app 123, reviewed, "Submit Interview", "Cancel", "Reject"
  

Understanding how to encode an imperative SOA static model to a conversational document interchange is not really that well understood yet (certainly to me). I think someday it will be, but right now there is just the quotes from the REST thesis I commented on and a reference to "data reactive programming" from Roy Fielding, and I just found Mimosa.

Just need to make sure of something about the Job Application Service example: is this service the provider or consumer of the lifecycle of a job app? It clearly has an internal state machine for job apps, but what other machine process does that get shared with?

[1] Some discussion (still searching for link...) hinted that WADL could be simply returned like an HTML Form, and not used to statically generate code. I think avoids the coupling of code-gen, but I'm not sure how it solves the problem of shared understanding either.

Passing and Returning Nulls

Several blog posts about handling null values caught my attention.

The pair of posts from Marty Alchin on "Returning None is Evil" and "Except the Unexpected" offer these opinions:

• most methods can return null
• Java has lots of APIs that silently return null (java.util.Map.get(Object key))
• returned nulls are annoying and hard to debug
• exceptions should be preferred when a non-null value isn't available
• always consider when/how nulls should(n't) be used
  

Cedric Otaku in "It's okay to return null" chimes in with these valid points:

• don't return exceptions unless something really is exceptional
• use the Null Object pattern if you need it
  

Neither of those suggestions (exceptions or Null Object) really work out well for my taste. I really don't like these bad choices:

1. Write clear and direct code -- that neither shows where nulls could be hiding nor handles them well when they occur
2. Write lots of verbose if/then/else and try/catch blocks to deal with nulls everywhere
   

So what to do? Borrow a construct from somewhere else of course!

In the Scala programming language I was introduced to the Option class. I'm pretty sure that is comes from Haskell and other places, but I'm not really researching it.

An Option is like a list of zero or one elements; either a Some with a value, or a None.

Functions that return or receive "optional" things use an Option wrapper to both document that fact and promote cleaner code constructs. Here is an example.

Normal Java code with an override variable:

String overrideMessage;
final String defaultMessage="Howdy";

public String sayHello(String name) {
  String s=null;
  if (overrideMessage!=null)
      s = overrideMessage;
  else
      s = defaultMessage;

  return s+" "+name;
}

Yuck! Now let's uses Option to clean up that method:

Option<String> overrideMessage=none();
final String defaultMessage="Howdy";

public String sayHello(String name) {
  return overrideMessage.or(defaultMessage)+" "+name;
}

Much nicer. The Java version can accept alternate values (with the or() method) but not code blocks to execute like the Scala version can (unless Java gets closures...). The Scala version can participate in list comprehensions and all sorts of other things, but I'm not not talking about those things now.

Here is the full version of my Option port to Java:

public class Options {

  /**
   * An Option can be either full (Some<T>) or empty (None).
   * Instead of passing/returning nulls use an Option
   */
  public abstract static class Option<T> {
      public final boolean some;
      public final boolean none;
   
      private Option(boolean some) {
          this.some = some;
          this.none = !some;
      }
   
      abstract public T val();
   
      public T or(T defaultResult) {
          if (some)
              return val();
          else
              return defaultResult;
      }
  }

  /**
   * Placeholder for empty
   */
  public static class None<T> extends Option {
      private None() {
          super(false);
      }
      public T val() {
          throw new NullPointerException("Can't dereference a None value");
      }
  }

  /**
   * A Holder for some value (never null)
   */
  public static class Some<T> extends Option {
      private final T val;
      private Some(T val) {
          super(true);
          this.val = val;
      }
      public T val() {
          return val;
      }
  }

  /**
   * Use this when not sure if value is present or null
   */
  public static <T> Option<T> option(T value) {
      if (value==null)
          return none();
      else
          return some(value);
  }

  /**
   * Use this to wrap a non-null value
   */
  public static <T> Some<T> some(T value) {
      return new Some<T>(value);
  }

  @SuppressWarnings("unchecked")
  private final static None NONE=new None();

  /**
   * Use this when you don't have a value
   */
  @SuppressWarnings("unchecked")
  public static <T> Option<T> none() {
      return NONE;
  } 
}

Using a lazy proxy to avoid Spring dependency cycles

Ever gotten a dependency cycle in your Spring configuration?

How about using the Hibernate Annotations and @Configurable together?

1. Hibernate will instantiate a single instance of each Entity class and ask for the default key value.
   
2. But when the first instance is instantiated @Configurable will try to Spring configure that instance.
3. It that Entity depends on anything that leads back to Hibernate... Bang!

I like this quote from Rod Johnson: "Ouch: this is nastier." That about sums it up. Oh, and that quote is from 2004.

I created a LazyProxyFactoryBean to allow me the chance to not break the cycle, but at least lazy resolve it. Combining a Spring FactoryBean with a dynamic proxy does the trick. Hope this can help someone else out there.

Here is a failing example with cycles:

<bean id="serviceA" class="com.ServiceA">
<property name="serviceB" ref="serviceB"/>
</bean>

<bean id="serviceB" class="com.ServiceB">
<property name="serviceA" ref="serviceA"/>
</bean>

Here is a modified version with a lazy proxy inserted (notice that proxy property is a value, not a ref):

<bean id="serviceA" class="com.ServiceA">
<property name="serviceB" ref="serviceB"/>
</bean>

<bean id="serviceB" class="com.ServiceB">
<property name="serviceA" ref="serviceAProxy"/>
</bean>

<bean id="serviceAProxy" class="com.LazyProxyFactoryBean">
<property name="serviceA" value="serviceA"/>
</bean>

Finally, here is the source code:

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;

import org.springframework.beans.BeansException;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.BeanFactoryAware;
import org.springframework.beans.factory.FactoryBean;
import org.springframework.beans.factory.FactoryBeanNotInitializedException;

public class LazyProxyFactoryBean implements FactoryBean, BeanFactoryAware {

private String beanName;
private BeanFactory beanFactory;
private Object proxyObject;
private Object realObject;

public LazyProxyFactoryBean() {
}

public Object getRealObject() throws Exception {
 if (this.realObject == null)
  this.realObject = this.beanFactory.getBean(this.beanName);

 return this.realObject;
}

public Object getObject() throws Exception {
 Class[] ifcs = getProxyInterfaces();
 if (ifcs == null) {
  throw new FactoryBeanNotInitializedException(
    getClass().getName() + " does not support circular references");
 }
 if (this.proxyObject == null) {
  this.proxyObject = Proxy.newProxyInstance(getClass().getClassLoader(), ifcs,
   new InvocationHandler() {
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
     try {
      return method.invoke(getRealObject(), args);
     }
     catch (InvocationTargetException ex) {
      throw ex.getTargetException();
     }
    }
   });
 }
 return this.proxyObject;
}

public Class getObjectType() {
 return beanFactory.getType(beanName);
}

public boolean isSingleton() {
 return false;
}

public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
 this.beanFactory = beanFactory;
}

protected Class[] getProxyInterfaces() {
 Class type = getObjectType();

 if (type != null && type.isInterface()) {
  return new Class[] {type};
 } else if (type != null) {
  return type.getInterfaces();
 } else {
  return null;
 }
}

public void setBeanName(String beanName) {
 this.beanName = beanName;
}
}

Notes from NetObjectives Lean Webinar

These are notes from a NetObjectives Webinar on "Using Lean Thinking to Align People, Process and Practices".

This was a really succinct presentation that still covered a lot of ground. If you register it is still available for free viewing!

Lean Software Development:

Focuses on the ability to add value quickly now, while improving the ability to add value quickly in the future.

Edwards Deming:

• Plan-Do-Check-Act (PDSA)
  
• System of Profound Knowledge
  

Gary Hamel. "Management Innovation" in Harvard Business Review. Feb 2006

Only after American carmakers had exhausted every other explanation for Toyota's success - an undervalued yen, a docile workforce, Japanese culture, superior automation - were they finally able to admit that Toyota's real advantage was its ability to harness the intellect of 'ordinary' employees.

Lean - A New Paradigm

Engaged, thinking people

Referenced "Software by Numbers" (better graphics...)

Just In: REST can't handle state!!

Or, maybe Jean-Jacques Dubray, Ph.D. doesn't understand hypermedia. :(

In his post on the states and transitions of a resource he claims that REST has no way to let a client know what state transitions are available for a given resource....

Um, what is hypermedia?

Distributed hypermedia provides a uniform means of accessing services through the embedding of action controls within the presentation of information retrieved from remote sites.

This quote is just the introduction to hypermedia and REST, but we already have an answer.

Action controls are embedded. Action controls are embedded.

Embedding the action controls means

1. The client doesn't need to have generated (and hard-coded) client stubs.
2. The server can change it's rules and expand available states as needed.
3. The client does need to process a generically extensible content type.
   

That last point isn't free or trivial, but it enable the first two.

Perhaps more directly on point is this quote from 5.3.3 Data View:

The model application is therefore an engine that moves from one state to the next by examining and choosing from among the alternative state transitions in the current set of representations.

Jean-Jacques' answer to how to share types, constraints, correct states, and transitions is not suprisingly: a schema and a business interface. Let's take those four goals one at a time:

• Types: definitely covered by IDL (until the systems changes then you need to tell everyone to re-compile...)
• Constraints: Not even covered a little by todays SOA technologies. Until there is an OCL or Catalyis (or Z or ...) model that get's compiled with WSDL the best a schema and business interface will have for constraints is a few lines of prose... (in <!-- XML comments no less!! -->)
  
• States: WSDL is not much help, maybe you can infer the states from the operation names? SSDL I think actually does cover this...
• Transitions: Same as states; how do we deduce which operations are transitions?
  

I really can understand the challenge and barrier to hypermedia state control with generic clients. Machine processing and interoperability in this context is unfamiliar: an "engine ... choosing from among the alternate state transitions" is very declarative and make a historically imperative programmer (me!) feels out of control.

Static class diagrams (WSDL) and code-generation are so much more in my control and experience. If only they worked for distributed systems without so many negatives...

The complete ignorance of hypermedia and FUD in the blog posting compelled me to respond and put my $.02 in.