Blogs

A good friend has said to me again and again over the years.  "I don't see why I can't just serve up an entire web application right from the database using SQL." Granted not everyone  loves T-SQL as much as he does.

We've all been been pushing more and more of the development process into the browser. I've talked about this in terms interactivity and putting the end user experience in the hands of front-end developers. Lately it strikes me that at bottom what we are doing is pushing the templating process into the client (and by templating I mean merging dynamic data with HTML markup).

The more projects I do this way the more I realize that there doesn't currently seem to be any standard or even really good way of doing client side templating. Recently I was doing a quick and dirty project that served up an html web page from inside of the CouchDB. Specifically I was doing this on the Cloudant hosted service, essentially giving my client a free place to run an application. I wound up using icanhazjs.  I wound up having a template embedded in a script tag like so:

<script id="loans" type="text/html">
<table id="template" bgcolor="gray" class="couch-grid">
<tbody>
{{#rows}}
<tr  bgcolor="white" class="data-row">
<td>{{state}}</td>
<td>{{college}}</td>
.
.
 {{/rows}}
</tbody>
</table>
</script>

Then using the document.ready function to retrieve the data and merge the template.

$(document).ready(function(){

var $db = $.couch.db("loans_private");
var res = $db.view("app/by_state",{ success: function(data) {
var rows = [];
$.each(data.rows, function(i,o) {rows.push(o.value)} );
var html = ich.loans( {"rows": rows});
$('.container').html(html);
}, descending:true});
});

This all seems rather ass-backwards to me, especially having to wrap the template markup in a script tag. One can put the templates into external files, but either you wind up a big template for every html page, or you wind up with many small templates referenced from inside your HTMl page. javascript mvc works like this too, as does jquery templates which is no longer actively being developed. It seems to me that this breaks with the notion of the document which is still a guiding metaphor for the web. pure gets around this by working with existing markup and using jquery like DOM selectors to do the templating. It strikes me that there will be problems with naming conventions in the markup as designers and front end developers my have different ideas on how to use ids and classes. Plus we reproduce the Babel that we had on the server side with dozens or even hundreds of templating languages with huge or minor differences among them.

When I finally had a chance to look over the HTML 5 spec, I was keen to see what the authoritative solution would be.  While I found many very good things in the spec: local storage, cross-domain messaging, "background threads"; I was very surprised to see no simple and universal way to handle templating.  Did anyone else have the same experience? I for one would like to see a uniform templating language that worked naturally in the browser as soon as possible.

My last post got picked up by slashdot and it generated a  fair amount of discussion. This one  was already well on the way and I didn't want to rewrite  it completely, but I did want to address the criticism against mindless adoption of the latest set of acronyms, of innovating for the sake of innovation.

In my consultancy, all experimentation with  development practices is driven solely by the need to provide my clients with more and more value. I do this by increasing software quality and decreasing time to market. On a practical level I simplify and regularize the way my team works (this includes not writing code whenever possible). If I'm sharing these practices it's because they are helping me achieve these goals, and I think others may be interested.

There was another criticism having to do with pointless layers of abstraction, especially in java development.  I've seen a lot of pointless abstraction (and indirection) in my time.  I hope I successfully answer that charge in-line.

A company operates their primary line of business application in a highly automated environment.  One where it is as likely that the client for for the middle tier is a python or perl script as a browser based application. In the past they had made the mistake of creating a separate facade for each sort of client.  PHP for the web application, perl and python libraries each hitting the database and reproducing any required business logic.

It's not hard to imagine the problems this led to. So the first thing I promised was that when we rewrote the app there would be a single interface no matter the client (including the UI).  In 2009, it seemed reasonable to pursue a RESTful apporoach and indeed an extreme form of it since we decided that all interaction with the middle tier would be via http and JSON.

This approach had a number of results and consequences that I think are worth sharing in 2010.

By treating all clients equally, certain aspects of  controller on the server dwindle, the part that knows knows anything about a particular presentation technology like Struts, Wicket or an asp page.   The middle tier is a  RESTFUL wrapper around EVERY application feature.

For .net applications we can use the WCF REST Starter Kit.  For java applications there are a dozen excellent implementations of the JAX-RS (jsr311) specification.  The point is that the service layer of my application stay essentially the same, no matter how we build the presentation layer.

As a manager and a developer, I think this is great.  I'm always trying to make all my applications look the same no matter the technology stack. Using a common set of idioms makes it easier to support existing applications and to build new ones.

Rather than adding yet another layer of abstraction we make clear of a boundary between system components (what made this boundary ambiguous were the different types of "shim" code dealing with different presentation layer technologies).

What about the additional work needed to translate our domain objects to JSON, XML or some other client side format?  JAX-RS transparently marshalls and unmarshalls several flavors of JSON and XML for my service layer.  I write my service methods as I normally do, working with my domain objects.  Clients can even negotiate the type of content returned by specifying an html encoding.  True, marshalled domain classes have to be identified with the @XmlRootElement annotation and the REST URis are specified on the service interface (The WCF starter kit works in a similar fashion):

@Path("test")
@Produces("application/json"
@Produces("application/xml")
@Consumes("application/json")
public interface TestService {
 
 @PUT
 @Path("testput/{val}")
 public void testPut(@PathParam("val")String val);
 
 @GET
 @Path("test-error/")
 public void testError();
}

The service layer and the ways it is accessed is completely specified in one place and this seems like a very good thing to me. Java clients in the same JVM can call the service code directly. As an added bonus remote Java clients can also use a proxy system (you must have the source of the domain objects and the service layer interfaces) that allows for idiomatic and IDE assisted access to the services:

@Test
 public void testStartPocessNoPipeline() throws Exception {
 
 RegisterBuiltin.register(ResteasyProviderFactory.getInstance());
 DeployService client =  ProxyFactory.create(DeployService.class, "http://localhost:8080/ext-webapp/rs");
 SkylineProcessWrapper pipelineresults1 = client.startAssetPipeline(UUID.randomUUID().toString(), "NONE",
    "boss_animation", "something nice",new ArrayList<EntryWrapper>());
 
assertNotNull(pipelineresults1);
 
 }

It's even better if I use grails. The jaxrs plugin automagically generates a RESTFUL service layer for all CRUD and listing, filtering, sorting behaviors for my domain objects, eliminating huge swaths of normally hand-coded boiler-plate  service layer!  The anti-scaffolding folks may be right when they complain that the CRUD screens produced by (G)rails aren't suitable for production apps, but I think the service layer generated in this fashion will be.

I think it becomes harder to write a good  service layer though, since we really don't know exactly how the conversation with clients will go (this makes sense in the age of SOA and the mashup).  We have to embrace an  open-endedness which is difficult to do and that goes against many of our instincts as developers. And somtimes I wonder whether the RESTFUL grammar wll be rich enough to support all possible application semantics.

Test driven development (TDD) helped us. Our tests are the service layers first and best clients. We also prototyped a browser based interface using Ext JS. Our application is a work flow and job tracking  system, and I think we managed to put together a middle tier that will support significant variation of functionality without needing to change very much. It helps that it's a  wrapper around a mature and full-featured work flow library, JBoss jBPM. Wherever possible we pushed application variablity down into the work flow system.

I suspect that in coming years, the level of flexibility achieved in the service layer will be one of the key ways we distinguish developer excellence.

As the controller orchestrating the conversation with the user moves into the browser, client side coders will have to rise to the occasion by taking on the controller portion of MVC.   They will always pass an XML or more likely a JSON model back and forth to the server via a  RESTful interface. They may manipulate the DOM directly using javascript or use a rich client framework.  I think their work will become more valuable, including monetarily, as they take on the job of fundamentally driving user interaction. Certainly they will be less frustrated and more productive as a single way of doing things they control takes the place of dozens of server side templating systems.  But they will have to learn new tools techniques for orchestrating the conversation with users.

I've been very impressed with Ext JS, a client side RIA framework.  It provides an excellent suite of UI components, but is even better at providing tools for putting the controller part of the application in the browser though its Ext.data.store construct and its ability to save state on the client.  Plus, it's another one of those rare tools that just make sense, where you can try things and they just work.

I don't imagine swapping out presentation layers the same way as we sometimes swap databases for our Hibernate or JPA entities.  In every case client code is going to have to know something about the business at hand.  The users at the company I introduced in the beginning of this post are incredibly sophisticated.  If a group of them doesn't like the way part of our web based interface works, they'll write their own. Our flagship human interface is written in flex and they are unlikely to replace it, but we've given them the tools to do so but in a safe and controlled manner. The innovations they come up will improve the application for everyone.

Certainly this "let a thousand flowers bloom" philsophy may not always be appropriate. But as we work this way, there'll be fewer and fewer times when a small change request for the way a client works will require rework all the way down into into bowels of an application.  That's no small thing, especially in tightly controlled environments.  This should include changes to the relational database, but that's the subject for another post dealing with the goodness of nosql databases.

I moved the Most Media web site to the latest version of Plone this week.  It seems like a good opportunity to reflect on where the industry and my practice have moved in the last three years.

When I last posted, I was very excited about Apache Wicket as the way to develop line of business  applications with a domain model, CRUD screens for maintaining the model, and in the most interesting cases, doing something else useful besides.

I still like Wicket.  It has, as its web site says "a small conceptual surface area." It reminds me of python in that   "You try something it usually just works." In many respects, though, Wicket seems to be at the wrong level of abstraction for the for the sorts of line of business applications described above.  If your team is spending any time at all writing code to produce listing, filtering, and sorting behavior, not to mention creating CRUD screens and the back end logic for these operations, they are probably working at the wrong level of abstraction.

This is why I continue to move  my clients away from relational databases whenever it makes sense to, and it seems to make sense to more and more. It's just harder and harder justify creating a bespoke database schema to support a content management system or line of business application when mega frameworks like Plone, SharePoint, Drupal or Joomla are available.

Nevertheless, sometimes you just have to write against a relational database.  So I looked for a way forward among the frameworks that generate scaffolding for CRUD operations.  I was productive, but uncomfortable with Rails. Ruby is swell, but there is a real lack of third party libraries as compared to Java. And so to the extent that an application might have to do anything besides maintain a domain model, the pickings might be slim. For example there are no more than two or three work flow engines available in Ruby.  Whereas you can hardly count the full featured ones in Java.  SOA is another area in which it is interesting to compare.

One way around this is with  JRuby, running rails inside the JVM, which exposes the wealth of Java apis to Rails applications. But no one has pushed me down that path yet.

Recently I did a small project using Grails and was quite pleased.  Grails uses groovy a dynamic language compatible with Java and is based on the proven technologies that I know and love well: Spring, Hibernate, SiteMesh, Maven, etc. It is configured via convention, but still produces all the artifacts I understand in case I need to override default behaviors.

Not only did Grails produce nice scaffolding code that was easy to customize to support some persnickety client demands, but it afforded several surprisingly nice features:

GORM which is a simplified wrapper on top of Hibernate, which let me define my domain objects and validation rules simply and elegantly:

class Wine {
 static searchable=true;

 Distributor distributor;
 Importer importer;
 Producer producer;
 Region region;
 Region subRegion;
 String name;
 String notes;
 String internalNotes;
 String vineyard;
 Date dateCreated
 Date lastUpdated


 static constraints = {
 name(blank:false)
 vineyard(nullable:true,blank:false)
 region()
 subRegion(nullable:true)
 producer()
 importer()
 distributor()
 dateCreated()
 lastUpdated()
 internalNotes(nullable:true, maxSize:1500)
 notes(nullable:true, maxSize:1500)
 }

 String toString(){
 return name
 }
}

That little bit of code gives me a domain object, database persistance,  relations  with it's dependent objects, validation, and lucene full text indexing.

Groovy gives me closures, functions as first class objects, easy creation of DSLs, and meta-programming. Adding a method to the Java string class is as easy as:

String.metaClass.intro = {len ->
    return StringUtils.abbreviate(delegate, len) ?: ''
 }

I get all the power of the Java ecosystem without the fustiness and lack of expressivity of the core language (no more getters and setters, ever!).

I still have the problem I complained about a few years ago: the problems that arise from mixing client side and server side markup in the same artifact.  Whereas three years ago I thought "clients without code" was solution to this problem, whether using Wicket or something like GWT. I've come to realize that the problem was not having logic in the web page, but in the mixing of client and server side markup.   In fact, for my last few projects I've been pushing most of the presenation layer logic into the web browser, taking advantage of javascript and JSON. This style of development already has acronyms: SOFEA (Service-Oriented Front-End Architecture) and  SOUI (Service-Oriented User Interface).

In a future post I'll explain how this style of development has made my applications both simpler, more powerful, and even more flexible.  Hopefully it won't take me three years to do.