ESBs and the Last Mile of Integration

I’ve been following the “what is the value of ESBs” discussion kicked off by Patrick Logan, which then migrated over to Steve Vinoski’s blog.

Good stuff! Alas, the discussion seems to be degenerating into a REST versus WS-* debate, with many comments assuming Service Oriented Architecture must be WS-* based , and can’t possibly use REST. In the process, I think the original points have been getting lost.

Patrick says:

” If you need XML transformation, email, HTTP, file drop, etc. then why not just use the simplest dang library for the one or two of those needed for any given situation? If you need more than two of those in a single situation, then that’s a symptom rather than a need.

Steve Says:

So, why not just do things the way Patrick suggested: when faced with an integration problem, just grab the nearest dynamic language, such as Ruby or Python, grab the modules they provide that surely cover whatever protocols and formats are needed for the integration, and code it up in just a day or two?

I couldn’t agree more with both of them when it comes to the power of dynamic languages and their library support.

In my experience, many application ‘interfaces’ are single purpose, point to point. In additional to point to point interfaces, there are also a lot of ‘feeds’ or ‘reports’ that are often just limited extracts of data to staging tables in a database, or CSV files on a file server. These interfaces often go to desktops – imported into a spreadsheet or other application. Like it or not, this is they way a lot of integration is done in the enterprise – just get the job done, and don’t over engineer it.

These types of interfaces definitely benefit from the use of dynamic languages and basic libraries. In fact, a lot of developers figured that out already – most of these interfaces are written using PeopleTools, or PL/SQL, or Transact/SQL, or Perl, or VB. Whether or not you consider all of those ‘dynamic languages’, they share many common characteristics – they are tools for fast development and efficient problem solving.

The downside to the ad-hoc interface approach is that it can get out of hand quickly unless the efforts are coordinated. Management, upgrades, and compliance audits in our world of SOX can become a nightmare. The motivation to move to a better approach is clear when the complexity increases.

To me, the real question is how do you get the benefits of flexibility and dynamic languages, while avoiding ad-hoc interfaces? I think the solution is a services architecture – the problem is that we still have debate over what web services and a services architecture really is. I like Adam Bosworth’s definition of Web services from a 2005 interview in Queue:

KIRK MCKUSICK (KM) People sure talk a lot about Web Services, but it’s not clear they’re all talking about the same thing. How would you define “Web Services”?

ADAM BOSWORTH (AB) The term Web Services refers to an architecture that allows applications to talk to each other. Period. End of statement.

That’s about a straightforward as you can get. Given that definition, where do ESB’s fit in to the whole services architecture ?

Well, when I look at Mule or any other ESB, I don’t see a ‘Bus’. I see a hub – a powerful, multiprotocol router with transaction coordination and message brokering, and other functions. (The bus stuff is a conceptual thing, that seems to be driven by marketing and industry history more than any actual bus.) There’s an obvious need for this functionality when ‘applications talk to each other’, just like we need routers on our regular networks. But can you build an entire services architecture from this functionality ? I don’t think you can, any more than you can build a physical network from just routers. You need cables, switches, and network adapters. The core functionality of EBS products is useful, but there’s more to web services.

As we move to a true, distributed network computing model in the enterprise, we are in the midst of continuous change, just like early networking dealt with many different, competing physical and protocol variations. We had bridges, and different physical adapters, rings and buses, and 10base5 coax to drill holes into (carefully!) Twisted pair, Etherent and TCP/IP eventually rose to solve the bulk of our networking problems.

We have a similar situation with ‘Web services’ today – there are competing alternatives, and we are seeing the shakeout. Today we have two main competing models – RPC style services and REST style services. Prior experience tells us that RPC style services have scalability problems; The experience of the Web tells us the REST style avoids many of those problems.

When I look at where information is being consumed today, I see the web browser. Everywhere. (On the iPhone, it’s about your only option.) The browser is a REST services client. So, as far as I’m concerned, REST is the only client interface that will remain relevant.

When I look at where information is being generated, in applications, I still see a lot of protocols and styles. I think that will continue, just as we have specialized networking like fiber and ATM for specific purposes. Theres definitely a need for messaging and ESBs here, but it’s not the bulk of the interfaces.

Why not ?

Well, for one, because the primary consumers are REST. And because anyone writing a new application today is writing a webapp. Probably on a LAMP stack. They’re likely using a dynamic language, and a framework like Rails, Django, Zend, CherryPy, Turbo Gears, or Pylons. And if they’re paying attention, it supports REST services as well. so it can talk to the other web applications.

Given those trends, the forward looking comments from Peter and Steve about using dynamic languages and REST for integration make a lot of sense to me. I can see the last mile of integration, and it’s all about REST.