Drea Thomas is the engineering director responsible for the front end technology at SnapLogic – which explains his clear bias toward Designer and all things visual.

When you build a data integratio

n platform, it is easy to expect integrations to look like something from a demo:

Figure 1: DemoLand

But, as we’ve learned over the years, real-world integrations often are a lot messier and can look more like:

The Spring ‘12 release of the SnapLogic Server is all about improving the user experience dealing with large and complex pipelines. The goal is to reduce the time required to build a complex pipeline by an order of magnitude. Let me outline some of the improvements we’ve made to the product to address this.

Latency

In previous versions of the product, calculating pass-through fields (something that is quite complex) was done on the server side. This makes a lot of sense because you can centralize the code in one place and re-use it for Designer, or for any application that builds pipelines using SNAPI. But, when your pipelines look like RealWorld, the pipeline definition is quite large – sometimes several megabytes. Sending it back and forth over the wire introduces significant latency.

In Spring ‘12, we’ve moved some of the most frequently used server logic into the client to greatly reduce latency. In some cases, this leads to a 300% improvement in Designer performance.

Debugging

In DemoLand, every pipeline runs the first time. In the RealWorld, things don’t always work out that way. In previous versions of the product, when a pipeline resource was badly configured, or there was an error in a query or an expression, you would sometimes discover it only when you ran your pipeline. This required you to read an error log to identify which resource caused the problem and why.

In DemoLand identifying the offending resource and fixing it is fairly straightforward because you only have a handful of resources. In RealWorld, even finding the offending resource on the pipeline canvas is challenging.

In Spring ‘12 we’ve done a number of things to improve the situation.

Improved Validation

Components now catch many more problems at validation time – before you even run the pipeline. And the problems that are caught produce better error messages. This tells the developer which resource failed and why.

Resource Highlighting

When a resource fails during pipeline execution, we mark it red and call out the specific error in a tooltip.

Getting Unstuck

In DemoLand, all pipelines run quickly and produce the results you expect. In the RealWorld, pipelines can take far longer than anticipated or produce unexpected results. When you want to identify the bottleneck in a pipeline, you need to see how much time is being spent by which resource and what it is doing.