Parts 1 through 5 discussed the details of our application integration framework design pattern.  A Business Analyst (BA) uses our DSL Visual Designer (as a RIA) that has our framework design pattern implemented to declaratively model specific application integration scenarios.  There is one optional point in the declaration where a small amount of custom code may be required and that is if a custom application adapter is required.  Other than that option, any application integration scenario, regardless of size or complexity can be 100% declaratively modeled using our DSL Visual Designer.

 

This goes directly to the point of why model driven development using the Software Factory approach works which in my opinion, represents the state of the art in software engineering today.  My intent in writing Parts 1 through 5 was to walk through how our application integration framework pattern can be used to model any application integration scenario.  The output of our DSL modeling tool is an XML definition that declaratively defines a specific application integration solution as defined by a BA and validated by our Software Factory schema (i.e. our framework model that we just walked through in parts 1 through 5).

 

Next I will discuss our Software Factory in detail and how it works based on a particular application integration definition.  Have a look at this logical view of our Software Factory.  It contains 3 subsystems, they are: the DSL Visual Designer that we just walked through, our Software Factory, and our runtime system.

 

You already know what the DSL Designer does.  The Software Factory takes the DSL Designer XML tools output and configures our Software Factory template.  This means that all of the properties that were set by a BA in the DSL Designer are interpreted by our Visual Studio script and populates a Visual Studio solution with pre-built code artifacts, including source and destination XSD schemas, adapters, business rules, XSL maps, any custom assemblies uploaded by the customer, source and destination namespaces, BizTalk orchestrations, and our pre-built runtime applications are added to the solution.

 

Once the Visual Studio solution and projects have been populated with the specific XML configuration information, a deployment project is added to the solution and the solution is built and packaged as a .msi, ready for deployment.  The Customer is then notified that the package is ready to be downloaded, the Customer downloads the packaged solution and runs it on the target server. During installation, we ask the Customer for credentials for the solution to run along with the source and destination addresses of where the applications to be integrated exist on the network. In our .msi, we check for the existence of BizTalk Server, SQL Server, IIS, etc., to ensure all of the services are installed and running correctly.

 

Configuring and extending Visual Studio is a relatively simple process as the Visual Studio extensibility model is well documented and provides for easy automation.  After designing and developing 25 BizTalk integration solutions, setting the same old properties, adding pre-built orchestration schedules, schemas, etc. is easy.  Sure, some projects are more difficult than others and a little manual manipulation may be required, but generally the time it takes to configure and build an application integration solution using the Software Factory approach is a matter of days instead of months, and thats the point of this series of posts.

 

Parts 1 through 6 have been captured in a Live Meeting demonstration that can be viewed online.

 

Next post we wrap up our discussion on the evolution of software industrialization.

In part 4, we discussed our application integration framework design pattern at the level of the Connection object.  A Connection represents a logical connection to each Application, and physically to the Applications Adapter.  A Connection contains 1 or more DataFlows.  A DataFlow is a collection object that contains 1 to 4 schemas (i.e. XSDs), 0 or more rules, and 0 to 2 maps depending on DataFlow communication type.

 

With respect to rules, for anyone that has used Microsofts Outlook Rules and Alerts functionality will recognize our GUI Rules Designer.  As mentioned in another post, its been done before, you just have to go look for it .  The Rules Designer contains a Rules Manager that allows the BA to add, edit or delete rules in an ordered list, a Rules Editor that allows the BA to declaratively make rules, A Fact Editor that allows a BA to declaratively name a rule, and A Rule Expression Editor that allows the BA to define expressions, and if required, provides the facility to upload custom rules (i.e. .NET assemblies).

 

This diagram shows a screen shot of our Rules Designer:

 

 

At this level we also provide the capability for BAs to upload test XML documents that can be validated against the specified schemas.  The BA can also test the (XSL) map for proper transformations.  Any errors for schema validation and/or map transformation are returned to BA in the Designer.  In fact, if the BA provides a valid XML document with test values that represent what the real values are likely to be, business rules can also be executed, in which the return values are the resulting actions that would occur for the rule processing.

 

Thats it.  The BA has now declaratively configured a complete application integration scenario using our DSL Visual Designer tool that implements our application integration framework design pattern.  As mentioned before, our Visual Designer is exposed over the internet as a Rich Internet Application (RIA).  This allows a BA easy access to their application integration solutions from anywhere in the world, in a secured manner.  This approach models the software as a service scenario where there are no requirements for an organization to use the Visual Designer other than a browser.

 

Now what?  After the BA has completed defining the application integration model and saved the model, the BA can now order the integration definition to be built (i.e. code generated) from our Software Factory.  The order includes pricing based on the number of application integration connections modeled in the scenario.  Once the order has been placed, the turnaround time could be a few hours to a few days, depending on options ordered.  The BA then returns to the Visual Designer web site and can download the resulting MSI to be installed on their target system.  Once installed, the BA and/or System Administrator can run a self-test to verify and certify the application integration solution for correctness. 

 

The runtime installation also includes other tools to manage and monitor the application integration solution.  This includes Business Activity Monitoring for monitoring the application integration scenario, a Centralized Exception Manager that handles any runtime messaging exchange exceptions, complete with notifications, and an Integration Manager that allows runtime configuration of security credentials and application end point locations.

 

With respect to making any changes to the application integration scenario, the BA can launch the Visual Designer again, load up the existing model, that has been versioned, make modifications (to a schema for example) and save, then order and get another MSI that performs the upgrade process on the installed application integration solution.

 

Next post I will discuss our Software Factory for taking application integration scenarios based on our framework design pattern (as realized by our DSL Visual Designer and output as an XML definition) and walk through the steps on how we configure our Software Factory template to code generate the specified solution.

 

In part 3, I described our application integration framework pattern at the level of BusinessFlows.  A BusinessFlow contains 2 or more Applications and 1 or more Connections.  I described how the framework pattern works with Applications and in this post we discuss Connections between Applications.

 

A Connection represents a logical connection to each Application, and physically to each Applications Adapter.  A Connection contains 1 or more DataFlows.  A DataFlow is a collection object that contains other objects, which we will describe in a minute, but its construct allows us to contain one to many DataFlow rows.  Each DataFlow row specifies the direction in which the communication of messages (i.e. data) flow.  It also specifies which Application in the integration scenario has initiated the event in the communication, either in real-time or on a scheduled (i.e. batch) basis. 

 

For example, if I have MS Great Plains on one end of the connection and the Corporate HQ Financial Application on the other end of the connection, I can specify that MS Great Plains will be the initiator of the business event and the messages (i.e. financial data) will flow from MS Great Plains to the HQ Financial Application.  Further, I can specify the DataFlow row type of communication.  For example, one-way asynchronous, one-way asynchronous, two-way request response synchronous, and two-way request with delayed response (i.e. synch on aysnch).

 

As mentioned above, a DataFlow contains a collection of objects, specifically, 1 to 4 schemas (i.e. XSDs), 0 or more rules, and 0 to 2 maps.  A map is an XSL transformation, using a visual editor that allows a BA to map from one schema format to another.  Anyone familiar with the BizTalk Mapper tool will recognize our visual editor as the same representation, except we expose our Mapper tool over the internet in our Rich Internet Application (RIA) Visual Designer.

The reason for 1 to 4 schemas is that depending on the communication type and whether it is a pass-thru or not with respect to message flow.  For example, in our MS Great Plains to Corporate HQ Financial Application, we have specified 2 message schemas, one for the Great Plains format and the other for the HQ Financial Application, in a one-way asynchronous dataflow.  We also have 1 XSL map that transforms the source Great Plains message format into the destination HQ Financial Application format.  Further we may have specified certain rules to be executed either before and/or after the mapping transformation.

 

As mentioned above, our DSL Visual Designer is exposed over the internet as a Rich Internet Application (RIA).  We also provide a facility for a BA to upload any custom message schemas that have developed.

 

In my next post, we will discuss the Rules Designer, which is important to BAs and their business as this is where any data processing of the data occurs.

In part 2, I started describing our framework design pattern for defining application integration scenarios.  At the top level is our Model object that contains one or more Business Entities.  Each Business Entity can also contain one of more Business Entities to represent any hierarchical business entity structure that occurs in the real world.  Each Business Entity contains one or more BusinessFlows.  A BusinessFlow represents Business Processes which can be a simple A2A integration or a complex workflow process.  With respect to our DSL Visual Designer, each object in the model includes a wizard and/or property sheet for the Business Analyst (BA) to enter values that describe the object being configured.

 

Looking at the next level in the object model we can see that a BusinessFlow contains 2 or more Applications and 1 or more Connections.  An Application object represents a real-world computer application.  We have a number of Application objects pre-built (see Application Types in the model) allowing the BA to select from a list. If the Application required by the BA is not in the list, then a custom Application is entered which will require custom code to describe its interface (i.e. Adapter).  Each Application has an Adapter.  An Adapter is a plug-in connector where one end of the connector knows how to natively communicate with the Applications API and the other end exposes an interface that has been transformed into XML messages (including XSDs) that the Application can exchange with other Applications.
 
In our continuing example of financial applications exchanging data, the Application may be MS Great Plains which comes with a COTS adapter (i.e. connector) that performs this transformation in communicating with Great Plains native API and exposes an XML messaging interface.  Since we are using BizTalk as the underlying messaging engine, there are many COTS adapters available that provides this connector and transformation mechanism.

In the case of a custom Application where a COTS adapter is not available, we use the provided BizTalk Adapter SDK to write custom code to build an adapter.  Note this is the only variable point in the entire application integration scenario that requires custom code to be written, that is, if a COTS adapter is not available.  The custom code is limited in scope as we are extending a pre-existing adapter framework supplied by the middleware vendor for a specific Application in which we will implement only the messaging interfaces required for the application integration scenario.

 

This is a key point to keep in mind as everything else in the application integration scenario is defined declaratively.  Since everything else is declarative, this dramatically reduces the amount of custom code that has to be written for any given application integration scenario.  In fact, even with a custom adapter, this means that the entire application integration definition is syntactically and semantically complete, which means we can code generate the entire solution and the main reason why the Software Factory approach works: model-driven development, which dramatically raises the level of abstraction for developing application integration solutions to the point where a BA declaratively defines the complete solution through a modeling tool (i.e. our DSL Visual Designer) to be generated.

 

Using model-driven development, we can define a model up-front that is both complete and of high-fidelity. We can then validate the model against a known schema and generate the output.  Our application integration framework design pattern is this schema (i.e. Software Factory schema).  Using our DSL Visual Designer, the BA models a specific application integration scenario in which the Designers output produces a valid XML document definition that describes that specific application integration scenario.  The XML definition is used as input to our Software Factory template which configures and extends Visual Studio design time components and reuses our pre-built components that are checked out from our source control system.  This includes configuring BizTalk Server and our factory template includes the instruction set (i.e. scripts) to build the complete solution into an installable package (i.e. MSI package) which is then installed on the target system.

 

Returning to our framework design pattern, a Connection object represents the actual connection between Applications and will be the topic of my next post.

In part 1, I described my experience using Software Factories, including DSLs, to predictably and repeatedly solve application integration problems.  Our Software Factory framework pattern evolved over the course of 4 years and 25 application integration projects.  Essentially we developed a product that encapsulated our framework pattern that could solve any application integration problem with minimal coding.  This was done by analyzing what is common and what is variable to every application integration scenario and designing a framework model to cover these scenarios.  My intent here is to cover each aspect of the model and discuss how and why we arrived at our overall application integration framework pattern.

 

Referring to our framework object model, the top most container is the Model object in which it contains all of the objects that make up one or more application integration scenarios.  Essentially it is an abstract (root) container, which translates to the top-level drawing canvas in our (DSL) visual designer modeling tool.

 

The first common modeling construct is that all application integrations occur within a Business entity or between Business entities.  A Business entity is an abstract object that can represent an individual company, a division of a company, a remote office, an enterprise or small group within an organization.  A Business entity may also contain other Business entities.

 

It seems obvious or even taken for granted that a Business entity models a real world construct (i.e. a real business).  However, in my experience this construct is typically either overlooked or viewed as trivial.  The reality is that application integrations do not occur in thin air, they are tied to real businesses, thus part of the problem domain we are interested in modeling.  Having this construct allows us to model any application integration scenario within or between any number of uniquely named Business entities.  For example, lets say we have 50 satellite offices that are geographically dispersed that use various accounting software packages that need to exchange financial data with a centralized financial package (i.e. Corporate HQ). 

 

Without the capability to model this in a visual designer would make it very difficult to design and construct this application integration scenario, let alone actually visualize semantically what is actually going on. While we are talking about the business entities and the logical connections between the business entities, we are also describing the many properties of each unique business entity and its relationships to each other.  This translates into unique name spaces required to define the paths between (or within) these businesses in the models XML definition output.  In other words, we have declaratively captured all of the information required for each business and to navigate from one business entity to the next.

 

Looking at the framework design pattern at the next level, we can see that each business entity has one or more Business Flows.  A Business Flow is an abstract object contained in a collection whose parent is a Business entity. The Business Flow represents a particular Business Process which may be a simple A2A integration, or may be a complex workflow process.  For example, if one of our satellite offices (i.e. business entity) had three financial applications that exchanged data between each to form an aggregation, before it sent its aggregation to the centralized financial package, then we need the capability to model this. 

 

You can see an example of what this visually looks like in this layered view of our visual designer model at the canvas layer called Business Flow.  You can see how this visual representation conforms to our framework design pattern.  You can also see how this visual representation can scale to represent any number of Business Entities and the collections of Business Flows for each Business Entity.  While this occurs in the real world, few application integration vendors toolsets have the capability to capture this critical information.

 

In part 3, we will continue walking through our application integration framework pattern.

This blog is dedicated to the advancement of what I call the industrialization of software.  Software industrialization is about raising the level of abstraction for programmers to produce quality software products using modern software engineering approaches such as Software Factories. One aspect of the Software Factory approach employs model-driven development using Domain Specific Languages (DSLs) that code generates parts (or all) of the solution for a specific problem domain.  The Software Factory approach enables programmers to produce and assemble software in a predictable and repeatable manner something we dont do well as an industry, but is done well in other industrialized engineering disciplines like civil, electronics, electrical and mechanical domains.  Or at the very least, software engineering is nowhere near as evolved as these engineering disciplines.

 

The author of this blog lived the Software Factory approach while working at a company called 5by5 Software, now called Bridgewerx.  5by5 was initially a Microsoft Systems Integrator (SI) that offered professional services in the area of application integration, designing and constructing solutions using Microsofts BizTalk Server product.  Over a 4 year time frame, we designed and constructed over 25 application integrations solutions.  These solutions included messaging exchanges and workflow of all sizes, shapes and forms for various vertical industries.

 

During this 4 year time frame, we recognized several application integration design/code patterns.  Each pattern went through the process of discovery, reuse, refinement, and then iterated through this process again for each new project we undertook.  After designing and constructing 25 BizTalk integration projects, we found an overall framework design pattern that could be applied to all integration projects.  We decided to encapsulate this design pattern in a modeling tool, which really is a Domain Specific Language (DSL) for designing and generating application integration solutions sitting on top of BizTalk Server. 

 

The idea is that the DSL runtime Visual Designer would be used by a Business Analyst, who has the business domain knowledge to model an application integration scenario and the modeling tools output (XML) would supply the complete definition of this specific modeled application integration. We used this complete definition as input to our software factory template which then grabbed reusable components, other artifacts (with some custom code) and configure BizTalk Server with the specific application integration scenario, automatically build the solution in Visual Studio and produce a Microsoft installer package (MSI) which then can be installed on the customers target machine.  We called our Software Factory and DSL, Bridgewerx.

 

The point is that the software factory approach, incorporating DSLs works.  Our 4 years of application integration domain experience and subsequent ISV product is living proof of this fact.  We were pioneers in producing a software factory for the design of and code generating application integration solutions.

 

To further substantiate this claim, I will walk through a number of our projects and describe the patterns we discovered, designed, codified, categorized and subsequently automated, until we had the entire application integration framework pattern designed. Then we encapsulated the framework pattern into a DSL and built our software factory for code generating application integration solutions. 

 

From where I sit, this is industrializing the software development process for the application integration domain.  Another claim I make is that this same approach can be successfully applied to other software application domains of interest (e.g. software factory for generating families of e-commerce applications.)

 

In part 2, we will discuss what is common and what is variable to every application integration scenario as a way of describing our framework pattern.  Then we will look at some specific integration projects we undertook and how it fits the framework pattern.

While paging through a software industry mag, I came across an ad for Oracles middleware product.  The ads buy line was that by using their middleware product, application integration was hot pluggable.  Hot pluggable?  The inference is that you can plug-in applications to their integration bus as easily as one can plug-in a hot swappable hard drive into a computer without powering down or effecting other applications. I have been building application integration solutions for 5 years and my experience can attest that application integration is not hot pluggable.  In fact, it is an incredibly complex rats nest that you have to make sense of many different vendor applications proprietary data formats, using (mostly) proprietary APIs.  Most vendors middleware products also work at this complex level of abstraction, i.e. too low.  Also, while the middleware vendors help manual is 10,000 pages (talk about complexity!), no-where does it tell you in detail how to solve the actual application integration problem, which in itself, is just as complex as the vendors toolset.

 

No wonder software developers are a cynical bunch when it comes to working with most vendors products.  What the data sheet says and the marketecture talks about, makes no sense in the real world.  While one could say that for any type of advertising, our software industry seems the most plagued in my opinion.  Why?  As discussed in previous posts software development is a complete mystery to almost everyone but the programmers.  So who are the ads targeted for?  Decision makers, who are CIOs or purchasing agents, most of whom dont (really) understand software development (generalization) so all they have to go on is the marketecture.

 

I am not just picking on Oracle, I saw a Microsoft Office cartoon ad where the people in the ad had dinosaur heads on their bodies.  Dinosaur heads?  Huh?  Is MSFT inferring that their users are dinosaurs?  The buy line is that one dinosaur accidentally forwarded everyones salary to the entire company and had the dinosaur evolved to using Information Rights Technology in MS Office, then this would not have happened.  I suppose that it also means that one of the dinosaurs in the ad is about to become extinct. 

 

As an industry professional, I find this dinosaur ad - embarrassing.  And insulting to the user community it is targeted at.  Anyone that knows me personally wont accuse me for not having a sense of humor, but honestly this is lame.  Not only is the ad just plain dumb, but it deals with laying FUD on the target audience. This is unacceptable.  Do the marketing people for these companies know no bounds?  Why not say that by using our product you can prevent sensitive information from being read by unauthorized readers using Information Rights Technology and here is how easy it is to do it.  Right now you might be thinking, what a naive software programmer maybe so.  But then again, "All Marketers Are Liars".

 

Both ads remind me of an old Dudley Moore movie called, Crazy People where he was an alcoholic adman and came up with his best ads while being institutionalized at an insane asylum. Dudley had the inmates create ads that told it like it was i.e. the truth.  How is that for irony.  Maybe the marketing people in the software industry can learn a lesson from this?  My cynical programmer persona says I doubt it.  However, eventually customers will push back with another cycle of what happened in the dot com era and want demand software products that do exactly what the marketing materials says it does.  This would help the industrialization of software, but until then, would you like fries with your SOA?

The software industry has a long history of recreating incompatible solutions to problems that are already solved  Pattern Oriented Software Architecture Volume 2

 

In 2001, my good friend Barry Varga  and I co-founded this company for a variety of reasons.  One of those reasons was to make decisions for our own company as previously we had worked for a variety of other companies where we watched the people in charge make, well, lets just say questionable decisions.  We felt that we could make better decisions or at least, we would be accountable for our own decisions we thought how, could we do worse?  I am sure this is how MiniMicrosoft kinda feels.

 

One of those decisions directly affected our programming team, which was, dont write code at least initially.  It sounds strange as most programmers (read: all) love to write code including yours truly.  The issue is that designs precede code and requirements precede design.  Pretty fundamental one would think.  However, as programmers, we are all guilty (and still are) of not conforming to this basic tenet.  Oh to be sure, there are a whole pile of reasons behind this we are good at rationalizing that we dont have any requirements or market intelligence to tell us what to code.  We also have great imaginations and think we know what the customer wants.  We also believe that regardless of size, complexity and/or change, by coding first we can figure it out

 

Since we were writing code on the MSFT platform, and have been since VB1, we knew, as old school programmers that it all has been done before, you just have to find an example.  The World Wide Wasteland has many examples of whatever you are coding.  In fact, since some 50% of the 8 million programmers out there code on the MSFT platform, you are bound to find an example or ten (or hundreds!) that is pretty close to what you need for your component/project/product.  Of course, the cynical programmer and/or perfectionist programmer type (arent we all?) will want to write their own code because what they write will be better than whatever can be found on the internet- surely.  Sometimes this is true, but in most cases not, or a least in my 15 years experience in the field.  This is also sometimes called the Not Invented Here (NIH) syndrome.  This is one of the major barriers to reusing perfectly good code that already exists. Oh, did I mention that coding is fun?

 

We also got our teams to think of different ways to solve various problems (once we had some semblance of requirements to iterate on).  In other words, come up with a design first.  Then think of 3 ways to solve the design, including searching the internet to see how other people have solved similar problems.  Then present those findings in an objective manner.  Once people thought this way, we were able to leverage and reuse several pieces of existing code (and design patterns) to solve a much bigger puzzle.  This meant that we could solve the problem in a much more timely and realistic manner.

 

The key word being realistic.  Anyone that has been writing code for any number of years knows how hard it is to write code from scratch to solve problems of any real size or complexity.  Not only is it really hard work, but the amount of effort required sometimes seems insurmountable.  We do have best practice processes and tools that raise the level of abstraction, but it is still at such a low level it still sometime feels like assembly language programming (to me anyways).  Also, when you the programmer, are the owner of your own software shop, you get a bit concerned as to how much time and effort is required to solve various problems particularly when you know they have been solved before (many times!).  Here is where refactoring really helps find any way to get the problem solved, like right now (hopefully by reusing some already existing code) and we will come back (given time) to refactor the code to make the design better.

 

Why am I saying this?  I see programmers everyday still re-inventing the wheel or NIH or doing everything else but trying to reuse and leverage some chunk of code, component, design that has already been designed, tested and reused by many to solve the same problem.  I ponder the reasons to why that it is while I have mentioned a few in this article, it still amazes me.  Even seasoned pros fall into the trap.  Maybe it is just too easy?  Maybe it is a combination of a whole pile of things.  Maybe it is simply that our software industry still has not gone through the industrialization (i.e. maturity) process like other similar engineering industriesWhatever it is, next time you open your favorite code editor stop!  Ask yourself this question did I look on the internet to see that this problem has been solved before?  If not, ask yourself why maybe you have the answer.