 Wednesday, 11 July 2007
Oxymoron? Could be, but all too real in the world of software start-ups in my opinion. The following is a short story providing some lessons learned to software developers interested in starting up their own company. Why? I wish I had this advice when I started my own company in 2001, but it seemed difficult to find any material, specifically targeted for software developers that think they have a good product idea and want to bring it to market.
First let’s talk about product success. My team and I created a product called Bridgewerx that used BizTalk Server as its integration engine. BizTalk Server is a middleware product from Microsoft. Whether you call it an Enterprise Service Bus or Internet Service Bus or Internet Data Router (which we called our product back in 2002 when no such term existed) or a SOA “type” product, or delivered as an SaaS or packaged as an Integration Appliance or… really does not matter cause these are mostly marketing words that basically mean the same thing: loosely-coupled, message-based application integration server. Purists will disagree, but I can make BizTalk meet any of these terms.
A saying is practice makes perfect. When you perform similar tasks over and over again, generally speaking, you get good at it. As a Systems Integrator, we designed and delivered over 30 application integration solutions to our customers over a 4 year time span. We thought we were pretty smart about it, generalizing components and reusing them for the next job. In fact we got so “practiced” that we were more or less assembling reusable components and the only customizations were customer specific adapters, business rules, schemas and maps. Regardless of the type of application integration work we did, these patterns occurred every time. We became experts at developing BizTalk solutions using several design patterns and practices we had discovered and refined over this 4 year time period.
Then we thought, we could turn this into a product. We could take our design patterns, best practices and embody them into a graphical design time tool that allows Business Analysts to specify application integrations. The tool would capture all of the data (and metadata) necessary to fully describe an application integration scenario (even complicated ones). The output of the designer tool would populate and configure a Visual Studio solution with several projects that contained source artifacts from our library of reusable components, (e.g. Orchestration patterns similar to C# generics, runtime message envelope, utilities, etc.), plus user defined schemas, maps, business rules and even custom assemblies.
Through scripting, we automated the build and packaging process so the final deliverable was a MSI that a customer could install on their target infrastructure, run the installer and voila, a completed application integration project, now ready for operation.
Anyone that has been involved in any size application integration project knows how difficult it is. We called ourselves plumbers because it was a dirty and complicated job. Application integration inherently deals mostly with distributed systems, not just a single application, therefore complexity goes up, along with the myriad of file formats per application integration. If you have been there, you know what I am talking about.
So what? Well, our product code generates “complete” application integration solutions from a design time tool output specification. That’s the one sentence description. This means the amount of highly skilled, intensive manual labor effort in designing and coding the solution is dramatically reduced. At least by a factor of 10X, in some cases by 100X, and in the extreme cases where you are talking about very large scale application integration with +100 points of integration, it is not even feasible using traditional (i.e. manual labor) techniques.
We believe this to mean product success. It successfully answers a difficult problem in our software development world in the domain of application integrations. It provides at least a 10X improvement over current methods and tools.
If you want to read more about the design of the product, you can click on the Bridgewerx category on the right side of my blog. If you want to see a one hour movie demonstrating Bridgewerx, sponsored by Microsoft (thanks Jim!), you can see it here. Finally, if you want to see a live demo, contact me at my email address at the bottom of the page.
Forrester research says that “Application Integration is the top priority for 2007.” So we should be all set for business success, yes? No.
Business failure. As the President of the company for 4 years, a co-founder and co-creator of the product, I failed to realize the vision I had for my business. Why? It was not just one reason, but there were a few factors that occurred at key milestones in our company operations that really broke our company (figuratively and literally).
One mistake I made was not getting the right people aboard our company to help us from a business perspective. Involving business partners/advisors in a highly advanced technology business who have (very) limited understanding of software development and the target market is problematic. Or even worse, business partners/advisors that were successful in the software business with one type of product and applying the same principles to a product and market that is completely different is disastrous, as was the case in my company.
My business partners had backgrounds in “traditional” software business/marketing, i.e. put the product in the sales channel and build up a base of customers. On the surface this makes sense and were going with what they know. However, our product, using terminology from Geoffery Moore’s, "Crossing the Chasm”, was an innovative and disruptive product that required very early adopters. And from my perspective, once we had half dozen early adopters, the play was to sell our IP to a much larger player that could integrate our innovation into their product. In our case, one play was to sell directly to Microsoft and have the innovation built into BizTalk Server. Hard to believe no effort was to put into this when the company was operational. Another play was to sell to a much larger System Integrator, like Avanade, who perform large scale application integrations, to give them a competitive advantage. Yet another exit strategy was to take the innovation and apply it to another middleware or SOA product. Based on my latest research, it appears we still have a market advantage of being the only vendor to code generate “complete” application integration solutions from a design time tool output specification. Technically speaking, using Microsoft terminology, our design time tool is a graphical Domain Specific Language (DSL) and our code generation system is a Software Factory. Innovative and disruptive? Yes. Easily understood? No.
Lesson #1 Make sure your business partners and advisors fully understand your products marketplace and value proposition, especially if it is an innovative, disruptive product.
That was my vision and intent. However, dear fellow software developers looking to bring your product to market, I made one fatal mistake. I lost control of my company through share dilution. In order to acquire the people that could realize the vision I had, required giving up shares as, of course, there was little money to pay large salaries. Or at least so I thought. Once you lose your controlling interest in the company, well, you know, other people take advantage of this for their own self interest and the companies vision and product changes direction, and not necessarily for the better.
Lesson #2 Keep a controlling interest in your own software company at all costs.
Another mistake I made was not keeping the professional services division of the company running to bring in cash flow. It seems obvious enough that you cannot turn an SI into an ISV overnight. But this was lost on me and my business partners/advisors, who were pushing just to raise capital. We could have also acquired another professional services company that was similar to ours (Hi Mike and Jame!) to not only bring in additional pro services dollars, but also prime the pump for existing customers that may be interested in trying our innovation out. This would have reduced the dilution of our company before we even got started. Unfortunately, our advisory board and other business executives did not understand this and was an opportunity missed and may have been the tipping point looking back in hindsight.
Lesson #3 Find alternate methods of funding, particularly if you are an SI that already has a revenue channel.
Speaking of funding, if you need to raise funds, ensure that you can raise enough. Research shows that to fully capitalize a product as a SaaS requires $35 million to be raised over 5 to 7 years. Here in the Great White North, we were able to raise $5 million tops. If this were a car race, we did not even have a chance to qualify, let alone participate in the actual race.
Lesson #4 If you need to raise capital make sure you can raise enough.
Followed closely by:
Lesson #5 Ensure the people that are raising the money actually have a track record of raising that kind of money.
I know it sounds simple enough, but at the time… At the time there was lots of bravado and speculation, which on the bottom line means absolutely nothing.
The final lesson I have yet to figure out. Based on my market research, our company has a product innovation that does something no other product does in our market space of application integration. Market research shows that application integration is a top priority for 2007. What up? My guess is that we are yet another company that has fallen into the Chasm.
On a personal note, my hope is that in this short story provides a few tidbits of advice (and a little bit of hard earned wisdom) that other budding software development entrepreneurs will find useful, which is the reason I wrote this in the first place. Secondly, am I bitter? No, it was one of the most exciting and exhilarating business adventure that I have been on , Would I do it again, absolutely! Of course I would do it differently, no regrets.
For software developers wanting to unshackle themselves from the security blanket of the day job to develop their idea, vision, product. If you can’t stop thinking about it… Carpe diem baby!
Wednesday, 21 December 2005
If you have read this 7 part series, you will see how our DSL Visual Designer works for declaratively defining complete application integration models and see how our Software Factory works by taking the DSL Visual Designer tools output (an XML model definition) and using it to configure our Software Factory template, which is a set of scripts that configures a Visual Studio solution, including BizTalk server as the core integration engine, to code generate complete application integration solutions. Our framework design pattern can be applied to any application integration scenario. Further, I assert that the can be applied to any software problem domain just as successfully as we have applied it to the application integration domain.
Our DSL Visual Designer is very specific in the sense that its domain language is specific to application integration scenarios. We built our Visual Designer on top of BizTalk Servers integration engine to provide a higher level abstraction so that Business Analysts (BAs) can easily model their specific integration scenarios as BAs typically have the business domain knowledge and just need a modeling tool, specific to that business problem domain, to model the solution which is declaratively complete. Since the model is complete (i.e. high fidelity, meaning not missing any pertinent information) and that we are in the virtual world of software, we can code generate the entire output, meaning producing the solution in a predictable and repeatable way.
Thats the main point of software industrialization. As mentioned in the introduction to this series of blog posts, software industrialization means raising the level of abstraction to producing software solutions in a predictable and repeatable manner. Producing software solutions in a predictable and repeatable manner is the bane of our industry. In our case, using the Software Factory approach enables us to use model-driven development techniques using Domain Specific Languages (DSL) and assemble pre-built components based on a domain model pattern specifically designed for modeling application integration scenarios.
Because we designed and constructed over 25 application integration scenarios by hand over a course of 4 years we saw many application integration patterns based on producing these solutions for several different industry verticals and different types of integration scenarios (i.e. basic messaging exchanges to complicated, recursive workflows, all using orchestration). In fact, in almost every project we were seeing these patterns repeat themselves over and over again. Based on these repeating patterns (i.e. our evolution), we saw how we could embody these design patterns into a modeling tool (i.e. our DSL) and take the modeling tools output (XML definition of an application integration scenario) and configure a Software Factory to code generate the output, producing an installer package that can be easily installed on the target system, including a number of runtime applications that can monitor and manage the installed application integration solution.
I personally believe that this Software Factory approach will be the way that most (business) software applications will be designed (i.e. DSL models) and constructed (i.e. code generated) in the future. Why do I say future? Why has this approach not caught on (yet)? This blog has discussed many themes to the reasons why. Mostly it boils down to education. I work with developers today that do not see the value in interface-based programming let alone the value of model-driven development or code generation or the concept of a Software Factory.
The education/experience gap in our own software engineering community is quite large and when some programmers (new and seasoned alike) are introduced to these concepts well, lets say that there is considerable resistance to even thinking about using this approach.
Now throw into the mix the vendor community that insists on producing marketecture. Our very own software vendors are doing us a disservice by selling hype to our target business community. Now add the business community to the mix, who are the target audience of this marketecture to most, software is a complete mystery. All they know that it is an incredible time consuming, costly, painful process that mostly results in never ending work, endless upgrades and broken promises. Unfortunately, given the current state of the art in the software industry, this is all but true.
With a few exceptions, I consider the software engineering industry to be still in the dark ages, yet to go through the industrialization process that other, more mature, engineering disciplines have evolved to. This series of posts is intended to show, in one companys way, how we evolved the industrialization of software in the application integration domain using modern software engineering techniques and tools.
Tuesday, 20 December 2005
Parts 1 through 5 discussed the details of our application integration framework design pattern. A Business Analyst (BA) uses our DSL Visual Designer (as a RIA) that has our framework design pattern implemented to declaratively model specific application integration scenarios. There is one optional point in the declaration where a small amount of custom code may be required and that is if a custom application adapter is required. Other than that option, any application integration scenario, regardless of size or complexity can be 100% declaratively modeled using our DSL Visual Designer.
This goes directly to the point of why model driven development using the Software Factory approach works which in my opinion, represents the state of the art in software engineering today. My intent in writing Parts 1 through 5 was to walk through how our application integration framework pattern can be used to model any application integration scenario. The output of our DSL modeling tool is an XML definition that declaratively defines a specific application integration solution as defined by a BA and validated by our Software Factory schema (i.e. our framework model that we just walked through in parts 1 through 5).
Next I will discuss our Software Factory in detail and how it works based on a particular application integration definition. Have a look at this logical view of our Software Factory. It contains 3 subsystems, they are: the DSL Visual Designer that we just walked through, our Software Factory, and our runtime system.
You already know what the DSL Designer does. The Software Factory takes the DSL Designer XML tools output and configures our Software Factory template. This means that all of the properties that were set by a BA in the DSL Designer are interpreted by our Visual Studio script and populates a Visual Studio solution with pre-built code artifacts, including source and destination XSD schemas, adapters, business rules, XSL maps, any custom assemblies uploaded by the customer, source and destination namespaces, BizTalk orchestrations, and our pre-built runtime applications are added to the solution.
Once the Visual Studio solution and projects have been populated with the specific XML configuration information, a deployment project is added to the solution and the solution is built and packaged as a .msi, ready for deployment. The Customer is then notified that the package is ready to be downloaded, the Customer downloads the packaged solution and runs it on the target server. During installation, we ask the Customer for credentials for the solution to run along with the source and destination addresses of where the applications to be integrated exist on the network. In our .msi, we check for the existence of BizTalk Server, SQL Server, IIS, etc., to ensure all of the services are installed and running correctly.
Configuring and extending Visual Studio is a relatively simple process as the Visual Studio extensibility model is well documented and provides for easy automation. After designing and developing 25 BizTalk integration solutions, setting the same old properties, adding pre-built orchestration schedules, schemas, etc. is easy. Sure, some projects are more difficult than others and a little manual manipulation may be required, but generally the time it takes to configure and build an application integration solution using the Software Factory approach is a matter of days instead of months, and thats the point of this series of posts.
Parts 1 through 6 have been captured in a Live Meeting demonstration that can be viewed online.
Next post we wrap up our discussion on the evolution of software industrialization.
Monday, 19 December 2005
In part 4, we discussed our application integration framework design pattern at the level of the Connection object. A Connection represents a logical connection to each Application, and physically to the Applications Adapter. A Connection contains 1 or more DataFlows. A DataFlow is a collection object that contains 1 to 4 schemas (i.e. XSDs), 0 or more rules, and 0 to 2 maps depending on DataFlow communication type.
With respect to rules, for anyone that has used Microsofts Outlook Rules and Alerts functionality will recognize our GUI Rules Designer. As mentioned in another post, its been done before, you just have to go look for it  . The Rules Designer contains a Rules Manager that allows the BA to add, edit or delete rules in an ordered list, a Rules Editor that allows the BA to declaratively make rules, A Fact Editor that allows a BA to declaratively name a rule, and A Rule Expression Editor that allows the BA to define expressions, and if required, provides the facility to upload custom rules (i.e. .NET assemblies).
This diagram shows a screen shot of our Rules Designer:
At this level we also provide the capability for BAs to upload test XML documents that can be validated against the specified schemas. The BA can also test the (XSL) map for proper transformations. Any errors for schema validation and/or map transformation are returned to BA in the Designer. In fact, if the BA provides a valid XML document with test values that represent what the real values are likely to be, business rules can also be executed, in which the return values are the resulting actions that would occur for the rule processing.
Thats it. The BA has now declaratively configured a complete application integration scenario using our DSL Visual Designer tool that implements our application integration framework design pattern. As mentioned before, our Visual Designer is exposed over the internet as a Rich Internet Application (RIA). This allows a BA easy access to their application integration solutions from anywhere in the world, in a secured manner. This approach models the software as a service scenario where there are no requirements for an organization to use the Visual Designer other than a browser.
Now what? After the BA has completed defining the application integration model and saved the model, the BA can now order the integration definition to be built (i.e. code generated) from our Software Factory. The order includes pricing based on the number of application integration connections modeled in the scenario. Once the order has been placed, the turnaround time could be a few hours to a few days, depending on options ordered. The BA then returns to the Visual Designer web site and can download the resulting MSI to be installed on their target system. Once installed, the BA and/or System Administrator can run a self-test to verify and certify the application integration solution for correctness.
The runtime installation also includes other tools to manage and monitor the application integration solution. This includes Business Activity Monitoring for monitoring the application integration scenario, a Centralized Exception Manager that handles any runtime messaging exchange exceptions, complete with notifications, and an Integration Manager that allows runtime configuration of security credentials and application end point locations.
With respect to making any changes to the application integration scenario, the BA can launch the Visual Designer again, load up the existing model, that has been versioned, make modifications (to a schema for example) and save, then order and get another MSI that performs the upgrade process on the installed application integration solution.
Next post I will discuss our Software Factory for taking application integration scenarios based on our framework design pattern (as realized by our DSL Visual Designer and output as an XML definition) and walk through the steps on how we configure our Software Factory template to code generate the specified solution.
Thursday, 15 December 2005
In part 3, I described our application integration framework pattern at the level of BusinessFlows. A BusinessFlow contains 2 or more Applications and 1 or more Connections. I described how the framework pattern works with Applications and in this post we discuss Connections between Applications.
A Connection represents a logical connection to each Application, and physically to each Applications Adapter. A Connection contains 1 or more DataFlows. A DataFlow is a collection object that contains other objects, which we will describe in a minute, but its construct allows us to contain one to many DataFlow rows. Each DataFlow row specifies the direction in which the communication of messages (i.e. data) flow. It also specifies which Application in the integration scenario has initiated the event in the communication, either in real-time or on a scheduled (i.e. batch) basis.
For example, if I have MS Great Plains on one end of the connection and the Corporate HQ Financial Application on the other end of the connection, I can specify that MS Great Plains will be the initiator of the business event and the messages (i.e. financial data) will flow from MS Great Plains to the HQ Financial Application. Further, I can specify the DataFlow row type of communication. For example, one-way asynchronous, one-way asynchronous, two-way request response synchronous, and two-way request with delayed response (i.e. synch on aysnch).
As mentioned above, a DataFlow contains a collection of objects, specifically, 1 to 4 schemas (i.e. XSDs), 0 or more rules, and 0 to 2 maps. A map is an XSL transformation, using a visual editor that allows a BA to map from one schema format to another. Anyone familiar with the BizTalk Mapper tool will recognize our visual editor as the same representation, except we expose our Mapper tool over the internet in our Rich Internet Application (RIA) Visual Designer.
The reason for 1 to 4 schemas is that depending on the communication type and whether it is a pass-thru or not with respect to message flow. For example, in our MS Great Plains to Corporate HQ Financial Application, we have specified 2 message schemas, one for the Great Plains format and the other for the HQ Financial Application, in a one-way asynchronous dataflow. We also have 1 XSL map that transforms the source Great Plains message format into the destination HQ Financial Application format. Further we may have specified certain rules to be executed either before and/or after the mapping transformation.
As mentioned above, our DSL Visual Designer is exposed over the internet as a Rich Internet Application (RIA). We also provide a facility for a BA to upload any custom message schemas that have developed.
In my next post, we will discuss the Rules Designer, which is important to BAs and their business as this is where any data processing of the data occurs.
Thursday, 08 December 2005
In part 2, I started describing our framework design pattern for defining application integration scenarios. At the top level is our Model object that contains one or more Business Entities. Each Business Entity can also contain one of more Business Entities to represent any hierarchical business entity structure that occurs in the real world. Each Business Entity contains one or more BusinessFlows. A BusinessFlow represents Business Processes which can be a simple A2A integration or a complex workflow process. With respect to our DSL Visual Designer, each object in the model includes a wizard and/or property sheet for the Business Analyst (BA) to enter values that describe the object being configured.
Looking at the next level in the object model we can see that a BusinessFlow contains 2 or more Applications and 1 or more Connections. An Application object represents a real-world computer application. We have a number of Application objects pre-built (see Application Types in the model) allowing the BA to select from a list. If the Application required by the BA is not in the list, then a custom Application is entered which will require custom code to describe its interface (i.e. Adapter). Each Application has an Adapter. An Adapter is a plug-in connector where one end of the connector knows how to natively communicate with the Applications API and the other end exposes an interface that has been transformed into XML messages (including XSDs) that the Application can exchange with other Applications.
In our continuing example of financial applications exchanging data, the Application may be MS Great Plains which comes with a COTS adapter (i.e. connector) that performs this transformation in communicating with Great Plains native API and exposes an XML messaging interface. Since we are using BizTalk as the underlying messaging engine, there are many COTS adapters available that provides this connector and transformation mechanism.
In the case of a custom Application where a COTS adapter is not available, we use the provided BizTalk Adapter SDK to write custom code to build an adapter. Note this is the only variable point in the entire application integration scenario that requires custom code to be written, that is, if a COTS adapter is not available. The custom code is limited in scope as we are extending a pre-existing adapter framework supplied by the middleware vendor for a specific Application in which we will implement only the messaging interfaces required for the application integration scenario.
This is a key point to keep in mind as everything else in the application integration scenario is defined declaratively. Since everything else is declarative, this dramatically reduces the amount of custom code that has to be written for any given application integration scenario. In fact, even with a custom adapter, this means that the entire application integration definition is syntactically and semantically complete, which means we can code generate the entire solution and the main reason why the Software Factory approach works: model-driven development, which dramatically raises the level of abstraction for developing application integration solutions to the point where a BA declaratively defines the complete solution through a modeling tool (i.e. our DSL Visual Designer) to be generated.
Using model-driven development, we can define a model up-front that is both complete and of high-fidelity. We can then validate the model against a known schema and generate the output. Our application integration framework design pattern is this schema (i.e. Software Factory schema). Using our DSL Visual Designer, the BA models a specific application integration scenario in which the Designers output produces a valid XML document definition that describes that specific application integration scenario. The XML definition is used as input to our Software Factory template which configures and extends Visual Studio design time components and reuses our pre-built components that are checked out from our source control system. This includes configuring BizTalk Server and our factory template includes the instruction set (i.e. scripts) to build the complete solution into an installable package (i.e. MSI package) which is then installed on the target system.
Returning to our framework design pattern, a Connection object represents the actual connection between Applications and will be the topic of my next post.
Sunday, 04 December 2005
In part 1, I described my experience using Software Factories, including DSLs, to predictably and repeatedly solve application integration problems. Our Software Factory framework pattern evolved over the course of 4 years and 25 application integration projects. Essentially we developed a product that encapsulated our framework pattern that could solve any application integration problem with minimal coding. This was done by analyzing what is common and what is variable to every application integration scenario and designing a framework model to cover these scenarios. My intent here is to cover each aspect of the model and discuss how and why we arrived at our overall application integration framework pattern.
Referring to our framework object model, the top most container is the Model object in which it contains all of the objects that make up one or more application integration scenarios. Essentially it is an abstract (root) container, which translates to the top-level drawing canvas in our (DSL) visual designer modeling tool.
The first common modeling construct is that all application integrations occur within a Business entity or between Business entities. A Business entity is an abstract object that can represent an individual company, a division of a company, a remote office, an enterprise or small group within an organization. A Business entity may also contain other Business entities.
It seems obvious or even taken for granted that a Business entity models a real world construct (i.e. a real business). However, in my experience this construct is typically either overlooked or viewed as trivial. The reality is that application integrations do not occur in thin air, they are tied to real businesses, thus part of the problem domain we are interested in modeling. Having this construct allows us to model any application integration scenario within or between any number of uniquely named Business entities. For example, lets say we have 50 satellite offices that are geographically dispersed that use various accounting software packages that need to exchange financial data with a centralized financial package (i.e. Corporate HQ).
Without the capability to model this in a visual designer would make it very difficult to design and construct this application integration scenario, let alone actually visualize semantically what is actually going on. While we are talking about the business entities and the logical connections between the business entities, we are also describing the many properties of each unique business entity and its relationships to each other. This translates into unique name spaces required to define the paths between (or within) these businesses in the models XML definition output. In other words, we have declaratively captured all of the information required for each business and to navigate from one business entity to the next.
Looking at the framework design pattern at the next level, we can see that each business entity has one or more Business Flows. A Business Flow is an abstract object contained in a collection whose parent is a Business entity. The Business Flow represents a particular Business Process which may be a simple A2A integration, or may be a complex workflow process. For example, if one of our satellite offices (i.e. business entity) had three financial applications that exchanged data between each to form an aggregation, before it sent its aggregation to the centralized financial package, then we need the capability to model this.
You can see an example of what this visually looks like in this layered view of our visual designer model at the canvas layer called Business Flow. You can see how this visual representation conforms to our framework design pattern. You can also see how this visual representation can scale to represent any number of Business Entities and the collections of Business Flows for each Business Entity. While this occurs in the real world, few application integration vendors toolsets have the capability to capture this critical information.
In part 3, we will continue walking through our application integration framework pattern.
Wednesday, 30 November 2005
This blog is dedicated to the advancement of what I call the industrialization of software. Software industrialization is about raising the level of abstraction for programmers to produce quality software products using modern software engineering approaches such as Software Factories. One aspect of the Software Factory approach employs model-driven development using Domain Specific Languages (DSLs) that code generates parts (or all) of the solution for a specific problem domain. The Software Factory approach enables programmers to produce and assemble software in a predictable and repeatable manner something we dont do well as an industry, but is done well in other industrialized engineering disciplines like civil, electronics, electrical and mechanical domains. Or at the very least, software engineering is nowhere near as evolved as these engineering disciplines.
The author of this blog lived the Software Factory approach while working at a company called 5by5 Software, now called Bridgewerx. 5by5 was initially a Microsoft Systems Integrator (SI) that offered professional services in the area of application integration, designing and constructing solutions using Microsofts BizTalk Server product. Over a 4 year time frame, we designed and constructed over 25 application integrations solutions. These solutions included messaging exchanges and workflow of all sizes, shapes and forms for various vertical industries.
During this 4 year time frame, we recognized several application integration design/code patterns. Each pattern went through the process of discovery, reuse, refinement, and then iterated through this process again for each new project we undertook. After designing and constructing 25 BizTalk integration projects, we found an overall framework design pattern that could be applied to all integration projects. We decided to encapsulate this design pattern in a modeling tool, which really is a Domain Specific Language (DSL) for designing and generating application integration solutions sitting on top of BizTalk Server.
The idea is that the DSL runtime Visual Designer would be used by a Business Analyst, who has the business domain knowledge to model an application integration scenario and the modeling tools output (XML) would supply the complete definition of this specific modeled application integration. We used this complete definition as input to our software factory template which then grabbed reusable components, other artifacts (with some custom code) and configure BizTalk Server with the specific application integration scenario, automatically build the solution in Visual Studio and produce a Microsoft installer package (MSI) which then can be installed on the customers target machine. We called our Software Factory and DSL, Bridgewerx.
The point is that the software factory approach, incorporating DSLs works. Our 4 years of application integration domain experience and subsequent ISV product is living proof of this fact. We were pioneers in producing a software factory for the design of and code generating application integration solutions.
To further substantiate this claim, I will walk through a number of our projects and describe the patterns we discovered, designed, codified, categorized and subsequently automated, until we had the entire application integration framework pattern designed. Then we encapsulated the framework pattern into a DSL and built our software factory for code generating application integration solutions.
From where I sit, this is industrializing the software development process for the application integration domain. Another claim I make is that this same approach can be successfully applied to other software application domains of interest (e.g. software factory for generating families of e-commerce applications.)
In part 2, we will discuss what is common and what is variable to every application integration scenario as a way of describing our framework pattern. Then we will look at some specific integration projects we undertook and how it fits the framework pattern.
Wednesday, 12 October 2005
In previous posts, I discussed, What is a Software Factory, BRIDGEWERX and Domain Specific Languages (DSLs). DSLs are part of Microsofts Software Factory initiative using model driven development.
This is how Microsoft defines DSLs and model driven development, there is a belief among those involved in software development that somehow, modeling can be applied to make their lives easier. Our vision is to change the way developers perceive the value of modeling. To shift their perception that modeling is a marginally useful activity that precedes real development, to recognition that modeling is an important mainstream development task and not an activity primarily focused on documentation. When models are regarded as first-class development artifacts, developers write less conventional code since more powerful application abstractions can be employed. Model-driven development is thus inherently more productive and agile. Moreover, others involved in development, from business analysts, architects, designers to network staff, and system management specialists, will perceive modeling as adding value to the tasks for which they are responsible. When models span development and run-time activities in this way, communication between people can be optimized, and traceability enabled across the life cycle in any direction. We hold that making modeling mainstream in this way can ultimately change the economics of software development and ensure software systems meet the needs of a business. This approach to model-driven development is part of an initiative at Microsoft called Software Factories."
I am a big fan of Microsofts software factory initiative, model driven development and DSLs. Why? Because I believe that this approach drives the industrialization of software. What do I mean by software industrialization? In my view, it is making the software development process a predictable and repeatable activity, which given the state of the art today, it certainly is not. Also, as a 15 year veteran programmer, I am looking for anything that will make my life easier in developing software.
I believe in the software factory vision so much that Barry Varga and I invented a product called BRIDEGWERX, which is a software factory for generating application integrations. We built this product using Microsofts BizTalk Server product as the integration engine and developed a Domain Specific Language (DSL) for business analysts and developers to model application integration scenarios using a visual designer, which then outputs a factory schema, which is then used by our software factory template to code generate the resulting application integration solution.
While this may seem like a plug for our product (it is , I am also very concerned about the state of the art of software development in our industry. The last statement in Microsofts definition of DSLs states, We hold that making modeling mainstream in this way can ultimately change the economics of software development and ensure software systems meet the needs of a business. Isnt that what software is all about? Meeting the needs of the business?
In any other industrialized industry, such as building architecture, manufacturing, electronics engineering, bridge building, etc., modeling (blueprints) is a considered a first class development artifact. As the subtext to the title of this blog site states, if you cant model it, you cant build it. John Walker, the inventor of AutoCAD, figured this out in 1982! It seems in 2005 we are still trying to figure this out for our software industry. At least one major software corporation and our small company BRIDGEWERX, is trying to advance the industrialization of software.
Wednesday, 31 August 2005
Last post we defined what was meant by an on-demand application generator. Lets look at a real world product. This on-demand application generator for producing application integrations is called BRIDGEWERX (BW).
Have a look at BWs component architecture. It consists of 3 subsystems. These are an on-line Rich Internet Application (RIA) for defining (i.e. modeling) the application integration(s), an application generator for producing the application integration solution, and the runtime environment for executing the generated application integration(s).
The graphical user interface (BW Designer) is developed using Macromedias FLEX product. The only requirement for running the GUI is a Flash plug-in (which you no doubt already have installed in your browser today. BW Designer is a modeling/drawing tool that allows a Business Analyst (BA) to model, draw and define application integrations on a drawing canvas where various levels of detail are represented by canvas layers.
Several business objects can be dragged and dropped onto different canvas layers, with each object having a specific configuration wizard for the BA to configure for any application integration scenario. In addition to selecting XML message schemas, defining transport protocols, business rules, business event triggers, adapters, mapping definitions, etc., there are also facilities for uploading custom XML message schema definitions and even custom assemblies, if required.
BW Designer uses web services to communicate with a Windows 2003 Server using IIS to store the configuration data (i.e. factory schema instance) into a SQL Server repository. As discussed in a previous post, the amount of effort is directly related to the size and complexity of the application integration(s) being defined. The best news is with this new level of abstraction and the fact that the application integration is generated, with little to no source coding involved, the time to develop an integration solution is a matter of days or a week or two as opposed to several weeks or months and in some cases, years for a ground up totally manual labor custom coding job. See the sorry state of our software industry.
Once the BA has modeled, drawn and defined an application integration(s) using nothing but a browser on his/her desktop computer, anywhere in the world, the BA can simply order the application integration to be generated. The order system receives the customers order and extracts the customer information along with the factory schema instance that represents the specific application integration (i.e. software product line) to be generated.
The application generator, or software factory if you like, reads the customers factory schema instance from the BW config repository and along with a customized factory template, using Visual Studio script code, automatically gets latest versions of specified components (generic and/or custom) from source code control, along with other scripts and/or assemblies and/or any other Visual Studio artifact required, creates a Visual Studio solution and then builds (i.e. compiles) the solution.
The solution is compiled and packaged as a Microsoft Installer package. The customer is then notified that the solution is ready for download from BW Designer. The customer then downloads the MSI package and runs the installer on the target run-time environment. Alternatively, the customers order may have included an integration appliance that includes the hardware, software and the solution, already pre-installed on a computer and tested, to be shipped directly to the customer.
Once the customer has installed the solution, along with a few run-time parameters such as physical addresses of where the applications to be integrated reside and enough credentials to run the solution, a self-test is run and the customer is notified that the integration solution is now operational. Other administration, monitoring and exception handling management tools are also installed for complete, managed runtime operation of the generated solution.
Need changes to the application integration solution? The customer simply reopens the model (i.e. drawing), makes changes, re-orders the solution, the solution is generated on-demand and since the installer wizard knows that this is simply an upgrade, the generated application integration solution is updated in the same way as applying an upgrade to MS Word on the desktop for example.
Whew, long post! Nonetheless, I hope you can see that there are real-life on-demand application generators out there, BRIDGEWERX being one of them. If you want to see BRIDGEWERX in action, have a look at this 1 hour Live Meeting broadcast.
Products like BRIDGEWERX are greatly assisting the industrialization of software. We just need more of them!
Monday, 29 August 2005
In my last post, I discussed a model for sizing and estimating effort for building software applications. This model worked reasonably well based on some assumptions. One assumption was that customers actually saw the need and value in spending the time documenting use cases and going through an estimating process. That in itself is a hard thing to do. Another assumption was that this was not a net new, never been done before, application by the development team or using brand new technology to build the application. It also assumes a development team that has worked together on more than dozen projects. Just like any team sport, it takes several times playing together for the team to gel.
So what does this have to do with on-demand applications? Everything! Because the reality is that most customers dont understand the need for software sizing and estimation. Also, the application being developed, indeed is somewhat net new and maybe with new technologies and the fact that most software teams are nothing more than assembled bodies, thrown together to get a project done in the shortest time possible based on who is available at the time. The irony is that the project will likely cost (much) more and take (much) more time because the customer and the thrown together team fail to follow a sound engineering process for properly sizing the software and estimating the effort required to build the application.
Enter on-demand application generation. After Googling the phrase, I think some explanation is required to help set the context of what I mean. On-demand means I can get it now or in short order. Application Generation means that I can use a higher level of abstraction to define what the end application is supposed to be without resorting (or at least minimizing) source code programming.
Sound too good to be true? Lets take the company I used to work for as an example. We had a core, very experienced, software development team that worked together on roughly 25 projects over a 4-year time frame, specializing in application integration projects. Translation - we had a very high performance team focused on one type of application development, which meant that we leveraged very specific technologies and code libraries (some built, some acquired) and used a number of integration design patterns to mostly assemble our integration applications. Then we developed that higher level abstraction into an on-line drawing tool that allowed us to define the integration application to be built and then we built an application generator to take this definition to assemble and code generate the final application integration. This system is called BRIDGEWERX and is designed to generate application integrations on-demand.
I think our company is/was unique because we understood the business value of having a high-performance software development team where management did not view programmers as simply bodies that could be plug and played together with some wild expectation that somehow this was going to work. You dont just assemble a NFL football team overnight and expect to win the SuperBowl the same season. Ludicrous! Yet our business leaders expect just that. Its just software isnt it?
With respect to the estimation piece, using on-demand application generators dramatically reduces the effort required as most of the effort is up front, defining the application to be built using a high-level abstraction modeling tool as opposed to manual labor source coding. Oh to be sure, there will always be some level of custom programming, but the point is that 80% of it can be generated and the other 20% is highly focused specialized work. In the case of our application integration generators, the only custom development is in developing an interface to the application to be integrated if one could not be bought off the shelf. The rest of the effort was simply using the modeling/drawing tool to define the application integration to be generated. This could be measured in hours and days instead of weeks and months (or years) with the traditional manual labor approach. Even more important is that this approach is more predictable and repeatable and therefore, dramatically reduces risk, particularly for the customer. This is the beginning of the industrialization of software.
Next post we will look at BRIDGEWERX components to see how it qualifies as an on-demand application generator.
Friday, 05 August 2005
Barrys and my invention, and the specific implementation of that invention, is so new and abstract, that it is a real challenge to bring the software to market. As John Walker, inventor of AutoCAD has said, we need to find lunatic early adopters to try the software out and evolve quickly from there.
As discussed earlier, our software world is incredibly complex and highly abstract, so much so, it is virtually (no pun intended) impossible to describe our invention to business people and even to technical people in our own specific area of our industry. Why is that? There have been many reasons given over the last several posts, but it comes down to our invention is ahead of its time. I believe this is an inherent characteristic of any invention, because an invention is just that, it is brand new and requires people to think in ways that they never have thought of before. What makes our invention even harder to understand is that it works in the virtual world of software which is already a complete mystery to most people.
Think of it this way, we invented a standard way to design (i.e. draw and define) application integrations of any size and complexity. The resulting output definition can be read by other drawing tools and therefore can be reused to replicate or use as a template for other companies application integrations that have similar business scenarios. In other words, the drawings form a library of business process integrations that can be reused by anyone, much like the way AutoCAD has a library of pre-built designs that can also be leveraged by anyone. This, in my mind, is advancing the industrialization of software.
We also invented an intelligent messaging definition that can be used by any middleware engine, in our case we are using BizTalks middleware engine. However, our intelligent message is a complete abstraction away from any middleware engine implementation. This means that other middleware engines can use the same standard file definition for business process execution.
And thats the key to our invention, much like AutoCADs invention of the DXF universal file format which "is probably one of the most widely supported vector formats in the world today". We have invented design-time and run-time XML schema definitions that any drawing tool (design time) and middleware engine (run-time) can use as both definitions contain everything required to define and execute any business process integration scenario totally independent of programming languages, operating systems, middleware engines or any implementation details whatsoever. In my opinion, this is raising the level of abstraction to a new level our software development world has not yet seen. You heard it here first
How would other middleware engine manufacturers use our invention? For example, Sun acquired SeeBeyond, which is a middleware engine, therefore Sun could use our invention to not only automate the design and code generation of application integrations for SeeBeyond, but also, and more importantly, build a library of reusable business process integrations for their entire application suite of software products.
Microsoft Indigo is another middleware engine that could benefit from our invention. Microsoft could use BRIDGEWERX as part of their Service Oriented Architecture (SOA) strategy. SOA is the latest TLA in our industry.
You would think with the amount of words written in these posts that this was turning into a book. It is! I decided in January of this year to write a book titled, The Industrialization of Software (shameless plug) to help educate people (and inventors alike) about our strange software world and even more bizarre business world of software. It chronicles my 4 years as President of 5by5Software and details most of the topics you have read over the last several posts that Shane Schick has been kind enough to post on ITBusiness.ca
As I see it, we are in a software revolution where the last 4 years of IT has been the worst in the history of our industry. Businesses have revolted against the software world largely because of our abysmal track record of delivering quality software. Now the general public (with AntiSpyware, SPAM, etc.) have entered the fray. For those of you that use software in your daily life, you have the power to make (demand!) the necessary changes in our software industry to bring us out of the dark ages and force us into the industrialization of software.
Thursday, 04 August 2005
Have a look at this object model. This is our modeling design pattern that Barry and I figured out after designing and constructing 25 application integration solutions over a 4 year time frame. This design pattern raises the level of abstraction for designing and constructing application integrations into a fully automated process that is predictable and repeatable.
How does it work? We have taken our design pattern and implemented it as a drawing/modeling tool, much like AutoCAD, where you can draw your application integration scenarios on a canvas. The modeling tool produces a full-fidelity or complete design specification output in which a code generator can read the completed specification and automatically produce (i.e. code generate) the solution, wrapped as an installation package, much like using an installation wizard to install a program on your computer.
At run-time, there are several administrative, operational and monitoring tools that are provided to manage the solution that are installed on a pre-configured technology stack, including a middleware engine. In fact, everything can be pre-installed onto a DELL server and delivered as a complete integration appliance.
Here is another view of the object model as represented by the layers or levels in our modeling/drawing tool. This is how we visualize size and complexity. Just like how AutoCAD has many levels of detail, we do too. We can view the upper most level to see the big picture (which may contain dozens of application integration scenarios across several business entities) and yet drill down, level by level, to the lowest level of detail of an individual data element that is being mapped for example. We can see the entire size and complexity of the business process problem being solved.
Another key to our invention is the Intelligent Messaging format. The intelligent messaging format is intended to be used as a standard wrapper for all messages flowing through an integration at run-time. The actual message content (i.e. data) is contained within the envelope as a payload. The envelope also contains a number of other fields used to control its flow through the business process and to record the route and activities undertaken during that process.
The use of a standard schema allows a higher degree of abstraction within a business process as this standard schema contains (i.e. encapsulates) all business process rules and state information necessary to execute the business process integration without any knowledge of the rest of the system and vice versa (i.e. fully abstracted). The word standard is very important in our software world as there are few standards. We invented a standard messaging format for executing application integrations in addition to a standard object model (i.e. drawing tool) that allows business analysts to fully design and describe application integrations in hours/weeks that are code generated and installed in minutes.
You would think an invention like this would help, in some small way, industrialize our software world like AutoCAD did for the engineering design world in the early 80s. However, as with most inventions, including AutoCAD, getting to market is an entirely different story which I will describe in tomorrows post.
Tuesday, 19 July 2005
So how to introduce this technology? Lets start off with a quote from Craig McMurtry, Microsoft evangelist extraordinaire, who happened by one day to see BRIDGEWERX in action,
"BRIDGEWERX exemplifies a new breed of software vendor, best described as an on-demand software factory, that may well turn out to be the fabled next big thing in the Information Technology industry. On-demand software factories provide superior software design tools in the form of freely-accessible services over the Web, and then sell the software that one designs with those tools, software that plugs right into the target environment. In the case of BRIDGEWERX, that free designer is a masterfully simple, yet uniquely powerful facility for representing complex business-to-business, and application-to-application integrations in a practical and intuitive way. Anyone with an integration problem to solve should certainly consider BRIDGEWERX among the top few tools they could use."
Barry Varga and I, being the creators of BRIDGEWERX, were quite flattered by Craigs quote. For myself, it was a bit of a relief as we were (and still are) having a heck of time trying to describe our wares to people. Craig gets it, but not too many other people do. Why is that?
As Albert Einstein once said, the only source of knowledge is experience. As it turns out, not too many people have experience with application integration. For the ones that dont, they are quite surprised to find out that applications dont normally communicate with each other as a matter of course. For the ones that have application integration experience, they are extremely skeptical as they know what a mind numbing, labor intensive process it really is and cant believe one can draw an application integration scenario, let alone code generate the solution.
In some small way, I would like to think Barry and I contributed to the industrialization of software with our invention. If you want to see BRIDGEWERX in action, (it will only take an hour , grab a beer, some popcorn and treat yourself to a Live Meeting presentation (courtesy of Jim Bowyer - BizTalk guru from Microsoft).
In my next post, I will discuss the history of this invention. As the saying goes, ...I have not failed. I've just found 10,000 ways that won't work --Thomas Edison
© Copyright 2010 Mitch Barnett - Software Industrialization is the computerization of software design and function.
newtelligence dasBlog 2.2.8279.16125 Theme design by Bryan Bell
 | Page rendered at Wednesday, 18 August 2010 08:59:40 (Pacific Daylight Time, UTC-07:00)
|
On this page....
Search
Categories
Software Engineering Links
Sign In
|