Last post we defined what was meant by an on-demand application generator.  Lets look at a real world product.  This on-demand application generator for producing application integrations is called BRIDGEWERX (BW).

 

Have a look at BWs component architecture.  It consists of 3 subsystems.  These are an on-line Rich Internet Application (RIA) for defining (i.e. modeling) the application integration(s), an application generator for producing the application integration solution, and the runtime environment for executing the generated application integration(s).

 

 

The graphical user interface (BW Designer) is developed using Macromedias FLEX product.  The only requirement for running the GUI is a Flash plug-in (which you no doubt already have installed in your browser today. BW Designer is a modeling/drawing tool that allows a Business Analyst (BA) to model, draw and define application integrations on a drawing canvas where various levels of detail are represented by canvas layers. 

 

Several business objects can be dragged and dropped onto different canvas layers, with each object having a specific configuration wizard for the BA to configure for any application integration scenario.  In addition to selecting XML message schemas, defining transport protocols, business rules, business event triggers, adapters, mapping definitions, etc., there are also facilities for uploading custom XML message schema definitions and even custom assemblies, if required.

 

BW Designer uses web services to communicate with a Windows 2003 Server using IIS to store the configuration data (i.e. factory schema instance) into a SQL Server repository.  As discussed in a previous post, the amount of effort is directly related to the size and complexity of the application integration(s) being defined.  The best news is with this new level of abstraction and the fact that the application integration is generated, with little to no source coding involved, the time to develop an integration solution is a matter of days or a week or two as opposed to several weeks or months and in some cases, years for a ground up totally manual labor custom coding job. See the sorry state of our software industry.

 

Once the BA has modeled, drawn and defined an application integration(s) using nothing but a browser on his/her desktop computer, anywhere in the world, the BA can simply order the application integration to be generated.   The order system receives the customers order and extracts the customer information along with the factory schema instance that represents the specific application integration (i.e. software product line) to be generated. 

 

The application generator, or software factory if you like, reads the customers factory schema instance from the BW config repository and along with a customized factory template, using Visual Studio script code, automatically gets latest versions of specified components (generic and/or custom) from source code control, along with other scripts and/or assemblies and/or any other Visual Studio artifact required, creates a Visual Studio solution and then builds (i.e. compiles) the solution.

 

The solution is compiled and packaged as a Microsoft Installer package.  The customer is then notified that the solution is ready for download from BW Designer. The customer then downloads the MSI package and runs the installer on the target run-time environment.  Alternatively, the customers order may have included an integration appliance that includes the hardware, software and the solution, already pre-installed on a computer and tested, to be shipped directly to the customer.

 

Once the customer has installed the solution, along with a few run-time parameters such as physical addresses of where the applications to be integrated reside and enough credentials to run the solution, a self-test is run and the customer is notified that the integration solution is now operational.  Other administration, monitoring and exception handling management tools are also installed for complete, managed runtime operation of the generated solution.

 

Need changes to the application integration solution? The customer simply reopens the model (i.e. drawing), makes changes, re-orders the solution, the solution is generated on-demand and since the installer wizard knows that this is simply an upgrade, the generated application integration solution is updated in the same way as applying an upgrade to MS Word on the desktop for example.

 

Whew, long post!  Nonetheless, I hope you can see that there are real-life on-demand application generators out there, BRIDGEWERX being one of them.  If you want to see BRIDGEWERX in action, have a look at this 1 hour Live Meeting broadcast.

 

Products like BRIDGEWERX are greatly assisting the industrialization of software.  We just need more of them!

In my last post, I discussed a model for sizing and estimating effort for building software applications.  This model worked reasonably well based on some assumptions.  One assumption was that customers actually saw the need and value in spending the time documenting use cases and going through an estimating process.  That in itself is a hard thing to do.  Another assumption was that this was not a net new, never been done before, application by the development team or using brand new technology to build the application.  It also assumes a development team that has worked together on more than dozen projects.  Just like any team sport, it takes several times playing together for the team to gel.

 

So what does this have to do with on-demand applications?  Everything!  Because the reality is that most customers dont understand the need for software sizing and estimation.  Also, the application being developed, indeed is somewhat net new and maybe with new technologies and the fact that most software teams are nothing more than assembled bodies, thrown together to get a project done in the shortest time possible based on who is available at the time.  The irony is that the project will likely cost (much) more and take (much) more time because the customer and the thrown together team fail to follow a sound engineering process for properly sizing the software and estimating the effort required to build the application.

 

Enter on-demand application generation.  After Googling the phrase, I think some explanation is required to help set the context of what I mean.  On-demand means I can get it now or in short order.  Application Generation means that I can use a higher level of abstraction to define what the end application is supposed to be without resorting (or at least minimizing) source code programming.

 

Sound too good to be true?  Lets take the company I used to work for as an example.  We had a core, very experienced, software development team that worked together on roughly 25 projects over a 4-year time frame, specializing in application integration projects.  Translation - we had a very high performance team focused on one type of application development, which meant that we leveraged very specific technologies and code libraries (some built, some acquired) and used a number of integration design patterns to mostly assemble our integration applications.  Then we developed that higher level abstraction into an on-line drawing tool that allowed us to define the integration application to be built and then we built an application generator to take this definition to assemble and code generate the final application integration.  This system is called BRIDGEWERX and is designed to generate application integrations on-demand.

 

I think our company is/was unique because we understood the business value of having a high-performance software development team where management did not view programmers as simply bodies that could be plug and played together with some wild expectation that somehow this was going to work.  You dont just assemble a NFL football team overnight and expect to win the SuperBowl the same season.  Ludicrous!  Yet our business leaders expect just that.  Its just software isnt it?

 

With respect to the estimation piece, using on-demand application generators dramatically reduces the effort required as most of the effort is up front, defining the application to be built using a high-level abstraction modeling tool as opposed to manual labor source coding.  Oh to be sure, there will always be some level of custom programming, but the point is that 80% of it can be generated and the other 20% is highly focused specialized work.  In the case of our application integration generators, the only custom development is in developing an interface to the application to be integrated if one could not be bought off the shelf.  The rest of the effort was simply using the modeling/drawing tool to define the application integration to be generated.  This could be measured in hours and days instead of weeks and months (or years) with the traditional manual labor approach.  Even more important is that this approach is more predictable and repeatable and therefore, dramatically reduces risk, particularly for the customer.  This is the beginning of the industrialization of software.

 

Next post we will look at BRIDGEWERX components to see how it qualifies as an on-demand application generator.

Before we continue with on-demand application generators, lets look at something completely different, but related.  Software size and effort estimation (read: cost estimation).

 

There are several formal methods for estimating software size, complexity, effort and cost.  Function Point Analysis, Lines of Code, tools like CoCoMo, SLIM, etc., including the most common technique called Wideband Delphi Estimating where senior programmers, architects, etc., base their estimates on real world experiences while following a formal estimating process.

 

Why do we do estimate?  Every customer I have come across always asks, how much? And shortly followed up with, and when can I get it? And finally, can you make it a fixed price estimate? 

 

Its like getting a building constructed, you need a blueprint to estimate a price for the construction, whether the blueprint is custom designed or "off the shelf". Regardless, the customer had to spend money on acquiring the blueprint. The blueprint puts in scope whether it is a house the customer wants or maybe it turns out to be an apartment complex or skyscraper that the customer really wants, hence the blueprint.  Constructing software is no different, you need a blueprint.  The blueprint we require are Use Case scenarios in order to provide an accurate cost estimate.
 
If we had the use case scenarios written down at this stage, then it would be much easier to come up with an accurate estimate.  However, the customer typically does not understand, that the level of detail we need is way more than the customer has supplied.  Almost always.  And the level of detail is down to documenting use case scenarios, usually a dozen to two dozen scenarios on smaller projects.  The quantifiable output is usually represented as written documentation with an average of 3 pages written per use case scenario, which in our example, means roughly 15 use cases x 3 pages each = 45 pages of  use case documentation.  Based on 15 years in the biz, I can count on one hand where I actually got enough detailed and documented use case scenarios from the customer to come up with an accurate cost estimate, as is.  For people that know me, I usually get requirements written on a cocktail napkin.  Funny, but not enough detail to be sure.

 

Geri Schneider and Jason Winters wrote an excellent book called, Applying Use Cases: A Practical Guide  In the book is a cost estimation model based on using the number of use cases and the complexity rating for each use case as input to the estimation model.  The model outputs effort estimates, with upper and lower bounds and by project phase and iterations of phases, in person days along with the ability to put in labor rates, which then outputs a cost, which then you can apply a contingency factor and out comes the final price estimate.

 

You can calibrate the use case estimation model as you use it over time, based on comparing estimates to actuals.  You can adjust estimating factors around skill sets and culture, plus the ability to calibrate around toolsets employed.  Whats great about it is that it works!  My team and I have used it successfully on a number of projects of varying size and complexity. After several iterations of comparing actual to the estimates, we got our estimation model calibrated within 10 to 20% differential between estimates and actual.

 

How does this tie in with on-demand application generators?  Next post.

In case you need to get up to speed on SharePoint Products and Technologies.

 

A typical usage of SharePoint is to use its form or document libraries as steps in a workflow process, like an approval process for example, for any form or document. Each work flow step = one document library.  If I had a workflow that contained 10 steps, I will have 10 document or forms libraries.

 

Jan Tielen has built an excellent workflow engine, called, Workflow Lite for SharePoint that uses an XML configuration file to describe the number of workflow steps, plus other elements such as properties and actions.  Properties reference document library columns, in which you customize and actions describe, well, actions like copy this document from one workflow step (i.e. document or form library) to another.

 

The idea is that you install Jans workflow engine, which is a .NET assembly that hooks into SharePoints document library event handler (a service point).  Once installed, the assembly reads the (XML) configuration file which contains all of the named workflow steps, properties and actions.

 

Here comes the software factory bit, which can be on demand.  The XML configuration file can be viewed as one part of a software factory schema in which we can build a user interface to filling out the schema with workflow steps along with properties and actions.  Another XML configuration file, (actually a SharePoint site template) can be used to specify the overall look, feel and some functionality (i.e. document library columns) of the overall SharePoint site, in which the workflow executes inside of.  You know, your company name goes here, upload your logo, URL naming conventions, etc.  We can also build a user interface to this part of the software factory schema.  Both XML documents represent the entire SharePoint workflow application, not only from a design point of view, but also used by the software factory template and at run-time.

 

In fact, I could set up an on-line web site with these two user interfaces and have anyone come to the site, fill out the forms that describe the SharePoint site definition and workflow schemas.  You could then press order, in which I will get your instance of a software factory schema. I can then use this customized factory schema in my software factory template, and with some Visual Studio scripting, I can check out Jans Workflow Lite from my source code control system, along with a SharePoint site generator, which will take the site definitions values from your order and build out a SharePoint site along (including the library columns) with the workflow, etc.  In fact, since I already know the steps in the workflow process, I can even write an automated test script to test out and certify.

 

I can then (automatically) back the site up and package the whole thing as an MSI (installer) file (from Visual Studio) and send it to you, an hour later after you ordered it. Running the installer deploys the solution to your environment, performs a self-test of the workflow steps and voila!  Hows that for an on-demand software factory?  Need changes? Go back to the web site, make them using your original specification files and re-order.  Just like that.  Sounds like an on-demand software product line for workflow applications in SharePoint.

Have you seen the movie Sideways?  It sometimes reminds me of the IT world where everything seems OK and then something goes so sideways that you wonder if you are in the right profession.

 

4 years ago, my previous company was hired to develop a Contracts Management System (CMS) for an Energy Utility company based in Western Canada.  The Energy Utility was going through the process of fitting into a recently deregulated industry.  Anyone that has been through this understands the extreme chaos involved.  Here was a Utility company whose customer base, for over 20 years as a public utility, was in the few hundreds servicing small to medium commercial businesses and now had to scale to hundreds of thousands for the retail market, hence the need for software automation.

 

The amount of business change served up so fast was a painful experience for many.  Most business processes had to be reengineered (read: discovered), every software application either had to be replaced or retooled, or a net new packaged or custom developed application was put into place to handle deregulated business processes.  With respect to CMS, the workflow for processing contracts was incredibly complex and recursive based on the deregulation rules.  We used BizTalk Server and its workflow capabilities to develop the solution and also used BizTalk from an application integration point of view to be able to send contracts (XML documents) to other applications.

 

The division we custom developed CMS for thought we did such a good job that an independent ROI case study was produced.  We were pretty happy with the work we did and felt like we provided real value to an incredibly difficult business problem.  So far so good.

 

Fast forward 4 years - I am working for a different Systems Integrator in a different location, but I get a call saying that there is this Utility Company that is looking for a document workflow solution for handling contracts. I think to myself, this cant be.  But of course it is!  It is indeed another Contracts Management System for the same Energy Utility.  What are the odds?  Or maybe this is a sure bet?  No-one knows what happened to the old CMS as there are all new employees in the IT department.

 

It kinda feels like Groundhog Day as everyday I wake up and feel like I am still working on CMS forever and ever.  The sad part is that the Energy Utility has already spent a $500,000 on a system that apparently does not exist anymore.  What happened to it?  No-one knows.  How do you lose $500,000 in software?

 

This reminds me of an excellent book by the Alan Cooper, father of Visual Basic, called, The Inmates are Running the Asylum.   If there ever was a truism in the IT business, I would say this one is it.

 

Someday, when software industrialization actually makes it way into our world, these types of sideways scenarios will become less prevalent in both the business and IT world.  In the meantime, its Groundhog Day!

Jack Greenfield and Keith Short wrote an excellent book called, Software Factories: Assembling Applications with Patterns, Models, Frameworks and Tools.  They define a Software Factory as, A software product line that provides a production facility by configuring extensible tools using a software template based on a software schema.

 

Wow!  I think we are going to need a few more definitions to understand what this means.  A definition of software product line is:  A software product line is a set of software-intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way.

 

A software schema is a document that categorizes and summarizes the artifacts used to build and maintain a system such as XML documents, models, configuration files, build scripts, sources code files, SQL files, localization files, deployment manifests and test case definitions, in an orderly way, and that defines relationships between them, so that we can maintain consistency among them.  A software factory schema essentially defines a recipe for building members of a software product family.

 

A software factory template includes code and metadata that can be loaded into extensible tools, like an IDE or an enterprise lifecycle tool suite, (author note: like BRIDGEWERX for example), to automate the development and maintenance of family members.  We call it a software factory template because it configures the tools to produce a specific type of software.

While customizing a software factory schema customizes the description of the software factory for the family member, however, customizing a template customizes the assets used to build the family member.

 

I will say it again, wow!  Jack and Keith are introducing patterns of industrialization to the software world.  Kudos to them. 

 

Lets look at what they are saying using a concrete example.  I can produce a software factory schema that will define a generic eCommerce web application.  Most people know what these are if you have ever purchased a book from Amazon or a product at Costco.com and oh and there it is.  Both Amazon and Costco are eCommerce web sites, and share many attributes from a definition (i.e. software factory schema) perspective.

 

Now based upon the needs and wants of both Costco and Amazon, we can customize the software factory schema (think of it like a bill of materials) for both.  For example, we define that for a transaction engine, Amazon is going to use a third-party component, but Costco is going to use their own source code for the transaction engine. All we are doing here is defining the recipe, or bill of materials, using a software factory schema, to determine what is going to be built.

 

The software factory template determines how the eCommerce web application is to be built as it configures the tools that build the solution.  As mentioned in the previous paragraph, while both companies accept credit card transactions, what components they are using for the transaction are different from an implementation point of view.

 

No problem, it is a customization of the software factory template, based upon the customized software factory schema, as an instantiated input to the factory template that determines which transaction engine asset is to be (re)used. In the case of Amazon, the factory template instructs the tools to go grab that named component from source control and include it in the build process for Amazons eCommerce product line.  In the case of Costco, the factory template provides the location of Costcos source code for their transaction engine, and using a specified script, gets latest version out of the code repository and inserts into the proper build order so when the solution is generated, we have the right transaction engine for Costcos eCommerce product line.

 

Pretty abstract stuff and a transmogrified process compared to the way we hand craft software today.  Have a look at a completed software factory for, "A Software Factory Approach To HL7 Version 3 Solutions".

According to my last post, I can draw a piston in a CAD program and have a (CAM) milling machine automatically produce the output.  Big deal, old hat, so what?  In the software world, we are dealing with sizes and complexity way beyond a piston.  There is no doubt of that.  However, in the design of a race car engine, it is not just the pistons being designed (and drawn), but also the aluminum engine block, rods, bearings, crankshaft, etc.  In fact, everything is designed and machined. And in the case of a race car engine, all of the parts are hand assembled in the end.

 

Thats because, people will say, that an engine is a well known entity and we have done them over and over again for what, 75 years?  I put it to you that a software invoicing application, is an invoicing application, is an invoicing application.  Just like the hundreds of thousands (millions?) of login dialog boxes that we, as programmers, continue to proliferate on an ad hoc, one off basis, we continue to maintain that every software job is 100% custom.  I call bullshit.  Btw, so does the CEO of SAP See slide 12.

At 5by5Software, we performed 25 application integration projects over a 4 year period, all of them very different for different vertical industries, yet they were all the same.  We knew the names of the applications (i.e. end points) to be connected, the names of the adapters, connectors or interfaces (whatever you want to call them) to each of these applications and whether we could purchase adapters off the shelf or we would have to build them (which was the only custom development piece in the entire project), the data (i.e. XML schemas) or messages that flowed to and from the application integrations, what type of communication it was (i.e. one-way asynch, one-way asynch with response, two-way request response, two-way synch on aysnch) and which application triggered the business event to kick-off the integration and whether it was a scheduled event or not.  Thats it.

 

In fact, that is it.  Sure I am leaving out some details, but those are just implementation choices and do not affect the fundamental design pattern as stated above.  In fact, every application integration job will have that same pattern, over and over again.  What does this have to do with code generation?  Well, everything, actually.  If the pattern is repeatable, then there must be a way to define all of the pieces or parts, just like in our race car engine up front.  If we can define (and draw) those parts (pieces, components, whatever) 100% complete up front, then we can code generate the solution, cause this is the same thing over and over again.  Some would call these software product lines (not what your think in the traditional sense of the term "product").

 

As discussed in the previous post, we seem to have software design patterns for everything, so why are we not using code generators with these patterns?  Do our tools suck?  Are we daft? Are we resistant to change?  Fearful that code generation is a career limiting move?  What then?

 

Next post, more code generation - software factories

I am going to discuss code generation over the next few posts for a couple of reasons.  One is that it is fundamental to the process of software industrialization and second, most programmers, that I know of anyway, dont like code generators, for a variety of reasons.

 

Maybe I should step back and give you my definition of what code generation means to me.  I am pretty sure we all have different ideas as to what this means.  Again, I will draw upon our analogy of AutoCAD.  Lets say I produce a drawing of a piston used for a race engine.  Once I have completed the drawing, I can then save the entire definition of that race piston in a universal file (i.e. DXF) format.  I can then take that piston file definition and feed it into a Computer Numerical Control (CNC) milling machine, which will produce (in our terms, code generate) the output, virtually 100% complete, save for a few finishing touches (i.e. polishing).

 

Thats the key.  I am required to produce a drawing that is 100% complete up front (i.e. design) before I can mill out the piston for my race engine.  When do we specify (i.e. design) anything that is 100% complete up front in the software world?  If I can design a software part or solution 100% up front, then I can code generate the solution, given the proper tools.

 

As a programmer, I would rather build the (code generator) tool that is going to produce the solution rather than the solution itself.  Why?  Because, in my experience, I know I am going to have to do it over and over again, just to get it right. 

 

Traditionally, I would incrementally iterate my way through the solution code several dozen (or even hundreds) of times which will ensure I have a POS by the time it is ready to deploy into production. 

 

So give me a design tool that allows me to draw and define what is supposed to be built (saved in a universal file format) and then I can build a code generator to read the design file and produce the solution output.  The effort is in the design and code generator, not in the solution code. It takes minutes to code generate a solution - every time.

 

Hey, I like to code as much as the next programmer.  But I am getting too old and tired and cant stay up to 4am for a week in a row when it comes down to crunch time for the final push to deliver I dont know what to the client.  If I am going to code anything, it is going to be a code generator.

 

If I build a code generator to read a file format that contains the definition (i.e. design) of a solution, then I can make changes to the design, just like when I want to change the shape or dimensions of the piston.  And if I am really smart, I would have a library of piston designs that I have accumulated over time, along with other reusable design pieces or parts (read: design patterns). 

 

We have several excellent books that document design patterns for virtually everything in software, including the infamous Gang of Four (Gamma et all), EAI design patterns, and in fact, if you type in software design patterns in Google, you get 18,900 hits.  So why dont we have code generators that take these design patterns as input?  Why not create a standardized library of design patterns that any IDE can read and produce the required output in your language of choice?  Oops, there is that word again, standard.

 

Next post will delve deeper into code generators.