In case you need to get up to speed on SharePoint Products and Technologies.

 

A typical usage of SharePoint is to use its form or document libraries as steps in a workflow process, like an approval process for example, for any form or document. Each work flow step = one document library.  If I had a workflow that contained 10 steps, I will have 10 document or forms libraries.

 

Jan Tielen has built an excellent workflow engine, called, Workflow Lite for SharePoint that uses an XML configuration file to describe the number of workflow steps, plus other elements such as properties and actions.  Properties reference document library columns, in which you customize and actions describe, well, actions like copy this document from one workflow step (i.e. document or form library) to another.

 

The idea is that you install Jans workflow engine, which is a .NET assembly that hooks into SharePoints document library event handler (a service point).  Once installed, the assembly reads the (XML) configuration file which contains all of the named workflow steps, properties and actions.

 

Here comes the software factory bit, which can be on demand.  The XML configuration file can be viewed as one part of a software factory schema in which we can build a user interface to filling out the schema with workflow steps along with properties and actions.  Another XML configuration file, (actually a SharePoint site template) can be used to specify the overall look, feel and some functionality (i.e. document library columns) of the overall SharePoint site, in which the workflow executes inside of.  You know, your company name goes here, upload your logo, URL naming conventions, etc.  We can also build a user interface to this part of the software factory schema.  Both XML documents represent the entire SharePoint workflow application, not only from a design point of view, but also used by the software factory template and at run-time.

 

In fact, I could set up an on-line web site with these two user interfaces and have anyone come to the site, fill out the forms that describe the SharePoint site definition and workflow schemas.  You could then press order, in which I will get your instance of a software factory schema. I can then use this customized factory schema in my software factory template, and with some Visual Studio scripting, I can check out Jans Workflow Lite from my source code control system, along with a SharePoint site generator, which will take the site definitions values from your order and build out a SharePoint site along (including the library columns) with the workflow, etc.  In fact, since I already know the steps in the workflow process, I can even write an automated test script to test out and certify.

 

I can then (automatically) back the site up and package the whole thing as an MSI (installer) file (from Visual Studio) and send it to you, an hour later after you ordered it. Running the installer deploys the solution to your environment, performs a self-test of the workflow steps and voila!  Hows that for an on-demand software factory?  Need changes? Go back to the web site, make them using your original specification files and re-order.  Just like that.  Sounds like an on-demand software product line for workflow applications in SharePoint.

Have you seen the movie Sideways?  It sometimes reminds me of the IT world where everything seems OK and then something goes so sideways that you wonder if you are in the right profession.

 

4 years ago, my previous company was hired to develop a Contracts Management System (CMS) for an Energy Utility company based in Western Canada.  The Energy Utility was going through the process of fitting into a recently deregulated industry.  Anyone that has been through this understands the extreme chaos involved.  Here was a Utility company whose customer base, for over 20 years as a public utility, was in the few hundreds servicing small to medium commercial businesses and now had to scale to hundreds of thousands for the retail market, hence the need for software automation.

 

The amount of business change served up so fast was a painful experience for many.  Most business processes had to be reengineered (read: discovered), every software application either had to be replaced or retooled, or a net new packaged or custom developed application was put into place to handle deregulated business processes.  With respect to CMS, the workflow for processing contracts was incredibly complex and recursive based on the deregulation rules.  We used BizTalk Server and its workflow capabilities to develop the solution and also used BizTalk from an application integration point of view to be able to send contracts (XML documents) to other applications.

 

The division we custom developed CMS for thought we did such a good job that an independent ROI case study was produced.  We were pretty happy with the work we did and felt like we provided real value to an incredibly difficult business problem.  So far so good.

 

Fast forward 4 years - I am working for a different Systems Integrator in a different location, but I get a call saying that there is this Utility Company that is looking for a document workflow solution for handling contracts. I think to myself, this cant be.  But of course it is!  It is indeed another Contracts Management System for the same Energy Utility.  What are the odds?  Or maybe this is a sure bet?  No-one knows what happened to the old CMS as there are all new employees in the IT department.

 

It kinda feels like Groundhog Day as everyday I wake up and feel like I am still working on CMS forever and ever.  The sad part is that the Energy Utility has already spent a $500,000 on a system that apparently does not exist anymore.  What happened to it?  No-one knows.  How do you lose $500,000 in software?

 

This reminds me of an excellent book by the Alan Cooper, father of Visual Basic, called, The Inmates are Running the Asylum.   If there ever was a truism in the IT business, I would say this one is it.

 

Someday, when software industrialization actually makes it way into our world, these types of sideways scenarios will become less prevalent in both the business and IT world.  In the meantime, its Groundhog Day!

Jack Greenfield and Keith Short wrote an excellent book called, .  They define a Software Factory as, A software product line that provides a production facility by configuring extensible tools using a software template based on a software schema.

 

Wow!  I think we are going to need a few more definitions to understand what this means.  A definition of software product line is:  A software product line is a set of software-intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way.

 

A software schema is a document that categorizes and summarizes the artifacts used to build and maintain a system such as XML documents, models, configuration files, build scripts, sources code files, SQL files, localization files, deployment manifests and test case definitions, in an orderly way, and that defines relationships between them, so that we can maintain consistency among them.  A software factory schema essentially defines a recipe for building members of a software product family.

 

A software factory template includes code and metadata that can be loaded into extensible tools, like an IDE or an enterprise lifecycle tool suite, (author note: like BRIDGEWERX for example), to automate the development and maintenance of family members.  We call it a software factory template because it configures the tools to produce a specific type of software.

While customizing a software factory schema customizes the description of the software factory for the family member, however, customizing a template customizes the assets used to build the family member.

 

I will say it again, wow!  Jack and Keith are introducing patterns of industrialization to the software world.  Kudos to them. 

 

Lets look at what they are saying using a concrete example.  I can produce a software factory schema that will define a generic eCommerce web application.  Most people know what these are if you have ever purchased a book from Amazon or a product at Costco.com and oh and there it is.  Both Amazon and Costco are eCommerce web sites, and share many attributes from a definition (i.e. software factory schema) perspective.

 

Now based upon the needs and wants of both Costco and Amazon, we can customize the software factory schema (think of it like a bill of materials) for both.  For example, we define that for a transaction engine, Amazon is going to use a third-party component, but Costco is going to use their own source code for the transaction engine. All we are doing here is defining the recipe, or bill of materials, using a software factory schema, to determine what is going to be built.

 

The software factory template determines how the eCommerce web application is to be built as it configures the tools that build the solution.  As mentioned in the previous paragraph, while both companies accept credit card transactions, what components they are using for the transaction are different from an implementation point of view.

 

No problem, it is a customization of the software factory template, based upon the customized software factory schema, as an instantiated input to the factory template that determines which transaction engine asset is to be (re)used. In the case of Amazon, the factory template instructs the tools to go grab that named component from source control and include it in the build process for Amazons eCommerce product line.  In the case of Costco, the factory template provides the location of Costcos source code for their transaction engine, and using a specified script, gets latest version out of the code repository and inserts into the proper build order so when the solution is generated, we have the right transaction engine for Costcos eCommerce product line.

 

Pretty abstract stuff and a transmogrified process compared to the way we hand craft software today.  Have a look at a completed software factory for, "A Software Factory Approach To HL7 Version 3 Solutions".

According to my last post, I can draw a piston in a CAD program and have a (CAM) milling machine automatically produce the output.  Big deal, old hat, so what?  In the software world, we are dealing with sizes and complexity way beyond a piston.  There is no doubt of that.  However, in the design of a race car engine, it is not just the pistons being designed (and drawn), but also the aluminum engine block, rods, bearings, crankshaft, etc.  In fact, everything is designed and machined. And in the case of a race car engine, all of the parts are hand assembled in the end.

 

Thats because, people will say, that an engine is a well known entity and we have done them over and over again for what, 75 years?  I put it to you that a software invoicing application, is an invoicing application, is an invoicing application.  Just like the hundreds of thousands (millions?) of login dialog boxes that we, as programmers, continue to proliferate on an ad hoc, one off basis, we continue to maintain that every software job is 100% custom.  I call bullshit.  Btw, so does the CEO of SAP See slide 12.

At 5by5Software, we performed 25 application integration projects over a 4 year period, all of them very different for different vertical industries, yet they were all the same.  We knew the names of the applications (i.e. end points) to be connected, the names of the adapters, connectors or interfaces (whatever you want to call them) to each of these applications and whether we could purchase adapters off the shelf or we would have to build them (which was the only custom development piece in the entire project), the data (i.e. XML schemas) or messages that flowed to and from the application integrations, what type of communication it was (i.e. one-way asynch, one-way asynch with response, two-way request response, two-way synch on aysnch) and which application triggered the business event to kick-off the integration and whether it was a scheduled event or not.  Thats it.

 

In fact, that is it.  Sure I am leaving out some details, but those are just implementation choices and do not affect the fundamental design pattern as stated above.  In fact, every application integration job will have that same pattern, over and over again.  What does this have to do with code generation?  Well, everything, actually.  If the pattern is repeatable, then there must be a way to define all of the pieces or parts, just like in our race car engine up front.  If we can define (and draw) those parts (pieces, components, whatever) 100% complete up front, then we can code generate the solution, cause this is the same thing over and over again.  Some would call these software product lines (not what your think in the traditional sense of the term "product").

 

As discussed in the previous post, we seem to have software design patterns for everything, so why are we not using code generators with these patterns?  Do our tools suck?  Are we daft? Are we resistant to change?  Fearful that code generation is a career limiting move?  What then?

 

Next post, more code generation - software factories

I am going to discuss code generation over the next few posts for a couple of reasons.  One is that it is fundamental to the process of software industrialization and second, most programmers, that I know of anyway, dont like code generators, for a variety of reasons.

 

Maybe I should step back and give you my definition of what code generation means to me.  I am pretty sure we all have different ideas as to what this means.  Again, I will draw upon our analogy of AutoCAD.  Lets say I produce a drawing of a piston used for a race engine.  Once I have completed the drawing, I can then save the entire definition of that race piston in a universal file (i.e. DXF) format.  I can then take that piston file definition and feed it into a Computer Numerical Control (CNC) milling machine, which will produce (in our terms, code generate) the output, virtually 100% complete, save for a few finishing touches (i.e. polishing).

 

Thats the key.  I am required to produce a drawing that is 100% complete up front (i.e. design) before I can mill out the piston for my race engine.  When do we specify (i.e. design) anything that is 100% complete up front in the software world?  If I can design a software part or solution 100% up front, then I can code generate the solution, given the proper tools.

 

As a programmer, I would rather build the (code generator) tool that is going to produce the solution rather than the solution itself.  Why?  Because, in my experience, I know I am going to have to do it over and over again, just to get it right. 

 

Traditionally, I would incrementally iterate my way through the solution code several dozen (or even hundreds) of times which will ensure I have a POS by the time it is ready to deploy into production. 

 

So give me a design tool that allows me to draw and define what is supposed to be built (saved in a universal file format) and then I can build a code generator to read the design file and produce the solution output.  The effort is in the design and code generator, not in the solution code. It takes minutes to code generate a solution - every time.

 

Hey, I like to code as much as the next programmer.  But I am getting too old and tired and cant stay up to 4am for a week in a row when it comes down to crunch time for the final push to deliver I dont know what to the client.  If I am going to code anything, it is going to be a code generator.

 

If I build a code generator to read a file format that contains the definition (i.e. design) of a solution, then I can make changes to the design, just like when I want to change the shape or dimensions of the piston.  And if I am really smart, I would have a library of piston designs that I have accumulated over time, along with other reusable design pieces or parts (read: design patterns). 

 

We have several excellent books that document design patterns for virtually everything in software, including the infamous (Gamma et all), EAI design patterns, and in fact, if you type in software design patterns in Google, you get 18,900 hits.  So why dont we have code generators that take these design patterns as input?  Why not create a standardized library of design patterns that any IDE can read and produce the required output in your language of choice?  Oops, there is that word again, standard.

 

Next post will delve deeper into code generators.

Barrys and my invention, and the specific implementation of that invention, is so new and abstract, that it is a real challenge to bring the software to market.  As John Walker, inventor of AutoCAD has said, we need to find lunatic early adopters to try the software out and evolve quickly from there.

 

As discussed earlier, our software world is incredibly complex and highly abstract, so much so, it is virtually (no pun intended) impossible to describe our invention to business people and even to technical people in our own specific area of our industry.  Why is that?  There have been many reasons given over the last several posts, but it comes down to our invention is ahead of its time.  I believe this is an inherent characteristic of any invention, because an invention is just that, it is brand new and requires people to think in ways that they never have thought of before.  What makes our invention even harder to understand is that it works in the virtual world of software which is already a complete mystery to most people.

 

Think of it this way, we invented a standard way to design (i.e. draw and define) application integrations of any size and complexity.  The resulting output definition can be read by other drawing tools and therefore can be reused to replicate or use as a template for other companies application integrations that have similar business scenarios.  In other words, the drawings form a library of business process integrations that can be reused by anyone, much like the way AutoCAD has a library of pre-built designs that can also be leveraged by anyone. This, in my mind, is advancing the industrialization of software.

 

We also invented an intelligent messaging definition that can be used by any middleware engine, in our case we are using BizTalks middleware engine.  However, our intelligent message is a complete abstraction away from any middleware engine implementation.  This means that other middleware engines can use the same standard file definition for business process execution.

 

And thats the key to our invention, much like AutoCADs invention of the DXF universal file format which "is probably one of the most widely supported vector formats in the world today".  We have invented design-time and run-time XML schema definitions that any drawing tool (design time) and middleware engine (run-time) can use as both definitions contain everything required to define and execute any business process integration scenario totally independent of programming languages, operating systems, middleware engines or any implementation details whatsoever.  In my opinion, this is raising the level of abstraction to a new level our software development world has not yet seen.  You heard it here first

 

How would other middleware engine manufacturers use our invention?  For example, Sun acquired SeeBeyond, which is a middleware engine, therefore Sun could use our invention to not only automate the design and code generation of application integrations for SeeBeyond, but also, and more importantly, build a library of reusable business process integrations for their entire application suite of software products.

 

Microsoft Indigo is another middleware engine that could benefit from our invention.  Microsoft could use BRIDGEWERX as part of their Service Oriented Architecture (SOA) strategy.  SOA is the latest TLA in our industry. 

 

You would think with the amount of words written in these posts that this was turning into a book.  It is!  I decided in January of this year to write a book titled, The Industrialization of Software (shameless plug) to help educate people (and inventors alike) about our strange software world and even more bizarre business world of software.  It chronicles my 4 years as President of 5by5Software and details most of the topics you have read over the last several posts that Shane Schick has been kind enough to post on ITBusiness.ca

 

As I see it, we are in a software revolution where the last 4 years of IT has been the worst in the history of our industry.  Businesses have revolted against the software world largely because of our abysmal track record of delivering quality software.  Now the general public (with AntiSpyware, SPAM, etc.) have entered the fray.  For those of you that use software in your daily life, you have the power to make (demand!) the necessary changes in our software industry to bring us out of the dark ages and force us into the industrialization of software.

Have a look at this object model.  This is our modeling design pattern that Barry and I figured out after designing and constructing 25 application integration solutions over a 4 year time frame.  This design pattern raises the level of abstraction for designing and constructing application integrations into a fully automated process that is predictable and repeatable.

 

 

How does it work?  We have taken our design pattern and implemented it as a drawing/modeling tool, much like AutoCAD, where you can draw your application integration scenarios on a canvas.  The modeling tool produces a full-fidelity or complete design specification output in which a code generator can read the completed specification and automatically produce (i.e. code generate) the solution, wrapped as an installation package, much like using an installation wizard to install a program on your computer.

 

At run-time, there are several administrative, operational and monitoring tools that are provided to manage the solution that are installed on a pre-configured technology stack, including a middleware engine.  In fact, everything can be pre-installed onto a DELL server and delivered as a complete integration appliance.

 

Here is another view of the object model as represented by the layers or levels in our modeling/drawing tool.  This is how we visualize size and complexity.  Just like how AutoCAD has many levels of detail, we do too.  We can view the upper most level to see the big picture (which may contain dozens of application integration scenarios across several business entities) and yet drill down, level by level, to the lowest level of detail of an individual data element that is being mapped for example. We can see the entire size and complexity of the business process problem being solved.

 

 

Another key to our invention is the Intelligent Messaging format.  The intelligent messaging format is intended to be used as a standard wrapper for all messages flowing through an integration at run-time.  The actual message content (i.e. data) is contained within the envelope as a payload.  The envelope also contains a number of other fields used to control its flow through the business process and to record the route and activities undertaken during that process.

 

The use of a standard schema allows a higher degree of abstraction within a business process as this standard schema contains (i.e. encapsulates) all business process rules and state information necessary to execute the business process integration without any knowledge of the rest of the system and vice versa (i.e. fully abstracted).  The word standard is very important in our software world as there are few standards.  We invented a standard messaging format for executing application integrations in addition to a standard object model (i.e. drawing tool) that allows business analysts to fully design and describe application integrations in hours/weeks that are code generated and installed in minutes. 

 

You would think an invention like this would help, in some small way, industrialize our software world like AutoCAD did for the engineering design world in the early 80s.  However, as with most inventions, including AutoCAD, getting to market is an entirely different story which I will describe in tomorrows post.

David Frankel wrote an excellent article called, Software Industrialization and the New IT, that describes what software abstractions are and why we want to raise the level.  First we started with 1s and 0s, which is ultimately what the computer understands, but as David points out, there has to be a better way.  There is, it was assembly language, ha ha, then third generation languages (3GLs) and then Model Driven Architecture (MDA) which represents the highest level of abstraction today.  As described in David's article, we continue to push the envelope with the next level of abstraction called, Computer-Assisted Business Process Management (CA-BPM) in which Barry Varga and I have invented one of these for the application integration world.

 

Ultimately, we want a drawing/modeling tool that allows a business analyst or a person that is not necessarily a programmer, to draw and describe the software to be built.  Since we are talking a virtual world here, the construction process (i.e. writing the code) can be fully automated using code generators that know how to read the output format of the drawing tool and code generate the solution to run on a target business process engine or server platform.  Therefore turning our incredibly labor intensive and error prone software development process into one that is predictable and repeatable.
 
Why dont these CA-BPM tools exist today?  There are several reasons for this, as discussed in earlier posts, but mostly due to the newness of our industry compared to other engineering disciplines.  I would like to introduce you to a CA-BPM tool that can fully describe size and complexity in the application integration domain. 

 

The need for application integration arose from executing business processes across software applications because no one application can do it all (unless it is SAP, right George   For example, when you order your computer from DELL, there are several business processes that are executed.  Placing the order, credit card transaction, procurement, inventory and order management, scheduling, back-orders, assembly, test, burn-in and finally, delivery.  I may be missing a few steps, but you get the idea.  It may come as a surprise that there could be a dozen computer applications/programs used in this end-to-end business process, with each application communicating data to one or more applications.  Or at least trying to communicate data, which is where all the integration issues are.

 

Application integration is considerably more complex, abstract and error prone then straight application development.  The middleware engine is what controls message flow (i.e. data) and orchestrates a business process (or multiple processes) between these applications.  Application integration development is very difficult to understand as it is akin to trying to connect wires together in a wiring closest that has thousands of colored coded wires, yet there is no wiring diagram with numbers or labels or an obvious meaning to the color codes.  This leads to a process of discovery that is long, painful, expensive, mostly trial and error and with an end result that may be less than the 20% success rate discussed in the CHAOS Study.

 

However, over time, eventually you figure it out and if you do it enough times you get good at it and the more you do, the more you see the patterns on how it is done and how to do it better each time.  This is how we evolved our invention, which after 25 application integration projects over a period of 4 years, Barry Varga and I arrived at BRIDGEWERX.  And by all accounts, it appears we were first to market with this invention.

 

Tomorrow we reveal our design pattern to show how we can model any application integration scenario that scales to any size or complexity.

Today, we are on a journey of the industrialization of software much the same way when Autodesk invented AutoCAD in 1982 and introduced it to the engineering design and manufacturing production world.  AutoCAD offered, for the first time, a full-fidelity drawing tool that enabled a predictable and repeatable way to produce engineering diagrams that could be saved electronically in a (publicly accessible) universal file format to be used by all sorts of other CAD/CAM devices. A standard was born. In my opinion, AutoCADs virtual drawing world has done more for the engineering design and manufacturing world then any other invention in the last 20 years.  Without it, we would still be forging hammers by hand at a rate of one a month and cost several thousands of dollars a piece to produce.

 

The same revolution is now taking place over 20 years later for the software design and code production world.  The only people that really know about it are software programmers from various tool vendors around the world.  Programmers are realizing that the days of hand crafting ever increasing complex software solutions are becoming too costly and taking too long in our internet time business world.  I predict in the next 5 years we are going to see the business world using software that is heading up the knee of the exponential curve and offer business services through software automation that are going to further increase the speed of which business is performed compared to today on a scale that is unfathomable.  Just as unfathomable as the railroad once was.  Just as unfathomable as electricity once was.

 

The software industry need tools that can visualize the size and complexity of the software structures to be constructed much like how the (traditional) building industry uses architectural blueprints to visualize the design of building structures and yet, are detailed enough to cover all aspects of the building to be constructed.  Not only will this give some sort of continuity to the world of software development, but will help non-technical people understand the (equivalent) difference between constructing their 3 bedroom bungalow house and the Empire State building.  Even a lay person with no level of expertise can visualize by looking at the blueprints and generally comprehend one is much bigger and more complex than the other.
 
Right now in the software industry we have no universal way of showing size and complexity differences even amongst the programming community.  The current state of the art is still so low level and specialized, that most programmers look at these crude architectural drawings and cant infer any meaning as to what is to be built, aside from understanding what the size and complexity is.  Sure we have software modeling languages like the Unified Modeling Language (UML) and the newly introduced Domain Specific Language (DSL), but these tools and languages are still very low level compared to a buildings architectural blueprint.

 

I am not trivializing these language inventions, in fact, they have done a world of good for the software world.  However, we need to raise the level of abstraction using better modeling tools and languages so we can get ourselves out of the dark ages and into the industrialization age.

 

Raising the level of abstraction is the topic of my next post.

Ok, we IT people in the software world have been known to take ourselves way too seriously (including myself) on occasion (how about all the time) so today we are going to have some nonsense.  The nonsense I am talking about is my washing machine.  Now my long time friend and co-founder of 5by5Software, Barry Varga, has asked me what I was doing around a washing machine in the first place.  Well, I was fixing it for my lovely wife, Lesley. Yah, thats the ticket.

 

Remember the annoying beep beep beep firmware problem?  Well, its back.  It seems that Lesley has pressed some magical key combination that caused the infernal machine to resume beeping, three beeps on the minute, every minute when the machine says its done.  Someone in the house has to physically go over to the machine and manually turn it off.  Sledgehammer anyone?

 

But there is more, and here is where the annoying crosses the boundary into the stupefying. Lesley likes to add things to the wash, whilst the washer is washing.  This according to her is common practice.  So she presses the pause button (no kidding, there is a pause button) and tries to open the door.  You guessed it, the door wont open.  Her words were, why wont the machine do what I want it to do? I used to do this with my old washer (not software controlled) and not only could I stop and add items at any time, I can even restart and the washer would already know what level the water was at and not add any more.  I would have liked to tell her that once we purchased a computer controlled machine that she is no longer in control, but I am sure she does not want to hear that   In the end, this means I get enlisted to fix it.

 

First, I do the unthinkable and consult the 27 page manual first (told ya, I am a geek).  Searching through the manual I come across the following statement, Adding laundry is not possible because the door is locked for safety reasons.  The very next line says, Laundry may be added after pressing the Start/Stop button.  Huh?  In fact, in several places these contradictory statements are made.  There is some (stupid computer) trick to make it work because we still cant open the door, even after pausing the machine and even after following the procedure(s) in the manual.  Obviously, as I told Lesley, we must call the manufacturer and request a secret decoder ring to figure out the magic computer logic sequence to unlock the door.

 

Lesley has something to say to the designers of this machine.  As far as she is concerned, the old washer lets you do this and the new one doesnt.  From her perspective, the new washer is a poor design as it does not meet her requirements and she is frustrated that the machine and the instruction manual are so complicated, that she gives up.  Someone at the new washer manufacturer has played a stupid computer trick on her.

 

And thats the point.  Something as simple as a computer controlled washing machine appears to be too complicated to get right in the way of features and user interface design.  How hard can this be?  What does that say about software programs that are 100 or 1000 times the size and complexity?

 

Next week we are continuing with the industrialization of software where the topic will be lessons in abstractions using modeling tools.