This blog is dedicated to the advancement of what I call the industrialization of software.  Software industrialization is about raising the level of abstraction for programmers to produce quality software products using modern software engineering approaches such as Software Factories. One aspect of the Software Factory approach employs model-driven development using Domain Specific Languages (DSLs) that code generates parts (or all) of the solution for a specific problem domain.  The Software Factory approach enables programmers to produce and assemble software in a predictable and repeatable manner something we dont do well as an industry, but is done well in other industrialized engineering disciplines like civil, electronics, electrical and mechanical domains.  Or at the very least, software engineering is nowhere near as evolved as these engineering disciplines.

 

The author of this blog lived the Software Factory approach while working at a company called 5by5 Software, now called Bridgewerx.  5by5 was initially a Microsoft Systems Integrator (SI) that offered professional services in the area of application integration, designing and constructing solutions using Microsofts BizTalk Server product.  Over a 4 year time frame, we designed and constructed over 25 application integrations solutions.  These solutions included messaging exchanges and workflow of all sizes, shapes and forms for various vertical industries.

 

During this 4 year time frame, we recognized several application integration design/code patterns.  Each pattern went through the process of discovery, reuse, refinement, and then iterated through this process again for each new project we undertook.  After designing and constructing 25 BizTalk integration projects, we found an overall framework design pattern that could be applied to all integration projects.  We decided to encapsulate this design pattern in a modeling tool, which really is a Domain Specific Language (DSL) for designing and generating application integration solutions sitting on top of BizTalk Server. 

 

The idea is that the DSL runtime Visual Designer would be used by a Business Analyst, who has the business domain knowledge to model an application integration scenario and the modeling tools output (XML) would supply the complete definition of this specific modeled application integration. We used this complete definition as input to our software factory template which then grabbed reusable components, other artifacts (with some custom code) and configure BizTalk Server with the specific application integration scenario, automatically build the solution in Visual Studio and produce a Microsoft installer package (MSI) which then can be installed on the customers target machine.  We called our Software Factory and DSL, Bridgewerx.

 

The point is that the software factory approach, incorporating DSLs works.  Our 4 years of application integration domain experience and subsequent ISV product is living proof of this fact.  We were pioneers in producing a software factory for the design of and code generating application integration solutions.

 

To further substantiate this claim, I will walk through a number of our projects and describe the patterns we discovered, designed, codified, categorized and subsequently automated, until we had the entire application integration framework pattern designed. Then we encapsulated the framework pattern into a DSL and built our software factory for code generating application integration solutions. 

 

From where I sit, this is industrializing the software development process for the application integration domain.  Another claim I make is that this same approach can be successfully applied to other software application domains of interest (e.g. software factory for generating families of e-commerce applications.)

 

In part 2, we will discuss what is common and what is variable to every application integration scenario as a way of describing our framework pattern.  Then we will look at some specific integration projects we undertook and how it fits the framework pattern.

While paging through a software industry mag, I came across an ad for Oracles middleware product.  The ads buy line was that by using their middleware product, application integration was hot pluggable.  Hot pluggable?  The inference is that you can plug-in applications to their integration bus as easily as one can plug-in a hot swappable hard drive into a computer without powering down or effecting other applications. I have been building application integration solutions for 5 years and my experience can attest that application integration is not hot pluggable.  In fact, it is an incredibly complex rats nest that you have to make sense of many different vendor applications proprietary data formats, using (mostly) proprietary APIs.  Most vendors middleware products also work at this complex level of abstraction, i.e. too low.  Also, while the middleware vendors help manual is 10,000 pages (talk about complexity!), no-where does it tell you in detail how to solve the actual application integration problem, which in itself, is just as complex as the vendors toolset.

 

No wonder software developers are a cynical bunch when it comes to working with most vendors products.  What the data sheet says and the marketecture talks about, makes no sense in the real world.  While one could say that for any type of advertising, our software industry seems the most plagued in my opinion.  Why?  As discussed in previous posts software development is a complete mystery to almost everyone but the programmers.  So who are the ads targeted for?  Decision makers, who are CIOs or purchasing agents, most of whom dont (really) understand software development (generalization) so all they have to go on is the marketecture.

 

I am not just picking on Oracle, I saw a Microsoft Office cartoon ad where the people in the ad had dinosaur heads on their bodies.  Dinosaur heads?  Huh?  Is MSFT inferring that their users are dinosaurs?  The buy line is that one dinosaur accidentally forwarded everyones salary to the entire company and had the dinosaur evolved to using Information Rights Technology in MS Office, then this would not have happened.  I suppose that it also means that one of the dinosaurs in the ad is about to become extinct. 

 

As an industry professional, I find this dinosaur ad - embarrassing.  And insulting to the user community it is targeted at.  Anyone that knows me personally wont accuse me for not having a sense of humor, but honestly this is lame.  Not only is the ad just plain dumb, but it deals with laying FUD on the target audience. This is unacceptable.  Do the marketing people for these companies know no bounds?  Why not say that by using our product you can prevent sensitive information from being read by unauthorized readers using Information Rights Technology and here is how easy it is to do it.  Right now you might be thinking, what a naive software programmer maybe so.  But then again, "All Marketers Are Liars".

 

Both ads remind me of an old Dudley Moore movie called, Crazy People where he was an alcoholic adman and came up with his best ads while being institutionalized at an insane asylum. Dudley had the inmates create ads that told it like it was i.e. the truth.  How is that for irony.  Maybe the marketing people in the software industry can learn a lesson from this?  My cynical programmer persona says I doubt it.  However, eventually customers will push back with another cycle of what happened in the dot com era and want demand software products that do exactly what the marketing materials says it does.  This would help the industrialization of software, but until then, would you like fries with your SOA?

The software industry has a long history of recreating incompatible solutions to problems that are already solved  Pattern Oriented Software Architecture Volume 2

 

In 2001, my good friend Barry Varga  and I co-founded this company for a variety of reasons.  One of those reasons was to make decisions for our own company as previously we had worked for a variety of other companies where we watched the people in charge make, well, lets just say questionable decisions.  We felt that we could make better decisions or at least, we would be accountable for our own decisions we thought how, could we do worse?  I am sure this is how MiniMicrosoft kinda feels.

 

One of those decisions directly affected our programming team, which was, dont write code at least initially.  It sounds strange as most programmers (read: all) love to write code including yours truly.  The issue is that designs precede code and requirements precede design.  Pretty fundamental one would think.  However, as programmers, we are all guilty (and still are) of not conforming to this basic tenet.  Oh to be sure, there are a whole pile of reasons behind this we are good at rationalizing that we dont have any requirements or market intelligence to tell us what to code.  We also have great imaginations and think we know what the customer wants.  We also believe that regardless of size, complexity and/or change, by coding first we can figure it out

 

Since we were writing code on the MSFT platform, and have been since VB1, we knew, as old school programmers that it all has been done before, you just have to find an example.  The World Wide Wasteland has many examples of whatever you are coding.  In fact, since some 50% of the 8 million programmers out there code on the MSFT platform, you are bound to find an example or ten (or hundreds!) that is pretty close to what you need for your component/project/product.  Of course, the cynical programmer and/or perfectionist programmer type (arent we all?) will want to write their own code because what they write will be better than whatever can be found on the internet- surely.  Sometimes this is true, but in most cases not, or a least in my 15 years experience in the field.  This is also sometimes called the Not Invented Here (NIH) syndrome.  This is one of the major barriers to reusing perfectly good code that already exists. Oh, did I mention that coding is fun?

 

We also got our teams to think of different ways to solve various problems (once we had some semblance of requirements to iterate on).  In other words, come up with a design first.  Then think of 3 ways to solve the design, including searching the internet to see how other people have solved similar problems.  Then present those findings in an objective manner.  Once people thought this way, we were able to leverage and reuse several pieces of existing code (and design patterns) to solve a much bigger puzzle.  This meant that we could solve the problem in a much more timely and realistic manner.

 

The key word being realistic.  Anyone that has been writing code for any number of years knows how hard it is to write code from scratch to solve problems of any real size or complexity.  Not only is it really hard work, but the amount of effort required sometimes seems insurmountable.  We do have best practice processes and tools that raise the level of abstraction, but it is still at such a low level it still sometime feels like assembly language programming (to me anyways).  Also, when you the programmer, are the owner of your own software shop, you get a bit concerned as to how much time and effort is required to solve various problems particularly when you know they have been solved before (many times!).  Here is where refactoring really helps find any way to get the problem solved, like right now (hopefully by reusing some already existing code) and we will come back (given time) to refactor the code to make the design better.

 

Why am I saying this?  I see programmers everyday still re-inventing the wheel or NIH or doing everything else but trying to reuse and leverage some chunk of code, component, design that has already been designed, tested and reused by many to solve the same problem.  I ponder the reasons to why that it is while I have mentioned a few in this article, it still amazes me.  Even seasoned pros fall into the trap.  Maybe it is just too easy?  Maybe it is a combination of a whole pile of things.  Maybe it is simply that our software industry still has not gone through the industrialization (i.e. maturity) process like other similar engineering industriesWhatever it is, next time you open your favorite code editor stop!  Ask yourself this question did I look on the internet to see that this problem has been solved before?  If not, ask yourself why maybe you have the answer.

Our human civilization rests on the power of abstraction, insofar as the volume of specific cases handled by an abstraction is typically much greater than the overhead of the abstraction mechanism itself.  Charles Simonyi

In part 1 of raising the level of abstraction, I discussed a computer-assisted business process management tool that provided Business Analysts the capability, through a modeling tool, to completely define an application integration solution.  This modeling tool sits on top of Microsofts BizTalk Server integration server product. 

 

The modeling tool is actually a Domain Specific Language (DSL) which is used to configure a software factory schema for a particular application integration scenario.  This software factory schema is then used as input to a software factory template, which configures MSFTs Visual Studio IDE to automatically generate (with minimal coding) and build the application integration solution.  This raising the level of abstraction increases programmer productivity through systematic reuse of pre-built components and for the customer, increases product time to market, increases product quality, and introduces predictability and repeatability to an otherwise CHAOS process.  I call this software industrialization.

 

Whats my point?  DSLs are a key enabler to raising the level of abstraction.  Yet, DSLs are not well known in the software development world.  In my observations as someone that has been in the software development industry for15 years, we typically still code (by hand) at a very low level.  My intent here is to raise the level of awareness of DSLs to programmers to help raise the level of abstraction for producing quality software products through programming with models.  As such, I will introduce some links to help facilitate this awareness.

 

I have a lot of respect for Martin Fowler.  He has written many great books and one of my favorites is, Refactoring: Improving the Design of Existing Code."  Martin has written an excellent paper called, Language Workbenches: The Killer-App for Domain Specific Languages?.  I would suggest this as a starting point for any programmer to grok the idea of what DSLs are and why we, as programmers, should be using them.

 

In Visual Studio 2005, MSFT have added a DSL Toolkit (SDK), which is the same infrastructure used for their Visual Class Designer and Distributed Designers.  With respect to the DSL Toolkit, there are samples that can be downloaded along with an on-line virtual lab  that allows you try it out for yourself at no cost.  Alan Cameron Wills has an excellent site for DSL FAQs  (thanks Alan!).

 

As a final thought to why we should be raising the level of abstraction:

 

The value of an abstraction increases with its specificity to some problem domain.  Michael Jackson 

I have been meaning to post a review of Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools for some time.

 

This book in my mind represents the state of the art in software engineering today.  The book is based upon the concept of building families of similar, but distinct products, which have been around for years in other engineering disciplines such as civil, mechanical and electronics engineering.  These concepts promote the systematic reuse of like components and factored out variable components for customization in order to produce products that were similar but yet each one being unique.  This is commonly known as mass customization, something that is very new to the software world, but old hat for other industrialized engineering industries.

 

I know Software Factories is an overloaded term, but consider this definition: a factory is a highly organized production facility that produces members of a product line using standardized parts, tools and production processes.  The factory term is common in the industrialized engineering world, but extremely uncommon in our un-industrialized software development world.

 

Jack and Keith initially introduce us to dealing with complexity and change, which are the two fundamental problems in designing and constructing quality software of any size.  Anyone that has read the Standish Groups CHAOS report understands our incredibly poor track record in dealing with these fundamental problems, regardless of programming languages, platforms or methodologies used.  The following chapter on Paradigm Shift assists the reader in understanding these problems as well as the critical innovations that solves these problems.

 

Software Factories goes on to explain their concept of what is a Software Factory within the context of economies of scale and scope.  This is the most critical point of the book to understand, Economies of scale arise in production, while economies of scope arise in development.  Economies of scale arise when multiple identical instances of a single design are produced collectively.  Economies of scope arise when multiple similar but distinct designs and prototypes are produced collectively rather than individually.  This fundamental concept is absolutely key in understanding the how Software Factories pave the road to the industrialization of software.  The authors could have spent more time on this subject at it is the most confusing concept for any software or non-software person to understand and represents the barrier to understanding that software development is no different than any other traditional engineering development process.

 

The next 3 chapters delve into Models and Patterns, Programming with Models and Language Anatomy and how these approaches raise the level of abstraction so that models can be used as first class development artifacts.  Essentially how Domain Specific Languages (as opposed to general purpose languages) converges the gap between requirements (problem input) and executables (the solution).

 

The following 7 chapters cover in detail the concepts above by discussing Families of Languages, Systematic Reuse, Software Product Lines, Platform-Based Abstractions, Components and Services, Mappings and Transformations, and Generating Implementations.  Incredibly well done and I cannot do these chapters justice in this short review.

 

Chapter 16 demonstrates a concrete example of a Software Factory using all of the concepts, techniques and best practices described previously in the book.  Jack and Keith show how the methodology can be implemented now, that it can be widely used to complement and eventually replace existing practices, and that it can help move the software industry toward maturity.  This chapter alone is worth the price of the book.

 

Finally, a section on Frequently Asked Questions compares Software Factory approach to what we know about software development, before Software Factories.  This puts into great perspective the differences between Software Factories and the current state of the art, which is commonly referred to as custom one-off software development.

 

I cannot recommend this book highly enough.  As someone who came from the R&D electronics world 20 years ago where I (and the rest of the electronics industry) routinely used product line engineering development practices, I thought when I joined the software world 15 years ago this approach would be the norm.  How nave was I.  This book should not only be required reading for anyone practicing software development, but also mandatory reading in every Computer Science program.  Then maybe we will see the industrialization of software development become common practice as it currently is in other industrialized engineering disciplines.

In my previous post, I commented on the 4 tenets (i.e. principles) of a SOA.  One of these tenets is, Services share schemas and contracts, not classes or types.  Most developers understand what this means, however, in Visual Studio (VS), it isnt so easy to make this so because VS assumes an XML-based RPC view of Web Services.  Here is what I mean.

 

The .NET Framework makes it extremely easy to create and deploy a Web Service, which is the primary delivery model for a SOA.  The process in VS is deceptively simple you add an .asmx file to your project, add a couple of methods, then build your solution, and voila, you have a Web Service.  In the background, VS generates a WSDL on the server side and a client proxy on the client side.  VS quietly handles all of this for you, which is not necessarily a good thing as mentioned above, VS assumes an XML-based RPC view of web services.

 

There is concept in Web Services development called contract-first or contract-driven development.  Basically, instead of letting Visual Studio generate a WSDL file for you, you should write this WSDL by hand using a WSDL, XML or text editor.  BizTalk developers get the virtue of contract-first development, we have been doing it for years.

 

Btw, WSDL stands for Web Services Description Language.  It is an XML format used to describe a Web Services interface by defining its operations and message exchange patterns.  The actual messages are defined using a schema definition language like XSD.

 

Why would you want to write your WSDL by hand when VS generates this for you?  Letting VS generate your WSDL for you is like picking up your cell phone from a phone company and then letting the phone company handle the contract for you.  Sure, you get your cell phone, but you dont know what you are committed to from a contract perspective.

 

By writing your own WSDL, you have complete control over how your Web Service will be seen to the outside world, including its interface and data structures.  In other words, you have designed it.  This is really a continuation of the idea of contract-driven development that has been around for years - the idea of first creating a contract that your object will implement and that your clients will consume.

 

However, writing WSDL to your Web Services by hand is not such an easy task and there are not really any great tools out there to help out.  In addition, the format takes a bit getting used to and then after you have created your WSDL, you then need to create the client-side proxy and the server-side Web Service code.

 

There is a great VS add-in called WSContractFirst that was written by Christian Weyer in an effort to make contract-first development easier.  Using this VS add-in, you can generate the code for client-side proxies and server-side Web Services based on the WSDL file right inside of VS.

 

I found this "hack" in James Averys most excellent book called, Visual Studio Hacks Tips and Tools for Turbocharging the IDE.  Thanks James!

Man, oh man, there is a ton of buzz (or is it hot air?) around this TLA.  I have been in the software industry as a programmer type for 15 years and have never seen so much marketing hype and confusion around this term.  Here is the technical truth.

 

What is SOA?

What is a Service?

What is a Web Service?

Key Attributes of a SOA

The four tenets of Service Orientation

 

What amazes me is that the SOA hype is pushed by vendors and analysts to mean something (everything!) else other than simply a type of technical software architecture design.  I have seen vendors brand their products as SOA products.  How can a type of software architecture design be a standalone product?  Makes no sense.  I have also seen SOA described as a technology.  Huh?  The whole point around SOA is that it is technology independent.

 

Further, we have technical people trying to describe SOA to business people.  Thats a mistake.  How would a business person know anything about modern software architecture design?  And why should they care?  Business people just want their IT problems solved.

 

Eventually people will figure out that SOA is yet another TLA buzzword invented by someone, somewhere whose only meaning is simply modern software architecture design baby!  Software Architects, including your truly, have been practicing the design of SOAs for some time and as written elsewhere, SOA This, SOA That, SOA What!?!

 

Thanks to Chris Pels of iDevTech for the no-nonsense technical definitions (a.k.a. the truth) of a SOA.

There is a great post by Harry Pierson over at his DevHawk blog site describing what can we learn from looking at the success of mainstream text-based programming languages to help us in the development of higher abstraction modeling languages that are actually useful.

 

As I have written elsewhere, I am a huge fan of raising the level of abstraction to deal with complexity in our software world.  This is part of what I call the industrialization of software.  No, I dont mean making programming fully automatic, as it will never be that way.  It is too large and complex to do so.  However, I am probably as frustrated as Mini-Microsofts quest to make Microsoft a leaner meaner machine in my quest to make software development more of a predictable and repeatable process.  It seemingly aint going to happen over night and may not happen in my lifetime!

 

John Walker, figured out the industrialization of the engineering design world when his invention, AutoCAD hit the market in 1982.  He said, if you cant model it, you cant build it.  Damn right!  In 2005 and in the software industry, we still have not figured it out, yet.  We are still bashing away with the stone age equivalent of hammers and chisels, whereas the engineering design world has AutoCAD to describe incredibly large and complex building structures, airplanes, engines, electronic circuit diagrams and just about everything else you can think of by modeling blueprints that are meaningful. People use these blueprints to turn models into real world constructs that you and I use every day.  How about that cell phone?  Or your iPOD? Or your car? Or that plane you just flew in on? Or this:

 

 

So whats up with our software world?  Why do we refuse to adapt the successes of other industries, like the engineering design world, and leverage those successes in our software world?  What are we afraid of? Why are we stuck using (still) low level programming languages that we toil with at all hours of the day and night to produce inferior software products?  If we were designing and constructing commodity cars by hand and even fabricating the tools used to build the cars by hand, we would be laughed out of the industry.  How come software development seems to be different?  Where is our AutoCAD for software development?

 

This may seem like a bit of a rant or maybe I have unrealistic expectations as to the maturity level of our software industry.  However, I still cant believe, yours truly after being in the software development business for 15 years, still have to manually add an imports or using source code statement every time I add a reference to an assembly in Visual Studio.  What the ??  Not that I am picking on Visual Studio I happen to think it is an excellent IDE along with introducing Software Factories, Domain Specific Languages, Guidance Automation Toolkits, etc.  While these tools and processes are definitely raising the level of abstraction in dealing with software complexity and advancing the industrialization of software, it still seems not enough to me.  We still lack standards like in the electronics design world for plug and play integrated circuits that I can order from a catalog. Or even standard electronic diagram symbols that describe everything electronic in which anyone trained in the industry can glean, in moments, what the circuit diagram is saying, with no ambiguity.  Are we there yet in our software world?

 

Moreover, we still have a world of programmers that refuse to even consider modeling as a first class software artifact they consider it pretty pictures.  I know we have had issues in the past with CASE tools and with earlier versions of UML and other code generation tools.  But, I have software developers that I have worked with that wont even give it a thought they immediately get out their favorite source code editor and start writing code - so much for design.  And it seems the younger they are, the more I see this behavior or even the old school guys who have become code crafters where they take their craft extremely seriously and are totally affronted in the thought of using a modeling tool.  I am not sure I understand either groups motivation for this behavior.  Where did it come from?  How come I (and a few others) dont have this behavior?

 

Whats my point?  I dont have one I am just pondering, out loud, what it is going to take to bring the industrialization of software into reality and how long?