Over the last 15 years as a software professional, I can count on one hand the leaders that I have worked for and that I would work for again.  There are four to be exact.

 

When I was at Eastman Kodak in New York, I worked for Mike Morba who was the VP of Business Development in the Digital Imaging Division.  Mike was responsible for a $30 million budget for Kodaks Picture Center product.  What made Mike a good leader?  First, he knew everything about digital imaging, not only from a software development perspective, but also from a marketing perspective.  He had keen insight into the marketplace and a hell of a vision for the product, which was way ahead of its time.  Mike also knew what motivated people.  It was quite simple really, he gave credit where credit was due.  So when his team did a good job, he made sure people knew about it, not only in his 300 person team, but everyone else in Kodak.  Mike was sincere and selfless in this respect.  He knew he was the man, and so did everyone else at Kodak, but he did not have to advertise it or take credit for what his team did.  A quality other prospective leaders should try and come to terms with, that is, if they can leave their egos behind.

 

Eastman Kodak was a huge company with over 40,000 employees.  At the other end of the spectrum, I worked for Shawn Abbott in a small 10 person start-up company that was more of an advanced R&D shop than a products company.  What made Shawn a good leader?  Not only the attributes mentioned above of Mike Morba, but in addition, Shawn had a vision for his company and never gave up control of his start up, even when times were tough and venture capitalists were hovering around ready to dish out cash but in return wanted control.  Shawns issue was what did these people have in the way of leadership other than cash?  Shawn stayed the course and has done very well for sticking to his vision.  I have a lot of respect for Shawn.

 

Steve Langley is another person I would work for again.  Not only did Steve have the attributes of the people above, Steve was also an extremely focused individual and the first person I met in the software world that said no to ridiculous schedules.  You see, Steve is one of the few people I know who was realistic as to what could be accomplished given the resources at hand.  Steve is a 25 year veteran programmer who knew the value of less code was a good thing, so when the young guys would crank out tons of code, Steve could do the same thing, but much better, with far less code.

 

And the fourth person I would work for again?  Myself.  While I dont think of myself as a good leader, I did start my own software company (because I was frustrated working for leaderless companies, other than the ones previously mentioned) and tried to embody the attributes of Mike, Shawn and Steve for the people that worked for me.  My company that I co-founded with Barry Varga was quite successful for a while.  However, unlike Shawn, I faltered at the choice of taking cash and giving up control, and I found out the hard way that leadership does not necessarily come with cash.  Lesson learned, and when I start up my next company, I wont make the same mistake twice.

 

Whats my point?  Its still the Wild West in the software world where anybody can slap on a set of six guns and call themselves a leader.  Real leaders are extremely hard to find, and in short supply, especially in the nebulous world of software.  Even in the largest software company in the world, the question of leadership is (still) a hot topic with Mini-Microsoft.  Think about the leader(s) in the software company you work for.  Count your blessings if you work for a leader that embodies the attributes mentioned above.  These are the types of individuals that know how to advance the industrialization of software.

Jochen Seemann presented a session on Visual Studio for Software Architects Future Directions in Modeling Tools to about 2000 people at PDC today.  This presentation covered topics of Domain Specific Languages (DSL), Software Factories and Visual Languages.  All of which I am interested in for advancing the industrialization of software.

 

Jochen put into context what a DSL meant by comparing it to an electronics circuit diagram.  In the electronics domain, this circuit diagram has specific meaning, but outside of that domain, it has no meaning whatsoever.  Hence the term, domain specific.  Continuing with the circuit diagram analogy, the language that is used, such as resistors, capacitors, transistors, etc. is specific to that domain.  Hence the term DSL.

 

He also compared DSL to UML where his view was that UML is a broad language for Object Oriented software engineering.  Sure you can customize UML to a large degree, but the problem is trying to use UML to describe something specific enough for a particular domain.  For example, he used the analogy of using use cases (text based) to describe the layout of the auditorium we were in and giving a 50 page use case document to construction workers to build the auditorium.  Using natural language (i.e. English) has a major problem with semantics.  That is, every person could read the use cases, including the construction workers, to build the auditorium and come up with different meaning as to what the words meant and therefore a different auditorium build out.  Hence the problem of using a generalized (read: unified) tool to describe something very (domain) specific.  Of course, no construction worker is going to read a 50 page use case document, they are looking for a blueprint that visualizes specifically what the auditorium is going to look like as the building blueprint is really a domain specific language and is actually a visual language since the blueprint is likely a CAD drawing.

 

This is where DSL comes in. DSL is customizing modeling to the extreme.  Jochen walked through a demo of building a DSL Visual Designer in Visual Studio, which basically consist of three steps.  First you define the domain model, second you define the notation for that model and third map the visualization of the domain model via notation elements.  All of which goes into a large XML schema file.

 

It is an incredibly powerful construct as not only does Visual Studio come with a number of (DSL) modeling templates, but the ability to add third party partner ISV templates and your own templates that all run on the same platform. This makes the interchangeability and reusability with a library of visual designers a reality.  Something that the software industry has been waiting for a long time.

 

Further the ability to link different model aspects and viewpoints means that the ability to code generate complete solutions a reality.  If you are interested in advancing the industrialization of software, I can only suggest that you have a real close look at Microsofts modeling strategy and download the latest DSL tools with the new release available tomorrow. 

The title is a quote from Don Box while presenting Windows Communication Foundation at Microsofts Professional Developers Conference in LA.  Many years ago I read some of Dons books and was always impressed on how he got Windows.  I had the pleasure of seeing him live at this years PDC and saw what a dynamic speaker he is, in addition to the fact that he probably knows more about Windows then any other person on the planet.

 

Dons SOA comment goes directly to how goofy our industry has become where everyone has latched onto this latest TLA.  I cant go anywhere in the software world without running across SOA.  Like Don says, SOA what!  To quote Don, Message, oh message, the truth is on the wire.  There is nothing else.  So much for SOA

 

I was able to catch Steven Sinofskys presentation on Office 12.  The big news is the investment Microsoft has made in SharePoint Version 3.  I thought SharePoint 2003 was pretty cool as that is how I make my living as an Architect, solving customer problems using SharePoint.  However, version 3 is incredible!  It is the center piece of Microsofts Enterprise Content Management (ECM) strategy for the Information Worker.  It represents the 5 pillars of connected systems, Interaction (using Atlas), Messaging and Services (using Windows Communication Foundation), Workflow (using Windows Workflow Foundation), Identity and Access (using InfoCard and AD), and Data (using SQL 2005).  Whats really surprising to me is that at this years PDC, nothing on ASP.NET web application development.  It looks like that SharePoint is the web application of choice.

 

Speaking of Workflow, I got to see Windows Workflow Foundation in detail.  Essentially it is a framework (i.e. a set of APIs) that allows you to embed workflow in your custom developed applications.  Since it is part of the Windows platform, it is available for anyone to use and has built in support in Office 12 and the new version of FrontPage.

 

One really cool feature of WWF is to be able to draw a workflow as a state diagram.  State diagrams are different than sequential workflows as they are more ad hoc in nature and makes a perfect match for many human business processes that are also ad hoc where workflow steps are skipped or even better, dynamic in nature.  Dynamic update meaning we now have the ability to change the workflow logic while the workflow is running.  This is pretty amazing and matches much closer to the real world then a static sequential workflow. This also means that you do not have to recompile.  Cool!

 

Now I have been using workflow (a.k.a. orchestration) in BizTalk 2004 for some time and wondered how the two technologies work together.  As I understand it, using WWF is great for workflow inside applications and BizTalk is great for using workflow across applications.  Makes sense, but I am sure we will see a version of BizTalk that will use WWF in the future.

 

I also wondered how BizTalk 2006 (BTS) will work with Windows Communication Foundation (WCF).  As explained to me, WCF is a great framework for building connected systems (Microsofts term for SOA), but has no facility for orchestration.  In other words, point to point communications.  Adding BTS works in concert with WCF for additional business process and integration server capabilities which adds another layer of abstraction in distributed systems design.  Again makes sense, but I am sure there are other plans for BizTalk in the future which I will get to hear about tomorrow.

 

Also tomorrow, I will get to see and hear a session on Visual Studio Team System for Software Architects and Future Directions in Modeling Tools.  I am quite excited by this as I believe this is another step in the industrialization of software, which is what my blog is all about.  Till tomorrow.

In part 1 of why software industrialization, I explained how even the simplest of end user software products still cant be designed right (i.e. washing machine firmware).

 

I have been a software professional for 15 years and I am still amazed at how little we have improved software engineering in our industry. The software development tools we use still feel like hammers and chisels to me.  The software development process, Agile or otherwise, seems overly complicated, convoluted and hardly predictable or repeatable, in addition to involving way too much effort and using people skills that vary wildly in knowledge and expertise.

 

I gave an example of how AutoCAD revolutionized/industrialized the engineering design world way back in 1982. We have nothing like this in our software development world.  In some respects, I think we have actually gone backwards instead of raising the level of abstraction to improve the process of software development, we are actually writing even more code as products and layers of interfaces are being added to our toolsets daily, plus watching vendor framework class libraries ever increasing in size, (literally 1000s of classes), but not really doing anything to raise the level of abstraction.

 

Here is another analogy I can give you to explain what I mean by the industrialization of software.  Too many years ago, I worked for an electronics company where I designed printed circuit boards by hand.  This meant hanging a circuit diagram (a real blueprint) on a wall and on a light table, with a Mylar grid 4 times the size of the physical circuit board, I laid down red and blue tape (for a double-sided board) of different thicknesses to represent electrical connections between components and different sticky symbols that represented resistors, capacitors, integrated circuit patterns, etc.  Once the layout was complete, we then had to shoot negatives (actually positives) of the layout (reduced by 4X), expose a blank copper plated printed circuit board in a UV oven, with the positive on top and then etched the unexposed copper away in a caustic chemical soup, which all tolled, took 2 weeks to complete.

 

I would work 8 hours a day times 6 to 8 weeks staring at a light table and as I made each connection (out of several hundred), I would highlight the connection on the circuit diagram (i.e. blueprint) to indicate a completed connection.  There were several design constraints, layout size of the circuit board, making sure power traces were not beside low level signal traces, etc.  Kinda like a chess game, except it took 6 weeks or so to see if you won or not.  If you painted yourself in a corner, it invariably meant a starting from scratch.  Management did not appreciate starting over, so we got real good at playing chess

 

Then industrialization hit the electronics/printed circuit board industry with the evolution of computer numerical control (CNC) that helped digitize the circuit diagram into an autorouter software program that automatically designed and laid out the printed circuit board from the digitized input by a light pen.

 

 

 

Today of course, this manual labor intensive process is fully automated.  The circuit diagram is now fully digital (remember AutoCAD) and the autorouter can simply read the circuit diagram and produce the circuit board directly with no human intervention or other intermediate steps.  In fact, the electronic circuit diagram is fed directly into this system and out comes the finished printed circuit boards (forget double sided, how about a dozen layers!), all in a matter of minutes to a few hours.  What used to take me 6 to 8 weeks or longer by hand is now reduced to minutes or hours, with no errors. How is that for industrialization?

 

What can we point to in our software industry, over the same time frame as our analogy above, that has reduced two months of manual labor intensive hand-coding effort into a matter of minutes or hours?  And I dont mean marketing words or the latest TLA buzzword SOA, or latest programming language, I mean some tool or generator that actually reduces the manual effort and coding required to produce a finished software product.  Can anyone point to anything that substantially has advanced the industrialization of software over the last 15 years?

 

In the software industry, we are still in the dark ages as far as I am concerned and my little blog here is a small attempt to try and help push the envelope of software industrialization from an educational perspective.  Hope you enjoy it.

In a previous post, I discussed a product called BRIDGEWERX, which uses BizTalk server as its COTS middleware engine.  In that post is a diagram of the BRIDGEWERX logical component architecture in which you may have noticed a term called dynamic interrogation.

 

One of the coolest features about BizTalk is that you can dynamically interrogate an application you wish to communicate with on a network and query it to return all of its public (XML) message schemas that can be exchanged with this application.  BizTalk uses an adapter that knows how to communicate with BizTalk on one end and the native interface of the application you want to communicate with on the other end.  Currently there are dozens of third-party adapters available for BizTalk that use this standard interface mechanism, plus several adapters that come out of the box with BizTalk.

 

For example, in BizTalk, through the Visual Studio IDE at design time, I can install a BizTalk adapter that can communicate with SAP.  Through a properties sheet in BizTalk, I can give the adapter a physical address that points to my SAP instance on my network.  I can fill in security credentials that will allow BizTalk to communicate with the SAP application, through the adapter.

 

Still at design time, I can then tell BizTalk, through the adapter, to return all of the message schemas that are available for this specific instance of SAP.  In other words, dynamically interrogate SAPs message interface.  Those message schemas are actually the public message types that I can use to map to other schemas as part of my application integration solution.  For example I can have SAP communicate with Great Plains and exchange financial data, even though the two message formats are different.  BizTalk has a mapping tool that graphically maps one file format to another, but thats another topic.

 

This is a great example of a service oriented architecture (SOA) as through BizTalk and an adapter (bought or built) that knows how to communicate natively with the application, I can expose the public schema messages (i.e. at run-time, instance data) as a service to be consumed by any other application completely location independent.

 

Using BRIDGEWERX (BW) Designer, I can extend this service oriented architecture approach to dynamically interrogate any application in the world.  Think of it this way, through BW Designer, as an on-line modeling tool, I can dynamically interrogate any application in which I have already installed an adapter on a BizTalk server for.  Of course, this means having the right physical addresses and security credentials in place. Even though this is across the web, any competent Network or Security analyst can lock it down and secure the connections without any worries of security breaches.  Its done all of the time.

 

The point is that dynamic interrogation using BRIDGEWERX, and BizTalks middleware engine, can allow any application to communicate messages to any other application, regardless of application type or geographical location.  The fact that I can do this at design time, and with just a browser, allows me to 100% specify up front all of the application integration interactions for my designed solution.  Because of this approach, we can code generate the application integration. Dynamic interrogation is another excellent example of advancing the industrialization of software.

Last post we defined what was meant by an on-demand application generator.  Lets look at a real world product.  This on-demand application generator for producing application integrations is called BRIDGEWERX (BW).

 

Have a look at BWs component architecture.  It consists of 3 subsystems.  These are an on-line Rich Internet Application (RIA) for defining (i.e. modeling) the application integration(s), an application generator for producing the application integration solution, and the runtime environment for executing the generated application integration(s).

 

 

The graphical user interface (BW Designer) is developed using Macromedias FLEX product.  The only requirement for running the GUI is a Flash plug-in (which you no doubt already have installed in your browser today. BW Designer is a modeling/drawing tool that allows a Business Analyst (BA) to model, draw and define application integrations on a drawing canvas where various levels of detail are represented by canvas layers. 

 

Several business objects can be dragged and dropped onto different canvas layers, with each object having a specific configuration wizard for the BA to configure for any application integration scenario.  In addition to selecting XML message schemas, defining transport protocols, business rules, business event triggers, adapters, mapping definitions, etc., there are also facilities for uploading custom XML message schema definitions and even custom assemblies, if required.

 

BW Designer uses web services to communicate with a Windows 2003 Server using IIS to store the configuration data (i.e. factory schema instance) into a SQL Server repository.  As discussed in a previous post, the amount of effort is directly related to the size and complexity of the application integration(s) being defined.  The best news is with this new level of abstraction and the fact that the application integration is generated, with little to no source coding involved, the time to develop an integration solution is a matter of days or a week or two as opposed to several weeks or months and in some cases, years for a ground up totally manual labor custom coding job. See the sorry state of our software industry.

 

Once the BA has modeled, drawn and defined an application integration(s) using nothing but a browser on his/her desktop computer, anywhere in the world, the BA can simply order the application integration to be generated.   The order system receives the customers order and extracts the customer information along with the factory schema instance that represents the specific application integration (i.e. software product line) to be generated. 

 

The application generator, or software factory if you like, reads the customers factory schema instance from the BW config repository and along with a customized factory template, using Visual Studio script code, automatically gets latest versions of specified components (generic and/or custom) from source code control, along with other scripts and/or assemblies and/or any other Visual Studio artifact required, creates a Visual Studio solution and then builds (i.e. compiles) the solution.

 

The solution is compiled and packaged as a Microsoft Installer package.  The customer is then notified that the solution is ready for download from BW Designer. The customer then downloads the MSI package and runs the installer on the target run-time environment.  Alternatively, the customers order may have included an integration appliance that includes the hardware, software and the solution, already pre-installed on a computer and tested, to be shipped directly to the customer.

 

Once the customer has installed the solution, along with a few run-time parameters such as physical addresses of where the applications to be integrated reside and enough credentials to run the solution, a self-test is run and the customer is notified that the integration solution is now operational.  Other administration, monitoring and exception handling management tools are also installed for complete, managed runtime operation of the generated solution.

 

Need changes to the application integration solution? The customer simply reopens the model (i.e. drawing), makes changes, re-orders the solution, the solution is generated on-demand and since the installer wizard knows that this is simply an upgrade, the generated application integration solution is updated in the same way as applying an upgrade to MS Word on the desktop for example.

 

Whew, long post!  Nonetheless, I hope you can see that there are real-life on-demand application generators out there, BRIDGEWERX being one of them.  If you want to see BRIDGEWERX in action, have a look at this 1 hour Live Meeting broadcast.

 

Products like BRIDGEWERX are greatly assisting the industrialization of software.  We just need more of them!

In my last post, I discussed a model for sizing and estimating effort for building software applications.  This model worked reasonably well based on some assumptions.  One assumption was that customers actually saw the need and value in spending the time documenting use cases and going through an estimating process.  That in itself is a hard thing to do.  Another assumption was that this was not a net new, never been done before, application by the development team or using brand new technology to build the application.  It also assumes a development team that has worked together on more than dozen projects.  Just like any team sport, it takes several times playing together for the team to gel.

 

So what does this have to do with on-demand applications?  Everything!  Because the reality is that most customers dont understand the need for software sizing and estimation.  Also, the application being developed, indeed is somewhat net new and maybe with new technologies and the fact that most software teams are nothing more than assembled bodies, thrown together to get a project done in the shortest time possible based on who is available at the time.  The irony is that the project will likely cost (much) more and take (much) more time because the customer and the thrown together team fail to follow a sound engineering process for properly sizing the software and estimating the effort required to build the application.

 

Enter on-demand application generation.  After Googling the phrase, I think some explanation is required to help set the context of what I mean.  On-demand means I can get it now or in short order.  Application Generation means that I can use a higher level of abstraction to define what the end application is supposed to be without resorting (or at least minimizing) source code programming.

 

Sound too good to be true?  Lets take the company I used to work for as an example.  We had a core, very experienced, software development team that worked together on roughly 25 projects over a 4-year time frame, specializing in application integration projects.  Translation - we had a very high performance team focused on one type of application development, which meant that we leveraged very specific technologies and code libraries (some built, some acquired) and used a number of integration design patterns to mostly assemble our integration applications.  Then we developed that higher level abstraction into an on-line drawing tool that allowed us to define the integration application to be built and then we built an application generator to take this definition to assemble and code generate the final application integration.  This system is called BRIDGEWERX and is designed to generate application integrations on-demand.

 

I think our company is/was unique because we understood the business value of having a high-performance software development team where management did not view programmers as simply bodies that could be plug and played together with some wild expectation that somehow this was going to work.  You dont just assemble a NFL football team overnight and expect to win the SuperBowl the same season.  Ludicrous!  Yet our business leaders expect just that.  Its just software isnt it?

 

With respect to the estimation piece, using on-demand application generators dramatically reduces the effort required as most of the effort is up front, defining the application to be built using a high-level abstraction modeling tool as opposed to manual labor source coding.  Oh to be sure, there will always be some level of custom programming, but the point is that 80% of it can be generated and the other 20% is highly focused specialized work.  In the case of our application integration generators, the only custom development is in developing an interface to the application to be integrated if one could not be bought off the shelf.  The rest of the effort was simply using the modeling/drawing tool to define the application integration to be generated.  This could be measured in hours and days instead of weeks and months (or years) with the traditional manual labor approach.  Even more important is that this approach is more predictable and repeatable and therefore, dramatically reduces risk, particularly for the customer.  This is the beginning of the industrialization of software.

 

Next post we will look at BRIDGEWERX components to see how it qualifies as an on-demand application generator.

Before we continue with on-demand application generators, lets look at something completely different, but related.  Software size and effort estimation (read: cost estimation).

 

There are several formal methods for estimating software size, complexity, effort and cost.  Function Point Analysis, Lines of Code, tools like CoCoMo, SLIM, etc., including the most common technique called Wideband Delphi Estimating where senior programmers, architects, etc., base their estimates on real world experiences while following a formal estimating process.

 

Why do we do estimate?  Every customer I have come across always asks, how much? And shortly followed up with, and when can I get it? And finally, can you make it a fixed price estimate? 

 

Its like getting a building constructed, you need a blueprint to estimate a price for the construction, whether the blueprint is custom designed or "off the shelf". Regardless, the customer had to spend money on acquiring the blueprint. The blueprint puts in scope whether it is a house the customer wants or maybe it turns out to be an apartment complex or skyscraper that the customer really wants, hence the blueprint.  Constructing software is no different, you need a blueprint.  The blueprint we require are Use Case scenarios in order to provide an accurate cost estimate.
 
If we had the use case scenarios written down at this stage, then it would be much easier to come up with an accurate estimate.  However, the customer typically does not understand, that the level of detail we need is way more than the customer has supplied.  Almost always.  And the level of detail is down to documenting use case scenarios, usually a dozen to two dozen scenarios on smaller projects.  The quantifiable output is usually represented as written documentation with an average of 3 pages written per use case scenario, which in our example, means roughly 15 use cases x 3 pages each = 45 pages of  use case documentation.  Based on 15 years in the biz, I can count on one hand where I actually got enough detailed and documented use case scenarios from the customer to come up with an accurate cost estimate, as is.  For people that know me, I usually get requirements written on a cocktail napkin.  Funny, but not enough detail to be sure.

 

Geri Schneider and Jason Winters wrote an excellent book called, Applying Use Cases: A Practical Guide  In the book is a cost estimation model based on using the number of use cases and the complexity rating for each use case as input to the estimation model.  The model outputs effort estimates, with upper and lower bounds and by project phase and iterations of phases, in person days along with the ability to put in labor rates, which then outputs a cost, which then you can apply a contingency factor and out comes the final price estimate.

 

You can calibrate the use case estimation model as you use it over time, based on comparing estimates to actuals.  You can adjust estimating factors around skill sets and culture, plus the ability to calibrate around toolsets employed.  Whats great about it is that it works!  My team and I have used it successfully on a number of projects of varying size and complexity. After several iterations of comparing actual to the estimates, we got our estimation model calibrated within 10 to 20% differential between estimates and actual.

 

How does this tie in with on-demand application generators?  Next post.