Today, we are on a journey of the industrialization of software much the same way when Autodesk invented AutoCAD in 1982 and introduced it to the engineering design and manufacturing production world.  AutoCAD offered, for the first time, a full-fidelity drawing tool that enabled a predictable and repeatable way to produce engineering diagrams that could be saved electronically in a (publicly accessible) universal file format to be used by all sorts of other CAD/CAM devices. A standard was born. In my opinion, AutoCADs virtual drawing world has done more for the engineering design and manufacturing world then any other invention in the last 20 years.  Without it, we would still be forging hammers by hand at a rate of one a month and cost several thousands of dollars a piece to produce.

 

The same revolution is now taking place over 20 years later for the software design and code production world.  The only people that really know about it are software programmers from various tool vendors around the world.  Programmers are realizing that the days of hand crafting ever increasing complex software solutions are becoming too costly and taking too long in our internet time business world.  I predict in the next 5 years we are going to see the business world using software that is heading up the knee of the exponential curve and offer business services through software automation that are going to further increase the speed of which business is performed compared to today on a scale that is unfathomable.  Just as unfathomable as the railroad once was.  Just as unfathomable as electricity once was.

 

The software industry need tools that can visualize the size and complexity of the software structures to be constructed much like how the (traditional) building industry uses architectural blueprints to visualize the design of building structures and yet, are detailed enough to cover all aspects of the building to be constructed.  Not only will this give some sort of continuity to the world of software development, but will help non-technical people understand the (equivalent) difference between constructing their 3 bedroom bungalow house and the Empire State building.  Even a lay person with no level of expertise can visualize by looking at the blueprints and generally comprehend one is much bigger and more complex than the other.
 
Right now in the software industry we have no universal way of showing size and complexity differences even amongst the programming community.  The current state of the art is still so low level and specialized, that most programmers look at these crude architectural drawings and cant infer any meaning as to what is to be built, aside from understanding what the size and complexity is.  Sure we have software modeling languages like the Unified Modeling Language (UML) and the newly introduced Domain Specific Language (DSL), but these tools and languages are still very low level compared to a buildings architectural blueprint.

 

I am not trivializing these language inventions, in fact, they have done a world of good for the software world.  However, we need to raise the level of abstraction using better modeling tools and languages so we can get ourselves out of the dark ages and into the industrialization age.

 

Raising the level of abstraction is the topic of my next post.

Ok, we IT people in the software world have been known to take ourselves way too seriously (including myself) on occasion (how about all the time) so today we are going to have some nonsense.  The nonsense I am talking about is my washing machine.  Now my long time friend and co-founder of 5by5Software, Barry Varga, has asked me what I was doing around a washing machine in the first place.  Well, I was fixing it for my lovely wife, Lesley. Yah, thats the ticket.

 

Remember the annoying beep beep beep firmware problem?  Well, its back.  It seems that Lesley has pressed some magical key combination that caused the infernal machine to resume beeping, three beeps on the minute, every minute when the machine says its done.  Someone in the house has to physically go over to the machine and manually turn it off.  Sledgehammer anyone?

 

But there is more, and here is where the annoying crosses the boundary into the stupefying. Lesley likes to add things to the wash, whilst the washer is washing.  This according to her is common practice.  So she presses the pause button (no kidding, there is a pause button) and tries to open the door.  You guessed it, the door wont open.  Her words were, why wont the machine do what I want it to do? I used to do this with my old washer (not software controlled) and not only could I stop and add items at any time, I can even restart and the washer would already know what level the water was at and not add any more.  I would have liked to tell her that once we purchased a computer controlled machine that she is no longer in control, but I am sure she does not want to hear that   In the end, this means I get enlisted to fix it.

 

First, I do the unthinkable and consult the 27 page manual first (told ya, I am a geek).  Searching through the manual I come across the following statement, Adding laundry is not possible because the door is locked for safety reasons.  The very next line says, Laundry may be added after pressing the Start/Stop button.  Huh?  In fact, in several places these contradictory statements are made.  There is some (stupid computer) trick to make it work because we still cant open the door, even after pausing the machine and even after following the procedure(s) in the manual.  Obviously, as I told Lesley, we must call the manufacturer and request a secret decoder ring to figure out the magic computer logic sequence to unlock the door.

 

Lesley has something to say to the designers of this machine.  As far as she is concerned, the old washer lets you do this and the new one doesnt.  From her perspective, the new washer is a poor design as it does not meet her requirements and she is frustrated that the machine and the instruction manual are so complicated, that she gives up.  Someone at the new washer manufacturer has played a stupid computer trick on her.

 

And thats the point.  Something as simple as a computer controlled washing machine appears to be too complicated to get right in the way of features and user interface design.  How hard can this be?  What does that say about software programs that are 100 or 1000 times the size and complexity?

 

Next week we are continuing with the industrialization of software where the topic will be lessons in abstractions using modeling tools.

Why do most people today view software as a commodity product when nothing could be farther from the truth?  As a software developer type, I could blame it on marketing, but that is not the whole story. 

 

Software is to most people, completely intangible.  That is to say nebulous.  The laws of physics seemingly dont apply to our world of software.  Software does not have a physical shape or a form that one can readily see or touch, other than on a computer screen.  And even on the computer screen, you do not see the actual size or complexity of the software because it isnt all on one screen.  In fact, you have no idea how many screens there are.  And even if you knew how many screens there were, it still gives you no indication to the size and complexity of the software program.  But once the computer is turned off, where did the software go?  Poof!  Its just like being at a magic show (and for some vendors of software, it truly is a magic show

 

Hundreds of billions of dollars are spent on software development, it permeates into almost everyones daily life, but it mostly does not occupy a physical space like the Empire state building for example.  The Empire State building cost $43 million dollars to design and construct, required 3600 people and 7 million person hours of effort and took one and half years to complete.  Most people can understand why once they see the size and complexity of the Empire State building. 

 

However, in the software world, we are routinely asked to build such structures, with respect to equivalent size and complexity, with a nothing more than a few hundred thousand dollars (at best) with a handful of people, whose programming skills vary wildly, (a topic for a future post), and guess what, can you deliver that software to us next week?  Ridiculous, yet it happens all the time.  It is still happening today.  Status quo continues.

 

Until we have a way of describing software size and complexity in the form of an architectural drawing or structural blueprint, that people can understand, we will continue to perpetuate software development as a massively labor intensive, non-predictable and non-repeatable, error prone process and remain in the pre-industrialization world forever.

 

Next week we will look at a new way of describing size and complexity using a modeling tool that produces architectural blueprints (and code generated solutions as it turns out). 
However, tomorrow is Friday and time for Stupid Computer Tricks!

It is no wonder to me why our software industry is in serious trouble.  FBI officials said they hope to award a contract by the year's end for a complex new software program to replace a failed project that was canceled this year at a cost of more than $100 million to taxpayers. 

 

Closer to home, the Canadian government has spent over $2 billion (yes, $2 billion!) on a gun registry that is nothing more than a giant database for storing information about legal owners of guns who have volunteered to register their weapons.  But $2 billion dollars? To coin a Dave Barry phrase, I am not making this stuff up.  It really happened.

 

The software that controls the baggage handling at the Denver airport where, due to a bug in the software, it was costing the airport $1.1 million dollars of lost operational cost per day!  At the time this was occurring, there was no predicted end in sight.

 

Finally, have a look at The CHAOS Study which reveals mind boggling statistics on software development failures.  The report summarizes that in 1995, the U.S. spends $250 billion per year on information technology for 175,000 software projects.  31% of those projects were canceled before completion. 53% of those projects cost 189% of original estimates.  $81 billion was spent on canceled projects.  Only 16% of software projects were complete on-time and on-budget.  16% wow!

 

Some of you may point out that this report is 10 years old.  I can tell from my own experience, and others, not much has changed since then.  In fact, an updated report verifies this.

 

For the experienced software developer, this does not come as a surprise.  Disheartening yes, surprising, not really we live it - everyday.  The CHAOS Report states that the top three causes for these failures are the lack of user input, incomplete requirements and changing requirements. No surprise here either, however, I would like to offer up what I would call the root cause of these failures, that is not categorized in the report, which is totally underestimating the size and complexity of any software development effort.

 

Software development is an incredibly complex, highly skilled, maximally labor intensive process, probably more so then any other professional human endeavor. Software design and programming is so complex and error prone that it is totally underestimated by everyone, including the majority of software programmers who are closest to truly understanding this size and complexity issue.  Further, the people that we are trying to develop the software for have no idea about size or complexity or how software is designed, built or even how it works.  It is a complete mystery to almost everyone.

 

Tomorrow we are going to delve deep into this size and complexity issue. 

In my discussion on certifying IT Architects, I came across this quote, But much of the work that architects do today is really an art form, not a certifiable set of practices, said James Barry, vice president of development for payroll and human resources applications at Automatic Data Processing Inc. (ADP) in Roseland, N.J. "The written communication and how they present their architecture would be mainly what we would look for in an architectural certification -- not the methodology that determines what to build," he said. "That would come from experience, not certification."

 

I am really beginning to wonder if we will ever get out of the dark ages in the software development world.  With all due respect to Mr. Barry, I must admit I am dumbfounded by his comments.  Art form?  How they present their architecture?  Lets look up the definition of Architect, One who designs and supervises the construction of buildings or other large structures I would say that definition applies to an IT architect as well except that our buildings are software structures.  The art in architecture is around the look and feel of the structure, much like that of the user interface in a software structure.

 

However, in my experience, the look and feel represents less than 10% or even 5% of what an IT Architect does.  We spend most of our time designing software structures so that they do not fall down!  We design software blueprints much like a building architect would design building blueprints for any size structure.   We design and construct software architectures indeed on a best set of practices, like Grady Boochs executable architecture which is an industry best practice and has been for over 10 years.  A comment from a reader of my previous post (thanks Brian Di Croce!) mentioned that there is a Software Engineering Body of Knowledge available, called SWEBOK.  Perfect!

 

In fact, it is indeed the methodology (I prefer practice or body of knowledge) that makes an IT Architect successful in the way that s/he can predict and repeat successes in an industry that isnt too successful.  Thats the premise behind the Software Engineering Institutes Capability Maturity Model.  You follow a prescriptive approach for increasing the maturity of your software development process by using sound engineering principles for developing software.  The mantra is, the quality of a software system is highly influenced by the quality of the processes used to develop and maintain it.

 

This is much in the same way a building architect follows, well known, defined processes for designing and constructing buildings.  Patrick MacLachlan, one of my co-workers at Burntsand is a real Architect.  I asked him what it took to get his Architects degree.  Patrick said it takes 8 years minimum and on average, 10 to 12 years!

 

Look at what Patrick had to do to obtain his Architects degree.  How can we take certifying our IT Architects seriously when there is no prescribed body of knowledge, no exams and takes 3 to 6 months to certify?  What a sorry state our software industry is in.  This sorry state will be the topic of my next post.

In my last post, I discussed the issues around using the title Software Engineer and how our educational system (i.e. Computer Science programs) needs to get into the software engineering game. While perusing about on this topic, I come across a posting on Grady Boochs site called, Certification of IT Architects.

 

Reading the Open Groups faqs, I came across this statement, 2.2 Will there be tests required to obtain the IT Architect certification?   Since there is no prescribed body of knowledge for the program, there is no test. Instead we will assess candidates experience and skills against the requirements of the program by evaluating their written applications and by a Certification Board interview.

 

The faqs go on to say how long it will take to obtain certification (3 to 6 months) and how much it costs ($1250 for the assessment plus $175 per annum to remain certified plus recertification every 3 years).

 

Now, I am all for industrializing our software world, but something bothers me here.  It is the statement that there is no prescribed body of knowledge and therefore no test.  No prescribed body of knowledge for an IT Architect?  Come on, of course there is.  Any seasoned IT Architect can pretty much tell you what the body of knowledge is required to be successful at the job.  It is the same body of knowledge that has been required since the first commercial software project was written some 40 years ago. 

 

It goes something like this: requirements, design, code and test.  Over and over again.  In fact, it has never changed, other than the fancy marketing names that have been attached to this process over the years.  Oh yes, you need some project management skills that you can find in the Project Management Book of Knowledge (been around for 20 years).  Specifically, I would also suggest Grady Boochs excellent book on Managing the Object Oriented Project (been around for 10 years) as a body of knowledge. You will need some body of knowledge on the process of quality as well, the Software Engineering Institute, has several bodies of knowledge on this subject area, including software engineering in general.  The SEI has been around for over 20 years.

 

10 years ago, I took a post-graduate, 2 year certificate program, called, Software Engineering Management at the University of Calgary taught by Karl Williams on behalf of Motorola University.  I can tell you there was a body of knowledge because I had to write 17 exams over those two years to prove I knew what it was. 

 

With respect to being a certified IT Architect, I am disappointed that our software industry would let an organization be accredited to issue certificates without any requirements for a prescribed body of knowledge and no written exams.  How is this advancing the industrialization of software development?

Shane Schick wrote this great article on why no-one wants to be a programmer  As an old-school programmer, I laughed out loud about being a cereal inventor, ha ha!  Thanks to George Rafael for sending this to me.

 

I think Shane is bang-on about Microsoft and other vendors have to take some time out from patting themselves on the back for making great products to show prospective computer science students that it stands to solve still more. They have to talk about what those problems are, and why they will be worth spending hours staring at a screen trying to combat them. They have to explain why the only thing better than living in an software-driven world is programming it in the first place.

 

I can tell you that we are still in the stone ages with respect to software development compared to other engineering disciplines.  The industrial revolution has not occurred yet in the software world.  As a programmer, I cant go to a software catalog and order a login component like I can order an integrated circuit in the electronics world.  How many login boxes do you think have been invented over and over again?  Millions!  And we as software developers continue to code these from scratch even today.

 

It would be great if the young bright minds of our computer science students helped us old-school programmers in developing tools that bring us into modern times so that we dont keep coding login boxes over and over again.  In fact it would be great if our Universities actually offered Software Engineering degrees in addition to Computer Science degrees (arent these the same? See Software Engineering, Not Computer Science ), but this idea still seems to be in its infancy in our software world.  Unfortunately, in Canada we are still arguing over the title of Software Engineer 

 

Titles aside, I am hoping that the professional engineering bodies in Canada are trying to help our immature software development world by institutionalizing some solid engineering practices in our computer science programs.  Doing so will bring software development out of the dark ages and into the modern world where our chances of building successful projects on time and budget will increase beyond the less then 20% success rate (The CHAOS Report) we currently have today.

So how to introduce this technology?  Lets start off with a quote from Craig McMurtry, Microsoft evangelist extraordinaire, who happened by one day to see BRIDGEWERX in action,

 

"BRIDGEWERX exemplifies a new breed of software vendor, best described as an on-demand software factory, that may well turn out to be the fabled next big thing in the Information Technology industry.  On-demand software factories provide superior software design tools in the form of freely-accessible services over the Web, and then sell the software that one designs with those tools, software that plugs right into the target environment.  In the case of BRIDGEWERX, that free designer is a masterfully simple, yet uniquely powerful facility for representing complex business-to-business, and application-to-application integrations in a practical and intuitive way.  Anyone with an integration problem to solve should certainly consider BRIDGEWERX among the top few tools they could use."

 

Barry Varga and I, being the creators of BRIDGEWERX, were quite flattered by Craigs quote.  For myself, it was a bit of a relief as we were (and still are) having a heck of time trying to describe our wares to people.  Craig gets it, but not too many other people do.  Why is that?

 

As Albert Einstein once said, the only source of knowledge is experience.  As it turns out, not too many people have experience with application integration.  For the ones that dont, they are quite surprised to find out that applications dont normally communicate with each other as a matter of course.  For the ones that have application integration experience, they are extremely skeptical as they know what a mind numbing, labor intensive process it really is and cant believe one can draw an application integration scenario, let alone code generate the solution.

 

In some small way, I would like to think Barry and I contributed to the industrialization of software with our invention.  If you want to see BRIDGEWERX in action, (it will only take an hour, grab a beer, some popcorn and treat yourself to a Live Meeting presentation (courtesy of Jim Bowyer - BizTalk guru from Microsoft).

 

In my next post, I will discuss the history of this invention.  As the saying goes, ...I have not failed. I've just found 10,000 ways that won't work --Thomas Edison